kpihx-web 2.2.0__tar.gz

kpihx_web-2.2.0/PKG-INFO

@@ -0,0 +1,275 @@
+Metadata-Version: 2.3
+Name: kpihx-web
+Version: 2.2.0
+Summary: Private CLI for metasearch, verified retrieval, bounded crawling and media investigation
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Dist: beautifulsoup4
+Requires-Dist: defusedxml>=0.7.1
+Requires-Dist: fpdf2>=2.8.9
+Requires-Dist: httpx[socks]
+Requires-Dist: lxml
+Requires-Dist: phonenumbers>=9.0.41
+Requires-Dist: picimagesearch
+Requires-Dist: pillow
+Requires-Dist: pypdf
+Requires-Dist: python-docx
+Requires-Dist: regex>=2026.9.29
+Requires-Dist: requests[socks]
+Requires-Dist: rich
+Requires-Dist: typer
+Requires-Python: >=3.12
+Project-URL: Repository, https://github.com/kpihx-labs/web
+Project-URL: GitLab, https://gitlab.com/kpihx-labs/web
+Project-URL: Issues, https://github.com/kpihx-labs/web/issues
+Provides-Extra: dev
+Description-Content-Type: text/markdown
+
+# web (kpihx-web)
+
+Local-first research CLI: discover indexed sources, retrieve content, verify
+evidence and preserve provenance. Python distribution: kpihx-web; executable: web.
+The previous implementation and documentation remain under references/ for audit.
+
+## Installation
+
+Use `make install` from a clone, or `uv tool install kpihx-web` then `web setup`.
+Run `make help` for dependency installation guidance, backend health/recovery,
+testing and uninstall/purge targets. uv manages Python dependencies and uv_build
+builds the package. No Hatch backend, paid recognition provider or transcription.
+
+Clone either SSH remote, keeping the projects directly under your project root:
+
+```sh
+git clone git@github.com:kpihx-labs/web.git
+# Alternative: git clone git@gitlab.com:kpihx-labs/web.git
+make -C web install
+make -C web backend-clone HOST=github  # or HOST=gitlab; retains an existing sibling
+make -C web backend-up
+web doctor --live
+```
+
+## Discover the CLI without guessing (2.2)
+
+Every command level has a purpose/limits explanation, at least three varied
+examples, and descriptions with examples for every argument and long/short flag.
+The help is offline: it does not query providers, record audio, inspect files or
+purge state. Explore the whole tree, including map's distinct API operations:
+
+```sh
+web --help
+web search --help
+web doc --help
+web map --help
+web map search --help
+web map query --help
+web map route --help
+web map request --help
+```
+
+The recursive help contract and example parser tests fail if a new command ships
+an undocumented option, missing short/long alias, or fewer than three examples.
+Generated help remains English, like the code. Detailed shared descriptions live
+in help_text.py, with explicit references at each Typer declaration.
+
+## Search precision, budgets and provenance
+
+```sh
+web search '"Lado Saha" site:lig-membres.imag.fr -annuaire' -e 'google cse' -P 2 -n 30
+web search '("memory scheduling" OR "unified memory") filetype:pdf after:2023' -P 3 -n 0 -f json
+web search @query.txt -X '{"time_range":"year"}' -c science -l en -m 1
+web search 'python logo' -c images -f json
+web engines -r 'google|bing|duck'
+```
+
+Quotes, Boolean operators, exclusions, site:, filetype:, intitle:, inurl: and date
+operators pass through unchanged. Support depends on the selected engine; there
+is no universal Google syntax translator. `-X/--params` accepts arbitrary
+SearXNG JSON or @file, while preserving the query/JSON response contract.
+Unknown engine names fail rather than silently querying defaults. `provider_fields`
+retains raw result metadata, including image URLs and provider-specific fields.
+
+`-n 30` is a best-effort collection target: fetch further pages until 30 unique
+filtered candidates, source exhaustion/failure or the page-request budget. Both
+`-p` and legacy `-P` now mean --max-pages (default 3); --start-page/-s is separate.
+This corrects 2.0's confusing -p=start/-P=count distinction. `-n 0` collects within
+the full explicit budget. Output reports actual requests, counts and stop_reason;
+it never pads or invents results when the target cannot be reached.
+Default relevance ranking orders term coverage then SearXNG's numeric aggregate
+score. `-k provider` sorts by that score alone (not API array/arrival order);
+`-k lexical` uses term coverage only. None establishes identity or semantic truth.
+Regex/domain filtering
+precedes the display limit. Regular expressions use a configurable regex_timeout
+budget rather than unbounded backtracking. URL dedup preserves meaningful query/path case.
+SafeSearch `-m 0/1/2` is requested upstream; off does not remove upstream censorship.
+
+Repeated identical searches reuse a labelled private snapshot for 300 seconds,
+independent of table/JSON format. --fresh/-F forces live collection and forbids
+stale fallback. --no-save also disables query-cache persistence/reuse. On failure,
+a default request may return a clearly marked stale snapshot up to 24 hours old,
+with partial status and the error; it is never represented as a new live result.
+TTL, max pages, page delay and fallback_engines are plain configurable SETTINGS.
+If defaults stall and a page-request budget remains, keyless Yahoo/Yep/MWMBL are
+queried once as an explicit alternative batch. No IP/account switching, CAPTCHA
+bypass or rate-limit evasion occurs. Explicit -e choices are not silently replaced.
+
+The table shows **Sources**, a full contributor list, and folds complete URLs
+without ellipses. Engine coverage distinguishes contributions, CAPTCHA, rate
+limits and no-results-reported; the latter is not proof an engine was queried or
+is healthy. Provider refusal does not recommend restarting Docker/VPN. Search
+reads index snippets; use fetch separately to inspect actual page content.
+
+```sh
+web search 'KAMDEM Ivann' -n 30 -p 5
+web show /tmp/web-results/SEARCH_SNAPSHOT.json -f table
+web show /tmp/web-results/SEARCH_SNAPSHOT.json -f json
+```
+`show` renders exactly the saved snapshot without any network request, allowing
+an honest table/JSON comparison. Different live requests may receive different
+provider responses, even when their query text is identical.
+
+## Fetch, inspection, sessions and exact downloads
+
+```sh
+web fetch https://example.org -s 'article h2' -r 'privacy|research' -f json
+web fetch https://example.org/document.pdf -d /tmp/document.pdf
+web fetch https://authorized.example/private -C /private/cookies.txt -H @/private/headers.json
+web crawl https://quotes.toscrape.com -D 3 -n 4 -d /tmp/quotes-pages
+```
+
+Fetch downloads once, records MIME/byte SHA256 and extracts links from those same
+bytes. Binary responses are not parsed as HTML. `--download` preserves exact
+response bytes; existing files are refused. CSS and regex inspection use the
+retrieved body, without extra requests. HTTP 200 login/registration/challenge
+pages are classified as partial access, not useful source reads. These heuristics
+can miss unusual walls; inspect access_state and content, not status alone.
+
+Cookies accept Netscape including HttpOnly comments, browser JSON records or a
+flat dictionary scoped to the seed host. Domain/path/secure/expiry restrictions
+are honored. Custom headers are limited to the original origin across redirects;
+header files avoid leaking tokens in shell history. Never send actual session
+values to agents or paste them into reports. Requests ignore environment proxies;
+`--proxy` explicitly selects HTTP/SOCKS. No TLS verification bypass is provided.
+
+Crawl is same-host, paced (`--delay`), depth/page bounded and robots-aware by
+default. `--no-robots` is explicit, not a bypass of authorization or terms.
+Next/rel=next links are prioritized; JS scroll/click/API pagination is not executed.
+`--match` restricts downloads, not traversal. Errors and remaining frontier are
+retained in the manifest. New output directories avoid collisions on reruns.
+Private/local destinations need `--allow-private` for your own services.
+`--exclude` is a URL-frontier regex; its configurable default guards common
+logout/delete/checkout GET links. An explicit empty expression disables it after
+review. This heuristic is not proof GET is side-effect-free: use a dedicated
+authorized session and inspect the frontier before crawling sensitive accounts.
+DNS checks are best effort, not a security sandbox against hostile DNS rebinding.
+
+For JS, authenticated DOM or browser challenges, use browser-proxy independently
+with your authorization. A cookie handoff can help ordinary sessions but cannot
+reproduce every browser/TLS fingerprint. This CLI never promises to bypass all
+CAPTCHAs, paywalls, permissions or account login.
+
+## Documents, images and audio
+
+```sh
+web doc report.pdf --compare other.pdf --ocr --pages 10 --report /tmp/evidence.pdf
+web doc public-report.md --online --samples 3 --limit 5
+web image photo.jpg                       # local metadata only
+web image https://public.example/logo.png # remote input, still no reverse upload
+web image public-logo.png --online -e yandex,google,bing --text-search
+web audio excerpt.mp3
+web audio --record 12 --mode hum
+web archive https://public.example/report.pdf -P '{"timestamp":"20240101"}'
+```
+
+Doc is offline by default: --online discloses sampled quotes to search engines.
+PDF/DOCX tables/ODT/PPTX and UTF-8 TXT/MD/CSV/JSON/HTML/XML/RST/TeX are supported;
+unknown binary must be exported, not decoded as garbage. Scanned PDFs require
+explicit OCR and are rendered to temporary images before Tesseract.
+These parsers are not a malware sandbox; use OS isolation for hostile files.
+Page limit 0 selects all pages and can be expensive. Comparisons provide exact byte equality
+and lexical matching spans, NOT semantic equivalence or a plagiarism verdict.
+Online links are candidates; downloaded content is checked separately. A hash
+search result alone is not proof a file leaked. Archive uses Wayback's closest
+snapshot API only; no Archive.today API or exhaustive historical search is claimed.
+
+Image uploads require --online and omit original EXIF/GPS. Providers are fragile
+unofficial adapters: failures or empty responses are not proof of absent copies.
+--text-search explicitly discloses local OCR words as a complementary text search;
+those results are not verified visual matches. GPS/device/time are local metadata.
+
+Audio uses free SongRec/Shazam; FFmpeg normalizes/records an excerpt. `hum` is a
+qualified best-effort fingerprint attempt and returns partial status: SongRec is
+not a melody recognizer and reliable free humming is not established. ACRCloud,
+paid API logic and credential loading are removed. No transcription is performed.
+
+## Hidden networks and limits
+
+Tor fetch uses SOCKS5h (managed optional profile: 127.0.0.1:19050; override
+tor_proxy for an existing external daemon, commonly 9050); once selected, redirects do not
+downgrade to direct or resolve through local DNS. I2P fetch requires an actual
+HTTP proxy (default 127.0.0.1:4444). There is no fake I2P search adapter. For
+indexed onion discovery use `web search QUERY -e ahmia -c onions`; coverage and
+uptime depend on the index and Tor. Private/unindexed content has no universal
+search engine. No tool can enumerate your complete digital footprint or prove
+absence of a leak from absent results. No arbitrary third-party kill switch
+compatibility or perfect anonymity is promised.
+
+Surface SearXNG startup is independent of Tor. `make -C ../searxng onion-up`
+explicitly starts an isolated optional onion worker; a failed Tor check does not
+crash the surface worker. `web engines -z onion` inspects it. `web search QUERY
+-e bing,ahmia` partitions those engines between backends and reports missing
+coverage rather than losing working surface results. `-z surface,onion` explicitly
+selects both, or use custom `SETTINGS['search_backends']` entries with url/socket
+fields for another SearXNG-compatible backend. Budgets apply per backend, so total
+requests grow with the number of selected backends. Protocols other than SearXNG
+still need an adapter; arbitrary services are not interchangeable by naming them.
+
+## Configuration, dependencies and lifecycle
+
+`web setup` creates ~/.config/web/config.py only if absent. SETTINGS is plain
+trusted Python, not a loader framework. WEB_CONFIG_DIR and standard XDG roots
+control configuration where supported. Existing xsearch configuration/cache were
+moved, not destroyed; dormant old .env credentials are never loaded. Defaults in
+config.py include response_max_bytes, local_max_bytes, xml_max_bytes and
+max_redirects; put overrides in SETTINGS.
+Set phone_region (for example FR) to validate national-format numbers; without
+a region hint, international numbers are preferred rather than guessing dates.
+Local SearXNG prefers its owner-only
+Unix socket to avoid host VPN TCP routing; remote endpoints remain explicit.
+
+Python libraries are resolved with bare uv add and uv.lock, never pip or manual
+dependency arrays. System dependencies are deliberately separate/shared:
+
+| Feature | Native dependency / installation guidance |
+| --- | --- |
+| Audio | ffmpeg; SongRec via Cargo: cargo install songrec --no-default-features --features ffmpeg |
+| SongRec source build | Rust/Cargo, pkg-config, Clang/libclang, libsoup-3.0-dev, ALSA/PipeWire development headers as required upstream |
+| PDF OCR | poppler-utils (pdftoppm), tesseract-ocr and desired language data |
+| Unicode PDF report | fontconfig (fc-match), fonts-dejavu-core; fpdf2 is managed by uv |
+| Backend | Docker Engine and Compose plugin; sibling searxng repository documents images/ports/volumes |
+| Tor/I2P | Your own reachable Tor SOCKS5h/I2P HTTP proxy; never silently started |
+
+Use upstream package instructions for your OS; dependency install commands evolve.
+`make deps` reports installed native capabilities and guidance rather than a rigid
+root installer. Cold builds need network or cached packages/images. Docker logs
+are bounded by the backend; web has no background daemon or persistent log file.
+JSON evidence is not silently rotated/deleted; /tmp is volatile. Explicit
+download/report destinations are user evidence and not automatically purged.
+
+`make uninstall` removes the uv tool only. `make purge` stops the sibling backend,
+trashes owned runtime paths, uninstalls the tool, and retains source/shared native
+tools. `make backend-purge CONFIRM=searxng` backs up named volumes under a private
+/tmp directory (.bak files), then removes only that stack's volumes. Review its
+Make recipe first. Never run global Docker prune. A clone is retained so reinstall
+works from either GitHub or GitLab. `make purge-all CONFIRM=searxng` sequences
+the confirmed backend backup/removal and app purge. Install from PyPI using
+uv tool install kpihx-web.
+
+All commands use an action/timestamp/status/data envelope. Formats: table, json,
+jsonl, csv, md; --report creates escaped offline HTML or a real PDF. Autosaves are
+unique 0600 JSON in /tmp/web-results; -S overrides and -N disables. Exit 2 is
+failure, exit 3 is partial/access-wall coverage, exit 0 is completion. Inspect
+non_exhaustive, budgets and source errors before drawing conclusions. Source
+quality, identity attribution and correctness require human corroboration.

kpihx_web-2.2.0/README.md

@@ -0,0 +1,246 @@
+# web (kpihx-web)
+
+Local-first research CLI: discover indexed sources, retrieve content, verify
+evidence and preserve provenance. Python distribution: kpihx-web; executable: web.
+The previous implementation and documentation remain under references/ for audit.
+
+## Installation
+
+Use `make install` from a clone, or `uv tool install kpihx-web` then `web setup`.
+Run `make help` for dependency installation guidance, backend health/recovery,
+testing and uninstall/purge targets. uv manages Python dependencies and uv_build
+builds the package. No Hatch backend, paid recognition provider or transcription.
+
+Clone either SSH remote, keeping the projects directly under your project root:
+
+```sh
+git clone git@github.com:kpihx-labs/web.git
+# Alternative: git clone git@gitlab.com:kpihx-labs/web.git
+make -C web install
+make -C web backend-clone HOST=github  # or HOST=gitlab; retains an existing sibling
+make -C web backend-up
+web doctor --live
+```
+
+## Discover the CLI without guessing (2.2)
+
+Every command level has a purpose/limits explanation, at least three varied
+examples, and descriptions with examples for every argument and long/short flag.
+The help is offline: it does not query providers, record audio, inspect files or
+purge state. Explore the whole tree, including map's distinct API operations:
+
+```sh
+web --help
+web search --help
+web doc --help
+web map --help
+web map search --help
+web map query --help
+web map route --help
+web map request --help
+```
+
+The recursive help contract and example parser tests fail if a new command ships
+an undocumented option, missing short/long alias, or fewer than three examples.
+Generated help remains English, like the code. Detailed shared descriptions live
+in help_text.py, with explicit references at each Typer declaration.
+
+## Search precision, budgets and provenance
+
+```sh
+web search '"Lado Saha" site:lig-membres.imag.fr -annuaire' -e 'google cse' -P 2 -n 30
+web search '("memory scheduling" OR "unified memory") filetype:pdf after:2023' -P 3 -n 0 -f json
+web search @query.txt -X '{"time_range":"year"}' -c science -l en -m 1
+web search 'python logo' -c images -f json
+web engines -r 'google|bing|duck'
+```
+
+Quotes, Boolean operators, exclusions, site:, filetype:, intitle:, inurl: and date
+operators pass through unchanged. Support depends on the selected engine; there
+is no universal Google syntax translator. `-X/--params` accepts arbitrary
+SearXNG JSON or @file, while preserving the query/JSON response contract.
+Unknown engine names fail rather than silently querying defaults. `provider_fields`
+retains raw result metadata, including image URLs and provider-specific fields.
+
+`-n 30` is a best-effort collection target: fetch further pages until 30 unique
+filtered candidates, source exhaustion/failure or the page-request budget. Both
+`-p` and legacy `-P` now mean --max-pages (default 3); --start-page/-s is separate.
+This corrects 2.0's confusing -p=start/-P=count distinction. `-n 0` collects within
+the full explicit budget. Output reports actual requests, counts and stop_reason;
+it never pads or invents results when the target cannot be reached.
+Default relevance ranking orders term coverage then SearXNG's numeric aggregate
+score. `-k provider` sorts by that score alone (not API array/arrival order);
+`-k lexical` uses term coverage only. None establishes identity or semantic truth.
+Regex/domain filtering
+precedes the display limit. Regular expressions use a configurable regex_timeout
+budget rather than unbounded backtracking. URL dedup preserves meaningful query/path case.
+SafeSearch `-m 0/1/2` is requested upstream; off does not remove upstream censorship.
+
+Repeated identical searches reuse a labelled private snapshot for 300 seconds,
+independent of table/JSON format. --fresh/-F forces live collection and forbids
+stale fallback. --no-save also disables query-cache persistence/reuse. On failure,
+a default request may return a clearly marked stale snapshot up to 24 hours old,
+with partial status and the error; it is never represented as a new live result.
+TTL, max pages, page delay and fallback_engines are plain configurable SETTINGS.
+If defaults stall and a page-request budget remains, keyless Yahoo/Yep/MWMBL are
+queried once as an explicit alternative batch. No IP/account switching, CAPTCHA
+bypass or rate-limit evasion occurs. Explicit -e choices are not silently replaced.
+
+The table shows **Sources**, a full contributor list, and folds complete URLs
+without ellipses. Engine coverage distinguishes contributions, CAPTCHA, rate
+limits and no-results-reported; the latter is not proof an engine was queried or
+is healthy. Provider refusal does not recommend restarting Docker/VPN. Search
+reads index snippets; use fetch separately to inspect actual page content.
+
+```sh
+web search 'KAMDEM Ivann' -n 30 -p 5
+web show /tmp/web-results/SEARCH_SNAPSHOT.json -f table
+web show /tmp/web-results/SEARCH_SNAPSHOT.json -f json
+```
+`show` renders exactly the saved snapshot without any network request, allowing
+an honest table/JSON comparison. Different live requests may receive different
+provider responses, even when their query text is identical.
+
+## Fetch, inspection, sessions and exact downloads
+
+```sh
+web fetch https://example.org -s 'article h2' -r 'privacy|research' -f json
+web fetch https://example.org/document.pdf -d /tmp/document.pdf
+web fetch https://authorized.example/private -C /private/cookies.txt -H @/private/headers.json
+web crawl https://quotes.toscrape.com -D 3 -n 4 -d /tmp/quotes-pages
+```
+
+Fetch downloads once, records MIME/byte SHA256 and extracts links from those same
+bytes. Binary responses are not parsed as HTML. `--download` preserves exact
+response bytes; existing files are refused. CSS and regex inspection use the
+retrieved body, without extra requests. HTTP 200 login/registration/challenge
+pages are classified as partial access, not useful source reads. These heuristics
+can miss unusual walls; inspect access_state and content, not status alone.
+
+Cookies accept Netscape including HttpOnly comments, browser JSON records or a
+flat dictionary scoped to the seed host. Domain/path/secure/expiry restrictions
+are honored. Custom headers are limited to the original origin across redirects;
+header files avoid leaking tokens in shell history. Never send actual session
+values to agents or paste them into reports. Requests ignore environment proxies;
+`--proxy` explicitly selects HTTP/SOCKS. No TLS verification bypass is provided.
+
+Crawl is same-host, paced (`--delay`), depth/page bounded and robots-aware by
+default. `--no-robots` is explicit, not a bypass of authorization or terms.
+Next/rel=next links are prioritized; JS scroll/click/API pagination is not executed.
+`--match` restricts downloads, not traversal. Errors and remaining frontier are
+retained in the manifest. New output directories avoid collisions on reruns.
+Private/local destinations need `--allow-private` for your own services.
+`--exclude` is a URL-frontier regex; its configurable default guards common
+logout/delete/checkout GET links. An explicit empty expression disables it after
+review. This heuristic is not proof GET is side-effect-free: use a dedicated
+authorized session and inspect the frontier before crawling sensitive accounts.
+DNS checks are best effort, not a security sandbox against hostile DNS rebinding.
+
+For JS, authenticated DOM or browser challenges, use browser-proxy independently
+with your authorization. A cookie handoff can help ordinary sessions but cannot
+reproduce every browser/TLS fingerprint. This CLI never promises to bypass all
+CAPTCHAs, paywalls, permissions or account login.
+
+## Documents, images and audio
+
+```sh
+web doc report.pdf --compare other.pdf --ocr --pages 10 --report /tmp/evidence.pdf
+web doc public-report.md --online --samples 3 --limit 5
+web image photo.jpg                       # local metadata only
+web image https://public.example/logo.png # remote input, still no reverse upload
+web image public-logo.png --online -e yandex,google,bing --text-search
+web audio excerpt.mp3
+web audio --record 12 --mode hum
+web archive https://public.example/report.pdf -P '{"timestamp":"20240101"}'
+```
+
+Doc is offline by default: --online discloses sampled quotes to search engines.
+PDF/DOCX tables/ODT/PPTX and UTF-8 TXT/MD/CSV/JSON/HTML/XML/RST/TeX are supported;
+unknown binary must be exported, not decoded as garbage. Scanned PDFs require
+explicit OCR and are rendered to temporary images before Tesseract.
+These parsers are not a malware sandbox; use OS isolation for hostile files.
+Page limit 0 selects all pages and can be expensive. Comparisons provide exact byte equality
+and lexical matching spans, NOT semantic equivalence or a plagiarism verdict.
+Online links are candidates; downloaded content is checked separately. A hash
+search result alone is not proof a file leaked. Archive uses Wayback's closest
+snapshot API only; no Archive.today API or exhaustive historical search is claimed.
+
+Image uploads require --online and omit original EXIF/GPS. Providers are fragile
+unofficial adapters: failures or empty responses are not proof of absent copies.
+--text-search explicitly discloses local OCR words as a complementary text search;
+those results are not verified visual matches. GPS/device/time are local metadata.
+
+Audio uses free SongRec/Shazam; FFmpeg normalizes/records an excerpt. `hum` is a
+qualified best-effort fingerprint attempt and returns partial status: SongRec is
+not a melody recognizer and reliable free humming is not established. ACRCloud,
+paid API logic and credential loading are removed. No transcription is performed.
+
+## Hidden networks and limits
+
+Tor fetch uses SOCKS5h (managed optional profile: 127.0.0.1:19050; override
+tor_proxy for an existing external daemon, commonly 9050); once selected, redirects do not
+downgrade to direct or resolve through local DNS. I2P fetch requires an actual
+HTTP proxy (default 127.0.0.1:4444). There is no fake I2P search adapter. For
+indexed onion discovery use `web search QUERY -e ahmia -c onions`; coverage and
+uptime depend on the index and Tor. Private/unindexed content has no universal
+search engine. No tool can enumerate your complete digital footprint or prove
+absence of a leak from absent results. No arbitrary third-party kill switch
+compatibility or perfect anonymity is promised.
+
+Surface SearXNG startup is independent of Tor. `make -C ../searxng onion-up`
+explicitly starts an isolated optional onion worker; a failed Tor check does not
+crash the surface worker. `web engines -z onion` inspects it. `web search QUERY
+-e bing,ahmia` partitions those engines between backends and reports missing
+coverage rather than losing working surface results. `-z surface,onion` explicitly
+selects both, or use custom `SETTINGS['search_backends']` entries with url/socket
+fields for another SearXNG-compatible backend. Budgets apply per backend, so total
+requests grow with the number of selected backends. Protocols other than SearXNG
+still need an adapter; arbitrary services are not interchangeable by naming them.
+
+## Configuration, dependencies and lifecycle
+
+`web setup` creates ~/.config/web/config.py only if absent. SETTINGS is plain
+trusted Python, not a loader framework. WEB_CONFIG_DIR and standard XDG roots
+control configuration where supported. Existing xsearch configuration/cache were
+moved, not destroyed; dormant old .env credentials are never loaded. Defaults in
+config.py include response_max_bytes, local_max_bytes, xml_max_bytes and
+max_redirects; put overrides in SETTINGS.
+Set phone_region (for example FR) to validate national-format numbers; without
+a region hint, international numbers are preferred rather than guessing dates.
+Local SearXNG prefers its owner-only
+Unix socket to avoid host VPN TCP routing; remote endpoints remain explicit.
+
+Python libraries are resolved with bare uv add and uv.lock, never pip or manual
+dependency arrays. System dependencies are deliberately separate/shared:
+
+| Feature | Native dependency / installation guidance |
+| --- | --- |
+| Audio | ffmpeg; SongRec via Cargo: cargo install songrec --no-default-features --features ffmpeg |
+| SongRec source build | Rust/Cargo, pkg-config, Clang/libclang, libsoup-3.0-dev, ALSA/PipeWire development headers as required upstream |
+| PDF OCR | poppler-utils (pdftoppm), tesseract-ocr and desired language data |
+| Unicode PDF report | fontconfig (fc-match), fonts-dejavu-core; fpdf2 is managed by uv |
+| Backend | Docker Engine and Compose plugin; sibling searxng repository documents images/ports/volumes |
+| Tor/I2P | Your own reachable Tor SOCKS5h/I2P HTTP proxy; never silently started |
+
+Use upstream package instructions for your OS; dependency install commands evolve.
+`make deps` reports installed native capabilities and guidance rather than a rigid
+root installer. Cold builds need network or cached packages/images. Docker logs
+are bounded by the backend; web has no background daemon or persistent log file.
+JSON evidence is not silently rotated/deleted; /tmp is volatile. Explicit
+download/report destinations are user evidence and not automatically purged.
+
+`make uninstall` removes the uv tool only. `make purge` stops the sibling backend,
+trashes owned runtime paths, uninstalls the tool, and retains source/shared native
+tools. `make backend-purge CONFIRM=searxng` backs up named volumes under a private
+/tmp directory (.bak files), then removes only that stack's volumes. Review its
+Make recipe first. Never run global Docker prune. A clone is retained so reinstall
+works from either GitHub or GitLab. `make purge-all CONFIRM=searxng` sequences
+the confirmed backend backup/removal and app purge. Install from PyPI using
+uv tool install kpihx-web.
+
+All commands use an action/timestamp/status/data envelope. Formats: table, json,
+jsonl, csv, md; --report creates escaped offline HTML or a real PDF. Autosaves are
+unique 0600 JSON in /tmp/web-results; -S overrides and -N disables. Exit 2 is
+failure, exit 3 is partial/access-wall coverage, exit 0 is completion. Inspect
+non_exhaustive, budgets and source errors before drawing conclusions. Source
+quality, identity attribution and correctness require human corroboration.

kpihx_web-2.2.0/pyproject.toml

@@ -0,0 +1,61 @@
+[project]
+name = "kpihx-web"
+version = "2.2.0"
+description = "Private CLI for metasearch, verified retrieval, bounded crawling and media investigation"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.12"
+dependencies = [
+    "beautifulsoup4",
+    "defusedxml>=0.7.1",
+    "fpdf2>=2.8.9",
+    "httpx[socks]",
+    "lxml",
+    "phonenumbers>=9.0.41",
+    "picimagesearch",
+    "pillow",
+    "pypdf",
+    "python-docx",
+    "regex>=2026.9.29",
+    "requests[socks]",
+    "rich",
+    "typer",
+]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+
+[project.optional-dependencies]
+dev = []
+
+[project.scripts]
+web = "k_web.cli:main"
+
+[project.urls]
+Repository = "https://github.com/kpihx-labs/web"
+GitLab = "https://gitlab.com/kpihx-labs/web"
+Issues = "https://github.com/kpihx-labs/web/issues"
+
+[build-system]
+requires = ["uv_build>=0.10.3,<0.11.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "k_web"
+
+[tool.ruff.lint.flake8-bugbear]
+extend-immutable-calls = ["typer.Argument", "typer.Option"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff.lint]
+select = ["E4", "E7", "E9", "F", "I", "B", "UP"]
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "reportlab>=5.1.0",
+]

kpihx_web-2.2.0/src/k_web/__init__.py

@@ -0,0 +1,8 @@
+"""Local-first web investigation primitives."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("kpihx-web")
+except PackageNotFoundError:
+    __version__ = "uninstalled"

kpihx_web-2.2.0/src/k_web/audio.py

@@ -0,0 +1,65 @@
+"""Free SongRec fingerprint adapter; humming is explicitly best-effort, not STT."""
+
+import json
+import math
+import shutil
+import subprocess
+import tempfile
+from pathlib import Path
+
+from .http import WebError
+
+
+def identify(
+    path, record=None, mode="recording", duration=12.0, source="default", offset=0.0
+):
+    if mode not in {"recording", "hum"}:
+        raise WebError("Mode must be recording or hum")
+    if not math.isfinite(offset) or offset < 0 or (record is not None and offset):
+        raise WebError("Offset must be finite, nonnegative and used only with a file")
+    if not math.isfinite(duration) or (
+        record is not None and not math.isfinite(record)
+    ):
+        raise WebError("Audio durations must be finite")
+    if (
+        (path is None) == (record is None)
+        or duration <= 0
+        or (record is not None and record <= 0)
+    ):
+        raise WebError("Provide one file or --record SECONDS, with positive duration")
+    for command in ("ffmpeg", "songrec"):
+        if not shutil.which(command):
+            raise WebError(
+                f"Missing {command}; run make deps for installation guidance"
+            )
+    with tempfile.TemporaryDirectory(prefix="web-audio-") as folder:
+        sample = Path(folder) / "sample.wav"
+        args = ["ffmpeg", "-nostdin", "-v", "error"]
+        args += (
+            ["-f", "pulse", "-i", source, "-t", str(record)]
+            if record
+            else ["-ss", str(offset), "-i", str(path), "-t", str(duration)]
+        )
+        subprocess.run(
+            [*args, "-ac", "1", "-ar", "16000", str(sample)],
+            check=True,
+            capture_output=True,
+            timeout=60 + (record or 0),
+        )
+        process = subprocess.run(
+            ["songrec", "audio-file-to-recognized-song", str(sample)],
+            capture_output=True,
+            text=True,
+            timeout=30,
+            check=True,
+        )
+        result = json.loads(process.stdout)
+    return {
+        "requested_offset": offset,
+        "provider": "songrec",
+        "mode": mode,
+        "matched": bool(result.get("track")),
+        "result": result,
+        "partial": mode == "hum",
+        "capability": "Original-recording fingerprint, not query-by-humming; melody-only matches are not reliable",
+    }