webless 0.1.0__py3-none-any.whl

webless-0.1.0.dist-info/METADATA

@@ -0,0 +1,241 @@
+Metadata-Version: 2.4
+Name: webless
+Version: 0.1.0
+Summary: Web search and page fetching with no API keys: multi-engine search with rank fusion, and HTML to Markdown
+Keywords: search,web-search,scraping,duckduckgo,markdown,html-to-markdown,no-api-key
+Author: Ahmed Saqr
+Author-email: Ahmed Saqr <ahmedhassansaqr28@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Typing :: Typed
+Requires-Dist: httpx>=0.27
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/ahmedhassan456/webless
+Project-URL: Repository, https://github.com/ahmedhassan456/webless
+Project-URL: Issues, https://github.com/ahmedhassan456/webless/issues
+Description-Content-Type: text/markdown
+
+# webless
+
+Web search and page fetching for Python, with **no API keys**.
+
+`pip install webless` and the first call works. There is no account to
+create, no key to configure, no quota to track, and no paid tier — every
+back end is a public endpoint.
+
+```python
+import asyncio
+from webless import search, fetch
+
+async def main():
+    result = await search("reciprocal rank fusion")
+    for hit in result[:3]:
+        print(hit.title, "—", hit.url)
+
+    page = await fetch(result[0].url)
+    print(page.content[:500])
+
+asyncio.run(main())
+```
+
+No event loop of your own? Use `search_sync` and `fetch_sync`, which take the
+same arguments.
+
+## Install
+
+```bash
+pip install webless
+```
+
+The only dependency is `httpx`. HTML parsing, content extraction, and Markdown
+rendering are all written against the standard library, so there is no
+`lxml`, no `beautifulsoup4`, and no compiler needed to install.
+
+## `search`
+
+```python
+result = await search(
+    "sqlite write-ahead logging",
+    limit=10,
+    wide=False,
+    allowed_domains=["sqlite.org"],
+    blocked_domains=["pinterest.com"],
+    timeout=12.0,
+)
+```
+
+`search` queries several engines **concurrently** and fuses their rankings.
+The general set is DuckDuckGo, Mojeek, Bing, and Wikipedia; `wide=True` adds
+Marginalia's small-web index and Hacker News, which helps on niche or
+discussion-oriented queries and costs some latency on ordinary ones.
+
+Fusion is Reciprocal Rank Fusion: a result scores `Σ 1 / (60 + rank)` over the
+engines that returned it. Rank is all it needs, which is the point — the
+engines disagree about scoring and none exposes a comparable number, but they
+all produce an ordered list, and a page several independent indexes rank
+highly is a better answer than any one index's favourite. The fused ranking is
+then reweighted by how much of the query actually appears in each result, so
+an engine answering loosely cannot outrank one answering the question.
+
+`allowed_domains` and `blocked_domains` filter on host, and an entry covers
+its subdomains: `python.org` keeps `docs.python.org`.
+
+### Results
+
+`SearchResult` iterates, indexes, and measures like the list of hits it
+carries, and also tells you what happened:
+
+```python
+result.hits        # list[SearchHit]
+result.succeeded   # ["duckduckgo", "bing"]
+result.failed      # {"mojeek": "returned a page with no results, likely a soft block."}
+result.ok          # at least one engine answered
+
+hit.title, hit.url, hit.snippet, hit.host, hit.engines, hit.score
+```
+
+**A blocked engine is reported, not raised.** Public endpoints rate-limit,
+soft-block, and serve challenge pages, and they do it differently from
+different IPs. Every engine is given the same deadline and none can hold up
+the others: a failure is recorded against its name and the rest of the ranking
+stands. Only when all of them fail do you get an empty ranking — with the
+reasons attached.
+
+A soft block is detected rather than mistaken for an empty index. A results
+page that returns HTTP 200 and parses to zero rows, or answers with a
+challenge status, is reported as that engine failing.
+
+## `fetch`
+
+```python
+page = await fetch(
+    "https://docs.python.org/3/library/asyncio.html",
+    format="markdown",
+    full_page=False,
+    max_chars=None,
+    offset=0,
+)
+```
+
+`fetch` returns the page's readable content, not its markup. A page's HTML is
+mostly navigation, scripts, and layout, so the default trims to the article
+body and renders headings, lists, code blocks, tables, and links as Markdown —
+small enough to read or feed to a model, structured enough to still quote from
+and follow links out of. Relative links are absolutised against the final URL.
+
+- `format="markdown"` — main content as Markdown (default)
+- `format="text"` — plain text, no markup
+- `format="raw"` — the untouched response body
+- `full_page=True` — keep navigation, sidebars, and footers, for when the part
+  you want *is* the chrome (a docs index, a link list)
+- `max_chars` / `offset` — read a long page in sections; `page.truncated` says
+  whether anything was left behind
+
+JSON responses are returned pretty-printed.
+
+```python
+page.content       # the rendered text
+page.title         # <og:title>, else <title>
+page.description   # meta description
+page.url           # what you asked for
+page.final_url     # where you ended up
+page.redirected    # whether those differ
+page.status, page.content_type, page.host, page.truncated
+```
+
+Already holding a response from your own client or a cache? `render(response,
+format=..., full_page=...)` does the same conversion with no network call.
+
+## Not blocking
+
+Two senses, both deliberate.
+
+**Nothing blocks anything else.** Every call is async with a hard timeout. A
+search is several concurrent requests, so a slow engine delays its own result
+and nothing more.
+
+**Requests are hard to block.** Hosts refuse on fingerprint reputation, so a
+retry after a 403 or 429 presents a *different* browser user agent rather than
+repeating the one just refused — which clears most transient blocks on its
+own. Retries back off exponentially with jitter.
+
+## Not reachable inward
+
+A URL that resolves to a private, loopback, link-local, or reserved address is
+refused before the request is sent — and again on every redirect hop, since
+redirects are followed by hand for exactly that reason. A URL from a search
+result, a user, or a model cannot be steered into your own network.
+
+```python
+await fetch("http://169.254.169.254/latest/meta-data/")
+# WebError: ... resolves to a private or loopback address; refusing to fetch it
+```
+
+Response bodies are capped, and redirect chains are limited to five hops.
+
+## Command line
+
+```bash
+webless search "structured concurrency python" -n 5
+webless search "rust async traits" --wide --allow rust-lang.org
+webless fetch https://peps.python.org/pep-3156/ > pep.md
+webless fetch https://api.github.com/repos/python/cpython --json | jq .status
+```
+
+`--json` on either subcommand prints the full record, so it composes with
+`jq`. `search` exits non-zero when every engine failed.
+
+## Engines
+
+| engine | how | notes |
+|---|---|---|
+| DuckDuckGo | scrape | lite view, falls back to the html view |
+| Mojeek | scrape | independent index |
+| Bing | RSS, then scrape | tracker URLs decoded |
+| Wikipedia | OpenSearch API | |
+| Marginalia | public API | `wide` only; small-web index |
+| Hacker News | Algolia API | `wide` only; discussion |
+
+Pick your own set with `engines=`:
+
+```python
+from webless import search, DuckDuckGoEngine, MojeekEngine
+
+result = await search("query", engines=(DuckDuckGoEngine, MojeekEngine))
+```
+
+A custom engine is any subclass of `Engine` with a `name` and an
+`async search(query, limit, timeout) -> list[SearchHit]`.
+
+## Honest caveats
+
+These are public endpoints being read by a program, which has consequences
+worth knowing before you build on them:
+
+- **Availability varies by IP.** Mojeek soft-blocks some address ranges
+  outright; Marginalia's public key is shared and rate-limits. This is why
+  failures are per-engine and reported rather than fatal.
+- **Bing occasionally serves results for an unrelated query** — the same
+  response carries the right query in its metadata and the wrong results in
+  its body. The relevance reweighting demotes them, but on a query where Bing
+  is the only engine answering you may see them.
+- **Scrapers track markup.** When an engine changes its results page, its
+  parser needs updating; the soft-block detection turns that into a reported
+  failure rather than silence.
+
+## Requirements
+
+Python 3.10+. `httpx` 0.27+.
+
+## License
+
+MIT

webless-0.1.0.dist-info/RECORD

@@ -0,0 +1,14 @@
+webless/__init__.py,sha256=4Kva_7KtdQJ5yiehKZQU4OnoziQSbDhQsWLGPZpL9Z8,2056
+webless/_engines.py,sha256=PHNXWxifdwYb3q219DSpWWi8-5a5-lbGoeRbGonbbS4,21497
+webless/_fetch.py,sha256=A2vnwKJAzzapZETGbUJlYSW8XjX6UJBANRKawdZHXRs,4501
+webless/_html.py,sha256=Lj3bVVWsV2PPDfJpvaw7h2vlAePVU17lPUjwFt1q1Jw,16995
+webless/_http.py,sha256=GJ8gd05xXuqo9NGdQQBYpBASS5apW49iFF5KLN_-FLk,9568
+webless/_search.py,sha256=nL19gri01A9lWbUMSBRt9PZ0zIMM574YIm8RkU-pRlU,2771
+webless/_sync.py,sha256=OsioPbu1koSWzHlxk1Li4D8kQkW1u99GDql5LZWtI5k,1404
+webless/cli.py,sha256=HH_3GJ9iGq3E-TEJ05rbmbixC_4mSexYOdMfdCXcJfw,4128
+webless/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+webless-0.1.0.dist-info/licenses/LICENSE,sha256=oAwCYlBHADbqgD77B-tz1XlWRsrvn_0qcSBHCkZROjg,1067
+webless-0.1.0.dist-info/WHEEL,sha256=0ksFqNncQTsCUyOQQNfyYN3tci86LvOLl92eMRFTR70,81
+webless-0.1.0.dist-info/entry_points.txt,sha256=KiV26Yf63G_wDh80NkTxDQ48UDKH6Fy1tQU2FfEz38Y,46
+webless-0.1.0.dist-info/METADATA,sha256=wFTcd-sFaEedA05aM69WMh0BXAH3TyPYpo2niBmdBaU,8725
+webless-0.1.0.dist-info/RECORD,,

webless-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: uv 0.11.22
+Root-Is-Purelib: true
+Tag: py3-none-any

webless-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+webless = webless.cli:main
+

webless-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ahmed Saqr
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.