webless 0.1.0__tar.gz

webless-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ahmed Saqr
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

webless-0.1.0/PKG-INFO

@@ -0,0 +1,241 @@
+Metadata-Version: 2.4
+Name: webless
+Version: 0.1.0
+Summary: Web search and page fetching with no API keys: multi-engine search with rank fusion, and HTML to Markdown
+Keywords: search,web-search,scraping,duckduckgo,markdown,html-to-markdown,no-api-key
+Author: Ahmed Saqr
+Author-email: Ahmed Saqr <ahmedhassansaqr28@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Typing :: Typed
+Requires-Dist: httpx>=0.27
+Requires-Python: >=3.10
+Project-URL: Homepage, https://github.com/ahmedhassan456/webless
+Project-URL: Repository, https://github.com/ahmedhassan456/webless
+Project-URL: Issues, https://github.com/ahmedhassan456/webless/issues
+Description-Content-Type: text/markdown
+
+# webless
+
+Web search and page fetching for Python, with **no API keys**.
+
+`pip install webless` and the first call works. There is no account to
+create, no key to configure, no quota to track, and no paid tier — every
+back end is a public endpoint.
+
+```python
+import asyncio
+from webless import search, fetch
+
+async def main():
+    result = await search("reciprocal rank fusion")
+    for hit in result[:3]:
+        print(hit.title, "—", hit.url)
+
+    page = await fetch(result[0].url)
+    print(page.content[:500])
+
+asyncio.run(main())
+```
+
+No event loop of your own? Use `search_sync` and `fetch_sync`, which take the
+same arguments.
+
+## Install
+
+```bash
+pip install webless
+```
+
+The only dependency is `httpx`. HTML parsing, content extraction, and Markdown
+rendering are all written against the standard library, so there is no
+`lxml`, no `beautifulsoup4`, and no compiler needed to install.
+
+## `search`
+
+```python
+result = await search(
+    "sqlite write-ahead logging",
+    limit=10,
+    wide=False,
+    allowed_domains=["sqlite.org"],
+    blocked_domains=["pinterest.com"],
+    timeout=12.0,
+)
+```
+
+`search` queries several engines **concurrently** and fuses their rankings.
+The general set is DuckDuckGo, Mojeek, Bing, and Wikipedia; `wide=True` adds
+Marginalia's small-web index and Hacker News, which helps on niche or
+discussion-oriented queries and costs some latency on ordinary ones.
+
+Fusion is Reciprocal Rank Fusion: a result scores `Σ 1 / (60 + rank)` over the
+engines that returned it. Rank is all it needs, which is the point — the
+engines disagree about scoring and none exposes a comparable number, but they
+all produce an ordered list, and a page several independent indexes rank
+highly is a better answer than any one index's favourite. The fused ranking is
+then reweighted by how much of the query actually appears in each result, so
+an engine answering loosely cannot outrank one answering the question.
+
+`allowed_domains` and `blocked_domains` filter on host, and an entry covers
+its subdomains: `python.org` keeps `docs.python.org`.
+
+### Results
+
+`SearchResult` iterates, indexes, and measures like the list of hits it
+carries, and also tells you what happened:
+
+```python
+result.hits        # list[SearchHit]
+result.succeeded   # ["duckduckgo", "bing"]
+result.failed      # {"mojeek": "returned a page with no results, likely a soft block."}
+result.ok          # at least one engine answered
+
+hit.title, hit.url, hit.snippet, hit.host, hit.engines, hit.score
+```
+
+**A blocked engine is reported, not raised.** Public endpoints rate-limit,
+soft-block, and serve challenge pages, and they do it differently from
+different IPs. Every engine is given the same deadline and none can hold up
+the others: a failure is recorded against its name and the rest of the ranking
+stands. Only when all of them fail do you get an empty ranking — with the
+reasons attached.
+
+A soft block is detected rather than mistaken for an empty index. A results
+page that returns HTTP 200 and parses to zero rows, or answers with a
+challenge status, is reported as that engine failing.
+
+## `fetch`
+
+```python
+page = await fetch(
+    "https://docs.python.org/3/library/asyncio.html",
+    format="markdown",
+    full_page=False,
+    max_chars=None,
+    offset=0,
+)
+```
+
+`fetch` returns the page's readable content, not its markup. A page's HTML is
+mostly navigation, scripts, and layout, so the default trims to the article
+body and renders headings, lists, code blocks, tables, and links as Markdown —
+small enough to read or feed to a model, structured enough to still quote from
+and follow links out of. Relative links are absolutised against the final URL.
+
+- `format="markdown"` — main content as Markdown (default)
+- `format="text"` — plain text, no markup
+- `format="raw"` — the untouched response body
+- `full_page=True` — keep navigation, sidebars, and footers, for when the part
+  you want *is* the chrome (a docs index, a link list)
+- `max_chars` / `offset` — read a long page in sections; `page.truncated` says
+  whether anything was left behind
+
+JSON responses are returned pretty-printed.
+
+```python
+page.content       # the rendered text
+page.title         # <og:title>, else <title>
+page.description   # meta description
+page.url           # what you asked for
+page.final_url     # where you ended up
+page.redirected    # whether those differ
+page.status, page.content_type, page.host, page.truncated
+```
+
+Already holding a response from your own client or a cache? `render(response,
+format=..., full_page=...)` does the same conversion with no network call.
+
+## Not blocking
+
+Two senses, both deliberate.
+
+**Nothing blocks anything else.** Every call is async with a hard timeout. A
+search is several concurrent requests, so a slow engine delays its own result
+and nothing more.
+
+**Requests are hard to block.** Hosts refuse on fingerprint reputation, so a
+retry after a 403 or 429 presents a *different* browser user agent rather than
+repeating the one just refused — which clears most transient blocks on its
+own. Retries back off exponentially with jitter.
+
+## Not reachable inward
+
+A URL that resolves to a private, loopback, link-local, or reserved address is
+refused before the request is sent — and again on every redirect hop, since
+redirects are followed by hand for exactly that reason. A URL from a search
+result, a user, or a model cannot be steered into your own network.
+
+```python
+await fetch("http://169.254.169.254/latest/meta-data/")
+# WebError: ... resolves to a private or loopback address; refusing to fetch it
+```
+
+Response bodies are capped, and redirect chains are limited to five hops.
+
+## Command line
+
+```bash
+webless search "structured concurrency python" -n 5
+webless search "rust async traits" --wide --allow rust-lang.org
+webless fetch https://peps.python.org/pep-3156/ > pep.md
+webless fetch https://api.github.com/repos/python/cpython --json | jq .status
+```
+
+`--json` on either subcommand prints the full record, so it composes with
+`jq`. `search` exits non-zero when every engine failed.
+
+## Engines
+
+| engine | how | notes |
+|---|---|---|
+| DuckDuckGo | scrape | lite view, falls back to the html view |
+| Mojeek | scrape | independent index |
+| Bing | RSS, then scrape | tracker URLs decoded |
+| Wikipedia | OpenSearch API | |
+| Marginalia | public API | `wide` only; small-web index |
+| Hacker News | Algolia API | `wide` only; discussion |
+
+Pick your own set with `engines=`:
+
+```python
+from webless import search, DuckDuckGoEngine, MojeekEngine
+
+result = await search("query", engines=(DuckDuckGoEngine, MojeekEngine))
+```
+
+A custom engine is any subclass of `Engine` with a `name` and an
+`async search(query, limit, timeout) -> list[SearchHit]`.
+
+## Honest caveats
+
+These are public endpoints being read by a program, which has consequences
+worth knowing before you build on them:
+
+- **Availability varies by IP.** Mojeek soft-blocks some address ranges
+  outright; Marginalia's public key is shared and rate-limits. This is why
+  failures are per-engine and reported rather than fatal.
+- **Bing occasionally serves results for an unrelated query** — the same
+  response carries the right query in its metadata and the wrong results in
+  its body. The relevance reweighting demotes them, but on a query where Bing
+  is the only engine answering you may see them.
+- **Scrapers track markup.** When an engine changes its results page, its
+  parser needs updating; the soft-block detection turns that into a reported
+  failure rather than silence.
+
+## Requirements
+
+Python 3.10+. `httpx` 0.27+.
+
+## License
+
+MIT

webless-0.1.0/README.md

@@ -0,0 +1,215 @@
+# webless
+
+Web search and page fetching for Python, with **no API keys**.
+
+`pip install webless` and the first call works. There is no account to
+create, no key to configure, no quota to track, and no paid tier — every
+back end is a public endpoint.
+
+```python
+import asyncio
+from webless import search, fetch
+
+async def main():
+    result = await search("reciprocal rank fusion")
+    for hit in result[:3]:
+        print(hit.title, "—", hit.url)
+
+    page = await fetch(result[0].url)
+    print(page.content[:500])
+
+asyncio.run(main())
+```
+
+No event loop of your own? Use `search_sync` and `fetch_sync`, which take the
+same arguments.
+
+## Install
+
+```bash
+pip install webless
+```
+
+The only dependency is `httpx`. HTML parsing, content extraction, and Markdown
+rendering are all written against the standard library, so there is no
+`lxml`, no `beautifulsoup4`, and no compiler needed to install.
+
+## `search`
+
+```python
+result = await search(
+    "sqlite write-ahead logging",
+    limit=10,
+    wide=False,
+    allowed_domains=["sqlite.org"],
+    blocked_domains=["pinterest.com"],
+    timeout=12.0,
+)
+```
+
+`search` queries several engines **concurrently** and fuses their rankings.
+The general set is DuckDuckGo, Mojeek, Bing, and Wikipedia; `wide=True` adds
+Marginalia's small-web index and Hacker News, which helps on niche or
+discussion-oriented queries and costs some latency on ordinary ones.
+
+Fusion is Reciprocal Rank Fusion: a result scores `Σ 1 / (60 + rank)` over the
+engines that returned it. Rank is all it needs, which is the point — the
+engines disagree about scoring and none exposes a comparable number, but they
+all produce an ordered list, and a page several independent indexes rank
+highly is a better answer than any one index's favourite. The fused ranking is
+then reweighted by how much of the query actually appears in each result, so
+an engine answering loosely cannot outrank one answering the question.
+
+`allowed_domains` and `blocked_domains` filter on host, and an entry covers
+its subdomains: `python.org` keeps `docs.python.org`.
+
+### Results
+
+`SearchResult` iterates, indexes, and measures like the list of hits it
+carries, and also tells you what happened:
+
+```python
+result.hits        # list[SearchHit]
+result.succeeded   # ["duckduckgo", "bing"]
+result.failed      # {"mojeek": "returned a page with no results, likely a soft block."}
+result.ok          # at least one engine answered
+
+hit.title, hit.url, hit.snippet, hit.host, hit.engines, hit.score
+```
+
+**A blocked engine is reported, not raised.** Public endpoints rate-limit,
+soft-block, and serve challenge pages, and they do it differently from
+different IPs. Every engine is given the same deadline and none can hold up
+the others: a failure is recorded against its name and the rest of the ranking
+stands. Only when all of them fail do you get an empty ranking — with the
+reasons attached.
+
+A soft block is detected rather than mistaken for an empty index. A results
+page that returns HTTP 200 and parses to zero rows, or answers with a
+challenge status, is reported as that engine failing.
+
+## `fetch`
+
+```python
+page = await fetch(
+    "https://docs.python.org/3/library/asyncio.html",
+    format="markdown",
+    full_page=False,
+    max_chars=None,
+    offset=0,
+)
+```
+
+`fetch` returns the page's readable content, not its markup. A page's HTML is
+mostly navigation, scripts, and layout, so the default trims to the article
+body and renders headings, lists, code blocks, tables, and links as Markdown —
+small enough to read or feed to a model, structured enough to still quote from
+and follow links out of. Relative links are absolutised against the final URL.
+
+- `format="markdown"` — main content as Markdown (default)
+- `format="text"` — plain text, no markup
+- `format="raw"` — the untouched response body
+- `full_page=True` — keep navigation, sidebars, and footers, for when the part
+  you want *is* the chrome (a docs index, a link list)
+- `max_chars` / `offset` — read a long page in sections; `page.truncated` says
+  whether anything was left behind
+
+JSON responses are returned pretty-printed.
+
+```python
+page.content       # the rendered text
+page.title         # <og:title>, else <title>
+page.description   # meta description
+page.url           # what you asked for
+page.final_url     # where you ended up
+page.redirected    # whether those differ
+page.status, page.content_type, page.host, page.truncated
+```
+
+Already holding a response from your own client or a cache? `render(response,
+format=..., full_page=...)` does the same conversion with no network call.
+
+## Not blocking
+
+Two senses, both deliberate.
+
+**Nothing blocks anything else.** Every call is async with a hard timeout. A
+search is several concurrent requests, so a slow engine delays its own result
+and nothing more.
+
+**Requests are hard to block.** Hosts refuse on fingerprint reputation, so a
+retry after a 403 or 429 presents a *different* browser user agent rather than
+repeating the one just refused — which clears most transient blocks on its
+own. Retries back off exponentially with jitter.
+
+## Not reachable inward
+
+A URL that resolves to a private, loopback, link-local, or reserved address is
+refused before the request is sent — and again on every redirect hop, since
+redirects are followed by hand for exactly that reason. A URL from a search
+result, a user, or a model cannot be steered into your own network.
+
+```python
+await fetch("http://169.254.169.254/latest/meta-data/")
+# WebError: ... resolves to a private or loopback address; refusing to fetch it
+```
+
+Response bodies are capped, and redirect chains are limited to five hops.
+
+## Command line
+
+```bash
+webless search "structured concurrency python" -n 5
+webless search "rust async traits" --wide --allow rust-lang.org
+webless fetch https://peps.python.org/pep-3156/ > pep.md
+webless fetch https://api.github.com/repos/python/cpython --json | jq .status
+```
+
+`--json` on either subcommand prints the full record, so it composes with
+`jq`. `search` exits non-zero when every engine failed.
+
+## Engines
+
+| engine | how | notes |
+|---|---|---|
+| DuckDuckGo | scrape | lite view, falls back to the html view |
+| Mojeek | scrape | independent index |
+| Bing | RSS, then scrape | tracker URLs decoded |
+| Wikipedia | OpenSearch API | |
+| Marginalia | public API | `wide` only; small-web index |
+| Hacker News | Algolia API | `wide` only; discussion |
+
+Pick your own set with `engines=`:
+
+```python
+from webless import search, DuckDuckGoEngine, MojeekEngine
+
+result = await search("query", engines=(DuckDuckGoEngine, MojeekEngine))
+```
+
+A custom engine is any subclass of `Engine` with a `name` and an
+`async search(query, limit, timeout) -> list[SearchHit]`.
+
+## Honest caveats
+
+These are public endpoints being read by a program, which has consequences
+worth knowing before you build on them:
+
+- **Availability varies by IP.** Mojeek soft-blocks some address ranges
+  outright; Marginalia's public key is shared and rate-limits. This is why
+  failures are per-engine and reported rather than fatal.
+- **Bing occasionally serves results for an unrelated query** — the same
+  response carries the right query in its metadata and the wrong results in
+  its body. The relevance reweighting demotes them, but on a query where Bing
+  is the only engine answering you may see them.
+- **Scrapers track markup.** When an engine changes its results page, its
+  parser needs updating; the soft-block detection turns that into a reported
+  failure rather than silence.
+
+## Requirements
+
+Python 3.10+. `httpx` 0.27+.
+
+## License
+
+MIT

webless-0.1.0/pyproject.toml

@@ -0,0 +1,53 @@
+[project]
+name = "webless"
+version = "0.1.0"
+description = "Web search and page fetching with no API keys: multi-engine search with rank fusion, and HTML to Markdown"
+readme = "README.md"
+authors = [
+    { name = "Ahmed Saqr", email = "ahmedhassansaqr28@gmail.com" }
+]
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = ["search", "web-search", "scraping", "duckduckgo", "markdown", "html-to-markdown", "no-api-key"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Topic :: Internet :: WWW/HTTP",
+    "Topic :: Text Processing :: Markup :: HTML",
+    "Typing :: Typed",
+]
+dependencies = [
+    "httpx>=0.27",
+]
+
+[project.scripts]
+webless = "webless.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/ahmedhassan456/webless"
+Repository = "https://github.com/ahmedhassan456/webless"
+Issues = "https://github.com/ahmedhassan456/webless/issues"
+
+[build-system]
+requires = ["uv_build>=0.11.22,<0.12.0"]
+build-backend = "uv_build"
+
+[tool.uv.build-backend]
+module-name = "webless"
+module-root = "src"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+    "pytest-asyncio>=0.24",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]

webless-0.1.0/src/webless/__init__.py

@@ -0,0 +1,85 @@
+"""webless: web search and page fetching with no API keys.
+
+Two calls. `search` queries several public search engines at once and fuses
+their rankings; `fetch` retrieves a URL and returns its readable content as
+Markdown. Nothing has to be configured first -- there is no account, no key,
+and no per-request quota to manage.
+
+Example:
+    import asyncio
+    from webless import search, fetch
+
+    async def main():
+        result = await search("reciprocal rank fusion")
+        for hit in result[:3]:
+            print(hit.title, hit.url)
+
+        page = await fetch(result[0].url)
+        print(page.content[:500])
+
+    asyncio.run(main())
+
+The same two calls exist as `search_sync` and `fetch_sync` for code with no
+event loop, and as a `webless` command on the terminal.
+"""
+
+from ._engines import (
+    ENGINES_BY_NAME,
+    GENERAL_ENGINES,
+    WIDE_ENGINES,
+    BingEngine,
+    DuckDuckGoEngine,
+    Engine,
+    HackerNewsEngine,
+    MarginaliaEngine,
+    MojeekEngine,
+    WikipediaEngine,
+    alignment,
+    fuse,
+    query_terms,
+    rank_by_relevance,
+)
+from ._html import html_to_markdown, main_content, page_description, page_title, parse_html
+from ._http import USER_AGENTS, Response, WebError, fetch_url, host_of, normalize_url
+from ._sync import fetch_sync, search_sync
+from ._fetch import Page, fetch, render
+from ._search import SearchHit, SearchResult, search
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "search",
+    "fetch",
+    "search_sync",
+    "fetch_sync",
+    "SearchResult",
+    "SearchHit",
+    "Page",
+    "WebError",
+    "render",
+    "Engine",
+    "DuckDuckGoEngine",
+    "MojeekEngine",
+    "BingEngine",
+    "WikipediaEngine",
+    "MarginaliaEngine",
+    "HackerNewsEngine",
+    "GENERAL_ENGINES",
+    "WIDE_ENGINES",
+    "ENGINES_BY_NAME",
+    "fuse",
+    "rank_by_relevance",
+    "query_terms",
+    "alignment",
+    "fetch_url",
+    "Response",
+    "normalize_url",
+    "host_of",
+    "USER_AGENTS",
+    "parse_html",
+    "html_to_markdown",
+    "main_content",
+    "page_title",
+    "page_description",
+    "__version__",
+]