webget-cli 0.14.0__tar.gz → 0.16.0__tar.gz

{webget_cli-0.14.0 → webget_cli-0.16.0}/PKG-INFO

@@ -1,16 +1,15 @@
 Metadata-Version: 2.4
 Name: webget-cli
-Version: 0.14.0
+Version: 0.16.0
 Summary: Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys.
 Author-email: David Tarigan <tarigansdavid@gmail.com>
-License: Apache-2.0
+License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/DavidPandleton/webget
 Project-URL: Repository, https://github.com/DavidPandleton/webget
 Project-URL: Changelog, https://github.com/DavidPandleton/webget/blob/main/CHANGELOG.md
 Keywords: scraping,crawling,cli,search,duckduckgo,http
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -103,7 +102,7 @@ webget --help
 
 ### Browser runtime (optional)
 
-Crawl4AI drives a Playwright Chromium. `pip install "webget-cli[browser]"`
+Crawl4AI drives a Playwright browser. `pip install "webget-cli[browser]"`
 installs the Python packages; the browser binary itself is downloaded
 separately:
 
@@ -111,6 +110,41 @@ separately:
 python -m playwright install chromium
 ```
 
+**You may not need that download.** Before reaching for Playwright's bundled
+Chromium, webget looks for a Chromium-family browser already installed on the
+machine and uses it when it can actually be driven. Run `webget doctor` to see
+what was found and what will be used:
+
+```bash
+webget doctor
+```
+
+Resolution order (first match wins):
+
+| Step | Source | Notes |
+|---|---|---|
+| 1 | `WEBGET_BROWSER_CDP` | Attach to a running browser over CDP. The only route that works for browsers with no Playwright channel. |
+| 2 | `WEBGET_BROWSER_CHANNEL` | A Playwright channel: `chrome`, `msedge`, or `chromium`. |
+| 3 | `WEBGET_BROWSER_PATH` | An explicit binary. Accepted, but see the limitation below. |
+| 4 | Auto-detected browser | Chrome, Edge, Chromium, Brave, Vivaldi or Opera found on the machine. |
+| 5 | Playwright bundled Chromium | The fallback when nothing usable is found. |
+
+Two limitations worth knowing, both reported honestly by `webget doctor`
+instead of failing quietly:
+
+- **Brave, Vivaldi and Opera cannot be launched by crawl4ai.** Playwright has no
+  channel for them, and crawl4ai 0.9.2 does not forward `executable_path` to
+  Playwright, so there is no path to the binary. They are detected and reported
+  as NOT USABLE. The workaround is CDP: start the browser with
+  `--remote-debugging-port=9222`, then set
+  `WEBGET_BROWSER_CDP=http://127.0.0.1:9222`.
+- **CDP is opt-in on purpose.** Attaching to a browser you are already logged
+  into mixes your personal session cookies into crawl output, so webget never
+  auto-detects an open debugging port.
+
+System Firefox is never selected: Playwright drives Firefox over WebDriver BiDi
+with its own patched build.
+
 Without the browser extra, `webget` still works for search and plain HTTP
 fetches. A fetch that needs the browser (JS rendering, `--profile` sessions,
 `login`) prints a clear warning telling you how to install it.
@@ -132,11 +166,27 @@ webget s "rust async runtime" --json      # machine-readable, includes engine pr
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
+webget crawl https://example.com crawl.db --limit 20 --json  # resumable local crawl
 webget fetch https://example.com --json   # machine-readable result
 ```
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+### Resumable local crawl
+
+The crawler is local and bounded: it uses SQLite for durable frontier state
+and page results, stays on the seed hostname, and can resume after a process
+restart. It does not require Docker, Redis, Postgres, or a browser.
+
+```bash
+webget crawl https://example.com ./crawl.db --limit 20 --timeout 20 --json
+```
+
+The same operation is available through Python as `webget.crawl_site(...)` and
+through MCP as the `crawl` tool. Results can also be exported with the Python
+API to JSONL or Markdown. Stale in-progress leases are recovered using a
+bounded timeout.
+
 ## Search engines
 
 webget searches through `ddgs`, a metasearch library that aggregates several
@@ -161,6 +211,31 @@ warning: unknown search engine(s): bogus - using auto
 Unknown names degrade to `auto` with a warning instead of failing, so a
 typo never kills a search.
 
+### Optional SearXNG provider
+
+SearXNG is not bundled or required. Point the optional HTTP adapter at an
+existing instance using an environment variable:
+
+```bash
+export WEBGET_SEARXNG_URL=http://localhost:8080
+```
+
+Then use the provider from Python:
+
+```python
+from webget import SearxngSearchProvider
+from webget.search import search_with_provenance
+
+provider = SearxngSearchProvider()
+results, provenance = search_with_provenance(
+    "query", n=5, engine="searxng", provider=provider
+)
+```
+
+The adapter calls `/search?format=json`, normalizes result fields, and keeps
+the usual per-call provenance. Live availability depends on the configured
+SearXNG instance; the core package does not start one.
+
 **Failover is automatic.** If the engine you named fails, or returns zero
 results, webget tries the remaining engines until a time budget is spent
 (15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
@@ -347,7 +422,11 @@ standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 ### Breaking change in 0.13.0: search returns an object, not a list
 
 `search` and `search_fetch` used to return a bare JSON array of results.
-They now return an object, because a list has nowhere to carry provenance:
+They now return an object, because a list has nowhere to carry provenance.
+The MCP result-count argument is `limit` (default 5 for `search`, 3 for
+`search_fetch`), matching the CLI vocabulary. The legacy MCP argument `n`
+remains accepted as an explicit compatibility alias; when both are supplied,
+`n` wins. The Python API continues to use `n` for backward compatibility.
 
 ```jsonc
 // <= 0.12.1  ->  a list

{webget_cli-0.14.0 → webget_cli-0.16.0}/README.md

@@ -66,7 +66,7 @@ webget --help
 
 ### Browser runtime (optional)
 
-Crawl4AI drives a Playwright Chromium. `pip install "webget-cli[browser]"`
+Crawl4AI drives a Playwright browser. `pip install "webget-cli[browser]"`
 installs the Python packages; the browser binary itself is downloaded
 separately:
 
@@ -74,6 +74,41 @@ separately:
 python -m playwright install chromium
 ```
 
+**You may not need that download.** Before reaching for Playwright's bundled
+Chromium, webget looks for a Chromium-family browser already installed on the
+machine and uses it when it can actually be driven. Run `webget doctor` to see
+what was found and what will be used:
+
+```bash
+webget doctor
+```
+
+Resolution order (first match wins):
+
+| Step | Source | Notes |
+|---|---|---|
+| 1 | `WEBGET_BROWSER_CDP` | Attach to a running browser over CDP. The only route that works for browsers with no Playwright channel. |
+| 2 | `WEBGET_BROWSER_CHANNEL` | A Playwright channel: `chrome`, `msedge`, or `chromium`. |
+| 3 | `WEBGET_BROWSER_PATH` | An explicit binary. Accepted, but see the limitation below. |
+| 4 | Auto-detected browser | Chrome, Edge, Chromium, Brave, Vivaldi or Opera found on the machine. |
+| 5 | Playwright bundled Chromium | The fallback when nothing usable is found. |
+
+Two limitations worth knowing, both reported honestly by `webget doctor`
+instead of failing quietly:
+
+- **Brave, Vivaldi and Opera cannot be launched by crawl4ai.** Playwright has no
+  channel for them, and crawl4ai 0.9.2 does not forward `executable_path` to
+  Playwright, so there is no path to the binary. They are detected and reported
+  as NOT USABLE. The workaround is CDP: start the browser with
+  `--remote-debugging-port=9222`, then set
+  `WEBGET_BROWSER_CDP=http://127.0.0.1:9222`.
+- **CDP is opt-in on purpose.** Attaching to a browser you are already logged
+  into mixes your personal session cookies into crawl output, so webget never
+  auto-detects an open debugging port.
+
+System Firefox is never selected: Playwright drives Firefox over WebDriver BiDi
+with its own patched build.
+
 Without the browser extra, `webget` still works for search and plain HTTP
 fetches. A fetch that needs the browser (JS rendering, `--profile` sessions,
 `login`) prints a clear warning telling you how to install it.
@@ -95,11 +130,27 @@ webget s "rust async runtime" --json      # machine-readable, includes engine pr
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
+webget crawl https://example.com crawl.db --limit 20 --json  # resumable local crawl
 webget fetch https://example.com --json   # machine-readable result
 ```
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+### Resumable local crawl
+
+The crawler is local and bounded: it uses SQLite for durable frontier state
+and page results, stays on the seed hostname, and can resume after a process
+restart. It does not require Docker, Redis, Postgres, or a browser.
+
+```bash
+webget crawl https://example.com ./crawl.db --limit 20 --timeout 20 --json
+```
+
+The same operation is available through Python as `webget.crawl_site(...)` and
+through MCP as the `crawl` tool. Results can also be exported with the Python
+API to JSONL or Markdown. Stale in-progress leases are recovered using a
+bounded timeout.
+
 ## Search engines
 
 webget searches through `ddgs`, a metasearch library that aggregates several
@@ -124,6 +175,31 @@ warning: unknown search engine(s): bogus - using auto
 Unknown names degrade to `auto` with a warning instead of failing, so a
 typo never kills a search.
 
+### Optional SearXNG provider
+
+SearXNG is not bundled or required. Point the optional HTTP adapter at an
+existing instance using an environment variable:
+
+```bash
+export WEBGET_SEARXNG_URL=http://localhost:8080
+```
+
+Then use the provider from Python:
+
+```python
+from webget import SearxngSearchProvider
+from webget.search import search_with_provenance
+
+provider = SearxngSearchProvider()
+results, provenance = search_with_provenance(
+    "query", n=5, engine="searxng", provider=provider
+)
+```
+
+The adapter calls `/search?format=json`, normalizes result fields, and keeps
+the usual per-call provenance. Live availability depends on the configured
+SearXNG instance; the core package does not start one.
+
 **Failover is automatic.** If the engine you named fails, or returns zero
 results, webget tries the remaining engines until a time budget is spent
 (15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
@@ -310,7 +386,11 @@ standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 ### Breaking change in 0.13.0: search returns an object, not a list
 
 `search` and `search_fetch` used to return a bare JSON array of results.
-They now return an object, because a list has nowhere to carry provenance:
+They now return an object, because a list has nowhere to carry provenance.
+The MCP result-count argument is `limit` (default 5 for `search`, 3 for
+`search_fetch`), matching the CLI vocabulary. The legacy MCP argument `n`
+remains accepted as an explicit compatibility alias; when both are supplied,
+`n` wins. The Python API continues to use `n` for backward compatibility.
 
 ```jsonc
 // <= 0.12.1  ->  a list

{webget_cli-0.14.0 → webget_cli-0.16.0}/pyproject.toml

@@ -4,17 +4,16 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "webget-cli"
-version = "0.14.0"
+version = "0.16.0"
 description = "Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys."
 readme = "README.md"
 requires-python = ">=3.11"
-license = { text = "Apache-2.0" }
+license = "Apache-2.0"
 authors = [{ name = "David Tarigan", email = "tarigansdavid@gmail.com" }]
 keywords = ["scraping", "crawling", "cli", "search", "duckduckgo", "http"]
 classifiers = [
     "Environment :: Console",
     "Intended Audience :: Developers",
-    "License :: OSI Approved :: Apache Software License",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",

{webget_cli-0.14.0 → webget_cli-0.16.0}/tests/test_adversarial_http.py

@@ -141,8 +141,11 @@ class TestResponseBodies:
     def test_empty_response(self, fresh_cache):
         server = fresh_cache
         res = asyncio.run(_fetch(server.url("/empty")))
-        # empty body -> markdown empty -> "content too thin" -> error terminal
-        assert _one(res)["status"] in ("error", "blocked")
+        # Body kosong: permintaan berhasil (HTTP 200) tapi kontennya
+        # ditolak karena tipis. Status akhirnya kini "thin", bukan
+        # "error" generik, supaya pemanggil bisa membedakan "server tidak
+        # bisa dihubungi" dari "konten ditolak, ubah ambang atau strategi".
+        assert _one(res)["status"] in ("thin", "error", "blocked")
 
     def test_malformed_html(self, fresh_cache):
         server = fresh_cache
@@ -166,9 +169,14 @@ class TestResponseBodies:
         assert _one(res)["status"] == "success"  # httpx auto-decompresses
 
     def test_huge_response_is_bounded(self, fresh_cache):
+        from webget.truncate import ELLIPSIS_MARKER
+
         server = fresh_cache
         # 5MB body; scrape_many must truncate, not blow memory or hang.
         res = asyncio.run(_fetch(server.url("/huge"), max_chars=1000))
         out = _one(res)
         assert out["status"] == "success"
-        assert len(out["markdown"]) <= 1000
+        # smart_truncate cuts at the limit and appends the ellipsis marker
+        # (this body has no clean boundary, so it is a hard cut at the limit).
+        assert len(out["markdown"]) <= 1000 + len(ELLIPSIS_MARKER)
+        assert out["markdown"].endswith(ELLIPSIS_MARKER)

webget_cli-0.16.0/tests/test_auth_state.py

@@ -0,0 +1,187 @@
+"""Tes profile._auth_state: deteksi login.
+
+Dua hal diuji di sini:
+
+1. PERBAIKAN yang gue commit: `login_words` memakai kata utuh, bukan
+   substring mentah. Sebelumnya "sign in" cocok dengan "design in", dan
+   setiap kata yang mengandung "login" dihitung. Sekarang regex kata utuh.
+
+2. FALSE POSITIVE YANG SENGAJA DIBIARKAN. Halaman publik yang punya menu
+   "Login" di navigasi DAN input password milik form lain tetap dilaporkan
+   "login_required". Tes terakhir mengunci perilaku itu supaya tidak
+   mengejutkan, dan docstring-nya menjelaskan mengapa.
+
+Gue sempat mencoba memperbaikinya dengan menuntut perintah login dan input
+password berada di <form> yang sama. Itu SALAH: halaman login sungguhan
+sering menaruh kata "Login" di <h1> di luar <form>, sehingga kontrol
+(portal yang benar-benar butuh login) ikut lolos jadi "success".
+
+Menandai halaman publik sebagai butuh login = pengguna mengejar sesi yang
+tidak ada. Melewatkan halaman login sungguhan = pengguna TIDAK diberi tahu
+sesinya mati, dan mengambil halaman login sebagai konten yang sah. Yang
+kedua lebih berbahaya, jadi bias sengaja diarahkan ke false positive dan
+bukan false negative.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from webget.profile import _auth_state
+
+
+def _hasil(markdown, html, status=200):
+    return {"markdown": markdown, "html": html, "status_code": status}
+
+
+ISI = "Konten yang cukup panjang untuk dianggap isi sungguhan. " * 10
+
+
+class TestPerbaikanKataUtuh:
+    def test_design_in_bukan_sign_in(self):
+        """'design in' tidak boleh dihitung sebagai perintah login."""
+        state, _ = _auth_state(
+            _hasil(f"Kami design in-house. {ISI}", "<p>Tentang kami</p>"), None
+        )
+        assert state == "success"
+
+    def test_sign_in_utuh_tetap_terdeteksi(self):
+        """Perbaikan tidak boleh menghilangkan deteksi yang sah.
+
+        Frasa "sign in to continue" termasuk penanda keharusan, jadi
+        terdeteksi lewat jalur login_phrases meskipun tanpa form password.
+        """
+        state, _ = _auth_state(
+            _hasil(f"Please sign in to continue. {ISI}", "<p>a</p>"), None
+        )
+        assert state == "login_required"
+
+
+class TestHalamanLoginTanpaForm:
+    """Celah yang diperbaiki: SPA tidak mengirim <input type=password>.
+
+    Halaman login yang di-render JavaScript tidak memuat tag form di HTML
+    mentah, sehingga pemeriksaan `has_password_input` meleset dan halaman
+    itu lolos sebagai "success". Pengguna tidak diberi tahu sesinya mati,
+    dan halaman login diambil sebagai konten sah.
+
+    Penanda frasa keharusan menutup celah ini tanpa menandai halaman
+    publik yang sekadar menyebut kata "login".
+    """
+
+    @pytest.mark.parametrize(
+        "teks",
+        [
+            "Please sign in to continue.",
+            "You must login first.",
+            "You must log in to view this page.",
+            "Login to continue.",
+            "Your session has timed out.",
+            "Session expired.",
+        ],
+    )
+    def test_frasa_keharusan_terdeteksi(self, teks):
+        state, auth = _auth_state(_hasil(f"{teks} {ISI}", "<p>JS app</p>"), None)
+        assert state == "login_required", f"{teks!r} tidak terdeteksi"
+        assert auth is False
+
+    def test_artikel_tentang_login_tidak_terdeteksi(self):
+        """Kata 'login' di artikel/navigasi bukan permintaan otentikasi."""
+        state, _ = _auth_state(
+            _hasil(f"Artikel ini membahas cara login ke sistem. {ISI}",
+                   "<nav>Login</nav>"),
+            None,
+        )
+        assert state == "success"
+
+
+class TestDeteksiLoginTetapKuat:
+    def test_form_login_sungguhan(self):
+        state, auth = _auth_state(
+            _hasil(
+                f"Silakan Login untuk melanjutkan. {ISI}",
+                '<form><input type="text" name="u">'
+                '<input type="password" name="p"></form>',
+            ),
+            None,
+        )
+        assert state == "login_required"
+        assert auth is False
+
+    def test_login_di_luar_form_tetap_terdeteksi(self):
+        """Kontrol penting: kata Login di <h1>, input password di <form>.
+
+        Inilah kasus yang patah saat gue menuntut keduanya satu <form>.
+        Portal kampus sering berbentuk begini, jadi harus tetap terdeteksi.
+        """
+        state, _ = _auth_state(
+            _hasil(
+                f"<h1>Login</h1> {ISI}",
+                '<form><input type="password" name="p"></form>',
+            ),
+            None,
+        )
+        assert state == "login_required"
+
+    def test_status_401(self):
+        state, auth = _auth_state(_hasil(ISI, "<p>a</p>", status=401), None)
+        assert state == "login_required"
+        assert auth is False
+
+    def test_label_kredensial_nim(self):
+        """SION memakai label NIM/username tanpa type=password."""
+        state, _ = _auth_state(
+            _hasil(
+                f"Masukkan password dan NIM Anda. {ISI}",
+                "<form><input name='nim'></form>",
+            ),
+            None,
+        )
+        assert state == "login_required"
+
+
+class TestFalsePositiveYangDibiarkan:
+    def test_menu_login_plus_input_password_lain(self):
+        """False positive yang sengaja dibiarkan, dikunci di sini.
+
+        Halaman publik dengan menu "Login" di navigasi DAN input password
+        milik form lain (demo/pencarian) dilaporkan butuh login. Ini tidak
+        ideal, tapi memperbaikinya tanpa kehilangan kasus
+        test_login_di_luar_form_tetap_terdeteksi belum bisa dilakukan
+        dengan sinyal murni teks. Lebih baik terkunci dan terlihat daripada
+        menjadi kejutan.
+        """
+        state, _ = _auth_state(
+            _hasil(
+                f"Selamat datang di dokumentasi publik. {ISI}",
+                '<nav><a href="/login">Login</a></nav>'
+                '<form action="/cari"><input type="text" name="q">'
+                '<input type="password" name="demo"></form>',
+            ),
+            None,
+        )
+        assert state == "login_required", (
+            "perilaku berubah; kalau false positive ini akhirnya diperbaiki "
+            "dengan cara yang aman, ubah tes ini dan hapus docstring-nya"
+        )
+
+
+class TestHalamanPublikNormal:
+    def test_input_password_tanpa_kata_login(self):
+        state, _ = _auth_state(
+            _hasil(f"Alat ini butuh kata sandi untuk demo. {ISI}",
+                   '<input type="password" name="demo">'),
+            None,
+        )
+        assert state == "success"
+
+    def test_halaman_bersih_dengan_profil(self):
+        """Dengan profil, halaman sukses berarti authenticated=True."""
+        state, auth = _auth_state(_hasil(ISI, "<p>a</p>"), "kampus")
+        assert state == "success"
+        assert auth is True
+
+    def test_halaman_bersih_tanpa_profil(self):
+        state, auth = _auth_state(_hasil(ISI, "<p>a</p>"), None)
+        assert state == "success"
+        assert auth is None