webget-cli 0.15.0__tar.gz → 0.16.0__tar.gz

{webget_cli-0.15.0 → webget_cli-0.16.0}/PKG-INFO

@@ -1,16 +1,15 @@
 Metadata-Version: 2.4
 Name: webget-cli
-Version: 0.15.0
+Version: 0.16.0
 Summary: Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys.
 Author-email: David Tarigan <tarigansdavid@gmail.com>
-License: Apache-2.0
+License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/DavidPandleton/webget
 Project-URL: Repository, https://github.com/DavidPandleton/webget
 Project-URL: Changelog, https://github.com/DavidPandleton/webget/blob/main/CHANGELOG.md
 Keywords: scraping,crawling,cli,search,duckduckgo,http
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -167,11 +166,27 @@ webget s "rust async runtime" --json      # machine-readable, includes engine pr
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
+webget crawl https://example.com crawl.db --limit 20 --json  # resumable local crawl
 webget fetch https://example.com --json   # machine-readable result
 ```
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+### Resumable local crawl
+
+The crawler is local and bounded: it uses SQLite for durable frontier state
+and page results, stays on the seed hostname, and can resume after a process
+restart. It does not require Docker, Redis, Postgres, or a browser.
+
+```bash
+webget crawl https://example.com ./crawl.db --limit 20 --timeout 20 --json
+```
+
+The same operation is available through Python as `webget.crawl_site(...)` and
+through MCP as the `crawl` tool. Results can also be exported with the Python
+API to JSONL or Markdown. Stale in-progress leases are recovered using a
+bounded timeout.
+
 ## Search engines
 
 webget searches through `ddgs`, a metasearch library that aggregates several
@@ -196,6 +211,31 @@ warning: unknown search engine(s): bogus - using auto
 Unknown names degrade to `auto` with a warning instead of failing, so a
 typo never kills a search.
 
+### Optional SearXNG provider
+
+SearXNG is not bundled or required. Point the optional HTTP adapter at an
+existing instance using an environment variable:
+
+```bash
+export WEBGET_SEARXNG_URL=http://localhost:8080
+```
+
+Then use the provider from Python:
+
+```python
+from webget import SearxngSearchProvider
+from webget.search import search_with_provenance
+
+provider = SearxngSearchProvider()
+results, provenance = search_with_provenance(
+    "query", n=5, engine="searxng", provider=provider
+)
+```
+
+The adapter calls `/search?format=json`, normalizes result fields, and keeps
+the usual per-call provenance. Live availability depends on the configured
+SearXNG instance; the core package does not start one.
+
 **Failover is automatic.** If the engine you named fails, or returns zero
 results, webget tries the remaining engines until a time budget is spent
 (15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
@@ -382,7 +422,11 @@ standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 ### Breaking change in 0.13.0: search returns an object, not a list
 
 `search` and `search_fetch` used to return a bare JSON array of results.
-They now return an object, because a list has nowhere to carry provenance:
+They now return an object, because a list has nowhere to carry provenance.
+The MCP result-count argument is `limit` (default 5 for `search`, 3 for
+`search_fetch`), matching the CLI vocabulary. The legacy MCP argument `n`
+remains accepted as an explicit compatibility alias; when both are supplied,
+`n` wins. The Python API continues to use `n` for backward compatibility.
 
 ```jsonc
 // <= 0.12.1  ->  a list

{webget_cli-0.15.0 → webget_cli-0.16.0}/README.md

@@ -130,11 +130,27 @@ webget s "rust async runtime" --json      # machine-readable, includes engine pr
 webget u https://example.com              # scrape (auto: http -> crawl4ai)
 webget su "llm inference" 5               # search + scrape top 5, parallel
 cat urls.txt | webget u -                 # batch scrape, one browser instance
+webget crawl https://example.com crawl.db --limit 20 --json  # resumable local crawl
 webget fetch https://example.com --json   # machine-readable result
 ```
 
 Long aliases: `search` = `s`, `fetch` = `u`, `search-fetch` = `su`.
 
+### Resumable local crawl
+
+The crawler is local and bounded: it uses SQLite for durable frontier state
+and page results, stays on the seed hostname, and can resume after a process
+restart. It does not require Docker, Redis, Postgres, or a browser.
+
+```bash
+webget crawl https://example.com ./crawl.db --limit 20 --timeout 20 --json
+```
+
+The same operation is available through Python as `webget.crawl_site(...)` and
+through MCP as the `crawl` tool. Results can also be exported with the Python
+API to JSONL or Markdown. Stale in-progress leases are recovered using a
+bounded timeout.
+
 ## Search engines
 
 webget searches through `ddgs`, a metasearch library that aggregates several
@@ -159,6 +175,31 @@ warning: unknown search engine(s): bogus - using auto
 Unknown names degrade to `auto` with a warning instead of failing, so a
 typo never kills a search.
 
+### Optional SearXNG provider
+
+SearXNG is not bundled or required. Point the optional HTTP adapter at an
+existing instance using an environment variable:
+
+```bash
+export WEBGET_SEARXNG_URL=http://localhost:8080
+```
+
+Then use the provider from Python:
+
+```python
+from webget import SearxngSearchProvider
+from webget.search import search_with_provenance
+
+provider = SearxngSearchProvider()
+results, provenance = search_with_provenance(
+    "query", n=5, engine="searxng", provider=provider
+)
+```
+
+The adapter calls `/search?format=json`, normalizes result fields, and keeps
+the usual per-call provenance. Live availability depends on the configured
+SearXNG instance; the core package does not start one.
+
 **Failover is automatic.** If the engine you named fails, or returns zero
 results, webget tries the remaining engines until a time budget is spent
 (15s by default, `WEBGET_FAILOVER_BUDGET_S` to override) and returns the
@@ -345,7 +386,11 @@ standalone with `webget-mcp` (stdio transport) or `python webget_mcp.py`.
 ### Breaking change in 0.13.0: search returns an object, not a list
 
 `search` and `search_fetch` used to return a bare JSON array of results.
-They now return an object, because a list has nowhere to carry provenance:
+They now return an object, because a list has nowhere to carry provenance.
+The MCP result-count argument is `limit` (default 5 for `search`, 3 for
+`search_fetch`), matching the CLI vocabulary. The legacy MCP argument `n`
+remains accepted as an explicit compatibility alias; when both are supplied,
+`n` wins. The Python API continues to use `n` for backward compatibility.
 
 ```jsonc
 // <= 0.12.1  ->  a list

{webget_cli-0.15.0 → webget_cli-0.16.0}/pyproject.toml

@@ -4,17 +4,16 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "webget-cli"
-version = "0.15.0"
+version = "0.16.0"
 description = "Local search + scrape CLI with an HTTP fast path, optional browser fallback, and authenticated session profiles. Zero API keys."
 readme = "README.md"
 requires-python = ">=3.11"
-license = { text = "Apache-2.0" }
+license = "Apache-2.0"
 authors = [{ name = "David Tarigan", email = "tarigansdavid@gmail.com" }]
 keywords = ["scraping", "crawling", "cli", "search", "duckduckgo", "http"]
 classifiers = [
     "Environment :: Console",
     "Intended Audience :: Developers",
-    "License :: OSI Approved :: Apache Software License",
     "Programming Language :: Python :: 3",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",

{webget_cli-0.15.0 → webget_cli-0.16.0}/tests/test_adversarial_http.py

@@ -141,8 +141,11 @@ class TestResponseBodies:
     def test_empty_response(self, fresh_cache):
         server = fresh_cache
         res = asyncio.run(_fetch(server.url("/empty")))
-        # empty body -> markdown empty -> "content too thin" -> error terminal
-        assert _one(res)["status"] in ("error", "blocked")
+        # Body kosong: permintaan berhasil (HTTP 200) tapi kontennya
+        # ditolak karena tipis. Status akhirnya kini "thin", bukan
+        # "error" generik, supaya pemanggil bisa membedakan "server tidak
+        # bisa dihubungi" dari "konten ditolak, ubah ambang atau strategi".
+        assert _one(res)["status"] in ("thin", "error", "blocked")
 
     def test_malformed_html(self, fresh_cache):
         server = fresh_cache

webget_cli-0.16.0/tests/test_auth_state.py

@@ -0,0 +1,187 @@
+"""Tes profile._auth_state: deteksi login.
+
+Dua hal diuji di sini:
+
+1. PERBAIKAN yang gue commit: `login_words` memakai kata utuh, bukan
+   substring mentah. Sebelumnya "sign in" cocok dengan "design in", dan
+   setiap kata yang mengandung "login" dihitung. Sekarang regex kata utuh.
+
+2. FALSE POSITIVE YANG SENGAJA DIBIARKAN. Halaman publik yang punya menu
+   "Login" di navigasi DAN input password milik form lain tetap dilaporkan
+   "login_required". Tes terakhir mengunci perilaku itu supaya tidak
+   mengejutkan, dan docstring-nya menjelaskan mengapa.
+
+Gue sempat mencoba memperbaikinya dengan menuntut perintah login dan input
+password berada di <form> yang sama. Itu SALAH: halaman login sungguhan
+sering menaruh kata "Login" di <h1> di luar <form>, sehingga kontrol
+(portal yang benar-benar butuh login) ikut lolos jadi "success".
+
+Menandai halaman publik sebagai butuh login = pengguna mengejar sesi yang
+tidak ada. Melewatkan halaman login sungguhan = pengguna TIDAK diberi tahu
+sesinya mati, dan mengambil halaman login sebagai konten yang sah. Yang
+kedua lebih berbahaya, jadi bias sengaja diarahkan ke false positive dan
+bukan false negative.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from webget.profile import _auth_state
+
+
+def _hasil(markdown, html, status=200):
+    return {"markdown": markdown, "html": html, "status_code": status}
+
+
+ISI = "Konten yang cukup panjang untuk dianggap isi sungguhan. " * 10
+
+
+class TestPerbaikanKataUtuh:
+    def test_design_in_bukan_sign_in(self):
+        """'design in' tidak boleh dihitung sebagai perintah login."""
+        state, _ = _auth_state(
+            _hasil(f"Kami design in-house. {ISI}", "<p>Tentang kami</p>"), None
+        )
+        assert state == "success"
+
+    def test_sign_in_utuh_tetap_terdeteksi(self):
+        """Perbaikan tidak boleh menghilangkan deteksi yang sah.
+
+        Frasa "sign in to continue" termasuk penanda keharusan, jadi
+        terdeteksi lewat jalur login_phrases meskipun tanpa form password.
+        """
+        state, _ = _auth_state(
+            _hasil(f"Please sign in to continue. {ISI}", "<p>a</p>"), None
+        )
+        assert state == "login_required"
+
+
+class TestHalamanLoginTanpaForm:
+    """Celah yang diperbaiki: SPA tidak mengirim <input type=password>.
+
+    Halaman login yang di-render JavaScript tidak memuat tag form di HTML
+    mentah, sehingga pemeriksaan `has_password_input` meleset dan halaman
+    itu lolos sebagai "success". Pengguna tidak diberi tahu sesinya mati,
+    dan halaman login diambil sebagai konten sah.
+
+    Penanda frasa keharusan menutup celah ini tanpa menandai halaman
+    publik yang sekadar menyebut kata "login".
+    """
+
+    @pytest.mark.parametrize(
+        "teks",
+        [
+            "Please sign in to continue.",
+            "You must login first.",
+            "You must log in to view this page.",
+            "Login to continue.",
+            "Your session has timed out.",
+            "Session expired.",
+        ],
+    )
+    def test_frasa_keharusan_terdeteksi(self, teks):
+        state, auth = _auth_state(_hasil(f"{teks} {ISI}", "<p>JS app</p>"), None)
+        assert state == "login_required", f"{teks!r} tidak terdeteksi"
+        assert auth is False
+
+    def test_artikel_tentang_login_tidak_terdeteksi(self):
+        """Kata 'login' di artikel/navigasi bukan permintaan otentikasi."""
+        state, _ = _auth_state(
+            _hasil(f"Artikel ini membahas cara login ke sistem. {ISI}",
+                   "<nav>Login</nav>"),
+            None,
+        )
+        assert state == "success"
+
+
+class TestDeteksiLoginTetapKuat:
+    def test_form_login_sungguhan(self):
+        state, auth = _auth_state(
+            _hasil(
+                f"Silakan Login untuk melanjutkan. {ISI}",
+                '<form><input type="text" name="u">'
+                '<input type="password" name="p"></form>',
+            ),
+            None,
+        )
+        assert state == "login_required"
+        assert auth is False
+
+    def test_login_di_luar_form_tetap_terdeteksi(self):
+        """Kontrol penting: kata Login di <h1>, input password di <form>.
+
+        Inilah kasus yang patah saat gue menuntut keduanya satu <form>.
+        Portal kampus sering berbentuk begini, jadi harus tetap terdeteksi.
+        """
+        state, _ = _auth_state(
+            _hasil(
+                f"<h1>Login</h1> {ISI}",
+                '<form><input type="password" name="p"></form>',
+            ),
+            None,
+        )
+        assert state == "login_required"
+
+    def test_status_401(self):
+        state, auth = _auth_state(_hasil(ISI, "<p>a</p>", status=401), None)
+        assert state == "login_required"
+        assert auth is False
+
+    def test_label_kredensial_nim(self):
+        """SION memakai label NIM/username tanpa type=password."""
+        state, _ = _auth_state(
+            _hasil(
+                f"Masukkan password dan NIM Anda. {ISI}",
+                "<form><input name='nim'></form>",
+            ),
+            None,
+        )
+        assert state == "login_required"
+
+
+class TestFalsePositiveYangDibiarkan:
+    def test_menu_login_plus_input_password_lain(self):
+        """False positive yang sengaja dibiarkan, dikunci di sini.
+
+        Halaman publik dengan menu "Login" di navigasi DAN input password
+        milik form lain (demo/pencarian) dilaporkan butuh login. Ini tidak
+        ideal, tapi memperbaikinya tanpa kehilangan kasus
+        test_login_di_luar_form_tetap_terdeteksi belum bisa dilakukan
+        dengan sinyal murni teks. Lebih baik terkunci dan terlihat daripada
+        menjadi kejutan.
+        """
+        state, _ = _auth_state(
+            _hasil(
+                f"Selamat datang di dokumentasi publik. {ISI}",
+                '<nav><a href="/login">Login</a></nav>'
+                '<form action="/cari"><input type="text" name="q">'
+                '<input type="password" name="demo"></form>',
+            ),
+            None,
+        )
+        assert state == "login_required", (
+            "perilaku berubah; kalau false positive ini akhirnya diperbaiki "
+            "dengan cara yang aman, ubah tes ini dan hapus docstring-nya"
+        )
+
+
+class TestHalamanPublikNormal:
+    def test_input_password_tanpa_kata_login(self):
+        state, _ = _auth_state(
+            _hasil(f"Alat ini butuh kata sandi untuk demo. {ISI}",
+                   '<input type="password" name="demo">'),
+            None,
+        )
+        assert state == "success"
+
+    def test_halaman_bersih_dengan_profil(self):
+        """Dengan profil, halaman sukses berarti authenticated=True."""
+        state, auth = _auth_state(_hasil(ISI, "<p>a</p>"), "kampus")
+        assert state == "success"
+        assert auth is True
+
+    def test_halaman_bersih_tanpa_profil(self):
+        state, auth = _auth_state(_hasil(ISI, "<p>a</p>"), None)
+        assert state == "success"
+        assert auth is None

webget_cli-0.16.0/tests/test_cache_cap.py

@@ -0,0 +1,115 @@
+"""Tes cache: WEBGET_CACHE_MAX tidak valid tidak boleh menghapus segalanya.
+
+Eviction di cache_put:
+
+    cap = int(os.environ.get("WEBGET_CACHE_MAX", "5000"))
+    ...
+    if len(files) > cap:
+        ... buang yang kedaluwarsa ...
+        if len(live) > cap:
+            live.sort(key=os.path.getmtime)
+            for f in live[: len(live) - int(cap * 0.8)]:
+                os.remove(f)
+
+Dengan cap <= 0, irisan itu mencakup SELURUH daftar, sehingga satu
+penulisan menghapus seluruh cache:
+
+    cap = 0   -> int(0*0.8) == 0,        live[:len(live)]     -> semua
+    cap = -5  -> int(-5*0.8) == -4,      live[:len(live)+4]   -> semua
+
+Sebelumnya hanya ValueError yang ditangani, jadi "abc" aman tapi "0" dan
+"-5" justru paling merusak. Sekarang cap <= 0 juga kembali ke bawaan.
+"""
+
+from __future__ import annotations
+
+import os
+
+import pytest
+
+from webget import cache
+
+
+@pytest.fixture
+def cache_dir(tmp_path, monkeypatch):
+    """Arahkan cache ke tmp_path SEBELUM penulisan apa pun.
+
+    Hook override ada di SHIM `webget_cli.CACHE_DIR`, bukan
+    `webget.cache.CACHE_DIR`: _cache_dir() mengimpor webget_cli dan membaca
+    atributnya, sehingga menetapkan atribut di modul cache sendiri tidak
+    berpengaruh. Versi pertama tes ini menambal nama yang salah, portanto
+    nol file tertulis di tmp dan assertion-nya lolos/gagal karena salah
+    alasan.
+    """
+    import webget_cli
+
+    d = tmp_path / "cache"
+    d.mkdir()
+    monkeypatch.setattr(webget_cli, "CACHE_DIR", str(d))
+    return d
+
+
+def _isi(d, n):
+    for i in range(n):
+        cache.cache_put(
+            url=f"https://x.test/{i}",
+            cookies=None,
+            headers=None,
+            max_chars=1000,
+            data={"markdown": f"isi {i}", "status": "success"},
+        )
+    return len([f for f in os.listdir(d) if f.endswith(".json")])
+
+
+class TestCapTidakValid:
+    @pytest.mark.parametrize("nilai", ["0", "-5", "-1"])
+    def test_cap_nol_atau_negatif_tidak_menghapus_semua(self, cache_dir, monkeypatch, nilai):
+        """cap <= 0 harus jatuh ke bawaan, bukan menghapus seluruh cache."""
+        monkeypatch.setenv("WEBGET_CACHE_MAX", nilai)
+        sebelum = _isi(cache_dir, 10)
+        assert sebelum == 10, f"setup gagal: hanya {sebelum} entri tertulis"
+
+        # Satu penulisan tambahan memicu jalur eviction.
+        cache.cache_put(
+            url="https://x.test/baru",
+            cookies=None,
+            headers=None,
+            max_chars=1000,
+            data={"markdown": "baru", "status": "success"},
+        )
+        sesudah = len([f for f in os.listdir(cache_dir) if f.endswith(".json")])
+        assert sesudah == 11, (
+            f"WEBGET_CACHE_MAX={nilai} menghapus cache: {sebelum} -> {sesudah}. "
+            f"cap <= 0 membuat irisan eviction mencakup semua entri."
+        )
+
+    def test_cap_bukan_angka_jatuh_ke_bawaan(self, cache_dir, monkeypatch):
+        monkeypatch.setenv("WEBGET_CACHE_MAX", "abc")
+        sebelum = _isi(cache_dir, 5)
+        cache.cache_put(
+            url="https://x.test/baru", cookies=None, headers=None,
+            max_chars=1000, data={"markdown": "x", "status": "success"},
+        )
+        sesudah = len([f for f in os.listdir(cache_dir) if f.endswith(".json")])
+        assert sesudah >= sebelum, "nilai tidak valid tidak boleh menghapus apa pun"
+
+
+class TestEvictionNormal:
+    def test_cap_kecil_membatasi_jumlah_entri(self, cache_dir, monkeypatch):
+        """cap yang sah tetap mengecilkan cache, tapi tidak ke nol.
+
+        Ini bukan bug: cap=2 menyisakan int(2*0.8)==1 entri, jadi cache
+        memang agresif untuk cap sangat kecil. Yang diuji adalah bahwa
+        eviction benar-benar jalan dan menyisakan sesuatu, bukan menghapus
+        semuanya.
+        """
+        monkeypatch.setenv("WEBGET_CACHE_MAX", "5")
+        _isi(cache_dir, 12)
+        cache.cache_put(
+            url="https://x.test/picu", cookies=None, headers=None,
+            max_chars=1000, data={"markdown": "x", "status": "success"},
+        )
+        sisa = len([f for f in os.listdir(cache_dir) if f.endswith(".json")])
+        assert 0 < sisa <= 12, f"eviction menyisakan {sisa} entri"
+        # int(5*0.8) == 4, jadi harusnya menyisakan sekitar 4.
+        assert sisa <= 6, f"cap=5 seharusnya memangkas ke sekitar 4, dapat {sisa}"

webget_cli-0.16.0/tests/test_cli_crawl.py

@@ -0,0 +1,38 @@
+import io
+import json
+import sys
+from contextlib import redirect_stdout
+
+import pytest
+
+from webget import cli
+
+
+def test_cli_crawl_dispatches_and_emits_json(monkeypatch, tmp_path):
+    seen = {}
+
+    async def fake_crawl(seed, frontier, **kwargs):
+        seen.update(seed=seed, frontier=frontier, kwargs=kwargs)
+        return {"stats": {"done": 1, "failed": 0}, "results": []}
+
+    monkeypatch.setattr(cli, "crawl_site", fake_crawl)
+    monkeypatch.setattr(
+        sys,
+        "argv",
+        ["webget", "crawl", "https://example.com", str(tmp_path / "crawl.db"), "--json", "--limit", "1"],
+    )
+    output = io.StringIO()
+    with redirect_stdout(output):
+        cli.main()
+    assert json.loads(output.getvalue())["stats"]["done"] == 1
+    assert seen["seed"] == "https://example.com"
+    assert seen["kwargs"]["max_pages"] == 1
+
+
+def test_cli_crawl_requires_frontier_path(monkeypatch, capsys):
+    monkeypatch.setattr(sys, "argv", ["webget", "crawl", "https://example.com"])
+    with pytest.raises(SystemExit) as exc:
+        cli.main()
+    captured = capsys.readouterr()
+    assert exc.value.code == 2
+    assert "crawl expects URL and frontier SQLite path" in captured.out