beo-mcdp 0.2.0__py3-none-any.whl

beo_mcdp-0.2.0.dist-info/METADATA

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: beo-mcdp
+Version: 0.2.0
+Summary: Core CDP Browser Automation and Utils Provider
+Author: Dwi Hardiyanto
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: playwright>=1.57.0
+Provides-Extra: patchright
+Requires-Dist: patchright; extra == "patchright"
+Provides-Extra: kafka
+Requires-Dist: confluent-kafka; extra == "kafka"
+Dynamic: author
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CDP Scraper Provider
+
+Library *core* untuk ekstraksi data lewat browser Chrome (CDP). Satu mesin yang sama dipakai untuk
+banyak sosial media: platform baru cukup **dideskripsikan** (URL, pola API, mapping field), tanpa
+menulis ulang sesi browser, scroll, capture, maupun parser.
+
+## Instalasi
+
+```bash
+pip install git+https://github.com/dwihard/social_scrapers.git@v0.2.0
+# opsional: pip install "cdp_scraper[kafka]"  /  "cdp_scraper[patchright]"
+```
+
+## Cara kerja
+
+1. Browser membuka halaman (`Endpoint.url`).
+2. Library mendengarkan response API yang dipanggil halaman (`Endpoint.api_match`).
+3. JSON-nya diubah jadi list dict oleh `parser` (biasanya `FieldMapper`), di-dedupe lewat `key`, lalu halaman di-scroll sampai `limit`/data habis.
+
+## Menambah platform baru
+
+```python
+from cdp_scraper import (Endpoint, F, FieldMapper, Platform, Scraper,
+                         const, make_scroller, register_platform, end_when_falsy)
+
+parser = FieldMapper(
+    items="data.posts",                      # path ke list item
+    fields={
+        "id":    ("id", "str", ""),          # (path, transform, default)
+        "url":   F(template="https://site.com/{author.handle}/{id}"),
+        "likes": ("stats.likes", "int", 0),
+        "tags":  "tags[].name",              # semua elemen
+        "posted": F("created", transform="iso_utc"),
+        "platform": const("site"),
+    },
+)
+
+site = Platform(
+    name="site",                              # juga jadi label log: [site] ...
+    cdp={"cdp_url": "http://localhost:9223"}, # opsional: Chrome/slot terpisah per platform
+    overlay_selectors=('div.popup-close',),   # popup yang ditutup otomatis
+    login_url_hints=("/login",),              # deteksi browser belum login
+    endpoints=[Endpoint(
+        "search", url="https://site.com/search?q={keyword}", api_match="api/search",
+        parser=parser, params=("keyword",), label="keyword", limit=20, max_limit=50,
+        end=end_when_falsy("has_more"),
+        scroll=make_scroller(wheel=2500, click_pattern="load more"),
+    )],
+)
+register_platform(site)
+
+result = await Scraper("site").run("search", {"keyword": "kopi"}, limit=30)     # satu query
+results = await Scraper("site").run_many("search", [{"keyword": "a"}, {"keyword": "b"}])
+```
+
+`run()` melempar exception (`PageCrashed`, `BrowserBusy`, `ValueError` untuk parameter salah).
+`run_many()` memakai satu tab untuk banyak query dan menjadikan error per-item `status="error"`.
+Contoh lengkap TikTok dan X/Twitter ada di `examples/`.
+
+## Mapping parsing (`FieldMapper`)
+
+| Sintaks path | Arti |
+|---|---|
+| `a.b.c` | nilai bersarang |
+| `a[0].b`, `a[-1]` | indeks |
+| `a[].b` | semua elemen (hasil list) |
+| `a[type=1].b` | elemen yang cocok filter (hasil list) |
+
+Bentuk spec field: `"path"` · `("path", "transform", default)` · `F(path | any_of=[...] | template=... | fn=...,
+default=, transform=, required=)` · `const(x)` · `callable(record)`.
+
+Opsi `FieldMapper`: `item_path` (sub-path record di tiap item), `where` / `record_where`
+(`{path: nilai | EXISTS | callable}` atau callable), `unwrap` (membuka pembungkus), `items` boleh
+list path atau callable.
+
+Transform bawaan: `int float str bool url lower upper strip clean_text username hashtag iso_utc
+date_id join first count unique drop_empty` dan `each:<transform>` untuk list. Argumen format tanggal:
+`iso_utc:%a %b %d %H:%M:%S %z %Y`. Tambah sendiri: `register_transform("nama", fn)`.
+
+Jika logika terlalu rumit untuk mapping, gunakan `F(fn=...)` atau parser Python biasa
+(`parser=` menerima callable apa pun yang mengembalikan list dict).
+
+## Hook & aksi halaman
+
+`prepare` (setelah halaman terbuka) dan `scroll` (tiap putaran) adalah fungsi `async def (page)`.
+Bawaan: `make_scroller(...)`, `make_wait_for(selector)`, `close_overlays(page, selectors)`.
+`Endpoint.capture_options` meneruskan kwargs tambahan ke `capture` (`poll_interval`, `settle`, dst).
+
+## Konfigurasi browser
+
+```python
+from cdp_scraper import CDPConfig, setup_cdp
+setup_cdp(cdp_url="http://localhost:9222", max_sessions=2, lock_timeout=60)   # default global
+cfg = CDPConfig.from_env("TWITTER_")   # TWITTER_CDP_URL, TWITTER_MAX_CONCURRENT_SESSIONS, ...
+```
+
+Setiap request memakai **tab baru sendiri** (ditutup saat selesai) dan dibatasi `max_sessions`
+bersamaan per konfigurasi. Konfigurasi yang sama berbagi slot; `cdp_url` berbeda = slot berbeda.
+Screenshot debug disimpan di `CDPConfig.debug_dir` (default folder kerja); matikan dengan
+`debug_screenshots=False`.
+
+## Logging
+
+Format seragam `[job] Status message: detail`, job = nama platform:
+
+```
+[2026-11-10 08:19:38] [INFO] [site] Success message: Capture api/search selesai: 20 data
+```
+
+Library hanya memakai logger `cdp_scraper` (tanpa handler). Aktifkan dengan `setup_logging()`
+atau injeksi logger sendiri: `setup_cdp(custom_logger=...)`. `get_logger("kafka").error("...")`
+untuk log aplikasi dengan format yang sama.
+
+## Kafka / Redpanda
+
+```python
+from cdp_scraper import RedpandaProducer, publish_chunks
+producer = RedpandaProducer()            # KAFKA_BOOTSTRAP_SERVERS; koneksi dibuat saat dipakai
+if producer.available:
+    stats = publish_chunks(producer, "topic-x", keywords, chunk_size=2,
+                           build_payload=lambda chunk: {"keywords": chunk})
+    # {"berhasil": n, "gagal": n}
+```
+Kontrak sumber data (keyword/akun) ada di `cdp_scraper.utils.database` (`KeywordSource`,
+`AccountSource`); implementasinya milik aplikasi.
+
+## Pengujian
+
+```bash
+python -m unittest discover -s tests -t .
+```
+Tes memakai browser palsu, jadi tidak butuh Chrome/Kafka.
+
+## Perubahan dari 0.1.x (breaking)
+
+- Konfigurasi sekarang `CDPConfig` (immutable, per platform); `setup_cdp(...)` tetap ada. Variabel modul `config.CDP_URL` dst dihapus.
+- `get_session_slots` dan `PW_ENGINE` dihapus; pakai `CDPSessionManager` / `manager.engine`.
+- `BrowserBusy` (subclass `TimeoutError`) menggantikan `TimeoutError` polos.
+- `helpers.close_login_modal(page, selectors)` kini wajib diberi selector (tidak ada lagi selector TikTok bawaan).
+- `utils`: tidak lagi mengimpor Flask; `extract_challenges` (khusus TikTok) dihapus; `calculate_engagement_rate` menerima `interaction_keys` / `view_key` (default sama dengan perilaku lama).
+- `utils.database` kini berupa Protocol; `utils.redpanda_client` tidak lagi membuat producer saat import (gunakan `cdp_scraper.messaging`).

beo_mcdp-0.2.0.dist-info/RECORD

@@ -0,0 +1,31 @@
+cdp_scraper/__init__.py,sha256=N9umzREVvK7eTlSS6PSqLkjCV7Ha8amM-znhrkt7lB8,1549
+cdp_scraper/_version.py,sha256=Zn1KFblwuFHiDRdRAiRnDBRkbPttWh44jKa5zG2ov0E,22
+cdp_scraper/config.py,sha256=CBFFngFkZ5yh0IqkZB_S9jBstoCtvNZGNmakGH0Oemw,2525
+cdp_scraper/errors.py,sha256=RH7oZ77Mi_tv_WPNSpk7hsIZAz8Eq6WA2cw4nNo3Hk0,291
+cdp_scraper/logs.py,sha256=e7Xu9LuMrSQYGykXV6ROURWRBQLdKLqVYTE3bob8Hb8,1367
+cdp_scraper/parsers.py,sha256=ZdvFia4r1tsvN-sIg45AfQJ9j4xTXwW-170GLr8DaDw,114
+cdp_scraper/browser/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+cdp_scraper/browser/actions.py,sha256=5Qrmvn1K-wnCuBeRlXUeEWS99A8z_9VbTvvNVKNef0k,2900
+cdp_scraper/browser/capture.py,sha256=U5YkQZ4P34ZiN5Xb37COEbma7Xd5YGot3C3vywc87UI,6819
+cdp_scraper/browser/diagnostics.py,sha256=2vTyQ1-2U5lADTMHpOHkbS4yrE7PNExKaKfYVzy8wPc,1791
+cdp_scraper/browser/helpers.py,sha256=v8bQMYFXH2Q6jLBQUltpNS1nuLNLKgW53PEU6eD4Wr0,133
+cdp_scraper/browser/session.py,sha256=wHVi3iqB1NnM7LNQhqEA96gesk8e4-EqzThMnJgnptc,5127
+cdp_scraper/browser/state.py,sha256=ewQR29KRYjtnwa7c_RLfXcuwspsuJ9DABIUte9fN4Dk,101
+cdp_scraper/messaging/__init__.py,sha256=AMBdGZuzhgO9d3dXzCHy6nOWiqNyNInlpdUhUvBgrlQ,253
+cdp_scraper/messaging/redpanda.py,sha256=6M1ifGVRHzv7xb3jdX5dB0b2a4w8Cd1iUPPpayGTyXc,3450
+cdp_scraper/parsing/__init__.py,sha256=q_QBPjNYD9ypeQOHQ4Nud3jdxdMCqrzmh4MQDZINNi8,313
+cdp_scraper/parsing/mapper.py,sha256=Mj1mMQuZ0AcijXw_kyWPGaOSKiZhtvrLIiaiyRoXT5M,6849
+cdp_scraper/parsing/paths.py,sha256=WAMZqdLjUL3dVhNuj2r1XXnCmzvtxyYwIZEhxMxXLgA,2468
+cdp_scraper/parsing/transforms.py,sha256=Ez4XVQMxCULD62OP1kodKtA8WJPcHx6MbA570Kx_HiE,4044
+cdp_scraper/platforms/__init__.py,sha256=Ce9MUVPNMaCsvrTybHVEon3Q54VrhVwDm6yO7LdmYL8,401
+cdp_scraper/platforms/registry.py,sha256=XU4nrMD4tGQMeWmaszRkAP-HIkJmsPMptH1XyhhAx4A,673
+cdp_scraper/platforms/runner.py,sha256=traqQBjTMTgYZCkwwsyCJcFkoeahRsrjuy5wWzWzxyA,5293
+cdp_scraper/platforms/spec.py,sha256=we1BMN03sMZF4xgGR2FTf1GWFxF3fsUh40RF322-gEw,4838
+cdp_scraper/utils/__init__.py,sha256=NJAlMwoaEEHRRS5TH-4JUePYzR7n8VPtne6fiqNWWgc,671
+cdp_scraper/utils/database.py,sha256=9oiw9ppN51R4RJRdd7y-BxNvFY0XHKSttbuPceLNFgA,333
+cdp_scraper/utils/redpanda_client.py,sha256=HoEoNSUb38ljTsJMvpub51c5tzePWduuPmPWOEZq-jM,283
+cdp_scraper/utils/utils.py,sha256=48ONTL8OKVHlCMVQ32dgV7_WcsnhnrIsQRVtNH_jiyc,3694
+beo_mcdp-0.2.0.dist-info/METADATA,sha256=sAY9_sc30ShpRmsfJqLWvQsJmCNrk5llk2VbeTxQ2F4,6865
+beo_mcdp-0.2.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+beo_mcdp-0.2.0.dist-info/top_level.txt,sha256=NliR-0bvpE9-A1PTF-2gSom_tWGAa_gaOePosP27G1Q,12
+beo_mcdp-0.2.0.dist-info/RECORD,,

beo_mcdp-0.2.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

beo_mcdp-0.2.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+cdp_scraper

cdp_scraper/__init__.py

@@ -0,0 +1,24 @@
+from ._version import __version__
+from .browser.actions import close_overlays, make_scroller, make_wait_for, scroll_page_down
+from .browser.capture import CaptureResult, capture, capture_api
+from .browser.session import (CDPSessionManager, default_manager, get_manager,
+                              is_browser_busy, playwright_session)
+from .config import CDPConfig, get_default_config, set_default_config, setup_cdp
+from .errors import BrowserBusy, CDPError, PageCrashed
+from .logs import JobLogger, get_logger, setup_logging
+from .messaging import RedpandaProducer, chunk_list, publish_chunks
+from .parsing import EXISTS, F, FieldMapper, const, get_path, register_transform
+from .platforms import (Endpoint, Platform, ScrapeResult, Scraper, end_when_empty,
+                        end_when_falsy, get_platform, list_platforms, register_platform)
+
+__all__ = [
+    "__version__", "CDPConfig", "setup_cdp", "get_default_config", "set_default_config",
+    "CDPError", "PageCrashed", "BrowserBusy", "JobLogger", "get_logger", "setup_logging",
+    "CDPSessionManager", "get_manager", "default_manager", "playwright_session", "is_browser_busy",
+    "capture", "capture_api", "CaptureResult",
+    "close_overlays", "make_scroller", "make_wait_for", "scroll_page_down",
+    "FieldMapper", "F", "const", "EXISTS", "get_path", "register_transform",
+    "Platform", "Endpoint", "Scraper", "ScrapeResult", "register_platform", "get_platform",
+    "list_platforms", "end_when_falsy", "end_when_empty",
+    "RedpandaProducer", "publish_chunks", "chunk_list",
+]

cdp_scraper/_version.py

@@ -0,0 +1 @@
+__version__ = "0.2.0"

cdp_scraper/browser/actions.py

@@ -0,0 +1,74 @@
+"""Aksi halaman yang bisa dirakit per platform: tutup overlay, scroll, tunggu elemen."""
+from ..logs import get_logger
+
+_SCROLL_JS = """(o) => {
+    if (o.window === 'bottom') window.scrollTo(0, document.documentElement.scrollHeight);
+    else if (o.window === 'viewport') window.scrollBy(0, window.innerHeight * o.factor);
+    if (o.containers) {
+        for (const el of document.querySelectorAll('div')) {
+            const st = getComputedStyle(el);
+            if ((st.overflowY === 'auto' || st.overflowY === 'scroll')
+                && el.scrollHeight > el.clientHeight + 50) {
+                el.scrollTop = el.scrollHeight;
+            }
+        }
+    }
+    if (o.pattern) {
+        const re = new RegExp(o.pattern, 'i');
+        const btn = [...document.querySelectorAll('button, [role="button"]')]
+            .find(b => re.test((b.innerText || '').trim()));
+        if (btn) btn.click();
+    }
+}"""
+
+
+async def close_overlays(page, selectors):
+    """Klik tombol penutup popup/modal (selector CSS diberikan platform)."""
+    if not selectors:
+        return
+    try:
+        await page.evaluate(
+            "(sels) => document.querySelectorAll(sels.join(',')).forEach(b => b.click())",
+            list(selectors))
+    except Exception:
+        pass
+
+
+async def scroll_page_down(page):
+    await page.keyboard.press("PageDown")
+
+
+def make_scroller(*, wheel=2500, window_scroll="bottom", viewport_factor=2.0,
+                  containers=False, click_pattern=None, mouse=(0.5, 0.5), key=None):
+    """Bangun fungsi scroll generik.
+
+    wheel: delta mouse wheel (0 = nonaktif); window_scroll: 'bottom' | 'viewport' | None;
+    containers: scroll semua div overflow; click_pattern: regex teks tombol 'load more'/'retry';
+    mouse: posisi kursor (fraksi viewport); key: tombol keyboard tambahan (mis. 'End').
+    """
+    options = {"window": window_scroll, "factor": viewport_factor,
+               "containers": containers, "pattern": click_pattern}
+    use_js = bool(window_scroll or containers or click_pattern)
+
+    async def _scroll(page):
+        if wheel:
+            vp = page.viewport_size or {"width": 1280, "height": 800}
+            await page.mouse.move(vp["width"] * mouse[0], vp["height"] * mouse[1])
+            await page.mouse.wheel(0, wheel)
+        if use_js:
+            await page.evaluate(_SCROLL_JS, options)
+        if key:
+            await page.keyboard.press(key)
+
+    return _scroll
+
+
+def make_wait_for(selector, timeout_ms=15000, message=None):
+    """Fungsi prepare: tunggu elemen pertama muncul (hanya warning jika tidak muncul)."""
+    async def _wait(page, log=None):
+        log = log or get_logger()
+        try:
+            await page.wait_for_selector(selector, state="attached", timeout=timeout_ms)
+        except Exception:
+            log.warning(message or f"Elemen '{selector}' belum muncul setelah {timeout_ms // 1000} detik.")
+    return _wait

cdp_scraper/browser/capture.py

@@ -0,0 +1,191 @@
+"""Capture response API generik (smart scroll & wait)."""
+import asyncio
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional
+
+from ..errors import PageCrashed
+from ..logs import get_logger
+from .diagnostics import DebugOptions, page_diag, save_debug_shot
+
+
+def compile_matcher(match):
+    """api_match bisa: substring (str), regex terkompilasi, atau callable(url) -> bool."""
+    if hasattr(match, "search"):
+        return (lambda url: bool(match.search(url))), match.pattern
+    if callable(match):
+        return match, getattr(match, "__name__", "callable")
+    text = str(match)
+    return (lambda url: text in url), text
+
+
+def key_getter(key):
+    if callable(key):
+        return key
+    return lambda item: item.get(key) if isinstance(item, dict) else None
+
+
+async def default_scroll(page):
+    await page.keyboard.press("PageDown")
+
+
+@dataclass
+class CaptureResult:
+    items: List[Any] = field(default_factory=list)
+    stopped_by: str = "max_scrolls"
+    scrolls: int = 0
+    diagnostics: Dict[str, Any] = field(default_factory=dict)
+
+
+async def capture(page, url, api_match, extract_fn, key_fn, limit,
+                  max_scrolls=8, max_wait=8.0, stall_limit=3,
+                  initial_sleep=5, scroll_fn=None, prepare_fn=None, end_fn=None,
+                  nav_timeout_ms=60000, wait_until="domcontentloaded",
+                  poll_interval=0.5, settle=1.0, final_sleep=1.0,
+                  log=None, diag=None, debug=None):
+    log = log or get_logger()
+    diag = {} if diag is None else diag
+    debug = debug or DebugOptions()
+    matches, match_label = compile_matcher(api_match)
+    get_key = key_getter(key_fn)
+
+    items = []
+    seen = set()
+    state = {"limit_reached": False, "exhausted": False, "crashed": False}
+
+    def check_crash():
+        if state["crashed"]:
+            raise PageCrashed("Tab Chrome crash (cek --shm-size / memori container)")
+
+    def on_crash(_):
+        state["crashed"] = True
+        log.error("Tab crash terdeteksi (Target crashed).")
+
+    async def handle_resp(res):
+        if state["limit_reached"] or res.request.method == "OPTIONS":
+            return
+        if not matches(res.url):
+            return
+        if res.status != 200:
+            log.warning(f"API: {match_label} status {res.status}: {res.url[:150]}")
+            return
+        try:
+            data_json = await res.json()
+        except Exception as e:
+            log.warning(f"API: Gagal parse JSON ({match_label}): {e}")
+            return
+        try:
+            if isinstance(data_json, dict):
+                log.info(f"API: {match_label} keys={list(data_json.keys())[:8]}")
+            for item in extract_fn(data_json):
+                k = get_key(item)
+                if not k or k in seen:
+                    continue
+                seen.add(k)
+                items.append(item)
+                if len(items) >= limit:
+                    state["limit_reached"] = True
+                    break
+            if end_fn and end_fn(data_json):
+                state["exhausted"] = True
+                log.info("API: Server menandai data sudah habis.")
+            log.info(f"API: Total tertangkap: {len(items)}")
+        except Exception as e:
+            log.warning(f"API: Gagal ekstrak data: {e}")
+
+    stopped_by = "max_scrolls"
+    scrolls = 0
+    stalled = 0
+    page.on("response", handle_resp)
+    page.on("crash", on_crash)
+    try:
+        log.info(f"Membuka link: {url}")
+        try:
+            nav_resp = await page.goto(url, wait_until=wait_until, timeout=nav_timeout_ms)
+        except Exception as e:
+            if "crash" in str(e).lower():
+                state["crashed"] = True
+            raise
+        check_crash()
+        await asyncio.sleep(initial_sleep)
+        check_crash()
+        await page_diag(page, nav_resp, "after_goto", diag=diag, log=log)
+        check_crash()
+
+        if prepare_fn:
+            await prepare_fn(page)
+            check_crash()
+
+        log.info(f"Data awal tertangkap: {len(items)}")
+
+        scroller = scroll_fn or default_scroll
+        for i in range(max_scrolls):
+            check_crash()
+            if state["limit_reached"]:
+                log.info("Limit tercapai, berhenti scroll.")
+                break
+            if state["exhausted"]:
+                log.info("Tidak ada data lagi, berhenti scroll.")
+                break
+
+            before = len(items)
+            scrolls = i + 1
+            log.info(f"Scroll {scrolls}/{max_scrolls}...")
+            try:
+                await scroller(page)
+            except Exception as e:
+                if "crash" in str(e).lower():
+                    state["crashed"] = True
+                    check_crash()
+                log.warning(f"Scroll gagal: {e}")
+
+            waited = 0.0
+            while (len(items) == before and waited < max_wait
+                   and not state["limit_reached"] and not state["exhausted"]
+                   and not state["crashed"]):
+                await asyncio.sleep(poll_interval)
+                waited += poll_interval
+            check_crash()
+
+            if len(items) == before:
+                if state["exhausted"]:
+                    break
+                log.warning("Data baru tidak terdeteksi (mungkin mentok/lambat).")
+                stalled += 1
+            else:
+                stalled = 0
+                if state["exhausted"]:
+                    break
+                await asyncio.sleep(settle)
+
+            if stalled >= stall_limit:
+                log.info(f"{stall_limit}x scroll tanpa data baru. Berhenti.")
+                break
+
+        await asyncio.sleep(final_sleep)
+        if state["limit_reached"]:
+            stopped_by = "limit"
+        elif state["exhausted"]:
+            stopped_by = "exhausted"
+        elif stalled >= stall_limit:
+            stopped_by = "stalled"
+
+        if not items:
+            await page_diag(page, None, "no_items", diag=diag, log=log)
+            await save_debug_shot(page, "debug_last.png", debug=debug, log=log)
+        elif stalled >= stall_limit:
+            await page_diag(page, None, "stalled", diag=diag, log=log)
+            await save_debug_shot(page, "debug_stalled.png", debug=debug, log=log)
+    finally:
+        try:
+            page.remove_listener("response", handle_resp)
+            page.remove_listener("crash", on_crash)
+        except Exception:
+            pass
+
+    log.success(f"Capture {match_label} selesai: {len(items)} data")
+    return CaptureResult(items=items, stopped_by=stopped_by, scrolls=scrolls, diagnostics=diag)
+
+
+async def capture_api(page, url, api_match, extract_fn, key_fn, limit, **kwargs):
+    """Versi ringkas: hanya mengembalikan list item (kompatibel 0.1.x)."""
+    return (await capture(page, url, api_match, extract_fn, key_fn, limit, **kwargs)).items

cdp_scraper/browser/diagnostics.py

@@ -0,0 +1,57 @@
+"""Diagnostik halaman: fingerprint, isi body, screenshot debug."""
+import os
+from dataclasses import dataclass
+from typing import Optional
+
+from ..logs import get_logger
+from .state import LAST_DIAG
+
+
+@dataclass
+class DebugOptions:
+    enabled: bool = True
+    directory: Optional[str] = None
+
+
+async def page_diag(page, nav_resp, tag, diag=None, log=None):
+    log = log or get_logger()
+    diag = {} if diag is None else diag
+    info = {"tag": tag, "engine": diag.get("engine") or LAST_DIAG.get("engine"), "url": page.url,
+            "nav_status": nav_resp.status if nav_resp else None}
+    try:
+        info["title"] = await page.title()
+    except Exception:
+        pass
+    try:
+        info.update(await page.evaluate("""() => ({
+            webdriver: navigator.webdriver,
+            ua: navigator.userAgent,
+            languages: navigator.languages,
+            plugins: navigator.plugins.length,
+            has_window_chrome: !!window.chrome
+        })"""))
+    except Exception as e:
+        info["eval_error"] = str(e)
+    try:
+        info["body_snippet"] = (await page.inner_text("body", timeout=3000))[:200].replace("\n", " ")
+    except Exception:
+        pass
+    diag.update(info)
+    LAST_DIAG.update(info)
+    log.info(f"Diagnostik: {info}")
+    return info
+
+
+async def save_debug_shot(page, name="debug_last.png", debug=None, log=None):
+    log = log or get_logger()
+    debug = debug or DebugOptions()
+    if not debug.enabled:
+        return None
+    path = os.path.join(debug.directory or os.getcwd(), name)
+    try:
+        await page.screenshot(path=path, timeout=5000)
+        log.success(f"Diagnostik: Screenshot disimpan: {path}")
+        return path
+    except Exception as e:
+        log.warning(f"Diagnostik: Gagal screenshot: {e}")
+        return None

cdp_scraper/browser/helpers.py

@@ -0,0 +1,2 @@
+"""Alias kompatibilitas; gunakan `actions.close_overlays`."""
+from .actions import close_overlays as close_login_modal  # noqa: F401