beo-mcdp 0.2.0__tar.gz

beo_mcdp-0.2.0/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: beo-mcdp
+Version: 0.2.0
+Summary: Core CDP Browser Automation and Utils Provider
+Author: Dwi Hardiyanto
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: playwright>=1.57.0
+Provides-Extra: patchright
+Requires-Dist: patchright; extra == "patchright"
+Provides-Extra: kafka
+Requires-Dist: confluent-kafka; extra == "kafka"
+Dynamic: author
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CDP Scraper Provider
+
+Library *core* untuk ekstraksi data lewat browser Chrome (CDP). Satu mesin yang sama dipakai untuk
+banyak sosial media: platform baru cukup **dideskripsikan** (URL, pola API, mapping field), tanpa
+menulis ulang sesi browser, scroll, capture, maupun parser.
+
+## Instalasi
+
+```bash
+pip install git+https://github.com/dwihard/social_scrapers.git@v0.2.0
+# opsional: pip install "cdp_scraper[kafka]"  /  "cdp_scraper[patchright]"
+```
+
+## Cara kerja
+
+1. Browser membuka halaman (`Endpoint.url`).
+2. Library mendengarkan response API yang dipanggil halaman (`Endpoint.api_match`).
+3. JSON-nya diubah jadi list dict oleh `parser` (biasanya `FieldMapper`), di-dedupe lewat `key`, lalu halaman di-scroll sampai `limit`/data habis.
+
+## Menambah platform baru
+
+```python
+from cdp_scraper import (Endpoint, F, FieldMapper, Platform, Scraper,
+                         const, make_scroller, register_platform, end_when_falsy)
+
+parser = FieldMapper(
+    items="data.posts",                      # path ke list item
+    fields={
+        "id":    ("id", "str", ""),          # (path, transform, default)
+        "url":   F(template="https://site.com/{author.handle}/{id}"),
+        "likes": ("stats.likes", "int", 0),
+        "tags":  "tags[].name",              # semua elemen
+        "posted": F("created", transform="iso_utc"),
+        "platform": const("site"),
+    },
+)
+
+site = Platform(
+    name="site",                              # juga jadi label log: [site] ...
+    cdp={"cdp_url": "http://localhost:9223"}, # opsional: Chrome/slot terpisah per platform
+    overlay_selectors=('div.popup-close',),   # popup yang ditutup otomatis
+    login_url_hints=("/login",),              # deteksi browser belum login
+    endpoints=[Endpoint(
+        "search", url="https://site.com/search?q={keyword}", api_match="api/search",
+        parser=parser, params=("keyword",), label="keyword", limit=20, max_limit=50,
+        end=end_when_falsy("has_more"),
+        scroll=make_scroller(wheel=2500, click_pattern="load more"),
+    )],
+)
+register_platform(site)
+
+result = await Scraper("site").run("search", {"keyword": "kopi"}, limit=30)     # satu query
+results = await Scraper("site").run_many("search", [{"keyword": "a"}, {"keyword": "b"}])
+```
+
+`run()` melempar exception (`PageCrashed`, `BrowserBusy`, `ValueError` untuk parameter salah).
+`run_many()` memakai satu tab untuk banyak query dan menjadikan error per-item `status="error"`.
+Contoh lengkap TikTok dan X/Twitter ada di `examples/`.
+
+## Mapping parsing (`FieldMapper`)
+
+| Sintaks path | Arti |
+|---|---|
+| `a.b.c` | nilai bersarang |
+| `a[0].b`, `a[-1]` | indeks |
+| `a[].b` | semua elemen (hasil list) |
+| `a[type=1].b` | elemen yang cocok filter (hasil list) |
+
+Bentuk spec field: `"path"` · `("path", "transform", default)` · `F(path | any_of=[...] | template=... | fn=...,
+default=, transform=, required=)` · `const(x)` · `callable(record)`.
+
+Opsi `FieldMapper`: `item_path` (sub-path record di tiap item), `where` / `record_where`
+(`{path: nilai | EXISTS | callable}` atau callable), `unwrap` (membuka pembungkus), `items` boleh
+list path atau callable.
+
+Transform bawaan: `int float str bool url lower upper strip clean_text username hashtag iso_utc
+date_id join first count unique drop_empty` dan `each:<transform>` untuk list. Argumen format tanggal:
+`iso_utc:%a %b %d %H:%M:%S %z %Y`. Tambah sendiri: `register_transform("nama", fn)`.
+
+Jika logika terlalu rumit untuk mapping, gunakan `F(fn=...)` atau parser Python biasa
+(`parser=` menerima callable apa pun yang mengembalikan list dict).
+
+## Hook & aksi halaman
+
+`prepare` (setelah halaman terbuka) dan `scroll` (tiap putaran) adalah fungsi `async def (page)`.
+Bawaan: `make_scroller(...)`, `make_wait_for(selector)`, `close_overlays(page, selectors)`.
+`Endpoint.capture_options` meneruskan kwargs tambahan ke `capture` (`poll_interval`, `settle`, dst).
+
+## Konfigurasi browser
+
+```python
+from cdp_scraper import CDPConfig, setup_cdp
+setup_cdp(cdp_url="http://localhost:9222", max_sessions=2, lock_timeout=60)   # default global
+cfg = CDPConfig.from_env("TWITTER_")   # TWITTER_CDP_URL, TWITTER_MAX_CONCURRENT_SESSIONS, ...
+```
+
+Setiap request memakai **tab baru sendiri** (ditutup saat selesai) dan dibatasi `max_sessions`
+bersamaan per konfigurasi. Konfigurasi yang sama berbagi slot; `cdp_url` berbeda = slot berbeda.
+Screenshot debug disimpan di `CDPConfig.debug_dir` (default folder kerja); matikan dengan
+`debug_screenshots=False`.
+
+## Logging
+
+Format seragam `[job] Status message: detail`, job = nama platform:
+
+```
+[2026-11-10 08:19:38] [INFO] [site] Success message: Capture api/search selesai: 20 data
+```
+
+Library hanya memakai logger `cdp_scraper` (tanpa handler). Aktifkan dengan `setup_logging()`
+atau injeksi logger sendiri: `setup_cdp(custom_logger=...)`. `get_logger("kafka").error("...")`
+untuk log aplikasi dengan format yang sama.
+
+## Kafka / Redpanda
+
+```python
+from cdp_scraper import RedpandaProducer, publish_chunks
+producer = RedpandaProducer()            # KAFKA_BOOTSTRAP_SERVERS; koneksi dibuat saat dipakai
+if producer.available:
+    stats = publish_chunks(producer, "topic-x", keywords, chunk_size=2,
+                           build_payload=lambda chunk: {"keywords": chunk})
+    # {"berhasil": n, "gagal": n}
+```
+Kontrak sumber data (keyword/akun) ada di `cdp_scraper.utils.database` (`KeywordSource`,
+`AccountSource`); implementasinya milik aplikasi.
+
+## Pengujian
+
+```bash
+python -m unittest discover -s tests -t .
+```
+Tes memakai browser palsu, jadi tidak butuh Chrome/Kafka.
+
+## Perubahan dari 0.1.x (breaking)
+
+- Konfigurasi sekarang `CDPConfig` (immutable, per platform); `setup_cdp(...)` tetap ada. Variabel modul `config.CDP_URL` dst dihapus.
+- `get_session_slots` dan `PW_ENGINE` dihapus; pakai `CDPSessionManager` / `manager.engine`.
+- `BrowserBusy` (subclass `TimeoutError`) menggantikan `TimeoutError` polos.
+- `helpers.close_login_modal(page, selectors)` kini wajib diberi selector (tidak ada lagi selector TikTok bawaan).
+- `utils`: tidak lagi mengimpor Flask; `extract_challenges` (khusus TikTok) dihapus; `calculate_engagement_rate` menerima `interaction_keys` / `view_key` (default sama dengan perilaku lama).
+- `utils.database` kini berupa Protocol; `utils.redpanda_client` tidak lagi membuat producer saat import (gunakan `cdp_scraper.messaging`).

beo_mcdp-0.2.0/README.md

@@ -0,0 +1,141 @@
+# CDP Scraper Provider
+
+Library *core* untuk ekstraksi data lewat browser Chrome (CDP). Satu mesin yang sama dipakai untuk
+banyak sosial media: platform baru cukup **dideskripsikan** (URL, pola API, mapping field), tanpa
+menulis ulang sesi browser, scroll, capture, maupun parser.
+
+## Instalasi
+
+```bash
+pip install git+https://github.com/dwihard/social_scrapers.git@v0.2.0
+# opsional: pip install "cdp_scraper[kafka]"  /  "cdp_scraper[patchright]"
+```
+
+## Cara kerja
+
+1. Browser membuka halaman (`Endpoint.url`).
+2. Library mendengarkan response API yang dipanggil halaman (`Endpoint.api_match`).
+3. JSON-nya diubah jadi list dict oleh `parser` (biasanya `FieldMapper`), di-dedupe lewat `key`, lalu halaman di-scroll sampai `limit`/data habis.
+
+## Menambah platform baru
+
+```python
+from cdp_scraper import (Endpoint, F, FieldMapper, Platform, Scraper,
+                         const, make_scroller, register_platform, end_when_falsy)
+
+parser = FieldMapper(
+    items="data.posts",                      # path ke list item
+    fields={
+        "id":    ("id", "str", ""),          # (path, transform, default)
+        "url":   F(template="https://site.com/{author.handle}/{id}"),
+        "likes": ("stats.likes", "int", 0),
+        "tags":  "tags[].name",              # semua elemen
+        "posted": F("created", transform="iso_utc"),
+        "platform": const("site"),
+    },
+)
+
+site = Platform(
+    name="site",                              # juga jadi label log: [site] ...
+    cdp={"cdp_url": "http://localhost:9223"}, # opsional: Chrome/slot terpisah per platform
+    overlay_selectors=('div.popup-close',),   # popup yang ditutup otomatis
+    login_url_hints=("/login",),              # deteksi browser belum login
+    endpoints=[Endpoint(
+        "search", url="https://site.com/search?q={keyword}", api_match="api/search",
+        parser=parser, params=("keyword",), label="keyword", limit=20, max_limit=50,
+        end=end_when_falsy("has_more"),
+        scroll=make_scroller(wheel=2500, click_pattern="load more"),
+    )],
+)
+register_platform(site)
+
+result = await Scraper("site").run("search", {"keyword": "kopi"}, limit=30)     # satu query
+results = await Scraper("site").run_many("search", [{"keyword": "a"}, {"keyword": "b"}])
+```
+
+`run()` melempar exception (`PageCrashed`, `BrowserBusy`, `ValueError` untuk parameter salah).
+`run_many()` memakai satu tab untuk banyak query dan menjadikan error per-item `status="error"`.
+Contoh lengkap TikTok dan X/Twitter ada di `examples/`.
+
+## Mapping parsing (`FieldMapper`)
+
+| Sintaks path | Arti |
+|---|---|
+| `a.b.c` | nilai bersarang |
+| `a[0].b`, `a[-1]` | indeks |
+| `a[].b` | semua elemen (hasil list) |
+| `a[type=1].b` | elemen yang cocok filter (hasil list) |
+
+Bentuk spec field: `"path"` · `("path", "transform", default)` · `F(path | any_of=[...] | template=... | fn=...,
+default=, transform=, required=)` · `const(x)` · `callable(record)`.
+
+Opsi `FieldMapper`: `item_path` (sub-path record di tiap item), `where` / `record_where`
+(`{path: nilai | EXISTS | callable}` atau callable), `unwrap` (membuka pembungkus), `items` boleh
+list path atau callable.
+
+Transform bawaan: `int float str bool url lower upper strip clean_text username hashtag iso_utc
+date_id join first count unique drop_empty` dan `each:<transform>` untuk list. Argumen format tanggal:
+`iso_utc:%a %b %d %H:%M:%S %z %Y`. Tambah sendiri: `register_transform("nama", fn)`.
+
+Jika logika terlalu rumit untuk mapping, gunakan `F(fn=...)` atau parser Python biasa
+(`parser=` menerima callable apa pun yang mengembalikan list dict).
+
+## Hook & aksi halaman
+
+`prepare` (setelah halaman terbuka) dan `scroll` (tiap putaran) adalah fungsi `async def (page)`.
+Bawaan: `make_scroller(...)`, `make_wait_for(selector)`, `close_overlays(page, selectors)`.
+`Endpoint.capture_options` meneruskan kwargs tambahan ke `capture` (`poll_interval`, `settle`, dst).
+
+## Konfigurasi browser
+
+```python
+from cdp_scraper import CDPConfig, setup_cdp
+setup_cdp(cdp_url="http://localhost:9222", max_sessions=2, lock_timeout=60)   # default global
+cfg = CDPConfig.from_env("TWITTER_")   # TWITTER_CDP_URL, TWITTER_MAX_CONCURRENT_SESSIONS, ...
+```
+
+Setiap request memakai **tab baru sendiri** (ditutup saat selesai) dan dibatasi `max_sessions`
+bersamaan per konfigurasi. Konfigurasi yang sama berbagi slot; `cdp_url` berbeda = slot berbeda.
+Screenshot debug disimpan di `CDPConfig.debug_dir` (default folder kerja); matikan dengan
+`debug_screenshots=False`.
+
+## Logging
+
+Format seragam `[job] Status message: detail`, job = nama platform:
+
+```
+[2026-11-10 08:19:38] [INFO] [site] Success message: Capture api/search selesai: 20 data
+```
+
+Library hanya memakai logger `cdp_scraper` (tanpa handler). Aktifkan dengan `setup_logging()`
+atau injeksi logger sendiri: `setup_cdp(custom_logger=...)`. `get_logger("kafka").error("...")`
+untuk log aplikasi dengan format yang sama.
+
+## Kafka / Redpanda
+
+```python
+from cdp_scraper import RedpandaProducer, publish_chunks
+producer = RedpandaProducer()            # KAFKA_BOOTSTRAP_SERVERS; koneksi dibuat saat dipakai
+if producer.available:
+    stats = publish_chunks(producer, "topic-x", keywords, chunk_size=2,
+                           build_payload=lambda chunk: {"keywords": chunk})
+    # {"berhasil": n, "gagal": n}
+```
+Kontrak sumber data (keyword/akun) ada di `cdp_scraper.utils.database` (`KeywordSource`,
+`AccountSource`); implementasinya milik aplikasi.
+
+## Pengujian
+
+```bash
+python -m unittest discover -s tests -t .
+```
+Tes memakai browser palsu, jadi tidak butuh Chrome/Kafka.
+
+## Perubahan dari 0.1.x (breaking)
+
+- Konfigurasi sekarang `CDPConfig` (immutable, per platform); `setup_cdp(...)` tetap ada. Variabel modul `config.CDP_URL` dst dihapus.
+- `get_session_slots` dan `PW_ENGINE` dihapus; pakai `CDPSessionManager` / `manager.engine`.
+- `BrowserBusy` (subclass `TimeoutError`) menggantikan `TimeoutError` polos.
+- `helpers.close_login_modal(page, selectors)` kini wajib diberi selector (tidak ada lagi selector TikTok bawaan).
+- `utils`: tidak lagi mengimpor Flask; `extract_challenges` (khusus TikTok) dihapus; `calculate_engagement_rate` menerima `interaction_keys` / `view_key` (default sama dengan perilaku lama).
+- `utils.database` kini berupa Protocol; `utils.redpanda_client` tidak lagi membuat producer saat import (gunakan `cdp_scraper.messaging`).

beo_mcdp-0.2.0/beo_mcdp.egg-info/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: beo-mcdp
+Version: 0.2.0
+Summary: Core CDP Browser Automation and Utils Provider
+Author: Dwi Hardiyanto
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: playwright>=1.57.0
+Provides-Extra: patchright
+Requires-Dist: patchright; extra == "patchright"
+Provides-Extra: kafka
+Requires-Dist: confluent-kafka; extra == "kafka"
+Dynamic: author
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CDP Scraper Provider
+
+Library *core* untuk ekstraksi data lewat browser Chrome (CDP). Satu mesin yang sama dipakai untuk
+banyak sosial media: platform baru cukup **dideskripsikan** (URL, pola API, mapping field), tanpa
+menulis ulang sesi browser, scroll, capture, maupun parser.
+
+## Instalasi
+
+```bash
+pip install git+https://github.com/dwihard/social_scrapers.git@v0.2.0
+# opsional: pip install "cdp_scraper[kafka]"  /  "cdp_scraper[patchright]"
+```
+
+## Cara kerja
+
+1. Browser membuka halaman (`Endpoint.url`).
+2. Library mendengarkan response API yang dipanggil halaman (`Endpoint.api_match`).
+3. JSON-nya diubah jadi list dict oleh `parser` (biasanya `FieldMapper`), di-dedupe lewat `key`, lalu halaman di-scroll sampai `limit`/data habis.
+
+## Menambah platform baru
+
+```python
+from cdp_scraper import (Endpoint, F, FieldMapper, Platform, Scraper,
+                         const, make_scroller, register_platform, end_when_falsy)
+
+parser = FieldMapper(
+    items="data.posts",                      # path ke list item
+    fields={
+        "id":    ("id", "str", ""),          # (path, transform, default)
+        "url":   F(template="https://site.com/{author.handle}/{id}"),
+        "likes": ("stats.likes", "int", 0),
+        "tags":  "tags[].name",              # semua elemen
+        "posted": F("created", transform="iso_utc"),
+        "platform": const("site"),
+    },
+)
+
+site = Platform(
+    name="site",                              # juga jadi label log: [site] ...
+    cdp={"cdp_url": "http://localhost:9223"}, # opsional: Chrome/slot terpisah per platform
+    overlay_selectors=('div.popup-close',),   # popup yang ditutup otomatis
+    login_url_hints=("/login",),              # deteksi browser belum login
+    endpoints=[Endpoint(
+        "search", url="https://site.com/search?q={keyword}", api_match="api/search",
+        parser=parser, params=("keyword",), label="keyword", limit=20, max_limit=50,
+        end=end_when_falsy("has_more"),
+        scroll=make_scroller(wheel=2500, click_pattern="load more"),
+    )],
+)
+register_platform(site)
+
+result = await Scraper("site").run("search", {"keyword": "kopi"}, limit=30)     # satu query
+results = await Scraper("site").run_many("search", [{"keyword": "a"}, {"keyword": "b"}])
+```
+
+`run()` melempar exception (`PageCrashed`, `BrowserBusy`, `ValueError` untuk parameter salah).
+`run_many()` memakai satu tab untuk banyak query dan menjadikan error per-item `status="error"`.
+Contoh lengkap TikTok dan X/Twitter ada di `examples/`.
+
+## Mapping parsing (`FieldMapper`)
+
+| Sintaks path | Arti |
+|---|---|
+| `a.b.c` | nilai bersarang |
+| `a[0].b`, `a[-1]` | indeks |
+| `a[].b` | semua elemen (hasil list) |
+| `a[type=1].b` | elemen yang cocok filter (hasil list) |
+
+Bentuk spec field: `"path"` · `("path", "transform", default)` · `F(path | any_of=[...] | template=... | fn=...,
+default=, transform=, required=)` · `const(x)` · `callable(record)`.
+
+Opsi `FieldMapper`: `item_path` (sub-path record di tiap item), `where` / `record_where`
+(`{path: nilai | EXISTS | callable}` atau callable), `unwrap` (membuka pembungkus), `items` boleh
+list path atau callable.
+
+Transform bawaan: `int float str bool url lower upper strip clean_text username hashtag iso_utc
+date_id join first count unique drop_empty` dan `each:<transform>` untuk list. Argumen format tanggal:
+`iso_utc:%a %b %d %H:%M:%S %z %Y`. Tambah sendiri: `register_transform("nama", fn)`.
+
+Jika logika terlalu rumit untuk mapping, gunakan `F(fn=...)` atau parser Python biasa
+(`parser=` menerima callable apa pun yang mengembalikan list dict).
+
+## Hook & aksi halaman
+
+`prepare` (setelah halaman terbuka) dan `scroll` (tiap putaran) adalah fungsi `async def (page)`.
+Bawaan: `make_scroller(...)`, `make_wait_for(selector)`, `close_overlays(page, selectors)`.
+`Endpoint.capture_options` meneruskan kwargs tambahan ke `capture` (`poll_interval`, `settle`, dst).
+
+## Konfigurasi browser
+
+```python
+from cdp_scraper import CDPConfig, setup_cdp
+setup_cdp(cdp_url="http://localhost:9222", max_sessions=2, lock_timeout=60)   # default global
+cfg = CDPConfig.from_env("TWITTER_")   # TWITTER_CDP_URL, TWITTER_MAX_CONCURRENT_SESSIONS, ...
+```
+
+Setiap request memakai **tab baru sendiri** (ditutup saat selesai) dan dibatasi `max_sessions`
+bersamaan per konfigurasi. Konfigurasi yang sama berbagi slot; `cdp_url` berbeda = slot berbeda.
+Screenshot debug disimpan di `CDPConfig.debug_dir` (default folder kerja); matikan dengan
+`debug_screenshots=False`.
+
+## Logging
+
+Format seragam `[job] Status message: detail`, job = nama platform:
+
+```
+[2026-11-10 08:19:38] [INFO] [site] Success message: Capture api/search selesai: 20 data
+```
+
+Library hanya memakai logger `cdp_scraper` (tanpa handler). Aktifkan dengan `setup_logging()`
+atau injeksi logger sendiri: `setup_cdp(custom_logger=...)`. `get_logger("kafka").error("...")`
+untuk log aplikasi dengan format yang sama.
+
+## Kafka / Redpanda
+
+```python
+from cdp_scraper import RedpandaProducer, publish_chunks
+producer = RedpandaProducer()            # KAFKA_BOOTSTRAP_SERVERS; koneksi dibuat saat dipakai
+if producer.available:
+    stats = publish_chunks(producer, "topic-x", keywords, chunk_size=2,
+                           build_payload=lambda chunk: {"keywords": chunk})
+    # {"berhasil": n, "gagal": n}
+```
+Kontrak sumber data (keyword/akun) ada di `cdp_scraper.utils.database` (`KeywordSource`,
+`AccountSource`); implementasinya milik aplikasi.
+
+## Pengujian
+
+```bash
+python -m unittest discover -s tests -t .
+```
+Tes memakai browser palsu, jadi tidak butuh Chrome/Kafka.
+
+## Perubahan dari 0.1.x (breaking)
+
+- Konfigurasi sekarang `CDPConfig` (immutable, per platform); `setup_cdp(...)` tetap ada. Variabel modul `config.CDP_URL` dst dihapus.
+- `get_session_slots` dan `PW_ENGINE` dihapus; pakai `CDPSessionManager` / `manager.engine`.
+- `BrowserBusy` (subclass `TimeoutError`) menggantikan `TimeoutError` polos.
+- `helpers.close_login_modal(page, selectors)` kini wajib diberi selector (tidak ada lagi selector TikTok bawaan).
+- `utils`: tidak lagi mengimpor Flask; `extract_challenges` (khusus TikTok) dihapus; `calculate_engagement_rate` menerima `interaction_keys` / `view_key` (default sama dengan perilaku lama).
+- `utils.database` kini berupa Protocol; `utils.redpanda_client` tidak lagi membuat producer saat import (gunakan `cdp_scraper.messaging`).

beo_mcdp-0.2.0/beo_mcdp.egg-info/SOURCES.txt

@@ -0,0 +1,37 @@
+README.md
+setup.py
+beo_mcdp.egg-info/PKG-INFO
+beo_mcdp.egg-info/SOURCES.txt
+beo_mcdp.egg-info/dependency_links.txt
+beo_mcdp.egg-info/requires.txt
+beo_mcdp.egg-info/top_level.txt
+cdp_scraper/__init__.py
+cdp_scraper/_version.py
+cdp_scraper/config.py
+cdp_scraper/errors.py
+cdp_scraper/logs.py
+cdp_scraper/parsers.py
+cdp_scraper/browser/__init__.py
+cdp_scraper/browser/actions.py
+cdp_scraper/browser/capture.py
+cdp_scraper/browser/diagnostics.py
+cdp_scraper/browser/helpers.py
+cdp_scraper/browser/session.py
+cdp_scraper/browser/state.py
+cdp_scraper/messaging/__init__.py
+cdp_scraper/messaging/redpanda.py
+cdp_scraper/parsing/__init__.py
+cdp_scraper/parsing/mapper.py
+cdp_scraper/parsing/paths.py
+cdp_scraper/parsing/transforms.py
+cdp_scraper/platforms/__init__.py
+cdp_scraper/platforms/registry.py
+cdp_scraper/platforms/runner.py
+cdp_scraper/platforms/spec.py
+cdp_scraper/utils/__init__.py
+cdp_scraper/utils/database.py
+cdp_scraper/utils/redpanda_client.py
+cdp_scraper/utils/utils.py
+tests/test_examples.py
+tests/test_parsing.py
+tests/test_runtime.py

beo_mcdp-0.2.0/beo_mcdp.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

beo_mcdp-0.2.0/beo_mcdp.egg-info/requires.txt

@@ -0,0 +1,7 @@
+playwright>=1.57.0
+
+[kafka]
+confluent-kafka
+
+[patchright]
+patchright

beo_mcdp-0.2.0/beo_mcdp.egg-info/top_level.txt

@@ -0,0 +1 @@
+cdp_scraper

beo_mcdp-0.2.0/cdp_scraper/__init__.py

@@ -0,0 +1,24 @@
+from ._version import __version__
+from .browser.actions import close_overlays, make_scroller, make_wait_for, scroll_page_down
+from .browser.capture import CaptureResult, capture, capture_api
+from .browser.session import (CDPSessionManager, default_manager, get_manager,
+                              is_browser_busy, playwright_session)
+from .config import CDPConfig, get_default_config, set_default_config, setup_cdp
+from .errors import BrowserBusy, CDPError, PageCrashed
+from .logs import JobLogger, get_logger, setup_logging
+from .messaging import RedpandaProducer, chunk_list, publish_chunks
+from .parsing import EXISTS, F, FieldMapper, const, get_path, register_transform
+from .platforms import (Endpoint, Platform, ScrapeResult, Scraper, end_when_empty,
+                        end_when_falsy, get_platform, list_platforms, register_platform)
+
+__all__ = [
+    "__version__", "CDPConfig", "setup_cdp", "get_default_config", "set_default_config",
+    "CDPError", "PageCrashed", "BrowserBusy", "JobLogger", "get_logger", "setup_logging",
+    "CDPSessionManager", "get_manager", "default_manager", "playwright_session", "is_browser_busy",
+    "capture", "capture_api", "CaptureResult",
+    "close_overlays", "make_scroller", "make_wait_for", "scroll_page_down",
+    "FieldMapper", "F", "const", "EXISTS", "get_path", "register_transform",
+    "Platform", "Endpoint", "Scraper", "ScrapeResult", "register_platform", "get_platform",
+    "list_platforms", "end_when_falsy", "end_when_empty",
+    "RedpandaProducer", "publish_chunks", "chunk_list",
+]

beo_mcdp-0.2.0/cdp_scraper/_version.py

@@ -0,0 +1 @@
+__version__ = "0.2.0"

beo_mcdp-0.2.0/cdp_scraper/browser/actions.py

@@ -0,0 +1,74 @@
+"""Aksi halaman yang bisa dirakit per platform: tutup overlay, scroll, tunggu elemen."""
+from ..logs import get_logger
+
+_SCROLL_JS = """(o) => {
+    if (o.window === 'bottom') window.scrollTo(0, document.documentElement.scrollHeight);
+    else if (o.window === 'viewport') window.scrollBy(0, window.innerHeight * o.factor);
+    if (o.containers) {
+        for (const el of document.querySelectorAll('div')) {
+            const st = getComputedStyle(el);
+            if ((st.overflowY === 'auto' || st.overflowY === 'scroll')
+                && el.scrollHeight > el.clientHeight + 50) {
+                el.scrollTop = el.scrollHeight;
+            }
+        }
+    }
+    if (o.pattern) {
+        const re = new RegExp(o.pattern, 'i');
+        const btn = [...document.querySelectorAll('button, [role="button"]')]
+            .find(b => re.test((b.innerText || '').trim()));
+        if (btn) btn.click();
+    }
+}"""
+
+
+async def close_overlays(page, selectors):
+    """Klik tombol penutup popup/modal (selector CSS diberikan platform)."""
+    if not selectors:
+        return
+    try:
+        await page.evaluate(
+            "(sels) => document.querySelectorAll(sels.join(',')).forEach(b => b.click())",
+            list(selectors))
+    except Exception:
+        pass
+
+
+async def scroll_page_down(page):
+    await page.keyboard.press("PageDown")
+
+
+def make_scroller(*, wheel=2500, window_scroll="bottom", viewport_factor=2.0,
+                  containers=False, click_pattern=None, mouse=(0.5, 0.5), key=None):
+    """Bangun fungsi scroll generik.
+
+    wheel: delta mouse wheel (0 = nonaktif); window_scroll: 'bottom' | 'viewport' | None;
+    containers: scroll semua div overflow; click_pattern: regex teks tombol 'load more'/'retry';
+    mouse: posisi kursor (fraksi viewport); key: tombol keyboard tambahan (mis. 'End').
+    """
+    options = {"window": window_scroll, "factor": viewport_factor,
+               "containers": containers, "pattern": click_pattern}
+    use_js = bool(window_scroll or containers or click_pattern)
+
+    async def _scroll(page):
+        if wheel:
+            vp = page.viewport_size or {"width": 1280, "height": 800}
+            await page.mouse.move(vp["width"] * mouse[0], vp["height"] * mouse[1])
+            await page.mouse.wheel(0, wheel)
+        if use_js:
+            await page.evaluate(_SCROLL_JS, options)
+        if key:
+            await page.keyboard.press(key)
+
+    return _scroll
+
+
+def make_wait_for(selector, timeout_ms=15000, message=None):
+    """Fungsi prepare: tunggu elemen pertama muncul (hanya warning jika tidak muncul)."""
+    async def _wait(page, log=None):
+        log = log or get_logger()
+        try:
+            await page.wait_for_selector(selector, state="attached", timeout=timeout_ms)
+        except Exception:
+            log.warning(message or f"Elemen '{selector}' belum muncul setelah {timeout_ms // 1000} detik.")
+    return _wait