zed-scrapy-playwright 0.1.0__tar.gz

zed_scrapy_playwright-0.1.0/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.3
+Name: zed-scrapy-playwright
+Version: 0.1.0
+Summary: Add your description here
+Author: zhouzhouos
+Author-email: zhouzhouos <leisure-zyf@qq.com>
+Requires-Dist: playwright>=1.62.0
+Requires-Dist: playwright-stealth>=2.0.3
+Requires-Dist: scrapy>=2.18.0
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+
+# zed-scrapy-playwright
+
+> Less meta config, more Python code.
+
+> 把 Scrapy 变成 Playwright 的事件调度引擎。
+
+[English](docs/README-en.md)
+
+---
+
+## 简介
+
+`zed-scrapy-playwright` 是一个 Scrapy 中间件，它重新定义了 Scrapy 与 Playwright 的集成方式——**让 Scrapy 成为事件调度器，让 Playwright 成为可编程的执行引擎。**
+
+不同于传统方案将每个请求限制为单次页面导航，本中间件允许你在一个 `execution` 中完成完整的业务流程：登录、搜索、详情、下单，一气呵成。
+
+---
+
+## 核心特性
+
+- **会话级控制** — 单次 execution 内多次 `goto`，登录态自动保持
+- **线性编码** — 复杂流程写在一个函数里，告别 callback 地狱
+- **精准数据传输** — 在 Page 中直接提取初筛数据，避免传输冗余 HTML
+- **完全开放** — 任意 Playwright API 均可使用，无预定义操作限制
+- **双向交叉检查** — 中间件与爬虫相互验证，配置错误快速失败
+- **Stealth 灵活可控** — 可全局注入，也可在每个 execution 中自定义
+- **类型安全** — 完整的类型提示，IDE 友好
+
+---
+
+## 快速开始
+
+### 安装
+
+```bash
+pip install zed-scrapy-playwright
+```
+
+### 配置 settings.py
+
+```python
+DOWNLOADER_MIDDLEWARES = {
+    "zed_scrapy_playwright.handler.PlaywrightDownloaderMiddleware": 300,
+}
+```
+
+### 编写爬虫
+
+```python
+import zed_scrapy_playwright as zsp
+from playwright.async_api import Page
+
+class MySpider(zsp.Spider):
+    name = "example"
+
+    async def start(self):
+        yield zsp.Request(
+            exec_type="NewPage",
+            execution=self.search_and_scrape,
+            callback=self.parse,
+            meta={"keyword": "laptop"},
+        )
+
+    @staticmethod
+    async def search_and_scrape(request: zsp.Request, page: Page):
+        keyword = request.meta["keyword"]
+
+        await page.goto("https://example.com")
+        await page.fill("#search", keyword)
+        await page.click("#search-btn")
+        await page.wait_for_selector(".results")
+
+        # 初筛数据：直接在页面中提取
+        items = await page.evaluate("""
+            Array.from(document.querySelectorAll('.item')).map(el => ({
+                title: el.querySelector('.title')?.innerText,
+                price: el.querySelector('.price')?.innerText,
+            }))
+        """)
+
+        return {"items": items, "keyword": keyword}
+
+    async def parse(self, response: zsp.Response):
+        data = response.data  # 初筛数据
+        for item in data["items"]:
+            yield {
+                "title": item["title"],
+                "price": float(item["price"].replace("$", "")),
+            }
+```
+
+---
+
+## 对比 scrapy-playwright
+
+| 维度 | scrapy-playwright | zed-scrapy-playwright |
+|------|:---:|:---:|
+| 操作自由度 | 有限 | 无限 |
+| 多次 goto | ❌ | ✅ |
+| 登录态保持 | 配置复杂 | 天然支持 |
+| 代码组织 | 分散 callback | 线性聚合 |
+| 数据传输 | 完整 HTML | 初筛数据 |
+| 防误用机制 | 无 | 双向交叉检查 |
+
+---
+
+## 许可证
+
+暂定：MIT

zed_scrapy_playwright-0.1.0/README.md

@@ -0,0 +1,109 @@
+# zed-scrapy-playwright
+
+> Less meta config, more Python code.
+
+> 把 Scrapy 变成 Playwright 的事件调度引擎。
+
+[English](docs/README-en.md)
+
+---
+
+## 简介
+
+`zed-scrapy-playwright` 是一个 Scrapy 中间件，它重新定义了 Scrapy 与 Playwright 的集成方式——**让 Scrapy 成为事件调度器，让 Playwright 成为可编程的执行引擎。**
+
+不同于传统方案将每个请求限制为单次页面导航，本中间件允许你在一个 `execution` 中完成完整的业务流程：登录、搜索、详情、下单，一气呵成。
+
+---
+
+## 核心特性
+
+- **会话级控制** — 单次 execution 内多次 `goto`，登录态自动保持
+- **线性编码** — 复杂流程写在一个函数里，告别 callback 地狱
+- **精准数据传输** — 在 Page 中直接提取初筛数据，避免传输冗余 HTML
+- **完全开放** — 任意 Playwright API 均可使用，无预定义操作限制
+- **双向交叉检查** — 中间件与爬虫相互验证，配置错误快速失败
+- **Stealth 灵活可控** — 可全局注入，也可在每个 execution 中自定义
+- **类型安全** — 完整的类型提示，IDE 友好
+
+---
+
+## 快速开始
+
+### 安装
+
+```bash
+pip install zed-scrapy-playwright
+```
+
+### 配置 settings.py
+
+```python
+DOWNLOADER_MIDDLEWARES = {
+    "zed_scrapy_playwright.handler.PlaywrightDownloaderMiddleware": 300,
+}
+```
+
+### 编写爬虫
+
+```python
+import zed_scrapy_playwright as zsp
+from playwright.async_api import Page
+
+class MySpider(zsp.Spider):
+    name = "example"
+
+    async def start(self):
+        yield zsp.Request(
+            exec_type="NewPage",
+            execution=self.search_and_scrape,
+            callback=self.parse,
+            meta={"keyword": "laptop"},
+        )
+
+    @staticmethod
+    async def search_and_scrape(request: zsp.Request, page: Page):
+        keyword = request.meta["keyword"]
+
+        await page.goto("https://example.com")
+        await page.fill("#search", keyword)
+        await page.click("#search-btn")
+        await page.wait_for_selector(".results")
+
+        # 初筛数据：直接在页面中提取
+        items = await page.evaluate("""
+            Array.from(document.querySelectorAll('.item')).map(el => ({
+                title: el.querySelector('.title')?.innerText,
+                price: el.querySelector('.price')?.innerText,
+            }))
+        """)
+
+        return {"items": items, "keyword": keyword}
+
+    async def parse(self, response: zsp.Response):
+        data = response.data  # 初筛数据
+        for item in data["items"]:
+            yield {
+                "title": item["title"],
+                "price": float(item["price"].replace("$", "")),
+            }
+```
+
+---
+
+## 对比 scrapy-playwright
+
+| 维度 | scrapy-playwright | zed-scrapy-playwright |
+|------|:---:|:---:|
+| 操作自由度 | 有限 | 无限 |
+| 多次 goto | ❌ | ✅ |
+| 登录态保持 | 配置复杂 | 天然支持 |
+| 代码组织 | 分散 callback | 线性聚合 |
+| 数据传输 | 完整 HTML | 初筛数据 |
+| 防误用机制 | 无 | 双向交叉检查 |
+
+---
+
+## 许可证
+
+暂定：MIT

zed_scrapy_playwright-0.1.0/pyproject.toml

@@ -0,0 +1,22 @@
+[project]
+name = "zed-scrapy-playwright"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "playwright>=1.62.0",
+    "playwright-stealth>=2.0.3",
+    "scrapy>=2.18.0",
+]
+
+[[project.authors]]
+name = "zhouzhouos"
+email = "leisure-zyf@qq.com"
+
+[project.scripts]
+zed-scrapy-playwright = "zed_scrapy_playwright:main"
+
+[build-system]
+requires = ["uv_build>=0.12.5,<0.13.0"]
+build-backend = "uv_build"

zed_scrapy_playwright-0.1.0/pyproject.toml.orig

@@ -0,0 +1,21 @@
+[project]
+name = "zed-scrapy-playwright"
+version = "0.1.0"
+description = "Add your description here"
+readme = "README.md"
+authors = [
+    { name = "zhouzhouos", email = "leisure-zyf@qq.com" }
+]
+requires-python = ">=3.12"
+dependencies = [
+    "playwright>=1.62.0",
+    "playwright-stealth>=2.0.3",
+    "scrapy>=2.18.0",
+]
+
+[project.scripts]
+zed-scrapy-playwright = "zed_scrapy_playwright:main"
+
+[build-system]
+requires = ["uv_build>=0.12.5,<0.13.0"]
+build-backend = "uv_build"

zed_scrapy_playwright-0.1.0/src/zed_scrapy_playwright/__init__.py

@@ -0,0 +1 @@
+from .definition import PREFIX, Request, Response, Spider

zed_scrapy_playwright-0.1.0/src/zed_scrapy_playwright/definition.py

@@ -0,0 +1,104 @@
+# from scrapy.http import Response, TextResponse
+from collections.abc import Awaitable, Callable
+from typing import Any, Literal, Self
+
+import scrapy
+import scrapy.http
+from playwright.async_api import Page
+from scrapy.crawler import Crawler
+from scrapy.exceptions import NotConfigured
+
+PREFIX = "playwright"
+ExecType = Literal["NewPage"]
+
+X = "zed_scrapy_playwright.handler.PlaywrightDownloaderMiddleware"
+
+
+class Response(scrapy.http.Response):
+    def __init__(
+        self,
+        result,
+        url=f"{PREFIX}://response",
+        status=200,
+        headers=None,
+        body=b"",
+        flags=None,
+        request=None,
+        certificate=None,
+        ip_address=None,
+        protocol=None,
+    ):
+        super().__init__(
+            url,
+            status,
+            headers,
+            body,
+            flags,
+            request,
+            certificate,
+            ip_address,
+            protocol,
+        )
+        self.result: Any = result
+
+
+class Request(scrapy.Request):
+    def __init__(
+        self,
+        exec_type: ExecType = "NewPage",
+        execution: Callable[["Request", Page], Awaitable[Any]] | None = None,
+        callback=None,
+        # method="GET",
+        # headers=None,
+        # body=None,
+        # cookies=None,
+        meta=None,
+        # encoding="utf-8",
+        # priority=0,
+        dont_filter=True,
+        # errback=None,
+        # flags=None,
+        # cb_kwargs=None,
+    ):
+        super().__init__(
+            url=f"{PREFIX}://{exec_type}",
+            callback=callback,
+            # method,
+            # headers,
+            # body,
+            # cookies,
+            meta=meta,
+            # encoding,
+            # priority,
+            # dont_filter,
+            # errback,
+            # flags,
+            # cb_kwargs,
+        )
+        self.exec_type: ExecType = exec_type
+        self.execution = execution if execution else self._default_execution
+
+    @staticmethod
+    async def _default_execution(request: "Request", page: Page):
+        print("Warning")
+
+
+class Spider(scrapy.Spider):
+    """用以判断是否启用此对应中间件的环境"""
+
+    info = {"headless": False, "executable_path": None}
+
+    @classmethod
+    def from_crawler(cls, crawler: Crawler, *args: Any, **kwargs: Any) -> Self:
+        spider = cls(*args, **kwargs)
+        spider._set_crawler(crawler)
+
+        if X not in spider.settings.getdict("DOWNLOADER_MIDDLEWARES"):
+            raise NotConfigured(
+                f"没有注册 {X}, 则不能启用该中间件及其对应的爬虫类 {type(spider)}。"
+            )
+
+        return spider
+
+    def __init__(self, name: str | None = None, **kwargs: dict):
+        super().__init__(name, **kwargs)

zed_scrapy_playwright-0.1.0/src/zed_scrapy_playwright/example.py

@@ -0,0 +1,25 @@
+from playwright.async_api import Page
+
+import zed_scrapy_playwright as zsp
+
+
+class ExampleExecutor(zsp.Spider):
+    name = "example"
+    info = {"headless": False, "executable_path": "/opt/google/chrome/chrome"}
+    # allowed_domains = ["example.com"]
+    # start_urls = ["https://example.com"]
+
+    async def start(self):
+
+        yield zsp.Request(
+            execution=self.exection,
+            callback=self.parse,
+        )
+
+    @staticmethod
+    async def exection(request: zsp.Request, page: Page):
+        await page.goto("https://example.com")
+        return await page.content()
+
+    def parse(self, response: zsp.Response):
+        print(response.result)

zed_scrapy_playwright-0.1.0/src/zed_scrapy_playwright/handler.py

@@ -0,0 +1,115 @@
+# Define here the models for your spider middleware
+#
+# See documentation in:
+# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
+
+# useful for handling different item types with a single interface
+# from itemadapter import ItemAdapter
+
+
+from playwright.async_api import async_playwright
+from playwright_stealth import Stealth
+from scrapy import Request, crawler, signals
+
+from . import definition as Zed
+
+
+class PlaywrightDownloaderMiddleware:
+    # Not all methods need to be defined. If a method is not defined,
+    # scrapy acts as if the downloader middleware does not modify the
+    # passed objects.
+
+    @classmethod
+    def from_crawler(cls, crawler: crawler.Crawler):
+        # This method is used by Scrapy to create your spiders.
+        s = cls()
+        s.c = crawler
+        # 连接中间件在爬虫开始时的触发事件
+        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
+        # 连接中间件在爬虫结束时的触发事件
+        crawler.signals.connect(s.spider_closed, signal=signals.spider_closed)
+        return s
+
+    async def process_request(self, request: Request, spider=None):
+        # Called for each request that goes through the downloader
+        # middleware.
+
+        # Must either:
+        # - return None: continue processing this request
+        # - or return a Response object
+        # - or return a Request object
+        # - or raise IgnoreRequest: process_exception() methods of
+        #   installed downloader middleware will be called
+
+        # 这里处理主动发起的自定义的 request 类型
+        if isinstance(request, Zed.Request):
+            print("主动发起", request.url, request.meta)
+
+            match request.exec_type:
+                case "NewPage":
+                    page = await self.context.new_page()
+
+                    if request.meta.get("no-stealth"):
+                        pass
+                    else:
+                        print("已为 new page 自动施加 Stealth.apply_stealth_async")
+                        await Stealth().apply_stealth_async(page)
+
+                    r = await request.execution(request, page)
+                    return Zed.Response(r, request=request)
+
+                case _:
+                    pass
+
+        # 这里处理自动发起的 scrapy.Request 类型，比如 <class 'scrapy.http.request.Request'> wpwp://nothing/robots.txt
+        if request.url.startswith(Zed.PREFIX):
+            print("自动请求", request.url)
+            return Zed.Response("", request=request)
+
+        # 其他的寻常的 request 不在这里截留，让其 continue
+
+    def process_response(self, request, response, spider=None):
+        # Called with the response returned from the downloader.
+
+        # Must either;
+        # - return a Response object
+        # - return a Request object
+        # - or raise IgnoreRequest
+        return response
+
+    def process_exception(self, request, exception, spider=None):
+        # Called when a download handler or a process_request()
+        # (from other downloader middleware) raises an exception.
+
+        # Must either:
+        # - return None: continue processing this exception
+        # - return a Response object: stops process_exception() chain
+        # - return a Request object: stops process_exception() chain
+        pass
+
+    async def spider_opened(self, spider=None):
+        self.c: crawler.Crawler
+        if not isinstance(self.c.spider, Zed.Spider):
+            print(
+                "Warning:",
+                f"该爬虫类没有继承于 {type(self)}，将不会进行此中间件环境的启用",
+            )
+            return
+
+        print("spider_opened, start the initialization of async playwright api object.")
+        self.pm = async_playwright()
+        self.pr = await self.pm.start()
+        self.browser = await self.pr.chromium.launch(**self.c.spider.info)
+        self.context = await self.browser.new_context(no_viewport=True)
+
+    async def spider_closed(self):
+        if not isinstance(self.c.spider, Zed.Spider):
+            return
+
+        print(
+            "spider_closed, start the finalizing work of async playwright api object."
+        )
+        await self.context.close()
+        await self.browser.close()
+        await self.pr.stop()
+        await self.pm.__aexit__()