aio-scrapy 2.1.6__tar.gz → 2.1.8__tar.gz

{aio_scrapy-2.1.6/aio_scrapy.egg-info → aio_scrapy-2.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: aio-scrapy
-Version: 2.1.6
+Version: 2.1.8
 Summary: A high-level Web Crawling and Web Scraping framework based on Asyncio
 Home-page: https://github.com/conlin-huang/aio-scrapy.git
 Author: conlin
@@ -76,115 +76,58 @@ Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
-<!--
-![aio-scrapy](./doc/images/aio-scrapy.png)
--->
-### aio-scrapy
+# AioScrapy
 
-An asyncio + aiolibs crawler  imitate scrapy framework
+AioScrapy是一个基于Python异步IO的强大网络爬虫框架。它的设计理念源自Scrapy，但完全基于异步IO实现，提供更高的性能和更灵活的配置选项。</br>
+AioScrapy is a powerful asynchronous web crawling framework built on Python's asyncio library. It is inspired by Scrapy but completely reimplemented with asynchronous IO, offering higher performance and more flexible configuration options.
 
-English | [中文](./doc/README_ZH.md)
+## 特性 | Features
 
-### Overview
-- aio-scrapy framework is base on opensource project Scrapy & scrapy_redis.
-- aio-scrapy implements compatibility with scrapyd.
-- aio-scrapy implements redis queue and rabbitmq queue.
-- aio-scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.
-- Distributed crawling/scraping.
-### Requirements
+- **完全异步**：基于Python的asyncio库，实现高效的并发爬取
+- **多种下载处理程序**：支持多种HTTP客户端，包括aiohttp、httpx、requests、pyhttpx、curl_cffi、DrissionPage和playwright
+- **灵活的中间件系统**：轻松添加自定义功能和处理逻辑
+- **强大的数据处理管道**：支持多种数据库存储选项
+- **内置信号系统**：方便的事件处理机制
+- **丰富的配置选项**：高度可定制的爬虫行为
+- **分布式爬取**：支持使用Redis和RabbitMQ进行分布式爬取
+- **数据库集成**：内置支持Redis、MySQL、MongoDB、PostgreSQL和RabbitMQ
 
-- Python 3.9+
-- Works on Linux, Windows, macOS, BSD
-
-### Install
-
-The quick way:
-
-```shell
-# Install the latest aio-scrapy
-pip install git+https://github.com/ConlinH/aio-scrapy
-
-# default
-pip install aio-scrapy
-
-# Install all dependencies
-pip install aio-scrapy[all]
-
-# When you need to use mysql/httpx/rabbitmq/mongo
-pip install aio-scrapy[aiomysql,httpx,aio-pika,mongo]
-```
-
-### Usage
-
-#### create project spider:
-
-```shell
-aioscrapy startproject project_quotes
-```
-
-```
-cd project_quotes
-aioscrapy genspider quotes
-```
-
-quotes.py
-
-```python
-from aioscrapy.spiders import Spider
-
-
-class QuotesMemorySpider(Spider):
-    name = 'QuotesMemorySpider'
-
-    start_urls = ['https://quotes.toscrape.com']
-
-    async def parse(self, response):
-        for quote in response.css('div.quote'):
-            yield {
-                'author': quote.xpath('span/small/text()').get(),
-                'text': quote.css('span.text::text').get(),
-            }
 
-        next_page = response.css('li.next a::attr("href")').get()
-        if next_page is not None:
-            yield response.follow(next_page, self.parse)
+- **Fully Asynchronous**: Built on Python's asyncio for efficient concurrent crawling
+- **Multiple Download Handlers**: Support for various HTTP clients including aiohttp, httpx, requests, pyhttpx, curl_cffi, DrissionPage and playwright
+- **Flexible Middleware System**: Easily add custom functionality and processing logic
+- **Powerful Data Processing Pipelines**: Support for various database storage options
+- **Built-in Signal System**: Convenient event handling mechanism
+- **Rich Configuration Options**: Highly customizable crawler behavior
+- **Distributed Crawling**: Support for distributed crawling using Redis and RabbitMQ
+- **Database Integration**: Built-in support for Redis, MySQL, MongoDB, PostgreSQL, and RabbitMQ
 
+## 安装 | Installation
 
-if __name__ == '__main__':
-    QuotesMemorySpider.start()
+### 要求 | Requirements
 
-```
-
-run the spider:
+- Python 3.9+
 
-```shell
-aioscrapy crawl quotes
-```
+### 使用pip安装 | Install with pip
 
-#### create single script spider:
+```bash
+pip install aio-scrapy
 
-```shell
-aioscrapy genspider single_quotes -t single
+# Install the latest aio-scrapy
+# pip install git+https://github.com/ConlinH/aio-scrapy
 ```
 
-single_quotes.py:
-
+### 开始 | Start
 ```python
-from aioscrapy.spiders import Spider
+from aioscrapy import Spider, logger
 
 
-class QuotesMemorySpider(Spider):
-    name = 'QuotesMemorySpider'
+class MyspiderSpider(Spider):
+    name = 'myspider'
     custom_settings = {
-        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
-        'CLOSE_SPIDER_ON_IDLE': True,
-        # 'DOWNLOAD_DELAY': 3,
-        # 'RANDOMIZE_DOWNLOAD_DELAY': True,
-        # 'CONCURRENT_REQUESTS': 1,
-        # 'LOG_LEVEL': 'INFO'
+        "CLOSE_SPIDER_ON_IDLE": True
     }
-
-    start_urls = ['https://quotes.toscrape.com']
+    start_urls = ["https://quotes.toscrape.com"]
 
     @staticmethod
     async def process_request(request, spider):
@@ -203,49 +146,45 @@ class QuotesMemorySpider(Spider):
 
     async def parse(self, response):
         for quote in response.css('div.quote'):
-            yield {
+            item = {
                 'author': quote.xpath('span/small/text()').get(),
                 'text': quote.css('span.text::text').get(),
             }
-
-        next_page = response.css('li.next a::attr("href")').get()
-        if next_page is not None:
-            yield response.follow(next_page, self.parse)
+            yield item
 
     async def process_item(self, item):
-        print(item)
+        logger.info(item)
 
 
 if __name__ == '__main__':
-    QuotesMemorySpider.start()
-
+    MyspiderSpider.start()
 ```
 
-run the spider:
-
-```shell
-aioscrapy runspider quotes.py
-```
-
-
-### more commands:
-
-```shell
-aioscrapy -h
-```
-
-#### [more example](./example)
-
-### Documentation
-[doc](./doc/documentation.md)
-
-### Ready
-
-Please submit your suggestions to the owner by creating an issue
-
-## Thanks
-
-[aiohttp](https://github.com/aio-libs/aiohttp/)
-
-[scrapy](https://github.com/scrapy/scrapy)
-
+## 文档 | Documentation
+
+## 文档目录 | Documentation Contents
+- [安装指南 | Installation Guide](docs/installation.md)
+- [快速入门 | Quick Start](docs/quickstart.md)
+- [核心概念 | Core Concepts](docs/concepts.md)
+- [爬虫指南 | Spider Guide](docs/spiders.md)
+- [下载器 | Downloaders](docs/downloaders.md)
+- [中间件 | Middlewares](docs/middlewares.md)
+- [管道 | Pipelines](docs/pipelines.md)
+- [队列 | Queues](docs/queues.md)
+- [请求过滤器 | Request Filters](docs/dupefilters.md)
+- [代理 | Proxy](docs/proxy.md)
+- [数据库连接 | Database Connections](docs/databases.md)
+- [分布式部署 | Distributed Deployment](docs/distributed.md)
+- [配置参考 | Settings Reference](docs/settings.md)
+- [API参考 | API Reference](docs/api.md)
+- [示例 | Example](example)
+
+## 许可证 | License
+
+本项目采用MIT许可证 - 详情请查看LICENSE文件。</br>
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+
+## 联系
+QQ: 995018884 </br>
+WeChat: h995018884

aio_scrapy-2.1.8/README.md

@@ -0,0 +1,112 @@
+# AioScrapy
+
+AioScrapy是一个基于Python异步IO的强大网络爬虫框架。它的设计理念源自Scrapy，但完全基于异步IO实现，提供更高的性能和更灵活的配置选项。</br>
+AioScrapy is a powerful asynchronous web crawling framework built on Python's asyncio library. It is inspired by Scrapy but completely reimplemented with asynchronous IO, offering higher performance and more flexible configuration options.
+
+## 特性 | Features
+
+- **完全异步**：基于Python的asyncio库，实现高效的并发爬取
+- **多种下载处理程序**：支持多种HTTP客户端，包括aiohttp、httpx、requests、pyhttpx、curl_cffi、DrissionPage和playwright
+- **灵活的中间件系统**：轻松添加自定义功能和处理逻辑
+- **强大的数据处理管道**：支持多种数据库存储选项
+- **内置信号系统**：方便的事件处理机制
+- **丰富的配置选项**：高度可定制的爬虫行为
+- **分布式爬取**：支持使用Redis和RabbitMQ进行分布式爬取
+- **数据库集成**：内置支持Redis、MySQL、MongoDB、PostgreSQL和RabbitMQ
+
+
+- **Fully Asynchronous**: Built on Python's asyncio for efficient concurrent crawling
+- **Multiple Download Handlers**: Support for various HTTP clients including aiohttp, httpx, requests, pyhttpx, curl_cffi, DrissionPage and playwright
+- **Flexible Middleware System**: Easily add custom functionality and processing logic
+- **Powerful Data Processing Pipelines**: Support for various database storage options
+- **Built-in Signal System**: Convenient event handling mechanism
+- **Rich Configuration Options**: Highly customizable crawler behavior
+- **Distributed Crawling**: Support for distributed crawling using Redis and RabbitMQ
+- **Database Integration**: Built-in support for Redis, MySQL, MongoDB, PostgreSQL, and RabbitMQ
+
+## 安装 | Installation
+
+### 要求 | Requirements
+
+- Python 3.9+
+
+### 使用pip安装 | Install with pip
+
+```bash
+pip install aio-scrapy
+
+# Install the latest aio-scrapy
+# pip install git+https://github.com/ConlinH/aio-scrapy
+```
+
+### 开始 | Start
+```python
+from aioscrapy import Spider, logger
+
+
+class MyspiderSpider(Spider):
+    name = 'myspider'
+    custom_settings = {
+        "CLOSE_SPIDER_ON_IDLE": True
+    }
+    start_urls = ["https://quotes.toscrape.com"]
+
+    @staticmethod
+    async def process_request(request, spider):
+        """ request middleware """
+        pass
+
+    @staticmethod
+    async def process_response(request, response, spider):
+        """ response middleware """
+        return response
+
+    @staticmethod
+    async def process_exception(request, exception, spider):
+        """ exception middleware """
+        pass
+
+    async def parse(self, response):
+        for quote in response.css('div.quote'):
+            item = {
+                'author': quote.xpath('span/small/text()').get(),
+                'text': quote.css('span.text::text').get(),
+            }
+            yield item
+
+    async def process_item(self, item):
+        logger.info(item)
+
+
+if __name__ == '__main__':
+    MyspiderSpider.start()
+```
+
+## 文档 | Documentation
+
+## 文档目录 | Documentation Contents
+- [安装指南 | Installation Guide](docs/installation.md)
+- [快速入门 | Quick Start](docs/quickstart.md)
+- [核心概念 | Core Concepts](docs/concepts.md)
+- [爬虫指南 | Spider Guide](docs/spiders.md)
+- [下载器 | Downloaders](docs/downloaders.md)
+- [中间件 | Middlewares](docs/middlewares.md)
+- [管道 | Pipelines](docs/pipelines.md)
+- [队列 | Queues](docs/queues.md)
+- [请求过滤器 | Request Filters](docs/dupefilters.md)
+- [代理 | Proxy](docs/proxy.md)
+- [数据库连接 | Database Connections](docs/databases.md)
+- [分布式部署 | Distributed Deployment](docs/distributed.md)
+- [配置参考 | Settings Reference](docs/settings.md)
+- [API参考 | API Reference](docs/api.md)
+- [示例 | Example](example)
+
+## 许可证 | License
+
+本项目采用MIT许可证 - 详情请查看LICENSE文件。</br>
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+
+## 联系
+QQ: 995018884 </br>
+WeChat: h995018884

{aio_scrapy-2.1.6 → aio_scrapy-2.1.8/aio_scrapy.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: aio-scrapy
-Version: 2.1.6
+Version: 2.1.8
 Summary: A high-level Web Crawling and Web Scraping framework based on Asyncio
 Home-page: https://github.com/conlin-huang/aio-scrapy.git
 Author: conlin
@@ -76,115 +76,58 @@ Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
-<!--
-![aio-scrapy](./doc/images/aio-scrapy.png)
--->
-### aio-scrapy
+# AioScrapy
 
-An asyncio + aiolibs crawler  imitate scrapy framework
+AioScrapy是一个基于Python异步IO的强大网络爬虫框架。它的设计理念源自Scrapy，但完全基于异步IO实现，提供更高的性能和更灵活的配置选项。</br>
+AioScrapy is a powerful asynchronous web crawling framework built on Python's asyncio library. It is inspired by Scrapy but completely reimplemented with asynchronous IO, offering higher performance and more flexible configuration options.
 
-English | [中文](./doc/README_ZH.md)
+## 特性 | Features
 
-### Overview
-- aio-scrapy framework is base on opensource project Scrapy & scrapy_redis.
-- aio-scrapy implements compatibility with scrapyd.
-- aio-scrapy implements redis queue and rabbitmq queue.
-- aio-scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages.
-- Distributed crawling/scraping.
-### Requirements
+- **完全异步**：基于Python的asyncio库，实现高效的并发爬取
+- **多种下载处理程序**：支持多种HTTP客户端，包括aiohttp、httpx、requests、pyhttpx、curl_cffi、DrissionPage和playwright
+- **灵活的中间件系统**：轻松添加自定义功能和处理逻辑
+- **强大的数据处理管道**：支持多种数据库存储选项
+- **内置信号系统**：方便的事件处理机制
+- **丰富的配置选项**：高度可定制的爬虫行为
+- **分布式爬取**：支持使用Redis和RabbitMQ进行分布式爬取
+- **数据库集成**：内置支持Redis、MySQL、MongoDB、PostgreSQL和RabbitMQ
 
-- Python 3.9+
-- Works on Linux, Windows, macOS, BSD
-
-### Install
-
-The quick way:
-
-```shell
-# Install the latest aio-scrapy
-pip install git+https://github.com/ConlinH/aio-scrapy
-
-# default
-pip install aio-scrapy
-
-# Install all dependencies
-pip install aio-scrapy[all]
-
-# When you need to use mysql/httpx/rabbitmq/mongo
-pip install aio-scrapy[aiomysql,httpx,aio-pika,mongo]
-```
-
-### Usage
-
-#### create project spider:
-
-```shell
-aioscrapy startproject project_quotes
-```
-
-```
-cd project_quotes
-aioscrapy genspider quotes
-```
-
-quotes.py
-
-```python
-from aioscrapy.spiders import Spider
-
-
-class QuotesMemorySpider(Spider):
-    name = 'QuotesMemorySpider'
-
-    start_urls = ['https://quotes.toscrape.com']
-
-    async def parse(self, response):
-        for quote in response.css('div.quote'):
-            yield {
-                'author': quote.xpath('span/small/text()').get(),
-                'text': quote.css('span.text::text').get(),
-            }
 
-        next_page = response.css('li.next a::attr("href")').get()
-        if next_page is not None:
-            yield response.follow(next_page, self.parse)
+- **Fully Asynchronous**: Built on Python's asyncio for efficient concurrent crawling
+- **Multiple Download Handlers**: Support for various HTTP clients including aiohttp, httpx, requests, pyhttpx, curl_cffi, DrissionPage and playwright
+- **Flexible Middleware System**: Easily add custom functionality and processing logic
+- **Powerful Data Processing Pipelines**: Support for various database storage options
+- **Built-in Signal System**: Convenient event handling mechanism
+- **Rich Configuration Options**: Highly customizable crawler behavior
+- **Distributed Crawling**: Support for distributed crawling using Redis and RabbitMQ
+- **Database Integration**: Built-in support for Redis, MySQL, MongoDB, PostgreSQL, and RabbitMQ
 
+## 安装 | Installation
 
-if __name__ == '__main__':
-    QuotesMemorySpider.start()
+### 要求 | Requirements
 
-```
-
-run the spider:
+- Python 3.9+
 
-```shell
-aioscrapy crawl quotes
-```
+### 使用pip安装 | Install with pip
 
-#### create single script spider:
+```bash
+pip install aio-scrapy
 
-```shell
-aioscrapy genspider single_quotes -t single
+# Install the latest aio-scrapy
+# pip install git+https://github.com/ConlinH/aio-scrapy
 ```
 
-single_quotes.py:
-
+### 开始 | Start
 ```python
-from aioscrapy.spiders import Spider
+from aioscrapy import Spider, logger
 
 
-class QuotesMemorySpider(Spider):
-    name = 'QuotesMemorySpider'
+class MyspiderSpider(Spider):
+    name = 'myspider'
     custom_settings = {
-        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36",
-        'CLOSE_SPIDER_ON_IDLE': True,
-        # 'DOWNLOAD_DELAY': 3,
-        # 'RANDOMIZE_DOWNLOAD_DELAY': True,
-        # 'CONCURRENT_REQUESTS': 1,
-        # 'LOG_LEVEL': 'INFO'
+        "CLOSE_SPIDER_ON_IDLE": True
     }
-
-    start_urls = ['https://quotes.toscrape.com']
+    start_urls = ["https://quotes.toscrape.com"]
 
     @staticmethod
     async def process_request(request, spider):
@@ -203,49 +146,45 @@ class QuotesMemorySpider(Spider):
 
     async def parse(self, response):
         for quote in response.css('div.quote'):
-            yield {
+            item = {
                 'author': quote.xpath('span/small/text()').get(),
                 'text': quote.css('span.text::text').get(),
             }
-
-        next_page = response.css('li.next a::attr("href")').get()
-        if next_page is not None:
-            yield response.follow(next_page, self.parse)
+            yield item
 
     async def process_item(self, item):
-        print(item)
+        logger.info(item)
 
 
 if __name__ == '__main__':
-    QuotesMemorySpider.start()
-
+    MyspiderSpider.start()
 ```
 
-run the spider:
-
-```shell
-aioscrapy runspider quotes.py
-```
-
-
-### more commands:
-
-```shell
-aioscrapy -h
-```
-
-#### [more example](./example)
-
-### Documentation
-[doc](./doc/documentation.md)
-
-### Ready
-
-Please submit your suggestions to the owner by creating an issue
-
-## Thanks
-
-[aiohttp](https://github.com/aio-libs/aiohttp/)
-
-[scrapy](https://github.com/scrapy/scrapy)
-
+## 文档 | Documentation
+
+## 文档目录 | Documentation Contents
+- [安装指南 | Installation Guide](docs/installation.md)
+- [快速入门 | Quick Start](docs/quickstart.md)
+- [核心概念 | Core Concepts](docs/concepts.md)
+- [爬虫指南 | Spider Guide](docs/spiders.md)
+- [下载器 | Downloaders](docs/downloaders.md)
+- [中间件 | Middlewares](docs/middlewares.md)
+- [管道 | Pipelines](docs/pipelines.md)
+- [队列 | Queues](docs/queues.md)
+- [请求过滤器 | Request Filters](docs/dupefilters.md)
+- [代理 | Proxy](docs/proxy.md)
+- [数据库连接 | Database Connections](docs/databases.md)
+- [分布式部署 | Distributed Deployment](docs/distributed.md)
+- [配置参考 | Settings Reference](docs/settings.md)
+- [API参考 | API Reference](docs/api.md)
+- [示例 | Example](example)
+
+## 许可证 | License
+
+本项目采用MIT许可证 - 详情请查看LICENSE文件。</br>
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+
+## 联系
+QQ: 995018884 </br>
+WeChat: h995018884

aio_scrapy-2.1.8/aioscrapy/VERSION

@@ -0,0 +1 @@
+2.1.8

{aio_scrapy-2.1.6 → aio_scrapy-2.1.8}/aioscrapy/core/downloader/handlers/aiohttp.py

@@ -50,7 +50,7 @@ class AioHttpDownloadHandler(BaseDownloadHandler):
 
         # Arguments to pass to aiohttp.ClientSession constructor
         # 传递给aiohttp.ClientSession构造函数的参数
-        self.aiohttp_client_session_args: dict = settings.getdict('AIOHTTP_CLIENT_SESSION_ARGS')
+        self.aiohttp_args: dict = settings.getdict('AIOHTTP_ARGS')
 
         # SSL verification setting
         # SSL验证设置
@@ -228,13 +228,13 @@ class AioHttpDownloadHandler(BaseDownloadHandler):
         if self.use_session:
             # Not recommended to use session, The abnormal phenomena will occurs when using tunnel proxy
             # 不建议使用会话，使用隧道代理时会出现异常现象
-            session = self.get_session(**self.aiohttp_client_session_args)
+            session = self.get_session(**self.aiohttp_args)
             async with session.request(request.method, request.url, **kwargs) as response:
                 content: bytes = await response.read()
         else:
             # Create a new session for each request (recommended)
             # 为每个请求创建一个新会话（推荐）
-            async with aiohttp.ClientSession(**self.aiohttp_client_session_args) as session:
+            async with aiohttp.ClientSession(**self.aiohttp_args) as session:
                 async with session.request(request.method, request.url, **kwargs) as response:
                     content: bytes = await response.read()
 

{aio_scrapy-2.1.6 → aio_scrapy-2.1.8}/aioscrapy/core/downloader/handlers/curl_cffi.py

@@ -44,7 +44,7 @@ class CurlCffiDownloadHandler(BaseDownloadHandler):
 
         # Arguments to pass to curl_cffi AsyncSession constructor
         # 传递给curl_cffi AsyncSession构造函数的参数
-        self.httpx_client_session_args: dict = self.settings.get('CURL_CFFI_CLIENT_SESSION_ARGS', {})
+        self.curl_cffi_args: dict = self.settings.get('CURL_CFFI_ARGS', {})
 
         # SSL verification setting
         # SSL验证设置
@@ -156,7 +156,7 @@ class CurlCffiDownloadHandler(BaseDownloadHandler):
 
         # Configure curl_cffi session
         # 配置curl_cffi会话
-        session_args = self.httpx_client_session_args.copy()
+        session_args = self.curl_cffi_args.copy()
 
         # Perform the request
         # 执行请求

{aio_scrapy-2.1.6 → aio_scrapy-2.1.8}/aioscrapy/core/downloader/handlers/httpx.py

@@ -46,7 +46,7 @@ class HttpxDownloadHandler(BaseDownloadHandler):
 
         # Arguments to pass to httpx AsyncClient constructor
         # 传递给httpx AsyncClient构造函数的参数
-        self.httpx_client_session_args: dict = self.settings.get('HTTPX_CLIENT_SESSION_ARGS', {})
+        self.httpx_args: dict = self.settings.get('HTTPX_ARGS', {})
 
         # SSL verification setting
         # SSL验证设置
@@ -147,7 +147,7 @@ class HttpxDownloadHandler(BaseDownloadHandler):
 
         # Configure httpx client session
         # 配置httpx客户端会话
-        session_args = self.httpx_client_session_args.copy()
+        session_args = self.httpx_args.copy()
         session_args.setdefault('http2', True)  # Enable HTTP/2 by default
                                                # 默认启用HTTP/2
         session_args.update({