yad2-scraper 0.4.0__tar.gz → 0.5.1__tar.gz

yad2_scraper-0.5.1/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.3
+Name: yad2-scraper
+Version: 0.5.1
+Summary: Scrape Yad2 in Python.
+License: LICENSE
+Author: dav ost
+Author-email: davidost2003@gmail.com
+Requires-Python: >=3.8
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: beautifulsoup4 (>=4.11.1,<5.0.0)
+Requires-Dist: fake-useragent (>=0.1.11,<0.2.0)
+Requires-Dist: httpcore (>=0.15.0)
+Requires-Dist: httpx (>=0.24.0,<0.25.0)
+Requires-Dist: pydantic (>=1.10.0,<2.0.0)
+Description-Content-Type: text/markdown
+
+# Yad2 Scraper
+
+A Python package for scraping listings from [Yad2](https://www.yad2.co.il/), Israel's leading classifieds platform.
+This package provides a simple and flexible interface to fetch data, filter results, and extract relevant information.
+
+__NOTE__: Currently, the package primarily supports the **vehicles category**.
+Support for additional categories may be added in future updates.
+
+---
+
+## Features
+
+- **Fetch Listings**: Retrieve listings by category (e.g., vehicles, real-estate, etc.).
+- **Filter Results**: Apply filters such as price range, year range, and sorting order.
+- **Dynamic URL Generation**: Generate URLs for specific categories and filters.
+- **Type-Safe API**: Uses Python type hints (`Literal`, `Optional`, etc.) for better code clarity and safety.
+- **Extensible**: Easily extendable to support additional categories and filters.
+
+---
+
+## Installation
+
+Install the package using `pip`:
+
+```bash
+pip install yad2-scraper
+```
+
+## Usage
+
+### Fetching Category Listings
+
+To fetch any category, use the `fetch_category` function:
+
+```python
+from yad2_scraper import fetch_category, Yad2Category
+
+# Fetch real estate category (returns a generic Yad2Category object)
+real_estate_category_page1 = fetch_category("https://www.yad2.co.il/realestate/forsale", page=1)
+...
+real_estate_category_page2 = fetch_category("https://www.yad2.co.il/realestate/forsale", page=2)
+...
+```
+
+### Fetching Vehicle Listings
+
+To fetch vehicle listings for a specific category, use the `fetch_vehicle_category` function:
+
+```python
+from yad2_scraper import fetch_vehicle_category, OrderVehiclesBy, Field
+
+# Fetch cars category
+cars_category = fetch_vehicle_category("cars")
+
+for car_data in cars_category.load_next_data().iterate_vehicles():
+    print(car_data.model(Field.ENGLISH_TEXT))
+    print(car_data.test_date)
+    print(car_data.price)
+    ...
+
+# Fetch motorcycles category
+motorcycle_categories = fetch_vehicle_category(
+    "motorcycles",
+    price_range=(5000, 15000),
+    year_range=(2010, 2020),
+    order_by=OrderVehiclesBy.PRICE_LOWEST_TO_HIGHEST
+)
+
+for motorcycle_tag in motorcycle_categories.get_vehicle_tags():
+    print(motorcycle_tag.page_link)
+    print(motorcycle_tag.hand)
+    print(motorcycle_tag.price)
+    ...
+```
+
+### The Scraper Object
+
+The `Yad2Scraper` class is the core of the package.
+It handles HTTP requests, parses responses, and provides methods to fetch and filter vehicle listings.
+
+#### Creating a Scraper Instance
+
+You can create a `Yad2Scraper` instance manually or use the default scraper provided by the package:
+
+```python
+from yad2_scraper import Yad2Scraper, get_default_scraper
+
+# Create a custom scraper instance
+scraper = Yad2Scraper()
+
+# Use the default scraper
+default_scraper = get_default_scraper()
+```
+
+#### Fetching Category Listings
+
+The `fetch_category` method is used to fetch listings for a specific category.
+It takes a URL, a `Category` type, and optionally query params as arguments:
+
+```python
+from yad2_scraper import Yad2Scraper, Yad2Category, QueryFilters, OrderBy
+from yad2_scraper.vehicles import (
+    Yad2VehiclesCategory,
+    VehiclesQueryFilters,
+    OrderVehiclesBy,
+    get_vehicle_category_url
+)
+
+# Fetch businesses for sale category with filters
+scraper = Yad2Scraper()
+url = "https://www.yad2.co.il/products/businesses-for-sale"
+query_filters = QueryFilters(price_range=(10000, 250000), order_by=OrderBy.PRICE_LOWEST_TO_HIGHEST)
+real_estate_category = scraper.fetch_category(url, Yad2Category, params=query_filters)
+
+# Fetch watercraft (vehicle) category with filters
+url = get_vehicle_category_url("watercraft")
+query_filters = VehiclesQueryFilters(year_range=(2010, 2020), order_by=OrderVehiclesBy.DATE)
+watercraft_category = scraper.fetch_category(url, Yad2VehiclesCategory, params=query_filters)
+```
+
+#### Attributes & Methods
+
+The `Yad2Scraper` object contains a lot of additional attributes & methods which you can use.
+Please check out the actual code documentation for more details.
+
+## Contributing
+
+Contributions are welcomed! Here’s how you can get started:
+
+1. Fork the repository.
+2. Create a new branch for your feature or bugfix.
+3. Write tests for your changes.
+4. Submit a pull request.
+
+## License
+
+This project is licensed under the MIT License. See the LICENSE file for details.
+
+## Support
+
+For questions, issues, or feature requests, please open an issue on the GitHub repository.

yad2_scraper-0.5.1/README.md

@@ -0,0 +1,141 @@
+# Yad2 Scraper
+
+A Python package for scraping listings from [Yad2](https://www.yad2.co.il/), Israel's leading classifieds platform.
+This package provides a simple and flexible interface to fetch data, filter results, and extract relevant information.
+
+__NOTE__: Currently, the package primarily supports the **vehicles category**.
+Support for additional categories may be added in future updates.
+
+---
+
+## Features
+
+- **Fetch Listings**: Retrieve listings by category (e.g., vehicles, real-estate, etc.).
+- **Filter Results**: Apply filters such as price range, year range, and sorting order.
+- **Dynamic URL Generation**: Generate URLs for specific categories and filters.
+- **Type-Safe API**: Uses Python type hints (`Literal`, `Optional`, etc.) for better code clarity and safety.
+- **Extensible**: Easily extendable to support additional categories and filters.
+
+---
+
+## Installation
+
+Install the package using `pip`:
+
+```bash
+pip install yad2-scraper
+```
+
+## Usage
+
+### Fetching Category Listings
+
+To fetch any category, use the `fetch_category` function:
+
+```python
+from yad2_scraper import fetch_category, Yad2Category
+
+# Fetch real estate category (returns a generic Yad2Category object)
+real_estate_category_page1 = fetch_category("https://www.yad2.co.il/realestate/forsale", page=1)
+...
+real_estate_category_page2 = fetch_category("https://www.yad2.co.il/realestate/forsale", page=2)
+...
+```
+
+### Fetching Vehicle Listings
+
+To fetch vehicle listings for a specific category, use the `fetch_vehicle_category` function:
+
+```python
+from yad2_scraper import fetch_vehicle_category, OrderVehiclesBy, Field
+
+# Fetch cars category
+cars_category = fetch_vehicle_category("cars")
+
+for car_data in cars_category.load_next_data().iterate_vehicles():
+    print(car_data.model(Field.ENGLISH_TEXT))
+    print(car_data.test_date)
+    print(car_data.price)
+    ...
+
+# Fetch motorcycles category
+motorcycle_categories = fetch_vehicle_category(
+    "motorcycles",
+    price_range=(5000, 15000),
+    year_range=(2010, 2020),
+    order_by=OrderVehiclesBy.PRICE_LOWEST_TO_HIGHEST
+)
+
+for motorcycle_tag in motorcycle_categories.get_vehicle_tags():
+    print(motorcycle_tag.page_link)
+    print(motorcycle_tag.hand)
+    print(motorcycle_tag.price)
+    ...
+```
+
+### The Scraper Object
+
+The `Yad2Scraper` class is the core of the package.
+It handles HTTP requests, parses responses, and provides methods to fetch and filter vehicle listings.
+
+#### Creating a Scraper Instance
+
+You can create a `Yad2Scraper` instance manually or use the default scraper provided by the package:
+
+```python
+from yad2_scraper import Yad2Scraper, get_default_scraper
+
+# Create a custom scraper instance
+scraper = Yad2Scraper()
+
+# Use the default scraper
+default_scraper = get_default_scraper()
+```
+
+#### Fetching Category Listings
+
+The `fetch_category` method is used to fetch listings for a specific category.
+It takes a URL, a `Category` type, and optionally query params as arguments:
+
+```python
+from yad2_scraper import Yad2Scraper, Yad2Category, QueryFilters, OrderBy
+from yad2_scraper.vehicles import (
+    Yad2VehiclesCategory,
+    VehiclesQueryFilters,
+    OrderVehiclesBy,
+    get_vehicle_category_url
+)
+
+# Fetch businesses for sale category with filters
+scraper = Yad2Scraper()
+url = "https://www.yad2.co.il/products/businesses-for-sale"
+query_filters = QueryFilters(price_range=(10000, 250000), order_by=OrderBy.PRICE_LOWEST_TO_HIGHEST)
+real_estate_category = scraper.fetch_category(url, Yad2Category, params=query_filters)
+
+# Fetch watercraft (vehicle) category with filters
+url = get_vehicle_category_url("watercraft")
+query_filters = VehiclesQueryFilters(year_range=(2010, 2020), order_by=OrderVehiclesBy.DATE)
+watercraft_category = scraper.fetch_category(url, Yad2VehiclesCategory, params=query_filters)
+```
+
+#### Attributes & Methods
+
+The `Yad2Scraper` object contains a lot of additional attributes & methods which you can use.
+Please check out the actual code documentation for more details.
+
+## Contributing
+
+Contributions are welcomed! Here’s how you can get started:
+
+1. Fork the repository.
+2. Create a new branch for your feature or bugfix.
+3. Write tests for your changes.
+4. Submit a pull request.
+
+## License
+
+This project is licensed under the MIT License. See the LICENSE file for details.
+
+## Support
+
+For questions, issues, or feature requests, please open an issue on the GitHub repository.

{yad2_scraper-0.4.0 → yad2_scraper-0.5.1}/pyproject.toml

@@ -1,13 +1,13 @@
 [tool.poetry]
 name = "yad2-scraper"
-version = "0.4.0"
+version = "0.5.1"
 description = "Scrape Yad2 in Python."
 authors = ["dav ost <davidost2003@gmail.com>"]
 license = "LICENSE"
 readme = "README.md"
 
 [tool.poetry.dependencies]
-python = ">=3.7"
+python = ">=3.8"
 httpx = "^0.24.0"
 httpcore = ">=0.15.0"
 fake-useragent = "^0.1.11"

yad2_scraper-0.5.1/yad2_scraper/__init__.py

@@ -0,0 +1,99 @@
+from typing import Optional, Type
+
+from .scraper import Yad2Scraper, Category
+from .query import QueryFilters, OrderBy, NumberRange
+from .category import Yad2Category
+from .next_data import NextData, Field
+from .utils import any_param_specified
+from .vehicles import (
+    Yad2VehiclesCategory,
+    VehiclesQueryFilters,
+    OrderVehiclesBy,
+    VehicleCategory,
+    get_vehicle_category_url
+)
+
+_default_scraper = None
+
+
+def get_default_scraper() -> Yad2Scraper:
+    """
+    Retrieves the default instance of the Yad2Scraper. If an instance does not already exist, it will be created.
+
+    Returns:
+        Yad2Scraper: The default instance of the Yad2Scraper.
+
+    Notes:
+        The default scraper is a singleton instance that is reused across multiple calls.
+    """
+    global _default_scraper
+
+    if not _default_scraper:
+        _default_scraper = Yad2Scraper()
+
+    return _default_scraper
+
+
+def fetch_category(
+        url: str,
+        category_type: Type[Category] = Yad2Category,
+        page: Optional[int] = None,
+        order_by: Optional[OrderBy] = None,
+        price_range: [NumberRange] = None
+) -> Category:
+    """
+    Fetches a specific category from the given URL, while applying optional filters.
+
+    Args:
+        url (str): The URL of the category to fetch.
+        category_type (Type[Category], optional): The type of category to return (default is `Yad2Category`).
+        page (Optional[int], optional): The page number for pagination (default is None).
+        order_by (Optional[OrderBy], optional): The sorting order for the results (default is None).
+        price_range (Optional[List[NumberRange]], optional): The price range filter for the results (default is None).
+
+    Returns:
+        Category: An instance of the specified `category_type`, populated with the fetched data.
+
+    Notes:
+        This method uses the default scraper to retrieve the category.
+    """
+    if any_param_specified(page, order_by, price_range):
+        params = QueryFilters(page=page, order_by=order_by, price_range=price_range)
+    else:
+        params = None
+
+    default_scraper = get_default_scraper()
+    return default_scraper.fetch_category(url, category_type, params=params)
+
+
+def fetch_vehicle_category(
+        vehicle_category: VehicleCategory,
+        page: Optional[int] = None,
+        order_by: Optional[OrderVehiclesBy] = None,
+        price_range: [NumberRange] = None,
+        year_range: [NumberRange] = None
+) -> Yad2VehiclesCategory:
+    """
+    Fetches a specific vehicle category, while applying optional filters.
+
+    Args:
+        vehicle_category (VehicleCategory): The vehicle category to fetch.
+        page (Optional[int], optional): The page number for pagination (default is None).
+        order_by (Optional[OrderVehiclesBy], optional): The sorting order for the results (default is None).
+        price_range (Optional[List[NumberRange]], optional): The price range filter for the results (default is None).
+        year_range (Optional[List[NumberRange]], optional): The year range filter for the results (default is None).
+
+    Returns:
+        Yad2VehiclesCategory: An instance of `Yad2VehiclesCategory`, populated with the fetched vehicle category data.
+
+    Notes:
+        This method uses the default scraper to fetch the vehicle category.
+    """
+    if any_param_specified(page, order_by, price_range, year_range):
+        params = VehiclesQueryFilters(page=page, order_by=order_by, price_range=price_range, year_range=year_range)
+    else:
+        params = None
+
+    url = get_vehicle_category_url(vehicle_category)
+    default_scraper = get_default_scraper()
+    return default_scraper.fetch_category(url, Yad2VehiclesCategory, params=params)

{yad2_scraper-0.4.0 → yad2_scraper-0.5.1}/yad2_scraper/category.py

@@ -8,18 +8,24 @@ from yad2_scraper.constants import NEXT_DATA_SCRIPT_ID
 
 
 class Yad2Category:
+    """Represents a Yad2 category parsed from an HTML page."""
+
     def __init__(self, soup: BeautifulSoup):
+        """Initialize with a BeautifulSoup object."""
         self.soup = soup
 
     @classmethod
     def from_html_io(cls, html_io: Union[TextIO, BinaryIO]):
+        """Create an instance from an HTML file-like object."""
         html = html_io.read()
         soup = BeautifulSoup(html, "html.parser")
         return cls(soup)
 
     def load_next_data(self) -> Optional[NextData]:
+        """Extract and parse Next.js data from the page."""
         tag = self.soup.find("script", id=NEXT_DATA_SCRIPT_ID)
         return NextData(json.loads(tag.string)) if tag else None
 
     def find_all_tags_by_class_substring(self, tag_name: str, substring: str) -> List[Tag]:
+        """Find all HTML tags with a class containing the given substring."""
         return find_all_html_tags_by_class_substring(self.soup, tag_name, substring)

{yad2_scraper-0.4.0 → yad2_scraper-0.5.1}/yad2_scraper/exceptions.py

@@ -3,29 +3,40 @@ from typing import List, Union
 
 
 class ResponseError(Exception):
+    """Represents an error response from an HTTP request."""
+
     def __init__(self, msg: str, request: httpx.Request, response: httpx.Response):
+        """Initialize with an error message, request, and response objects."""
         super().__init__(msg)
         self.request = request
         self.response = response
 
 
 class AntiBotDetectedError(ResponseError):
+    """Raised when an anti-bot mechanism is detected."""
     pass
 
 
 class UnexpectedContentError(ResponseError):
+    """Raised when the response content is not as expected."""
     pass
 
 
 class MaxAttemptsExceededError(Exception):
+    """Raised when the maximum number of attempts is exceeded."""
+
     def __init__(self, msg: str, max_attempts: int, errors: List[BaseException] = None):
+        """Initialize with an error message, max attempts, and optional errors."""
         super().__init__(msg)
         self.max_attempts = max_attempts
         self.errors = errors
 
 
 class MaxRequestAttemptsExceededError(MaxAttemptsExceededError):
+    """Raised when all HTTP request attempts fail."""
+
     def __init__(self, method: str, url: str, max_attempts: int, errors: List[Union[httpx.HTTPError, ResponseError]]):
+        """Initialize with request method, URL, max attempts, and error list."""
         msg = f"All {max_attempts} attempts for {method} request to '{url}' have failed"
         super().__init__(msg, max_attempts, errors)
         self.method = method

yad2_scraper-0.5.1/yad2_scraper/next_data.py

@@ -0,0 +1,59 @@
+from datetime import datetime
+from enum import Enum
+from typing import List, Union
+
+from yad2_scraper.utils import safe_access
+
+FieldTypes = Union[str, int]
+
+_safe_access_optional_keys = safe_access(exceptions=(KeyError, TypeError), default=None)
+
+
+class SafeAccessOptionalKeysMeta(type):
+    """Metaclass that wraps methods and properties with safe access handling."""
+
+    def __new__(cls, name, bases, dictionary):
+        for attr_name, attr_value in dictionary.items():
+            if callable(attr_value):  # Wrap methods
+                dictionary[attr_name] = _safe_access_optional_keys(attr_value)
+            elif isinstance(attr_value, property):  # Wrap properties
+                dictionary[attr_name] = property(
+                    _safe_access_optional_keys(attr_value.fget) if attr_value.fget else None,
+                    _safe_access_optional_keys(attr_value.fset) if attr_value.fset else None,
+                    _safe_access_optional_keys(attr_value.fdel) if attr_value.fdel else None,
+                    attr_value.__doc__,
+                )
+        return super().__new__(cls, name, bases, dictionary)
+
+
+class Field(str, Enum):
+    ID = "id"
+    TEXT = "text"
+    ENGLISH_TEXT = "textEng"
+
+
+def convert_string_date_to_datetime(date_string: str) -> datetime:
+    """Convert an ISO format string to a datetime object."""
+    return datetime.fromisoformat(date_string)
+
+
+class NextData:
+    """Represents structured Next.js data."""
+
+    def __init__(self, data: dict):
+        """Initialize with Next.js data dictionary."""
+        self.data = data
+
+    @property
+    def json(self) -> dict:
+        """Return raw JSON data."""
+        return self.data
+
+    @property
+    def queries(self) -> List[dict]:
+        """Extract query data from Next.js state."""
+        return self.data["props"]["pageProps"]["dehydratedState"]["queries"]
+
+    def __getitem__(self, item):
+        """Allow dictionary-style access to data."""
+        return self.data[item]

{yad2_scraper-0.4.0 → yad2_scraper-0.5.1}/yad2_scraper/query.py

@@ -2,7 +2,7 @@ from pydantic import BaseModel
 from enum import Enum
 from typing import Optional, Tuple
 
-PriceRange = Tuple[int, int]
+NumberRange = Tuple[int, int]
 
 
 class OrderBy(int, Enum):
@@ -13,6 +13,7 @@ class OrderBy(int, Enum):
 
 
 def format_number_range(number_range: Optional[Tuple[int, int]]) -> Optional[str]:
+    """Format a number range as 'min_value-max_value'."""
     if number_range is None:
         return None
 
@@ -25,12 +26,13 @@ def format_number_range(number_range: Optional[Tuple[int, int]]) -> Optional[str
 
 
 class QueryFilters(BaseModel):
+    """Pydantic model representing query filters for querying a resource."""
     page: Optional[int] = None
     order_by: Optional[OrderBy] = None
-    price_range: Optional[PriceRange] = None
-    ...
+    price_range: Optional[NumberRange] = None
 
     def to_params(self) -> dict:
+        """Convert filter fields to query parameters."""
         return {
             "page": self.page,
             "Order": self.order_by,
@@ -38,9 +40,9 @@ class QueryFilters(BaseModel):
         }
 
     def to_clean_params(self):
+        """Return query parameters excluding None values."""
         return {key: value for key, value in self.to_params().items() if value is not None}
 
-    # TODO: add helper methods for managing the attribute values
-
     def __iter__(self):
+        """Allow iteration over the clean query parameters."""
         yield from self.to_clean_params().items()