pyscrappy 1.0.0__tar.gz

pyscrappy-1.0.0/.github/workflows/ci.yml

@@ -0,0 +1,41 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install ruff
+      - run: ruff check src/
+
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e '.[all]' pytest
+      - run: pytest tests/ -v
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build

pyscrappy-1.0.0/.github/workflows/python-publish.yml

@@ -0,0 +1,24 @@
+name: Upload Python Package
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install build tools
+        run: pip install build
+      - name: Build package
+        run: python -m build
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          password: ${{ secrets.PYPI_API_TOKEN }}

pyscrappy-1.0.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+*.pyo
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+*.egg
+.mypy_cache/

pyscrappy-1.0.0/CONTRIBUTING.md

@@ -0,0 +1,312 @@
+# Contributing to PyScrappy
+
+Thank you for your interest in contributing to PyScrappy! We welcome contributions of all kinds — from bug fixes and documentation improvements to new scrapers and core features.
+
+## Checklist before submitting a PR
+
+Here are the core requirements for any PR submitted to PyScrappy:
+
+- [ ] **Keep scope isolated** — your changes should address 1 specific problem at a time
+- [ ] **Add tests** — adding at least 1 test is a hard requirement — [see details](#adding-tests)
+- [ ] **Ensure your PR passes all checks:**
+  - [ ] Unit tests — `pytest tests/ -v`
+  - [ ] Linting — `ruff check src/`
+
+## Quick Start
+
+### 1. Setup Your Local Development Environment
+
+```sh
+# Fork the repository on GitHub (click the Fork button at https://github.com/mldsveda/PyScrappy)
+# Then clone your fork locally
+git clone https://github.com/YOUR_USERNAME/PyScrappy.git
+cd PyScrappy
+
+# Create a new branch for your feature
+git checkout -b your-feature-branch
+
+# Install the package in editable mode with all extras
+pip install -e '.[all]'
+
+# Install development tools
+pip install pytest ruff mypy
+
+# Verify your setup works
+pytest tests/ -v
+```
+
+That's it! Your local development environment is ready.
+
+### 2. Development Workflow
+
+Here's the recommended workflow for making changes:
+
+```sh
+# Make your changes to the code
+# ...
+
+# Run linting to catch issues early
+ruff check src/
+
+# Run the full test suite
+pytest tests/ -v
+
+# Commit your changes
+git add .
+git commit -m "Your descriptive commit message"
+
+# Push and create a PR
+git push origin your-feature-branch
+```
+
+## Adding Tests
+
+Adding at least 1 test is a **hard requirement** for all PRs.
+
+### Where to Add Tests
+
+| What you changed | Where to add tests |
+|---|---|
+| `src/pyscrappy/core/` | `tests/test_core/` |
+| `src/pyscrappy/generic/` | `tests/test_generic/` |
+| `src/pyscrappy/scrapers/` | `tests/test_scrapers/` |
+| Package-level (`__init__.py`) | `tests/test_init.py` |
+
+### File Naming Convention
+
+The `tests/` directory mirrors the structure of `src/pyscrappy/`:
+
+| Source file | Test file |
+|---|---|
+| `src/pyscrappy/core/config.py` | `tests/test_core/test_config.py` |
+| `src/pyscrappy/core/http.py` | `tests/test_core/test_http.py` |
+| `src/pyscrappy/generic/extractors.py` | `tests/test_generic/test_extractors.py` |
+| `src/pyscrappy/scrapers/wikipedia.py` | `tests/test_scrapers/test_wikipedia.py` |
+
+### Key Testing Principles
+
+- **Mock HTTP calls** — never make real network requests in tests. Use `unittest.mock.MagicMock` to mock the `_http` attribute on scrapers.
+- **Test parsing logic** — provide realistic sample HTML/JSON and verify the scraper extracts the correct fields.
+- **Test edge cases** — empty responses, missing fields, malformed HTML.
+- **Test validation** — ensure proper errors are raised for invalid arguments.
+
+### Example Test
+
+```python
+from unittest.mock import MagicMock
+from pyscrappy.scrapers.wikipedia import WikipediaScraper
+
+SAMPLE_HTML = """
+<html><body>
+<div id="mw-content-text">
+<div class="mw-parser-output">
+    <p>Python is a high-level programming language.</p>
+</div>
+</div>
+</body></html>
+"""
+
+def test_wikipedia_scrape_returns_paragraphs():
+    """Test that WikipediaScraper extracts paragraph text."""
+    scraper = WikipediaScraper()
+    mock_http = MagicMock()
+    mock_http.get_html.return_value = SAMPLE_HTML
+    scraper._http = mock_http
+
+    result = scraper.scrape(query="Python", mode="paragraphs")
+
+    assert len(result.data) > 0
+    assert result.data[0]["type"] == "paragraph"
+    assert "Python" in result.data[0]["text"]
+    scraper.close()
+```
+
+## Running Tests and Checks
+
+### Running Unit Tests
+
+Run the full test suite:
+
+```sh
+pytest tests/ -v
+```
+
+Run a specific test file:
+
+```sh
+pytest tests/test_core/test_http.py -v
+```
+
+Run a specific test:
+
+```sh
+pytest tests/test_scrapers/test_wikipedia.py::TestWikipediaScraperFull::test_full_mode -v
+```
+
+### Running Linting
+
+Run Ruff linting (matches CI):
+
+```sh
+ruff check src/
+```
+
+Auto-fix linting issues:
+
+```sh
+ruff check src/ --fix
+```
+
+### Running Type Checks (optional)
+
+```sh
+mypy src/pyscrappy/
+```
+
+### CI Compatibility
+
+To ensure your changes will pass CI, run the same checks locally:
+
+```sh
+# These match the GitHub Actions workflows exactly
+ruff check src/
+pytest tests/ -v
+```
+
+CI runs tests across Python 3.9, 3.11, 3.12, and 3.13.
+
+## Project Structure
+
+```
+PyScrappy/
+├── src/pyscrappy/
+│   ├── __init__.py          # Package exports and convenience scrape() function
+│   ├── core/                # Core infrastructure
+│   │   ├── base.py          # BaseScraper abstract class
+│   │   ├── browser.py       # Playwright browser manager
+│   │   ├── config.py        # ScraperConfig dataclass
+│   │   ├── exceptions.py    # Custom exception hierarchy
+│   │   ├── http.py          # HTTP client with retries/rate-limiting
+│   │   └── models.py        # ScrapeResult, ScrapeMetadata, ScrapeError
+│   ├── generic/             # GenericScraper (works on any URL)
+│   │   ├── scraper.py       # Main GenericScraper class
+│   │   ├── extractors.py    # Metadata, Text, Link, Image, Table extractors
+│   │   └── pagination.py    # Auto-pagination detection
+│   └── scrapers/            # Site-specific scrapers (16 total)
+│       ├── wikipedia.py
+│       ├── imdb.py
+│       ├── stock.py
+│       └── ...
+├── tests/
+│   ├── test_core/           # Tests for core/
+│   ├── test_generic/        # Tests for generic/
+│   ├── test_scrapers/       # Tests for scrapers/
+│   └── test_init.py         # Package-level import tests
+└── pyproject.toml           # Build config, dependencies, tool settings
+```
+
+## Adding a New Scraper
+
+Want to add support for a new website? Here's how:
+
+### 1. Create the scraper
+
+Create a new file in `src/pyscrappy/scrapers/`, e.g. `mysite.py`:
+
+```python
+from __future__ import annotations
+from typing import Any
+from pyscrappy.core.base import BaseScraper
+from pyscrappy.core.models import ScrapeMetadata, ScrapeResult
+
+class MySiteScraper(BaseScraper):
+    name = "mysite"
+
+    def scrape(self, query: str, **kwargs: object) -> ScrapeResult:
+        url = f"https://mysite.com/search?q={query}"
+        soup = self.fetch_and_parse(url)
+
+        items: list[dict[str, Any]] = []
+        for card in soup.select(".result-card"):
+            items.append({
+                "title": card.select_one("h2").get_text(strip=True),
+                "url": card.select_one("a")["href"],
+            })
+
+        return ScrapeResult(
+            data=items,
+            metadata=ScrapeMetadata(source_urls=[url], scraper=self.name),
+        )
+```
+
+### 2. Register the export
+
+Add your scraper to:
+- `src/pyscrappy/scrapers/__init__.py`
+- `src/pyscrappy/__init__.py` (import + add to `__all__`)
+
+### 3. Add tests
+
+Create `tests/test_scrapers/test_mysite.py` with mock HTML and assertions.
+
+### 4. Submit your PR
+
+## Code Quality Standards
+
+- **Style** — enforced by [Ruff](https://docs.astral.sh/ruff/) with a 100-character line length
+- **Type hints** — all public APIs should have type annotations
+- **Python version** — must be compatible with Python 3.9+
+- **No real HTTP calls in tests** — always mock network requests
+- **Match existing patterns** — follow the conventions you see in existing scrapers
+
+## Common Issues and Solutions
+
+### Linting Failures
+
+If `ruff check src/` fails:
+- Run `ruff check src/ --fix` to auto-fix most issues
+- Check import ordering (Ruff enforces isort-compatible ordering)
+- Ensure lines are under 100 characters
+
+### Test Failures
+
+If `pytest tests/ -v` fails:
+- Check if you broke existing functionality
+- Ensure tests use mocks, not real API calls
+- Check test file naming conventions (`test_*.py`)
+- Make sure `__init__.py` exists in test directories
+
+### Import Errors
+
+If you get `ModuleNotFoundError: No module named 'pyscrappy'`:
+```sh
+pip install -e '.[all]'
+```
+
+## Submitting Your PR
+
+1. **Push your branch:** `git push origin your-feature-branch`
+2. **Create a PR:** go to GitHub and create a pull request against `main`
+3. **Fill out the description:** clearly explain what your changes do and why
+4. **Wait for CI:** ensure all checks pass (lint, tests across Python versions, build)
+5. **Address feedback:** make requested changes and push updates
+6. **Merge:** once approved, your PR will be merged!
+
+## Getting Help
+
+If you need help:
+
+- [Create an issue](https://github.com/mldsveda/PyScrappy/issues)
+- Check existing [discussions](https://github.com/mldsveda/PyScrappy/issues) for similar questions
+
+## What to Contribute
+
+Looking for ideas? Check out:
+
+- **Bug fixes** — check [open issues](https://github.com/mldsveda/PyScrappy/issues)
+- **New scrapers** — add support for a website you use
+- **Test coverage** — improve tests for existing scrapers
+- **Documentation** — improve docstrings, examples, or guides
+- **Core improvements** — better error handling, caching, async support
+
+Thank you for contributing to PyScrappy!

pyscrappy-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Vedant Tibrewal, Vedaant Singh.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pyscrappy-1.0.0/PKG-INFO

@@ -0,0 +1,262 @@
+Metadata-Version: 2.4
+Name: pyscrappy
+Version: 1.0.0
+Summary: A robust, all-in-one Python web scraping toolkit
+Project-URL: Homepage, https://github.com/mldsveda/PyScrappy
+Project-URL: Repository, https://github.com/mldsveda/PyScrappy
+Project-URL: Issues, https://github.com/mldsveda/PyScrappy/issues
+Author: Vedaant Singh
+Author-email: Vedant Tibrewal <mlds93363@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: beautifulsoup,crawler,data-extraction,httpx,playwright,scraping,web-scraping
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: httpx>=0.24
+Requires-Dist: lxml>=4.9
+Provides-Extra: all
+Requires-Dist: pandas>=1.5; extra == 'all'
+Requires-Dist: playwright>=1.40; extra == 'all'
+Provides-Extra: browser
+Requires-Dist: playwright>=1.40; extra == 'browser'
+Provides-Extra: dataframe
+Requires-Dist: pandas>=1.5; extra == 'dataframe'
+Description-Content-Type: text/markdown
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/mldsveda/PyScrappy/main/PyScrappy.png">
+  <hr>
+</div>
+
+## PyScrappy: robust, all-in-one Python web scraping toolkit
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![PyPI Latest Release](https://img.shields.io/pypi/v/PyScrappy.svg)](https://pypi.org/project/PyScrappy/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/mldsveda/PyScrappy/blob/main/LICENSE)
+
+PyScrappy is a Python toolkit for web scraping that works out of the box. Point it at any URL and get structured data back — or use built-in scrapers for Wikipedia, IMDB, Yahoo Finance, news feeds, and more.
+
+### Key features
+
+- **Generic scraper** — give it any URL, get back structured text, links, images, tables, and metadata
+- **Auto-pagination** — automatically follows "next page" links
+- **JS rendering** — optional Playwright backend for JavaScript-heavy sites
+- **Custom selectors** — pass CSS selectors to extract exactly what you need
+- **Built-in scrapers** — Wikipedia, IMDB, Yahoo Finance, news (RSS), image search, Amazon, LinkedIn
+- **Clean API** — every scraper returns a `ScrapeResult` with `.to_dataframe()` and `.to_json()`
+- **Retry & rate-limiting** — built-in exponential backoff and per-domain rate limiting
+- **Type-safe** — full type hints, `py.typed` marker
+
+## Installation
+
+```sh
+pip install pyscrappy
+```
+
+**Optional extras:**
+
+```sh
+# Browser support (for JS-rendered pages)
+pip install 'pyscrappy[browser]'
+playwright install chromium
+
+# DataFrame support
+pip install 'pyscrappy[dataframe]'
+
+# Everything
+pip install 'pyscrappy[all]'
+```
+
+## Quick start
+
+### Scrape any URL (one-liner)
+
+```python
+from pyscrappy import scrape
+
+result = scrape("https://en.wikipedia.org/wiki/Web_scraping")
+print(result.data[0]["metadata"]["title"])
+print(result.data[0]["text"]["word_count"])
+```
+
+### Custom CSS selectors
+
+```python
+from pyscrappy import GenericScraper
+
+with GenericScraper() as gs:
+    result = gs.scrape(
+        url="https://news.ycombinator.com",
+        selectors={"title": ".titleline a", "score": ".score"},
+    )
+    for item in result.data:
+        print(item["title"], item.get("score", ""))
+```
+
+### Wikipedia
+
+```python
+from pyscrappy import WikipediaScraper
+
+with WikipediaScraper() as ws:
+    result = ws.scrape(query="Python (programming language)", mode="summary")
+    print(result.data[0]["text"])
+```
+
+### Stock data
+
+```python
+from pyscrappy import StockScraper
+
+with StockScraper() as ss:
+    result = ss.scrape(symbol="AAPL", mode="history", period="1mo")
+    df = result.to_dataframe()
+    print(df.head())
+```
+
+### IMDB
+
+```python
+from pyscrappy import IMDBScraper
+
+with IMDBScraper() as scraper:
+    result = scraper.scrape(genre="sci-fi", max_pages=2)
+    df = result.to_dataframe()
+    print(df[["title", "year", "rating"]])
+```
+
+### News (RSS feeds)
+
+```python
+from pyscrappy import NewsScraper
+
+with NewsScraper() as ns:
+    result = ns.scrape(feed_url="https://rss.nytimes.com/services/xml/rss/nyt/World.xml")
+    for article in result.data[:5]:
+        print(article["title"])
+```
+
+### Image search
+
+```python
+from pyscrappy import ImageSearchScraper
+
+with ImageSearchScraper() as iss:
+    result = iss.scrape(query="golden retriever", max_images=10, download_to="./dogs")
+```
+
+## Configuration
+
+```python
+from pyscrappy import ScraperConfig, GenericScraper
+
+config = ScraperConfig(
+    timeout=20.0,            # request timeout in seconds
+    max_retries=3,           # retry failed requests
+    rate_limit=2.0,          # seconds between requests per domain
+    proxy="http://...",      # HTTP/SOCKS proxy
+    headless=True,           # browser runs headless
+    render_js="auto",        # auto-detect if JS rendering is needed
+)
+
+with GenericScraper(config) as gs:
+    result = gs.scrape(url="https://example.com")
+```
+
+### YouTube
+
+```python
+from pyscrappy import YouTubeScraper
+
+with YouTubeScraper() as scraper:
+    result = scraper.scrape(query="python tutorial", max_results=10)
+    for video in result.data:
+        print(video["title"], video.get("views", ""))
+```
+
+### SoundCloud
+
+```python
+from pyscrappy import SoundCloudScraper
+
+with SoundCloudScraper() as scraper:
+    result = scraper.scrape(query="lo-fi beats", max_results=10)
+```
+
+### E-Commerce (Alibaba, Flipkart, Snapdeal)
+
+```python
+from pyscrappy import AlibabaScraper, FlipkartScraper, SnapdealScraper
+
+with FlipkartScraper() as scraper:
+    result = scraper.scrape(query="laptop", max_pages=2)
+    df = result.to_dataframe()
+```
+
+### Food Delivery (Swiggy, Zomato)
+
+```python
+from pyscrappy import SwiggyScraper, ZomatoScraper
+
+# These are JS-heavy — use render_js=True for best results
+with SwiggyScraper() as scraper:
+    result = scraper.scrape(city="bangalore", render_js=True)
+```
+
+## Built-in scrapers
+
+| Scraper | What it does | Needs browser? |
+|---------|-------------|----------------|
+| `GenericScraper` | Scrape any URL with auto-extraction | Optional |
+| **Data / Research** | | |
+| `WikipediaScraper` | Articles, sections, infoboxes | No |
+| `IMDBScraper` | Movies by genre, search, charts | No |
+| `StockScraper` | Quotes, history, profiles (Yahoo Finance) | No |
+| `NewsScraper` | RSS/Atom feeds, article extraction | No |
+| `ImageSearchScraper` | Image search + download | No |
+| `LinkedInJobsScraper` | Public job listings | No |
+| **E-Commerce** | | |
+| `AmazonScraper` | Product search | No |
+| `AlibabaScraper` | Product search | No |
+| `FlipkartScraper` | Product search | No |
+| `SnapdealScraper` | Product search | No |
+| **Social Media** | | |
+| `YouTubeScraper` | Video search, channel scraping | Optional |
+| `InstagramScraper` | Profiles, hashtag posts | Recommended |
+| `TwitterScraper` | Tweet search | Recommended |
+| **Music** | | |
+| `SpotifyScraper` | Track/playlist search | Recommended |
+| `SoundCloudScraper` | Track search | Optional |
+| **Food Delivery** | | |
+| `SwiggyScraper` | Restaurant listings | Recommended |
+| `ZomatoScraper` | Restaurant listings | Recommended |
+
+## Dependencies
+
+**Required:** `httpx`, `beautifulsoup4`, `lxml`
+
+**Optional:** `playwright` (JS rendering), `pandas` (DataFrames)
+
+## License
+
+[MIT](https://github.com/mldsveda/PyScrappy/blob/main/LICENSE)
+
+## Contributing
+
+All contributions welcome. See [Issues](https://github.com/mldsveda/PyScrappy/issues).
+
+**This package is for educational and research purposes.**