gem-scrapscrap 1.0.0__tar.gz

gem_scrapscrap-1.0.0/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: gem-scrapscrap
+Version: 1.0.0
+Summary: CLI tool to scrape and download contract documents from GeM (Government e-Marketplace)
+Home-page: https://github.com/Wickcore/gem-scrapscrap
+Author: Wickcore
+License: MIT
+Project-URL: Homepage, https://github.com/Wickcore/gem-scrapscrap
+Project-URL: Repository, https://github.com/Wickcore/gem-scrapscrap
+Project-URL: Issues, https://github.com/Wickcore/gem-scrapscrap/issues
+Keywords: gem,government,scraping,contracts,tenders
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.28.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: rich>=13.0.0
+Dynamic: home-page
+Dynamic: requires-python
+
+# GeM Document Scraper
+
+A Python web app that scrapes and downloads contract documents from [GeM (Government e-Marketplace)](https://gem.gov.in). Enter a category, set a date range, and download all matching contract PDFs — with automatic CAPTCHA solving.
+
+## Features
+
+- **Web UI** — Clean dark-themed interface, no terminal needed
+- **Category Search** — Autocomplete from GeM's 13,000+ categories
+- **Auto CAPTCHA** — Solves GeM's image CAPTCHAs via Tesseract OCR
+- **PDF Downloads** — Downloads individual files or all as ZIP
+- **Real-time Progress** — Live logs and progress bar while scraping
+- **Background Processing** — Scraper runs in a thread, UI stays responsive
+
+## Demo
+
+1. Enter a category (e.g., "Ball Point Pens (V2) as per IS 3705")
+2. Set date range (DD-MM-YYYY)
+3. Click "Start Scraping"
+4. Watch live progress, download PDFs when done
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.9+
+
+### Install
+
+```bash
+pip3 install -r requirements.txt
+```
+
+### Run CLI (Interactive Terminal)
+
+```bash
+python3 gem_cli.py
+```
+
+### Run Web UI
+
+```bash
+python3 app.py
+```
+
+Open **http://localhost:5000** in your browser.
+
+## How It Works
+
+1. **Search** — User enters category + date range in the web UI
+2. **CAPTCHA** — Opens GeM's contract page, extracts the CAPTCHA image, reads it via Tesseract OCR, submits automatically (retries up to 15 times)
+3. **Scrape** — Collects all matching contract documents from the results
+4. **Download** — For each document, calls GeM's AJAX endpoint to get the download URL, then fetches the PDF via HTTP
+5. **Serve** — Downloaded PDFs are served back through the web UI for browsing/downloading
+
+## Project Structure
+
+```
+GeM-Document-Scraper/
+├── app.py              # Flask web server (routes, job management, threading)
+├── scraper.py          # Background scraping engine (Selenium + AJAX downloads)
+├── actions.py          # Selenium helper functions (category, dates, CAPTCHA)
+├── main.py             # Legacy CLI entry point (still works standalone)
+├── templates/
+│   └── index.html      # Web UI (HTML + CSS + JS, no frameworks)
+├── requirements.txt    # Python dependencies
+├── downloads/          # Downloaded PDFs (per job)
+└── captcha/            # CAPTCHA images (temporary)
+```
+
+## API Endpoints
+
+| Method | Route | Description |
+|--------|-------|-------------|
+| GET | `/` | Web UI |
+| POST | `/scrape` | Start a new scraping job |
+| GET | `/status/<job_id>` | Job status + logs + file list |
+| GET | `/download/<job_id>/<file>` | Download a single PDF |
+| GET | `/download_all/<job_id>` | Download all PDFs as ZIP |
+
+## Tested Categories
+
+These categories are known to have contracts on GeM:
+
+| Category | Notes |
+|----------|-------|
+| `Ball Point Pens (V2) as per IS 3705` | Office supplies, high volume |
+| `Note Sorting Machines (V2)` | Banking equipment |
+| `LED Bulb with Battery as per IS 16102` | Govt LED scheme |
+| `Split Air Conditioner, Wall Mount Type (V3) ISI Marked to IS 1391 (Part 2)` | HVAC |
+| `Nitrogen Tyre Inflators` | Automotive |
+
+Use a wide date range (e.g., `01-01-2025` to `07-07-2025`) for more results.
+
+## Date Format
+
+The web UI accepts multiple formats and auto-converts:
+- `DD-MM-YYYY` (e.g., 01-07-2025)
+- `DD/MM/YYYY` (e.g., 01/07/2025)
+- `YYYY-MM-DD` (e.g., 2025-07-01)
+
+## Dependencies
+
+| Package | Purpose |
+|---------|---------|
+| Flask | Web server |
+| Selenium | Browser automation |
+| Tesseract OCR (`pytesseract`) | CAPTCHA reading |
+| Pillow | Image processing |
+| Requests | HTTP file downloads |
+| Gunicorn | Production WSGI server |
+
+## Deploy to Railway
+
+1. Go to [railway.app](https://railway.app) → Sign up with GitHub
+2. Click **New Project** → **Deploy from GitHub repo**
+3. Select `Wickcore/gem-scrapscrap`
+4. Railway auto-detects the Dockerfile
+5. Click **Deploy**
+
+Your app will be live at `https://your-app.up.railway.app`
+
+**Note:** Railway gives $5 free credits/month — enough for this app.
+
+## Notes
+
+- CAPTCHA accuracy depends on image quality; the scraper retries up to 15 times
+- Headless Chrome is used — no browser window pops up
+- Each scraping job runs in its own thread with its own download directory
+- GeM may change their page structure, which could break selectors
+
+## License
+
+MIT

gem_scrapscrap-1.0.0/README.md

@@ -0,0 +1,133 @@
+# GeM Document Scraper
+
+A Python web app that scrapes and downloads contract documents from [GeM (Government e-Marketplace)](https://gem.gov.in). Enter a category, set a date range, and download all matching contract PDFs — with automatic CAPTCHA solving.
+
+## Features
+
+- **Web UI** — Clean dark-themed interface, no terminal needed
+- **Category Search** — Autocomplete from GeM's 13,000+ categories
+- **Auto CAPTCHA** — Solves GeM's image CAPTCHAs via Tesseract OCR
+- **PDF Downloads** — Downloads individual files or all as ZIP
+- **Real-time Progress** — Live logs and progress bar while scraping
+- **Background Processing** — Scraper runs in a thread, UI stays responsive
+
+## Demo
+
+1. Enter a category (e.g., "Ball Point Pens (V2) as per IS 3705")
+2. Set date range (DD-MM-YYYY)
+3. Click "Start Scraping"
+4. Watch live progress, download PDFs when done
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.9+
+
+### Install
+
+```bash
+pip3 install -r requirements.txt
+```
+
+### Run CLI (Interactive Terminal)
+
+```bash
+python3 gem_cli.py
+```
+
+### Run Web UI
+
+```bash
+python3 app.py
+```
+
+Open **http://localhost:5000** in your browser.
+
+## How It Works
+
+1. **Search** — User enters category + date range in the web UI
+2. **CAPTCHA** — Opens GeM's contract page, extracts the CAPTCHA image, reads it via Tesseract OCR, submits automatically (retries up to 15 times)
+3. **Scrape** — Collects all matching contract documents from the results
+4. **Download** — For each document, calls GeM's AJAX endpoint to get the download URL, then fetches the PDF via HTTP
+5. **Serve** — Downloaded PDFs are served back through the web UI for browsing/downloading
+
+## Project Structure
+
+```
+GeM-Document-Scraper/
+├── app.py              # Flask web server (routes, job management, threading)
+├── scraper.py          # Background scraping engine (Selenium + AJAX downloads)
+├── actions.py          # Selenium helper functions (category, dates, CAPTCHA)
+├── main.py             # Legacy CLI entry point (still works standalone)
+├── templates/
+│   └── index.html      # Web UI (HTML + CSS + JS, no frameworks)
+├── requirements.txt    # Python dependencies
+├── downloads/          # Downloaded PDFs (per job)
+└── captcha/            # CAPTCHA images (temporary)
+```
+
+## API Endpoints
+
+| Method | Route | Description |
+|--------|-------|-------------|
+| GET | `/` | Web UI |
+| POST | `/scrape` | Start a new scraping job |
+| GET | `/status/<job_id>` | Job status + logs + file list |
+| GET | `/download/<job_id>/<file>` | Download a single PDF |
+| GET | `/download_all/<job_id>` | Download all PDFs as ZIP |
+
+## Tested Categories
+
+These categories are known to have contracts on GeM:
+
+| Category | Notes |
+|----------|-------|
+| `Ball Point Pens (V2) as per IS 3705` | Office supplies, high volume |
+| `Note Sorting Machines (V2)` | Banking equipment |
+| `LED Bulb with Battery as per IS 16102` | Govt LED scheme |
+| `Split Air Conditioner, Wall Mount Type (V3) ISI Marked to IS 1391 (Part 2)` | HVAC |
+| `Nitrogen Tyre Inflators` | Automotive |
+
+Use a wide date range (e.g., `01-01-2025` to `07-07-2025`) for more results.
+
+## Date Format
+
+The web UI accepts multiple formats and auto-converts:
+- `DD-MM-YYYY` (e.g., 01-07-2025)
+- `DD/MM/YYYY` (e.g., 01/07/2025)
+- `YYYY-MM-DD` (e.g., 2025-07-01)
+
+## Dependencies
+
+| Package | Purpose |
+|---------|---------|
+| Flask | Web server |
+| Selenium | Browser automation |
+| Tesseract OCR (`pytesseract`) | CAPTCHA reading |
+| Pillow | Image processing |
+| Requests | HTTP file downloads |
+| Gunicorn | Production WSGI server |
+
+## Deploy to Railway
+
+1. Go to [railway.app](https://railway.app) → Sign up with GitHub
+2. Click **New Project** → **Deploy from GitHub repo**
+3. Select `Wickcore/gem-scrapscrap`
+4. Railway auto-detects the Dockerfile
+5. Click **Deploy**
+
+Your app will be live at `https://your-app.up.railway.app`
+
+**Note:** Railway gives $5 free credits/month — enough for this app.
+
+## Notes
+
+- CAPTCHA accuracy depends on image quality; the scraper retries up to 15 times
+- Headless Chrome is used — no browser window pops up
+- Each scraping job runs in its own thread with its own download directory
+- GeM may change their page structure, which could break selectors
+
+## License
+
+MIT

gem_scrapscrap-1.0.0/gem_cli.py

@@ -0,0 +1,422 @@
+#!/usr/bin/env python3
+"""GeM Document Scraper CLI — Interactive terminal tool."""
+
+import os
+import sys
+import time
+import re
+import json
+import zipfile
+import shutil
+import requests as req
+from bs4 import BeautifulSoup
+from rich.console import Console
+from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn, TimeElapsedColumn
+from rich.table import Table
+from rich.panel import Panel
+from rich.columns import Columns
+from rich.text import Text
+from rich.layout import Layout
+from rich.align import Align
+from rich.rule import Rule
+from rich import box
+
+console = Console()
+
+# ─── Helpers ────────────────────────────────────────────────────────────────
+
+def get_terminal_width():
+    return shutil.get_terminal_size((80, 24)).columns
+
+def get_terminal_height():
+    return shutil.get_terminal_size((80, 24)).lines
+
+def banner():
+    w = get_terminal_width()
+    lines = [
+        "  ╔═══════════════════════════════════════════╗  ",
+        "  ║       GeM Document Scraper CLI v1.0       ║  ",
+        "  ║   Government e-Marketplace Downloader     ║  ",
+        "  ╚═══════════════════════════════════════════╝  ",
+    ]
+    for line in lines:
+        console.print(Align.center(line, width=w), style="bold cyan")
+
+def divider():
+    console.print(Rule(style="dim blue"))
+
+def success(msg):
+    console.print(f"  [bold green]✓[/bold green] {msg}")
+
+def error(msg):
+    console.print(f"  [bold red]✗[/bold red] {msg}")
+
+def info(msg):
+    console.print(f"  [bold blue]i[/bold blue] {msg}")
+
+def section(title):
+    console.print()
+    console.print(f"  [bold yellow]{title}[/bold yellow]")
+    divider()
+
+def input_field(prompt, default="", validator=None):
+    while True:
+        try:
+            val = console.input(f"  [bold cyan]{prompt}:[/bold cyan] ").strip()
+        except EOFError:
+            return default
+        if not val and default:
+            return default
+        if validator and not validator(val):
+            continue
+        return val
+
+def confirm(prompt, default=True):
+    try:
+        ans = console.input(f"  [bold yellow]{prompt}[/bold yellow] [{'Y/n' if default else 'y/N'}]: ").strip().lower()
+    except EOFError:
+        return default
+    if not ans:
+        return default
+    return ans in ("y", "yes")
+
+def pick_from_list(items, label="option"):
+    """Let user pick from a numbered list."""
+    w = get_terminal_width()
+    max_name = max(len(c["text"]) for c in items) if items else 0
+    cols = max(1, (w - 4) // max(max_name + 8, 25))
+
+    for i, item in enumerate(items, 1):
+        console.print(f"  [green]{i:3d}.[/green] {item['text']}")
+
+    while True:
+        try:
+            choice = console.input(f"\n  [bold]Pick {label} (1-{len(items)}):[/bold] ").strip()
+        except EOFError:
+            return None
+        if choice.isdigit() and 1 <= int(choice) <= len(items):
+            return items[int(choice) - 1]
+        error("Invalid choice. Try again.")
+
+# ─── GeM API ────────────────────────────────────────────────────────────────
+
+def get_session():
+    s = req.Session()
+    s.headers.update({
+        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36"
+    })
+    return s
+
+def load_categories(session):
+    resp = session.get("https://gem.gov.in/view_contracts", timeout=60)
+    soup = BeautifulSoup(resp.text, "html.parser")
+    cat_select = soup.find("select", {"id": "buyer_category"})
+    categories = []
+    if cat_select:
+        for opt in cat_select.find_all("option"):
+            text = opt.text.strip()
+            value = opt["value"]
+            if text and text not in ("--Select Category--", "--Select--"):
+                categories.append({"text": text, "value": value})
+    return categories
+
+def search_contracts(session, cat_value, from_date, to_date, page=0):
+    search_data = {
+        "fromDate": from_date,
+        "toDate": to_date,
+        "department": "",
+        "bno": "",
+        "buyer_category": cat_value,
+        "page": page,
+    }
+    resp = session.post(
+        "https://gem.gov.in/view_contracts/contract_details",
+        data=search_data,
+        timeout=60,
+    )
+    soup = BeautifulSoup(resp.text, "html.parser")
+    return soup.find_all("div", class_="border")
+
+def load_all_pages(session, cat_value, from_date, to_date, max_pages=50):
+    all_docs = []
+    for page in range(max_pages):
+        docs = search_contracts(session, cat_value, from_date, to_date, page)
+        if not docs:
+            break
+        all_docs.extend(docs)
+        if len(docs) < 20:
+            break
+        time.sleep(0.3)
+    return all_docs
+
+def download_document(session, doc, index, download_dir):
+    link = doc.find("a")
+    if not link:
+        return None
+    onclick = link.get("onclick", "")
+    match = re.search(r"openCap\('([^']+)'\)", onclick)
+    if not match:
+        return None
+    contract_id = match.group(1)
+
+    dl_resp = session.post(
+        "https://gem.gov.in/view_contracts/sbtCaptcha",
+        data={"oid": contract_id},
+        timeout=60,
+    )
+    dl_data = json.loads(dl_resp.text)
+    if dl_data.get("status") != "1":
+        return None
+
+    code_html = dl_data.get("code", "")
+    url_match = re.search(r'href=["\']([^"\']+)["\']', code_html)
+    if not url_match:
+        return None
+
+    pdf_resp = session.get(url_match.group(1), timeout=60)
+    if len(pdf_resp.content) > 1000:
+        filename = f"document_{index}.pdf"
+        filepath = os.path.join(download_dir, filename)
+        with open(filepath, "wb") as f:
+            f.write(pdf_resp.content)
+        return {"filename": filename, "size": len(pdf_resp.content), "contract": contract_id}
+    return None
+
+def create_zip(download_dir, zip_path):
+    with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zf:
+        for f in sorted(os.listdir(download_dir)):
+            if f.endswith(".pdf"):
+                zf.write(os.path.join(download_dir, f), f)
+    return os.path.getsize(zip_path)
+
+# ─── Category Selection ─────────────────────────────────────────────────────
+
+def select_category(categories):
+    section("Select Category")
+    info(f"{len(categories)} categories available")
+
+    search = input_field("Search (type to filter)", default="")
+
+    matches = categories
+    if search:
+        matches = [c for c in categories if search.lower() in c["text"].lower()]
+
+    if not matches:
+        error("No categories match your search.")
+        return None
+
+    console.print()
+    if len(matches) > 30:
+        console.print(f"  [dim]Showing 30 of {len(matches)} matches[/dim]")
+        matches = matches[:30]
+
+    selected = pick_from_list(matches, "category")
+    if selected:
+        success(f"Selected: {selected['text']}")
+    return selected
+
+def select_dates():
+    section("Date Range")
+    info("Format: DD-MM-YYYY (e.g. 01-01-2025)")
+
+    def valid_date(d):
+        if not re.match(r"^\d{2}-\d{2}-\d{4}$", d):
+            error("Invalid format. Use DD-MM-YYYY")
+            return False
+        return True
+
+    from_date = input_field("From date", validator=valid_date)
+    to_date = input_field("To date", validator=valid_date)
+    return from_date, to_date
+
+def show_documents(docs):
+    section(f"Found {len(docs)} Documents")
+    w = get_terminal_width()
+
+    table = Table(box=box.ROUNDED, show_lines=False, expand=True)
+    table.add_column("#", style="green", width=4, no_wrap=True)
+    table.add_column("Contract", style="cyan", ratio=3)
+    table.add_column("Status", style="yellow", width=10, no_wrap=True)
+
+    shown = min(15, len(docs))
+    for i, doc in enumerate(docs[:shown], 1):
+        link = doc.find("a")
+        text = link.text.strip()[:w - 25] if link else "Unknown"
+        table.add_row(str(i), text, "Ready")
+
+    if len(docs) > shown:
+        table.add_row("...", f"[dim]{len(docs) - shown} more[/dim]", "")
+
+    console.print(table)
+
+def select_download_option():
+    section("Download Options")
+    console.print("  [green]1.[/green] Download ALL documents")
+    console.print("  [green]2.[/green] Download as ZIP")
+    console.print("  [green]3.[/green] Pick specific documents")
+    console.print("  [green]4.[/green] Skip")
+
+    choice = input_field("Choose", default="2", validator=lambda x: x in ("1","2","3","4"))
+    return {"1": "all", "2": "zip", "3": "pick", "4": "skip"}[choice]
+
+def pick_documents(docs):
+    section("Pick Documents")
+    info("Enter numbers separated by commas (e.g. 1,3,5-8)")
+    raw = input_field("Documents")
+
+    indices = []
+    for part in raw.split(","):
+        part = part.strip()
+        if "-" in part:
+            start, end = part.split("-", 1)
+            if start.isdigit() and end.isdigit():
+                indices.extend(range(int(start), int(end) + 1))
+        elif part.isdigit():
+            indices.append(int(part))
+
+    return [docs[i - 1] for i in indices if 1 <= i <= len(docs)]
+
+def download_with_progress(session, docs, download_dir):
+    section(f"Downloading {len(docs)} Documents")
+
+    results = []
+    with Progress(
+        SpinnerColumn(),
+        TextColumn("[progress.description]{task.description}"),
+        BarColumn(bar_width=get_terminal_width() - 30),
+        TaskProgressColumn(),
+        TimeElapsedColumn(),
+        console=console,
+    ) as progress:
+        task = progress.add_task("Downloading...", total=len(docs))
+
+        for i, doc in enumerate(docs):
+            progress.update(task, description=f"Document {i+1}/{len(docs)}")
+            result = download_document(session, doc, i + 1, download_dir)
+            if result:
+                results.append(result)
+            progress.advance(task)
+
+    return results
+
+def show_summary(results, download_dir):
+    section("Download Complete")
+
+    table = Table(box=box.ROUNDED, expand=True)
+    table.add_column("File", style="cyan", ratio=2)
+    table.add_column("Contract", style="dim", ratio=3)
+    table.add_column("Size", style="green", justify="right", width=10)
+
+    total_size = 0
+    for r in results:
+        size_kb = r["size"] / 1024
+        total_size += r["size"]
+        table.add_row(r["filename"], r.get("contract", ""), f"{size_kb:.1f} KB")
+
+    console.print(table)
+    console.print()
+    success(f"{len(results)} files saved to: [bold]{download_dir}[/bold]")
+    success(f"Total size: {total_size / 1024:.1f} KB")
+
+# ─── Main ───────────────────────────────────────────────────────────────────
+
+def main():
+    console.clear()
+    banner()
+    divider()
+
+    # Load categories
+    session = get_session()
+    with console.status("[bold green]Connecting to GeM...[/bold green]"):
+        categories = load_categories(session)
+
+    console.print()
+    success(f"Loaded {len(categories)} categories from GeM")
+
+    while True:
+        console.print()
+        divider()
+        console.print("  [bold]MAIN MENU[/bold]")
+        divider()
+        console.print("  [green]1.[/green]  Scrape & Download contracts")
+        console.print("  [green]2.[/green]  Search categories")
+        console.print("  [green]3.[/green]  Quit")
+        divider()
+
+        action = input_field("Choose", default="1", validator=lambda x: x in ("1","2","3"))
+
+        if action == "3":
+            console.print()
+            info("Goodbye!")
+            console.print()
+            break
+
+        if action == "2":
+            section("Category Search")
+            search = input_field("Search")
+            matches = [c for c in categories if search.lower() in c["text"].lower()]
+            if matches:
+                for c in matches[:15]:
+                    console.print(f"    {c['text']}")
+                if len(matches) > 15:
+                    console.print(f"    [dim]... {len(matches) - 15} more[/dim]")
+            else:
+                error("No matches found")
+            continue
+
+        # Full workflow
+        selected = select_category(categories)
+        if not selected:
+            continue
+
+        from_date, to_date = select_dates()
+        if not from_date:
+            continue
+
+        with console.status("[bold yellow]Searching GeM...[/bold yellow]"):
+            docs = load_all_pages(session, selected["value"], from_date, to_date)
+
+        if not docs:
+            error("No documents found. Try wider date range or different category.")
+            continue
+
+        show_documents(docs)
+
+        option = select_download_option()
+        if option == "skip":
+            continue
+
+        if option == "pick":
+            to_download = pick_documents(docs)
+            if not to_download:
+                error("No documents selected.")
+                continue
+        else:
+            to_download = docs
+
+        download_dir = os.path.join(os.getcwd(), "gem_downloads")
+        os.makedirs(download_dir, exist_ok=True)
+
+        results = download_with_progress(session, to_download, download_dir)
+
+        if not results:
+            error("No documents downloaded.")
+            continue
+
+        # ZIP
+        if option in ("all", "zip") and len(results) > 1:
+            if confirm("Create ZIP file?", default=True):
+                zip_path = os.path.join(download_dir, "all_documents.zip")
+                with console.status("[bold yellow]Creating ZIP...[/bold yellow]"):
+                    zip_size = create_zip(download_dir, zip_path)
+                success(f"ZIP created: {zip_path} ({zip_size / 1024:.0f} KB)")
+
+        show_summary(results, download_dir)
+
+if __name__ == "__main__":
+    try:
+        main()
+    except KeyboardInterrupt:
+        console.print("\n")
+        info("Interrupted. Bye!")
+        sys.exit(0)

gem_scrapscrap-1.0.0/gem_scrapscrap.egg-info/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: gem-scrapscrap
+Version: 1.0.0
+Summary: CLI tool to scrape and download contract documents from GeM (Government e-Marketplace)
+Home-page: https://github.com/Wickcore/gem-scrapscrap
+Author: Wickcore
+License: MIT
+Project-URL: Homepage, https://github.com/Wickcore/gem-scrapscrap
+Project-URL: Repository, https://github.com/Wickcore/gem-scrapscrap
+Project-URL: Issues, https://github.com/Wickcore/gem-scrapscrap/issues
+Keywords: gem,government,scraping,contracts,tenders
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.28.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: rich>=13.0.0
+Dynamic: home-page
+Dynamic: requires-python
+
+# GeM Document Scraper
+
+A Python web app that scrapes and downloads contract documents from [GeM (Government e-Marketplace)](https://gem.gov.in). Enter a category, set a date range, and download all matching contract PDFs — with automatic CAPTCHA solving.
+
+## Features
+
+- **Web UI** — Clean dark-themed interface, no terminal needed
+- **Category Search** — Autocomplete from GeM's 13,000+ categories
+- **Auto CAPTCHA** — Solves GeM's image CAPTCHAs via Tesseract OCR
+- **PDF Downloads** — Downloads individual files or all as ZIP
+- **Real-time Progress** — Live logs and progress bar while scraping
+- **Background Processing** — Scraper runs in a thread, UI stays responsive
+
+## Demo
+
+1. Enter a category (e.g., "Ball Point Pens (V2) as per IS 3705")
+2. Set date range (DD-MM-YYYY)
+3. Click "Start Scraping"
+4. Watch live progress, download PDFs when done
+
+## Quick Start
+
+### Prerequisites
+
+- Python 3.9+
+
+### Install
+
+```bash
+pip3 install -r requirements.txt
+```
+
+### Run CLI (Interactive Terminal)
+
+```bash
+python3 gem_cli.py
+```
+
+### Run Web UI
+
+```bash
+python3 app.py
+```
+
+Open **http://localhost:5000** in your browser.
+
+## How It Works
+
+1. **Search** — User enters category + date range in the web UI
+2. **CAPTCHA** — Opens GeM's contract page, extracts the CAPTCHA image, reads it via Tesseract OCR, submits automatically (retries up to 15 times)
+3. **Scrape** — Collects all matching contract documents from the results
+4. **Download** — For each document, calls GeM's AJAX endpoint to get the download URL, then fetches the PDF via HTTP
+5. **Serve** — Downloaded PDFs are served back through the web UI for browsing/downloading
+
+## Project Structure
+
+```
+GeM-Document-Scraper/
+├── app.py              # Flask web server (routes, job management, threading)
+├── scraper.py          # Background scraping engine (Selenium + AJAX downloads)
+├── actions.py          # Selenium helper functions (category, dates, CAPTCHA)
+├── main.py             # Legacy CLI entry point (still works standalone)
+├── templates/
+│   └── index.html      # Web UI (HTML + CSS + JS, no frameworks)
+├── requirements.txt    # Python dependencies
+├── downloads/          # Downloaded PDFs (per job)
+└── captcha/            # CAPTCHA images (temporary)
+```
+
+## API Endpoints
+
+| Method | Route | Description |
+|--------|-------|-------------|
+| GET | `/` | Web UI |
+| POST | `/scrape` | Start a new scraping job |
+| GET | `/status/<job_id>` | Job status + logs + file list |
+| GET | `/download/<job_id>/<file>` | Download a single PDF |
+| GET | `/download_all/<job_id>` | Download all PDFs as ZIP |
+
+## Tested Categories
+
+These categories are known to have contracts on GeM:
+
+| Category | Notes |
+|----------|-------|
+| `Ball Point Pens (V2) as per IS 3705` | Office supplies, high volume |
+| `Note Sorting Machines (V2)` | Banking equipment |
+| `LED Bulb with Battery as per IS 16102` | Govt LED scheme |
+| `Split Air Conditioner, Wall Mount Type (V3) ISI Marked to IS 1391 (Part 2)` | HVAC |
+| `Nitrogen Tyre Inflators` | Automotive |
+
+Use a wide date range (e.g., `01-01-2025` to `07-07-2025`) for more results.
+
+## Date Format
+
+The web UI accepts multiple formats and auto-converts:
+- `DD-MM-YYYY` (e.g., 01-07-2025)
+- `DD/MM/YYYY` (e.g., 01/07/2025)
+- `YYYY-MM-DD` (e.g., 2025-07-01)
+
+## Dependencies
+
+| Package | Purpose |
+|---------|---------|
+| Flask | Web server |
+| Selenium | Browser automation |
+| Tesseract OCR (`pytesseract`) | CAPTCHA reading |
+| Pillow | Image processing |
+| Requests | HTTP file downloads |
+| Gunicorn | Production WSGI server |
+
+## Deploy to Railway
+
+1. Go to [railway.app](https://railway.app) → Sign up with GitHub
+2. Click **New Project** → **Deploy from GitHub repo**
+3. Select `Wickcore/gem-scrapscrap`
+4. Railway auto-detects the Dockerfile
+5. Click **Deploy**
+
+Your app will be live at `https://your-app.up.railway.app`
+
+**Note:** Railway gives $5 free credits/month — enough for this app.
+
+## Notes
+
+- CAPTCHA accuracy depends on image quality; the scraper retries up to 15 times
+- Headless Chrome is used — no browser window pops up
+- Each scraping job runs in its own thread with its own download directory
+- GeM may change their page structure, which could break selectors
+
+## License
+
+MIT

gem_scrapscrap-1.0.0/gem_scrapscrap.egg-info/SOURCES.txt

@@ -0,0 +1,10 @@
+README.md
+gem_cli.py
+pyproject.toml
+setup.py
+gem_scrapscrap.egg-info/PKG-INFO
+gem_scrapscrap.egg-info/SOURCES.txt
+gem_scrapscrap.egg-info/dependency_links.txt
+gem_scrapscrap.egg-info/entry_points.txt
+gem_scrapscrap.egg-info/requires.txt
+gem_scrapscrap.egg-info/top_level.txt

gem_scrapscrap-1.0.0/gem_scrapscrap.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

gem_scrapscrap-1.0.0/gem_scrapscrap.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+gem-scrapscrap = gem_cli:main

gem_scrapscrap-1.0.0/gem_scrapscrap.egg-info/requires.txt

@@ -0,0 +1,3 @@
+requests>=2.28.0
+beautifulsoup4>=4.12.0
+rich>=13.0.0

gem_scrapscrap-1.0.0/gem_scrapscrap.egg-info/top_level.txt

@@ -0,0 +1 @@
+gem_cli

gem_scrapscrap-1.0.0/pyproject.toml

@@ -0,0 +1,45 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "gem-scrapscrap"
+version = "1.0.0"
+description = "CLI tool to scrape and download contract documents from GeM (Government e-Marketplace)"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Wickcore"}
+]
+keywords = ["gem", "government", "scraping", "contracts", "tenders"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Internet :: WWW/HTTP",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "requests>=2.28.0",
+    "beautifulsoup4>=4.12.0",
+    "rich>=13.0.0",
+]
+
+[project.scripts]
+gem-scrapscrap = "gem_cli:main"
+
+[project.urls]
+Homepage = "https://github.com/Wickcore/gem-scrapscrap"
+Repository = "https://github.com/Wickcore/gem-scrapscrap"
+Issues = "https://github.com/Wickcore/gem-scrapscrap/issues"
+
+[tool.setuptools]
+py-modules = ["gem_cli"]

gem_scrapscrap-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

gem_scrapscrap-1.0.0/setup.py

@@ -0,0 +1,26 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="gem-scrapscrap",
+    version="1.0.0",
+    description="GeM Document Scraper — Download contract documents from Government e-Marketplace",
+    author="Wickcore",
+    url="https://github.com/Wickcore/gem-scrapscrap",
+    py_modules=["gem_cli"],
+    install_requires=[
+        "requests",
+        "beautifulsoup4",
+        "rich",
+    ],
+    entry_points={
+        "console_scripts": [
+            "gem-scrapscrap=gem_cli:main",
+        ],
+    },
+    python_requires=">=3.9",
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+    ],
+)