scrapedesk 1.0.0__tar.gz

scrapedesk-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Engr. Mubashir Hussain
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

scrapedesk-1.0.0/PKG-INFO

@@ -0,0 +1,163 @@
+Metadata-Version: 2.4
+Name: scrapedesk
+Version: 1.0.0
+Summary: A friendly desktop web scraper: grab images, tables, links and text from any website and export to Excel, CSV, JSON or SQLite.
+Author-email: "Engr. Mubashir Hussain" <mubashir6u@gmail.com>
+License-Expression: MIT
+Keywords: web scraping,scraper,gui,tkinter,images,tables,excel,csv,crawler,beautifulsoup,data extraction,desktop
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: MacOS X
+Classifier: Environment :: Win32 (MS Windows)
+Classifier: Environment :: X11 Applications
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Office/Business
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.31
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: lxml>=5.0
+Requires-Dist: pandas>=2.0
+Requires-Dist: openpyxl>=3.1
+Requires-Dist: pillow>=10.0
+Provides-Extra: js
+Requires-Dist: playwright>=1.40; extra == "js"
+Provides-Extra: selenium
+Requires-Dist: selenium>=4.15; extra == "selenium"
+Provides-Extra: all
+Requires-Dist: playwright>=1.40; extra == "all"
+Requires-Dist: selenium>=4.15; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5; extra == "dev"
+Dynamic: license-file
+
+# ScrapeDesk
+
+**A friendly desktop web scraper.** Paste a link, choose what you want (images, tables, links, text, emails or custom fields) and export it to Excel, CSV, JSON, SQLite and more. You don't need to write any code.
+
+```bash
+pip install scrapedesk
+scrapedesk
+```
+
+Works on Windows, macOS and Linux with Python 3.9+.
+
+---
+
+## Highlights
+
+| | |
+|---|---|
+| **Easy to use** | Sidebar navigation, big toggle cards, live result counters, tooltips, notifications, light and dark mode, and keyboard shortcuts |
+| **Images** | Thumbnail gallery with click-to-select and a full-size viewer. Finds `img`, `srcset`, lazy-loaded (`data-src` and more), `<picture>`, CSS backgrounds, `og:image`, icons and linked image files |
+| **Smart downloads** | Filter by type, width/height and file size. Removes duplicates, downloads in parallel, renames, and converts to PNG/JPG/WEBP |
+| **Tables** | Every `<table>`, including merged cells (`colspan`/`rowspan`) and multi-row headers. Can also turn product grids and lists into tables. Sort, filter, merge, and copy to Excel |
+| **Exports** | Excel (one sheet per table), CSV, TSV, JSON, HTML, SQLite and Markdown, or *everything* in one workbook |
+| **Custom fields** | Pick exact values (title, price, rating, link…) with CSS or XPath and get one row per item |
+| **More pages** | Follows "Next page" pagination and crawls links with depth, page limits and include/exclude rules |
+| **Tough sites** | Browser identities, headers, cookies, proxy, retries, request delay, robots.txt, and optional JavaScript rendering |
+
+## Install
+
+```bash
+pip install scrapedesk
+```
+
+Start the app:
+
+```bash
+scrapedesk               # or:  python -m scrapedesk
+scrapedesk https://example.com   # open with a URL already filled in
+```
+
+> **Linux:** tkinter is sometimes packaged separately: `sudo apt install python3-tk`.
+
+### JavaScript-heavy websites (optional)
+
+If a site builds its content with JavaScript (React or Vue apps, infinite scroll), install the browser extra and turn on **Settings → JavaScript websites → Playwright**:
+
+```bash
+pip install "scrapedesk[js]"
+python -m playwright install chromium
+```
+
+If you skip the last step, ScrapeDesk falls back to an installed Google Chrome or Microsoft Edge.
+
+## How to use
+
+1. **Home:** paste a URL (or several), then click the cards for what you want to collect and press **Start**.
+2. Watch the counters update. When the job finishes, a notification takes you to the results.
+3. **Images:** click thumbnails to select them, double-click to see full size, then **Download selected**.
+4. **Tables:** pick a table, then **Export** it (Excel, CSV…) or **Copy** it straight into Excel.
+5. **Custom Fields:** press **Load example** to see how product names and prices are extracted.
+6. Press **Ctrl+E** to export everything in one Excel workbook.
+
+Keyboard shortcuts:
+
+| Keys | Action |
+|---|---|
+| Enter / Ctrl+Enter | Start |
+| Esc | Stop |
+| Ctrl+1…9 | Switch section |
+| Ctrl+E | Export all |
+| Ctrl+D | Dark mode |
+| F1 | Help |
+
+## Command line
+
+Everything also works without the window:
+
+```bash
+# all images wider than 300 px
+scrapedesk-cli https://example.com --images --download --min-width 300 --types jpg,png
+
+# every table to Excel
+scrapedesk-cli https://example.com/stats --tables -o stats.xlsx
+
+# one row per product across 50 pages
+scrapedesk-cli https://books.toscrape.com --paginate --max-pages 50 \
+    --row article.product_pod --css "title=h3 a@title" --css "price=.price_color" -o books.csv
+
+# crawl a site and export everything
+scrapedesk-cli https://example.com --crawl --depth 2 --max-pages 100 --report everything.xlsx
+```
+
+Run `scrapedesk-cli --help` for all options.
+
+## Use it from Python
+
+```python
+from scrapedesk import ScrapeSettings, WebScraper
+from scrapedesk.exporters import export_dataframes, tables_to_frames
+
+settings = ScrapeSettings(extract_images=True, extract_tables=True, min_width=200)
+scraper = WebScraper(settings)
+pages = scraper.run(["https://example.com"])
+
+tables = [t for p in pages for t in p.tables]
+export_dataframes(tables_to_frames(tables), "tables.xlsx")
+scraper.download_images([img for p in pages for img in p.images])
+```
+
+## Responsible use
+
+Only scrape content you are allowed to access. Respect each website's terms of service, robots.txt (there is a setting for it) and rate limits (use the delay setting), and follow privacy laws such as GDPR when you collect personal data. ScrapeDesk does not bypass logins, CAPTCHAs or anti-bot protection.
+
+## License
+
+MIT

scrapedesk-1.0.0/README.md

@@ -0,0 +1,115 @@
+# ScrapeDesk
+
+**A friendly desktop web scraper.** Paste a link, choose what you want (images, tables, links, text, emails or custom fields) and export it to Excel, CSV, JSON, SQLite and more. You don't need to write any code.
+
+```bash
+pip install scrapedesk
+scrapedesk
+```
+
+Works on Windows, macOS and Linux with Python 3.9+.
+
+---
+
+## Highlights
+
+| | |
+|---|---|
+| **Easy to use** | Sidebar navigation, big toggle cards, live result counters, tooltips, notifications, light and dark mode, and keyboard shortcuts |
+| **Images** | Thumbnail gallery with click-to-select and a full-size viewer. Finds `img`, `srcset`, lazy-loaded (`data-src` and more), `<picture>`, CSS backgrounds, `og:image`, icons and linked image files |
+| **Smart downloads** | Filter by type, width/height and file size. Removes duplicates, downloads in parallel, renames, and converts to PNG/JPG/WEBP |
+| **Tables** | Every `<table>`, including merged cells (`colspan`/`rowspan`) and multi-row headers. Can also turn product grids and lists into tables. Sort, filter, merge, and copy to Excel |
+| **Exports** | Excel (one sheet per table), CSV, TSV, JSON, HTML, SQLite and Markdown, or *everything* in one workbook |
+| **Custom fields** | Pick exact values (title, price, rating, link…) with CSS or XPath and get one row per item |
+| **More pages** | Follows "Next page" pagination and crawls links with depth, page limits and include/exclude rules |
+| **Tough sites** | Browser identities, headers, cookies, proxy, retries, request delay, robots.txt, and optional JavaScript rendering |
+
+## Install
+
+```bash
+pip install scrapedesk
+```
+
+Start the app:
+
+```bash
+scrapedesk               # or:  python -m scrapedesk
+scrapedesk https://example.com   # open with a URL already filled in
+```
+
+> **Linux:** tkinter is sometimes packaged separately: `sudo apt install python3-tk`.
+
+### JavaScript-heavy websites (optional)
+
+If a site builds its content with JavaScript (React or Vue apps, infinite scroll), install the browser extra and turn on **Settings → JavaScript websites → Playwright**:
+
+```bash
+pip install "scrapedesk[js]"
+python -m playwright install chromium
+```
+
+If you skip the last step, ScrapeDesk falls back to an installed Google Chrome or Microsoft Edge.
+
+## How to use
+
+1. **Home:** paste a URL (or several), then click the cards for what you want to collect and press **Start**.
+2. Watch the counters update. When the job finishes, a notification takes you to the results.
+3. **Images:** click thumbnails to select them, double-click to see full size, then **Download selected**.
+4. **Tables:** pick a table, then **Export** it (Excel, CSV…) or **Copy** it straight into Excel.
+5. **Custom Fields:** press **Load example** to see how product names and prices are extracted.
+6. Press **Ctrl+E** to export everything in one Excel workbook.
+
+Keyboard shortcuts:
+
+| Keys | Action |
+|---|---|
+| Enter / Ctrl+Enter | Start |
+| Esc | Stop |
+| Ctrl+1…9 | Switch section |
+| Ctrl+E | Export all |
+| Ctrl+D | Dark mode |
+| F1 | Help |
+
+## Command line
+
+Everything also works without the window:
+
+```bash
+# all images wider than 300 px
+scrapedesk-cli https://example.com --images --download --min-width 300 --types jpg,png
+
+# every table to Excel
+scrapedesk-cli https://example.com/stats --tables -o stats.xlsx
+
+# one row per product across 50 pages
+scrapedesk-cli https://books.toscrape.com --paginate --max-pages 50 \
+    --row article.product_pod --css "title=h3 a@title" --css "price=.price_color" -o books.csv
+
+# crawl a site and export everything
+scrapedesk-cli https://example.com --crawl --depth 2 --max-pages 100 --report everything.xlsx
+```
+
+Run `scrapedesk-cli --help` for all options.
+
+## Use it from Python
+
+```python
+from scrapedesk import ScrapeSettings, WebScraper
+from scrapedesk.exporters import export_dataframes, tables_to_frames
+
+settings = ScrapeSettings(extract_images=True, extract_tables=True, min_width=200)
+scraper = WebScraper(settings)
+pages = scraper.run(["https://example.com"])
+
+tables = [t for p in pages for t in p.tables]
+export_dataframes(tables_to_frames(tables), "tables.xlsx")
+scraper.download_images([img for p in pages for img in p.images])
+```
+
+## Responsible use
+
+Only scrape content you are allowed to access. Respect each website's terms of service, robots.txt (there is a setting for it) and rate limits (use the delay setting), and follow privacy laws such as GDPR when you collect personal data. ScrapeDesk does not bypass logins, CAPTCHAs or anti-bot protection.
+
+## License
+
+MIT

scrapedesk-1.0.0/pyproject.toml

@@ -0,0 +1,74 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scrapedesk"
+dynamic = ["version"]
+description = "A friendly desktop web scraper: grab images, tables, links and text from any website and export to Excel, CSV, JSON or SQLite."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Engr. Mubashir Hussain", email = "mubashir6u@gmail.com" }]
+keywords = ["web scraping", "scraper", "gui", "tkinter", "images", "tables", "excel", "csv", "crawler",
+            "beautifulsoup", "data extraction", "desktop"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: MacOS X",
+    "Environment :: Win32 (MS Windows)",
+    "Environment :: X11 Applications",
+    "Intended Audience :: End Users/Desktop",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Internet :: WWW/HTTP",
+    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
+    "Topic :: Office/Business",
+    "Topic :: Utilities",
+]
+dependencies = [
+    "requests>=2.31",
+    "beautifulsoup4>=4.12",
+    "lxml>=5.0",
+    "pandas>=2.0",
+    "openpyxl>=3.1",
+    "pillow>=10.0",
+]
+
+[project.optional-dependencies]
+js = ["playwright>=1.40"]
+selenium = ["selenium>=4.15"]
+all = ["playwright>=1.40", "selenium>=4.15"]
+dev = ["pytest>=7", "build>=1.0", "twine>=5"]
+
+[project.scripts]
+scrapedesk-cli = "scrapedesk.cli:main"
+
+[project.gui-scripts]
+scrapedesk = "scrapedesk.gui.app:main"
+
+# Fill these in once the code is on GitHub - they appear as links on the PyPI page.
+# [project.urls]
+# Homepage = "https://github.com/<your-user>/scrapedesk"
+# Issues = "https://github.com/<your-user>/scrapedesk/issues"
+# Changelog = "https://github.com/<your-user>/scrapedesk/blob/main/CHANGELOG.md"
+
+[tool.setuptools.dynamic]
+version = { attr = "scrapedesk.__version__" }
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+scrapedesk = ["assets/*.png", "assets/*.ico"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

scrapedesk-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

scrapedesk-1.0.0/src/scrapedesk/__init__.py

@@ -0,0 +1,7 @@
+"""ScrapeDesk - a friendly desktop web scraper for images, tables, links and text."""
+
+__version__ = "1.0.0"
+
+from .core import ImageItem, PageResult, ScrapeSettings, TableItem, WebScraper  # noqa: E402
+
+__all__ = ["ScrapeSettings", "WebScraper", "PageResult", "ImageItem", "TableItem", "__version__"]

scrapedesk-1.0.0/src/scrapedesk/__main__.py

@@ -0,0 +1,15 @@
+"""`python -m scrapedesk` opens the GUI; `python -m scrapedesk --cli ...` runs the command line."""
+
+import sys
+
+
+def main():
+    if len(sys.argv) > 1 and sys.argv[1] == "--cli":
+        from .cli import main as cli_main
+        return cli_main(sys.argv[2:])
+    from .gui.app import main as gui_main
+    return gui_main()
+
+
+if __name__ == "__main__":
+    sys.exit(main())

scrapedesk-1.0.0/src/scrapedesk/cli.py

@@ -0,0 +1,100 @@
+"""
+Command-line interface for ScrapeDesk (no GUI needed).
+
+Examples
+  scrapedesk-cli https://example.com --images --download
+  scrapedesk-cli https://en.wikipedia.org/wiki/List_of_largest_cities --tables -o cities.xlsx
+  scrapedesk-cli https://books.toscrape.com --crawl --max-pages 20 --report report.xlsx
+"""
+
+import argparse
+import sys
+
+from .core import ScrapeSettings, WebScraper, USER_AGENTS
+from . import exporters, __version__
+
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(prog="scrapedesk-cli", description="ScrapeDesk - scrape images, tables, "
+                                 "links and text from websites (command line)")
+    ap.add_argument("-V", "--version", action="version", version=f"%(prog)s {__version__}")
+    ap.add_argument("urls", nargs="+", help="URL(s) or local .html file(s)")
+    ap.add_argument("--images", action="store_true", help="extract images")
+    ap.add_argument("--tables", action="store_true", help="extract tables")
+    ap.add_argument("--links", action="store_true", help="extract links")
+    ap.add_argument("--text", action="store_true", help="extract text/meta/contacts")
+    ap.add_argument("--download", action="store_true", help="download the images found")
+    ap.add_argument("--dir", help="image download folder")
+    ap.add_argument("--min-width", type=int, default=0)
+    ap.add_argument("--min-height", type=int, default=0)
+    ap.add_argument("--types", help="comma separated image types, e.g. jpg,png,webp")
+    ap.add_argument("-o", "--output", help="export tables to this file (.xlsx .csv .json .html .db .md)")
+    ap.add_argument("--report", help="export EVERYTHING to .xlsx / .json / .db")
+    ap.add_argument("--crawl", action="store_true")
+    ap.add_argument("--depth", type=int, default=1)
+    ap.add_argument("--max-pages", type=int, default=20)
+    ap.add_argument("--paginate", action="store_true", help="follow 'next page' links")
+    ap.add_argument("--js", choices=["none", "playwright", "selenium"], default="none")
+    ap.add_argument("--delay", type=float, default=0.5)
+    ap.add_argument("--proxy", default="")
+    ap.add_argument("--ua", choices=list(USER_AGENTS), default="Chrome (Windows)")
+    ap.add_argument("--css", action="append", default=[], metavar="NAME=SELECTOR[@attr]",
+                    help="custom CSS field, e.g. price=.price_color or link='h3 a@href'")
+    ap.add_argument("--row", default="", help="record container CSS selector for --css fields")
+    ap.add_argument("--profile", help="load settings from a profile .json saved by the GUI")
+    a = ap.parse_args(argv)
+
+    if a.profile:
+        with open(a.profile, encoding="utf-8") as fh:
+            s = ScrapeSettings.from_json(fh.read())
+    else:
+        s = ScrapeSettings()
+    any_flag = a.images or a.tables or a.links or a.text
+    if any_flag or not a.profile:
+        everything = not any_flag
+        s.extract_images = a.images or everything
+        s.extract_tables = a.tables or everything
+        s.extract_links = a.links or everything
+        s.extract_text = s.extract_meta = s.extract_contacts = a.text or everything
+    s.crawl, s.max_depth, s.max_pages, s.follow_pagination = a.crawl, a.depth, a.max_pages, a.paginate
+    s.render_js, s.delay, s.proxy, s.user_agent = a.js, a.delay, a.proxy, USER_AGENTS[a.ua]
+    s.min_width, s.min_height = a.min_width, a.min_height
+    if a.dir:
+        s.download_dir = a.dir
+    if a.types:
+        s.image_types = [t.strip().lower() for t in a.types.split(",")]
+    for spec in a.css:
+        name, _, q = spec.partition("=")
+        q, _, attr = q.partition("@")
+        s.custom_selectors.append({"name": name, "type": "css", "query": q, "attr": attr or "text"})
+    s.custom_row_selector = a.row
+
+    sc = WebScraper(s, log=print)
+    results = sc.run(a.urls)
+    images = [i for r in results for i in r.images]
+    tables = [t for r in results for t in r.tables]
+    print(f"\nPages: {len(results)}  Images: {len(images)}  Tables: {len(tables)}  "
+          f"Links: {sum(len(r.links) for r in results)}  Custom rows: {sum(len(r.custom) for r in results)}")
+    for t in tables[:20]:
+        print(f"  - {t.name}: {len(t.rows)} rows x {max(len(x) for x in t.rows)} cols")
+    if a.download and images:
+        sc.download_images(images)
+    if a.output:
+        custom = [row for r in results for row in r.custom]
+        frames = exporters.tables_to_frames(tables)
+        if custom:
+            frames = {"Custom": exporters.records_to_df(custom), **frames}
+        if frames:
+            for f in exporters.export_dataframes(frames, a.output):
+                print("saved", f)
+        else:
+            print("No tables / custom data to export.")
+    if a.report:
+        for f in exporters.export_full_report(results, a.report):
+            print("saved", f)
+    sc.close()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())