seolens 0.1.0__tar.gz

seolens-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Supriyo Mallick
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

seolens-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: seolens
+Version: 0.1.0
+Summary: Free, open-source SEO crawler and audit tool (up to 10,000 URLs) with a Streamlit dashboard.
+Author: Supriyo Mallick
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/growthlab-supriyo/seolens
+Keywords: seo,crawler,seo-audit,technical-seo,streamlit
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Environment :: Web Environment
+Classifier: Intended Audience :: Information Technology
+Classifier: Topic :: Internet :: WWW/HTTP :: Site Management
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests<3,>=2.31
+Requires-Dist: regex>=2023.0
+Requires-Dist: openpyxl<4,>=3.1
+Requires-Dist: pandas<4,>=2
+Requires-Dist: streamlit<2,>=1.40
+Provides-Extra: js
+Requires-Dist: playwright<2,>=1.40; extra == "js"
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Dynamic: license-file
+
+# 🔎 SEO Lens
+
+**A free, open-source SEO crawler and site audit tool.** It crawls up to 10,000 URLs per run and gives you a health dashboard, an Excel report and a site architecture map. It runs on your own computer, so there's no account, no subscription and no tracking.
+
+Made by [Supriyo Mallick](https://www.linkedin.com/in/s-mallick) for SEOs, freelancers and small teams who can't justify a paid crawler licence.
+
+## Install
+
+You need Python 3.10 or newer.
+
+```bash
+pipx install seolens
+seolens
+```
+
+The app opens in your browser at `http://localhost:8501`. (`pip install seolens` works too.)
+
+**Optional JavaScript rendering**, for sites built with React, Vue and similar frameworks:
+
+```bash
+pipx install "seolens[js]"
+playwright install chromium
+```
+
+JavaScript rendering uses Microsoft Edge or Google Chrome if one is installed, and Chromium otherwise.
+
+**From source:**
+
+```bash
+git clone https://github.com/growthlab-supriyo/seolens && cd seolens
+pip install -e ".[dev]"
+seolens
+```
+
+## What it checks
+
+| Area | Checks |
+|---|---|
+| Crawlability & indexing | Status codes, redirect chains and loops, noindex, X-Robots-Tag, canonicals (missing, conflicting, broken, chained), robots.txt, XML sitemaps and sitemap indexes |
+| On-page | Titles, meta descriptions, H1–H3, heading levels, title vs H1, viewport, word count |
+| Content | Thin pages, exact duplicates, near-duplicates (MinHash), placeholder text, date consistency |
+| Internal linking | Inlinks and outlinks, anchors, broken links, click depth, orphan pages, Link Score (PageRank-style) |
+| Markup | JSON-LD, Microdata, RDFa, missing schema properties, Open Graph, Twitter cards |
+| International | hreflang (return links, codes, targets), `html lang`, og:locale |
+| Performance signals | Heavy images, render-blocking scripts, third-party scripts, page weight |
+
+**Dashboard tabs:** Overview (health score, crawled-pages breakdown, errors, warnings and notices, thematic scores, top issues, click depth), Issues, All pages (filter and export CSV), Response codes, Blocked & failed, Broken links and Site health. The **Excel report** has the same data plus a Summary sheet with quick wins.
+
+## Fast, and polite
+
+- **Adaptive speed (AIMD).** It starts at 10 pages at a time on Medium. It speeds up while the site answers quickly and slows down gradually only when the site *stays* slow. A few slow pages don't change anything.
+- When a site says it is overloaded (HTTP 429 or 503), the crawler halves its speed and waits for the `Retry-After` time the site asks for.
+- It follows robots.txt, including `Crawl-delay`.
+- **Pause and resume.** Every queued URL and crawled page is saved to SQLite as the crawl runs. You can pause a crawl, close the app, and resume it later. **Finish now** builds the report from the pages crawled so far.
+
+## Honest crawling: what it does with Cloudflare and firewalls
+
+SEO Lens **does not bypass** Cloudflare, CAPTCHAs or other bot protection, and it never disguises itself as a person's browser. It sends `SEOLens/<version>` in its User-Agent so site owners can see who is crawling them. What it does instead:
+
+- **Detects and labels** each blocked page: Cloudflare challenge, bot check or CAPTCHA, 403, rate limiting (429), server busy (503), timeouts and connection errors. Blocked pages appear in the **Blocked & failed** tab and the Excel summary, never silently missing, so an incomplete crawl can't pass for a complete audit.
+- **Doesn't hammer a blocked page.** A challenge page is not retried. A 403 is retried once, and only if the site was serving pages normally until then.
+- **Authorized access for sites you own.** In Cloudflare, add a WAF custom rule that skips security checks for a secret request header (or for the `SEOLens` User-Agent). Then add that header under *Authorized access* in the app. HTTP Basic login is supported too. Credentials are sent **only** to the site being audited, never to external links, images or redirects to other hosts. They are stored in the local database on your computer.
+
+## Command line
+
+```bash
+seolens crawl example.com --max 2000 --speed Medium --render off
+seolens crawl --list urls.txt
+```
+
+## Where your data lives
+
+The database and reports are stored in `~/.seolens/` (`C:\Users\<you>\.seolens\` on Windows). To use another folder, set `SEOLENS_HOME`. Nothing is sent anywhere else.
+
+## Limits
+
+- 10,000 URLs per crawl.
+- Performance checks are signals from the HTML, not measured Core Web Vitals. PageSpeed Insights integration is planned.
+- Word count is approximate on pages with a lot of interface text.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest -q tests
+```
+
+The tests run a small local website with redirects, a 404, a slow page, a rate-limited page, a Cloudflare-style challenge and a protected page.
+
+| File | What it does |
+|---|---|
+| `seolens/crawler.py` | Discovery, fetch loop, blocked-page classification, saving and resuming |
+| `seolens/throttle.py` | The adaptive concurrency controller |
+| `seolens/analyze.py`, `checks.py`, `sitehealth.py` | Per-page analysis, cross-page checks, site-wide checks |
+| `seolens/report.py` | Excel report and site architecture map |
+| `seolens/store.py` | SQLite (WAL) |
+| `seolens/worker.py` | Runs the crawl in its own process |
+| `seolens/app.py`, `overview.py` | Streamlit dashboard |
+
+**Public demo:** set `SEOLENS_DEMO=1` to cap crawls at 500 URLs and turn off JavaScript rendering (for example on Hugging Face Spaces).
+
+MIT licence.

seolens-0.1.0/README.md

@@ -0,0 +1,102 @@
+# 🔎 SEO Lens
+
+**A free, open-source SEO crawler and site audit tool.** It crawls up to 10,000 URLs per run and gives you a health dashboard, an Excel report and a site architecture map. It runs on your own computer, so there's no account, no subscription and no tracking.
+
+Made by [Supriyo Mallick](https://www.linkedin.com/in/s-mallick) for SEOs, freelancers and small teams who can't justify a paid crawler licence.
+
+## Install
+
+You need Python 3.10 or newer.
+
+```bash
+pipx install seolens
+seolens
+```
+
+The app opens in your browser at `http://localhost:8501`. (`pip install seolens` works too.)
+
+**Optional JavaScript rendering**, for sites built with React, Vue and similar frameworks:
+
+```bash
+pipx install "seolens[js]"
+playwright install chromium
+```
+
+JavaScript rendering uses Microsoft Edge or Google Chrome if one is installed, and Chromium otherwise.
+
+**From source:**
+
+```bash
+git clone https://github.com/growthlab-supriyo/seolens && cd seolens
+pip install -e ".[dev]"
+seolens
+```
+
+## What it checks
+
+| Area | Checks |
+|---|---|
+| Crawlability & indexing | Status codes, redirect chains and loops, noindex, X-Robots-Tag, canonicals (missing, conflicting, broken, chained), robots.txt, XML sitemaps and sitemap indexes |
+| On-page | Titles, meta descriptions, H1–H3, heading levels, title vs H1, viewport, word count |
+| Content | Thin pages, exact duplicates, near-duplicates (MinHash), placeholder text, date consistency |
+| Internal linking | Inlinks and outlinks, anchors, broken links, click depth, orphan pages, Link Score (PageRank-style) |
+| Markup | JSON-LD, Microdata, RDFa, missing schema properties, Open Graph, Twitter cards |
+| International | hreflang (return links, codes, targets), `html lang`, og:locale |
+| Performance signals | Heavy images, render-blocking scripts, third-party scripts, page weight |
+
+**Dashboard tabs:** Overview (health score, crawled-pages breakdown, errors, warnings and notices, thematic scores, top issues, click depth), Issues, All pages (filter and export CSV), Response codes, Blocked & failed, Broken links and Site health. The **Excel report** has the same data plus a Summary sheet with quick wins.
+
+## Fast, and polite
+
+- **Adaptive speed (AIMD).** It starts at 10 pages at a time on Medium. It speeds up while the site answers quickly and slows down gradually only when the site *stays* slow. A few slow pages don't change anything.
+- When a site says it is overloaded (HTTP 429 or 503), the crawler halves its speed and waits for the `Retry-After` time the site asks for.
+- It follows robots.txt, including `Crawl-delay`.
+- **Pause and resume.** Every queued URL and crawled page is saved to SQLite as the crawl runs. You can pause a crawl, close the app, and resume it later. **Finish now** builds the report from the pages crawled so far.
+
+## Honest crawling: what it does with Cloudflare and firewalls
+
+SEO Lens **does not bypass** Cloudflare, CAPTCHAs or other bot protection, and it never disguises itself as a person's browser. It sends `SEOLens/<version>` in its User-Agent so site owners can see who is crawling them. What it does instead:
+
+- **Detects and labels** each blocked page: Cloudflare challenge, bot check or CAPTCHA, 403, rate limiting (429), server busy (503), timeouts and connection errors. Blocked pages appear in the **Blocked & failed** tab and the Excel summary, never silently missing, so an incomplete crawl can't pass for a complete audit.
+- **Doesn't hammer a blocked page.** A challenge page is not retried. A 403 is retried once, and only if the site was serving pages normally until then.
+- **Authorized access for sites you own.** In Cloudflare, add a WAF custom rule that skips security checks for a secret request header (or for the `SEOLens` User-Agent). Then add that header under *Authorized access* in the app. HTTP Basic login is supported too. Credentials are sent **only** to the site being audited, never to external links, images or redirects to other hosts. They are stored in the local database on your computer.
+
+## Command line
+
+```bash
+seolens crawl example.com --max 2000 --speed Medium --render off
+seolens crawl --list urls.txt
+```
+
+## Where your data lives
+
+The database and reports are stored in `~/.seolens/` (`C:\Users\<you>\.seolens\` on Windows). To use another folder, set `SEOLENS_HOME`. Nothing is sent anywhere else.
+
+## Limits
+
+- 10,000 URLs per crawl.
+- Performance checks are signals from the HTML, not measured Core Web Vitals. PageSpeed Insights integration is planned.
+- Word count is approximate on pages with a lot of interface text.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest -q tests
+```
+
+The tests run a small local website with redirects, a 404, a slow page, a rate-limited page, a Cloudflare-style challenge and a protected page.
+
+| File | What it does |
+|---|---|
+| `seolens/crawler.py` | Discovery, fetch loop, blocked-page classification, saving and resuming |
+| `seolens/throttle.py` | The adaptive concurrency controller |
+| `seolens/analyze.py`, `checks.py`, `sitehealth.py` | Per-page analysis, cross-page checks, site-wide checks |
+| `seolens/report.py` | Excel report and site architecture map |
+| `seolens/store.py` | SQLite (WAL) |
+| `seolens/worker.py` | Runs the crawl in its own process |
+| `seolens/app.py`, `overview.py` | Streamlit dashboard |
+
+**Public demo:** set `SEOLENS_DEMO=1` to cap crawls at 500 URLs and turn off JavaScript rendering (for example on Hugging Face Spaces).
+
+MIT licence.

seolens-0.1.0/pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "seolens"
+version = "0.1.0"
+description = "Free, open-source SEO crawler and audit tool (up to 10,000 URLs) with a Streamlit dashboard."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.10"
+authors = [{ name = "Supriyo Mallick" }]
+keywords = ["seo", "crawler", "seo-audit", "technical-seo", "streamlit"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Environment :: Web Environment",
+    "Intended Audience :: Information Technology",
+    "Topic :: Internet :: WWW/HTTP :: Site Management",
+]
+dependencies = [
+    "requests>=2.31,<3",
+    "regex>=2023.0",
+    "openpyxl>=3.1,<4",
+    "pandas>=2,<4",
+    "streamlit>=1.40,<2",
+]
+
+[project.optional-dependencies]
+js = ["playwright>=1.40,<2"]
+dev = ["pytest>=8"]
+
+[project.scripts]
+seolens = "seolens.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/growthlab-supriyo/seolens"
+
+[tool.setuptools]
+packages = ["seolens", "seolens.data"]
+
+[tool.setuptools.package-data]
+seolens = ["data/*"]

seolens-0.1.0/seolens/__init__.py

@@ -0,0 +1,7 @@
+"""SEO Lens: a free, open-source SEO crawler and audit tool (up to 10,000 URLs per crawl)."""
+__version__ = "0.1.0"
+
+# Change this to the project's GitHub page before the first release: it is sent in the User-Agent so site owners can see who crawls them.
+PROJECT_URL = "https://github.com/growthlab-supriyo/seolens"
+MAX_URLS = 10000
+DEMO_MAX_URLS = 500  # when SEOLENS_DEMO=1 (the public online demo)