protor 2.0.0__tar.gz

protor-2.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Pulkit
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

protor-2.0.0/PKG-INFO

@@ -0,0 +1,297 @@
+Metadata-Version: 2.4
+Name: protor
+Version: 2.0.0
+Summary: AI-powered web scraper and analyzer — async, offline-friendly, Ollama-backed
+Author-email: Pulkit <work.pulkitpareek@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 Pulkit
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Project-URL: Homepage, https://wtfpulkit.xyz
+Project-URL: Repository, https://github.com/wtfpulkit/protor
+Project-URL: Issues, https://github.com/wtfpulkit/protor/issues
+Project-URL: Changelog, https://github.com/wtfpulkit/protor/blob/main/CHANGELOG.md
+Keywords: web-scraper,cli,ollama,ai,crawler
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Utilities
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: aiohttp>=3.9
+Requires-Dist: beautifulsoup4>=4.12
+Requires-Dist: lxml>=5.0
+Requires-Dist: rich>=13.7
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: aioresponses>=0.7; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: pre-commit>=3.7; extra == "dev"
+Dynamic: license-file
+
+# protor
+
+> scrape websites. analyze with ai. no bs.
+
+a cli tool that actually works. scrapes web content with curl, feeds it to your local ollama models, gets insights. that's it.
+
+## why this exists
+
+because paying for web scraping apis is kinda mid when you can just use curl and a local llm. also because sometimes you need to analyze a bunch of sites and doing it manually is literally painful.
+
+## what you need
+
+- python 3.8+ (obviously)
+- curl (you probably have it)
+- [ollama](https://ollama.ai) running locally
+
+### get ollama set up
+
+```bash
+# grab some models
+ollama pull llama3
+ollama pull mistral
+ollama pull codellama
+
+# start the server
+ollama serve
+```
+
+## install
+
+```bash
+# clone this
+git clone <your-repo-url>
+cd basic-llm-web-scraper
+
+# install it
+pip install -e .
+
+# or just
+pip install -r requirements.txt
+```
+
+## how to use
+
+### see what models you have
+
+```bash
+protor models
+```
+
+### scrape stuff
+
+```bash
+# one site
+protor scrape https://example.com
+
+# multiple sites
+protor scrape https://example.com https://another-site.com
+
+# skip the js files if you want
+protor scrape https://example.com --no-js
+
+# custom settings
+protor scrape https://example.com --output my_data --timeout 60
+```
+
+### analyze what you scraped
+
+```bash
+# general vibes check
+protor analyze
+
+# tech stack deep dive
+protor analyze --focus technical --model codellama
+
+# seo audit
+protor analyze --focus seo --model mistral
+
+# content analysis
+protor analyze --focus content
+```
+
+### do both at once (recommended)
+
+```bash
+# basic usage
+protor run https://example.com
+
+# with options
+protor run https://example.com https://another.com --model llama3 --focus technical
+
+# go crazy
+protor run https://site1.com https://site2.com https://site3.com \
+  --model mistral \
+  --focus seo \
+  --no-js
+```
+
+## what the focus modes do
+
+- **general** - overall content, main themes, what the site's about
+- **technical** - frameworks, tech stack, how it's built
+- **content** - writing quality, structure, how readable it is
+- **seo** - meta tags, optimization stuff, what needs fixing
+
+## what you get
+
+### after scraping
+
+```
+data/
+├── example_com/
+│   ├── index.html        # the actual html
+│   ├── manifest.json     # metadata and stuff
+│   └── js/               # javascript files
+└── sites_index.json      # summary of everything
+```
+
+### after analysis
+
+```
+analysis/
+├── README.md            # readable report
+└── analysis.json        # raw data
+```
+
+## real examples
+
+### quick content check
+
+```bash
+protor run https://blog.example.com --focus content
+```
+
+### technical audit
+
+```bash
+# grab everything including js
+protor scrape https://webapp.example.com
+
+# analyze the tech
+protor analyze --focus technical --model codellama
+```
+
+### competitor research
+
+```bash
+# scrape competitors
+protor scrape https://competitor1.com https://competitor2.com https://competitor3.com
+
+# get seo insights
+protor analyze --focus seo --model mistral
+```
+
+### batch analysis
+
+```bash
+protor run \
+  https://source1.com \
+  https://source2.com \
+  https://source3.com \
+  --model llama3
+```
+
+## when stuff breaks
+
+### ollama issues
+
+```bash
+# make sure it's running
+ollama serve
+
+# check your models
+ollama list
+
+# pull a model if needed
+ollama pull llama3
+```
+
+### curl failing
+
+```bash
+# test manually
+curl -sL https://example.com
+
+# try longer timeout
+protor scrape https://example.com --timeout 120
+```
+
+### analysis taking forever
+
+- use a smaller model
+- scrape fewer sites
+- use --no-js flag
+- get better hardware lol
+
+## pro tips
+
+- always check robots.txt before scraping (be respectful)
+- start with --no-js if you just need content
+- codellama is best for technical analysis
+- mistral is faster than llama3
+- use custom output dirs for different projects
+
+## what's inside
+
+```
+protor/
+├── cli.py          # command interface
+├── scraper.py      # does the scraping
+├── analyzer.py     # talks to ollama
+└── utils.py        # helper stuff
+```
+
+## customize it
+
+want different analysis prompts? edit the `ANALYSIS_PROMPTS` in `protor/analyzer.py`
+
+need different rate limits? check `protor/scraper.py` (0.3s between js files, 1s between sites)
+
+## legal stuff
+
+mit license. do whatever you want with it.
+
+just don't be weird and scrape sites that explicitly say no. respect robots.txt. don't ddos anyone. you know, basic internet etiquette.
+
+## tech stack
+
+- ollama (local llm inference)
+- beautifulsoup (html parsing)
+- requests (http stuff)
+- curl (the goat)
+
+---
+
+built because web scraping shouldn't require a phd or a credit card
+
+made with spite and caffeine
+
+star it if it's useful idk

protor-2.0.0/README.md

@@ -0,0 +1,240 @@
+# protor
+
+> scrape websites. analyze with ai. no bs.
+
+a cli tool that actually works. scrapes web content with curl, feeds it to your local ollama models, gets insights. that's it.
+
+## why this exists
+
+because paying for web scraping apis is kinda mid when you can just use curl and a local llm. also because sometimes you need to analyze a bunch of sites and doing it manually is literally painful.
+
+## what you need
+
+- python 3.8+ (obviously)
+- curl (you probably have it)
+- [ollama](https://ollama.ai) running locally
+
+### get ollama set up
+
+```bash
+# grab some models
+ollama pull llama3
+ollama pull mistral
+ollama pull codellama
+
+# start the server
+ollama serve
+```
+
+## install
+
+```bash
+# clone this
+git clone <your-repo-url>
+cd basic-llm-web-scraper
+
+# install it
+pip install -e .
+
+# or just
+pip install -r requirements.txt
+```
+
+## how to use
+
+### see what models you have
+
+```bash
+protor models
+```
+
+### scrape stuff
+
+```bash
+# one site
+protor scrape https://example.com
+
+# multiple sites
+protor scrape https://example.com https://another-site.com
+
+# skip the js files if you want
+protor scrape https://example.com --no-js
+
+# custom settings
+protor scrape https://example.com --output my_data --timeout 60
+```
+
+### analyze what you scraped
+
+```bash
+# general vibes check
+protor analyze
+
+# tech stack deep dive
+protor analyze --focus technical --model codellama
+
+# seo audit
+protor analyze --focus seo --model mistral
+
+# content analysis
+protor analyze --focus content
+```
+
+### do both at once (recommended)
+
+```bash
+# basic usage
+protor run https://example.com
+
+# with options
+protor run https://example.com https://another.com --model llama3 --focus technical
+
+# go crazy
+protor run https://site1.com https://site2.com https://site3.com \
+  --model mistral \
+  --focus seo \
+  --no-js
+```
+
+## what the focus modes do
+
+- **general** - overall content, main themes, what the site's about
+- **technical** - frameworks, tech stack, how it's built
+- **content** - writing quality, structure, how readable it is
+- **seo** - meta tags, optimization stuff, what needs fixing
+
+## what you get
+
+### after scraping
+
+```
+data/
+├── example_com/
+│   ├── index.html        # the actual html
+│   ├── manifest.json     # metadata and stuff
+│   └── js/               # javascript files
+└── sites_index.json      # summary of everything
+```
+
+### after analysis
+
+```
+analysis/
+├── README.md            # readable report
+└── analysis.json        # raw data
+```
+
+## real examples
+
+### quick content check
+
+```bash
+protor run https://blog.example.com --focus content
+```
+
+### technical audit
+
+```bash
+# grab everything including js
+protor scrape https://webapp.example.com
+
+# analyze the tech
+protor analyze --focus technical --model codellama
+```
+
+### competitor research
+
+```bash
+# scrape competitors
+protor scrape https://competitor1.com https://competitor2.com https://competitor3.com
+
+# get seo insights
+protor analyze --focus seo --model mistral
+```
+
+### batch analysis
+
+```bash
+protor run \
+  https://source1.com \
+  https://source2.com \
+  https://source3.com \
+  --model llama3
+```
+
+## when stuff breaks
+
+### ollama issues
+
+```bash
+# make sure it's running
+ollama serve
+
+# check your models
+ollama list
+
+# pull a model if needed
+ollama pull llama3
+```
+
+### curl failing
+
+```bash
+# test manually
+curl -sL https://example.com
+
+# try longer timeout
+protor scrape https://example.com --timeout 120
+```
+
+### analysis taking forever
+
+- use a smaller model
+- scrape fewer sites
+- use --no-js flag
+- get better hardware lol
+
+## pro tips
+
+- always check robots.txt before scraping (be respectful)
+- start with --no-js if you just need content
+- codellama is best for technical analysis
+- mistral is faster than llama3
+- use custom output dirs for different projects
+
+## what's inside
+
+```
+protor/
+├── cli.py          # command interface
+├── scraper.py      # does the scraping
+├── analyzer.py     # talks to ollama
+└── utils.py        # helper stuff
+```
+
+## customize it
+
+want different analysis prompts? edit the `ANALYSIS_PROMPTS` in `protor/analyzer.py`
+
+need different rate limits? check `protor/scraper.py` (0.3s between js files, 1s between sites)
+
+## legal stuff
+
+mit license. do whatever you want with it.
+
+just don't be weird and scrape sites that explicitly say no. respect robots.txt. don't ddos anyone. you know, basic internet etiquette.
+
+## tech stack
+
+- ollama (local llm inference)
+- beautifulsoup (html parsing)
+- requests (http stuff)
+- curl (the goat)
+
+---
+
+built because web scraping shouldn't require a phd or a credit card
+
+made with spite and caffeine
+
+star it if it's useful idk

protor-2.0.0/protor/__init__.py

@@ -0,0 +1,19 @@
+"""
+protor — async web scraper and AI analyzer.
+
+Quickstart
+----------
+    from protor import scrape_multiple, analyze_with_ollama
+
+    index = scrape_multiple(["https://example.com"])
+    analyze_with_ollama(index, model="llama3", focus="general")
+"""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("protor")
+except PackageNotFoundError:
+    __version__ = "dev"
+
+__all__ = ["__version__"]