mindforge-cc 11.5.1 → 11.7.0

package/.mindforge/skills/scrapling/SKILL.md

@@ -0,0 +1,345 @@
+---
+name: scrapling
+description: "Web scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python."
+version: 1.0.0
+status: stable
+min_mindforge_version: 11.5.1
+triggers: scrape website, web scraping, extract web content, scrape page, web page scraping, extract from website, html scraping, scrape data, web extraction, crawl page, scrape url, web content extraction
+---
+
+# Scrapling
+
+[Scrapling](https://github.com/D4Vinci/Scrapling) is a web scraping framework with anti-bot bypass, stealth browser automation, and a spider framework. It provides three fetching strategies (HTTP, dynamic JS, stealth/Cloudflare) and a full CLI.
+
+**This skill is for educational and research purposes only.** Users must comply with local/international data scraping laws and respect website Terms of Service.
+
+## When to Use
+
+- Scraping static HTML pages (faster than browser tools)
+- Scraping JS-rendered pages that need a real browser
+- Bypassing Cloudflare Turnstile or bot detection
+- Crawling multiple pages with a spider
+- When the built-in `web_extract` tool does not return the data you need
+
+## Installation
+
+```bash
+pip install "scrapling[all]"
+scrapling install
+```
+
+Minimal install (HTTP only, no browser):
+```bash
+pip install scrapling
+```
+
+With browser automation only:
+```bash
+pip install "scrapling[fetchers]"
+scrapling install
+```
+
+## Quick Reference
+
+| Approach | Class | Use When |
+|----------|-------|----------|
+| HTTP | `Fetcher` / `FetcherSession` | Static pages, APIs, fast bulk requests |
+| Dynamic | `DynamicFetcher` / `DynamicSession` | JS-rendered content, SPAs |
+| Stealth | `StealthyFetcher` / `StealthySession` | Cloudflare, anti-bot protected sites |
+| Spider | `Spider` | Multi-page crawling with link following |
+
+## CLI Usage
+
+### Extract Static Page
+
+```bash
+scrapling extract get 'https://example.com' output.md
+```
+
+With CSS selector and browser impersonation:
+
+```bash
+scrapling extract get 'https://example.com' output.md \
+  --css-selector '.content' \
+  --impersonate 'chrome'
+```
+
+### Extract JS-Rendered Page
+
+```bash
+scrapling extract fetch 'https://example.com' output.md \
+  --css-selector '.dynamic-content' \
+  --disable-resources \
+  --network-idle
+```
+
+### Extract Cloudflare-Protected Page
+
+```bash
+scrapling extract stealthy-fetch 'https://protected-site.com' output.html \
+  --solve-cloudflare \
+  --block-webrtc \
+  --hide-canvas
+```
+
+### POST Request
+
+```bash
+scrapling extract post 'https://example.com/api' output.json \
+  --json '{"query": "search term"}'
+```
+
+### Output Formats
+
+The output format is determined by the file extension:
+- `.html` -- raw HTML
+- `.md` -- converted to Markdown
+- `.txt` -- plain text
+- `.json` / `.jsonl` -- JSON
+
+## Python: HTTP Scraping
+
+### Single Request
+
+```python
+from scrapling.fetchers import Fetcher
+
+page = Fetcher.get('https://quotes.toscrape.com/')
+quotes = page.css('.quote .text::text').getall()
+for q in quotes:
+    print(q)
+```
+
+### Session (Persistent Cookies)
+
+```python
+from scrapling.fetchers import FetcherSession
+
+with FetcherSession(impersonate='chrome') as session:
+    page = session.get('https://example.com/', stealthy_headers=True)
+    links = page.css('a::attr(href)').getall()
+    for link in links[:5]:
+        sub = session.get(link)
+        print(sub.css('h1::text').get())
+```
+
+### POST / PUT / DELETE
+
+```python
+page = Fetcher.post('https://api.example.com/data', json={"key": "value"})
+page = Fetcher.put('https://api.example.com/item/1', data={"name": "updated"})
+page = Fetcher.delete('https://api.example.com/item/1')
+```
+
+### With Proxy
+
+```python
+page = Fetcher.get('https://example.com', proxy='http://user:pass@proxy:8080')
+```
+
+## Python: Dynamic Pages (JS-Rendered)
+
+For pages that require JavaScript execution (SPAs, lazy-loaded content):
+
+```python
+from scrapling.fetchers import DynamicFetcher
+
+page = DynamicFetcher.fetch('https://example.com', headless=True)
+data = page.css('.js-loaded-content::text').getall()
+```
+
+### Wait for Specific Element
+
+```python
+page = DynamicFetcher.fetch(
+    'https://example.com',
+    wait_selector=('.results', 'visible'),
+    network_idle=True,
+)
+```
+
+### Disable Resources for Speed
+
+Blocks fonts, images, media, stylesheets (~25% faster):
+
+```python
+from scrapling.fetchers import DynamicSession
+
+with DynamicSession(headless=True, disable_resources=True, network_idle=True) as session:
+    page = session.fetch('https://example.com')
+    items = page.css('.item::text').getall()
+```
+
+### Custom Page Automation
+
+```python
+from playwright.sync_api import Page
+from scrapling.fetchers import DynamicFetcher
+
+def scroll_and_click(page: Page):
+    page.mouse.wheel(0, 3000)
+    page.wait_for_timeout(1000)
+    page.click('button.load-more')
+    page.wait_for_selector('.extra-results')
+
+page = DynamicFetcher.fetch('https://example.com', page_action=scroll_and_click)
+results = page.css('.extra-results .item::text').getall()
+```
+
+## Python: Stealth Mode (Anti-Bot Bypass)
+
+For Cloudflare-protected or heavily fingerprinted sites:
+
+```python
+from scrapling.fetchers import StealthyFetcher
+
+page = StealthyFetcher.fetch(
+    'https://protected-site.com',
+    headless=True,
+    solve_cloudflare=True,
+    block_webrtc=True,
+    hide_canvas=True,
+)
+content = page.css('.protected-content::text').getall()
+```
+
+### Stealth Session
+
+```python
+from scrapling.fetchers import StealthySession
+
+with StealthySession(headless=True, solve_cloudflare=True) as session:
+    page1 = session.fetch('https://protected-site.com/page1')
+    page2 = session.fetch('https://protected-site.com/page2')
+```
+
+## Element Selection
+
+All fetchers return a `Selector` object with these methods:
+
+### CSS Selectors
+
+```python
+page.css('h1::text').get()              # First h1 text
+page.css('a::attr(href)').getall()      # All link hrefs
+page.css('.quote .text::text').getall() # Nested selection
+```
+
+### XPath
+
+```python
+page.xpath('//div[@class="content"]/text()').getall()
+page.xpath('//a/@href').getall()
+```
+
+### Find Methods
+
+```python
+page.find_all('div', class_='quote')       # By tag + attribute
+page.find_by_text('Read more', tag='a')    # By text content
+page.find_by_regex(r'\$\d+\.\d{2}')       # By regex pattern
+```
+
+### Similar Elements
+
+Find elements with similar structure (useful for product listings, etc.):
+
+```python
+first_product = page.css('.product')[0]
+all_similar = first_product.find_similar()
+```
+
+### Navigation
+
+```python
+el = page.css('.target')[0]
+el.parent                # Parent element
+el.children              # Child elements
+el.next_sibling          # Next sibling
+el.prev_sibling          # Previous sibling
+```
+
+## Python: Spider Framework
+
+For multi-page crawling with link following:
+
+```python
+from scrapling.spiders import Spider, Request, Response
+
+class QuotesSpider(Spider):
+    name = "quotes"
+    start_urls = ["https://quotes.toscrape.com/"]
+    concurrent_requests = 10
+    download_delay = 1
+
+    async def parse(self, response: Response):
+        for quote in response.css('.quote'):
+            yield {
+                "text": quote.css('.text::text').get(),
+                "author": quote.css('.author::text').get(),
+                "tags": quote.css('.tag::text').getall(),
+            }
+
+        next_page = response.css('.next a::attr(href)').get()
+        if next_page:
+            yield response.follow(next_page)
+
+result = QuotesSpider().start()
+print(f"Scraped {len(result.items)} quotes")
+result.items.to_json("quotes.json")
+```
+
+### Multi-Session Spider
+
+Route requests to different fetcher types:
+
+```python
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
+
+class SmartSpider(Spider):
+    name = "smart"
+    start_urls = ["https://example.com/"]
+
+    def configure_sessions(self, manager):
+        manager.add("fast", FetcherSession(impersonate="chrome"))
+        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
+
+    async def parse(self, response: Response):
+        for link in response.css('a::attr(href)').getall():
+            if "protected" in link:
+                yield Request(link, sid="stealth")
+            else:
+                yield Request(link, sid="fast", callback=self.parse)
+```
+
+### Pause/Resume Crawling
+
+```python
+spider = QuotesSpider(crawldir="./crawl_checkpoint")
+spider.start()  # Ctrl+C to pause, re-run to resume from checkpoint
+```
+
+## Pitfalls
+
+- **Browser install required**: run `scrapling install` after pip install -- without it, `DynamicFetcher` and `StealthyFetcher` will fail
+- **Timeouts**: DynamicFetcher/StealthyFetcher timeout is in **milliseconds** (default 30000), Fetcher timeout is in **seconds**
+- **Cloudflare bypass**: `solve_cloudflare=True` adds 5-15 seconds to fetch time -- only enable when needed
+- **Resource usage**: StealthyFetcher runs a real browser -- limit concurrent usage
+- **Legal**: always check robots.txt and website ToS before scraping. This library is for educational and research purposes
+- **Python version**: requires Python 3.10+
+
+## Mandatory actions when this skill is active
+
+Before applying this skill:
+- [ ] Read the task requirements fully before acting
+- [ ] Confirm you understand the goal and constraints
+- [ ] Check for existing work or prior context in the codebase
+
+While working:
+- [ ] Follow the methodology described above step by step
+- [ ] Document any decisions or findings as you go
+
+After completing:
+- [ ] Self-check: does the output satisfy the original requirement?
+- [ ] Verify no regressions or unintended side effects
+

package/.mindforge/skills/sherlock/SKILL.md

@@ -0,0 +1,203 @@
+---
+name: sherlock
+description: "OSINT username search across 400+ social networks. Hunt down social media accounts by username."
+version: 1.0.0
+status: stable
+min_mindforge_version: 11.5.1
+triggers: sherlock, username investigation, find accounts, OSINT username, social media investigation, find social accounts, username search, account discovery, username osint, find profiles, sherlock username, account investigation
+---
+
+# Sherlock OSINT Username Search
+
+Hunt down social media accounts by username across 400+ social networks using the [Sherlock Project](https://github.com/sherlock-project/sherlock).
+
+## When to Use
+
+- User asks to find accounts associated with a username
+- User wants to check username availability across platforms
+- User is conducting OSINT or reconnaissance research
+- User asks "where is this username registered?" or similar
+
+## Requirements
+
+- Sherlock CLI installed: `pipx install sherlock-project` or `pip install sherlock-project`
+- Alternatively: Docker available (`docker run -it --rm sherlock/sherlock`)
+- Network access to query social platforms
+
+## Procedure
+
+### 1. Check if Sherlock is Installed
+
+**Before doing anything else**, verify sherlock is available:
+
+```bash
+sherlock --version
+```
+
+If the command fails:
+- Offer to install: `pipx install sherlock-project` (recommended) or `pip install sherlock-project`
+- **Do NOT** try multiple installation methods — pick one and proceed
+- If installation fails, inform the user and stop
+
+### 2. Extract Username
+
+**Extract the username directly from the user's message if clearly stated.**
+
+Examples where you should **NOT** use clarify:
+- "Find accounts for nasa" → username is `nasa`
+- "Search for johndoe123" → username is `johndoe123`
+- "Check if alice exists on social media" → username is `alice`
+- "Look up user bob on social networks" → username is `bob`
+
+**Only use clarify if:**
+- Multiple potential usernames mentioned ("search for alice or bob")
+- Ambiguous phrasing ("search for my username" without specifying)
+- No username mentioned at all ("do an OSINT search")
+
+When extracting, take the **exact** username as stated — preserve case, numbers, underscores, etc.
+
+### 3. Build Command
+
+**Default command** (use this unless user specifically requests otherwise):
+```bash
+sherlock --print-found --no-color "<username>" --timeout 90
+```
+
+**Optional flags** (only add if user explicitly requests):
+- `--nsfw` — Include NSFW sites (only if user asks)
+- `--tor` — Route through Tor (only if user asks for anonymity)
+
+**Do NOT ask about options via clarify** — just run the default search. Users can request specific options if needed.
+
+### 4. Execute Search
+
+Run via the `terminal` tool. The command typically takes 30-120 seconds depending on network conditions and site count.
+
+**Example terminal call:**
+```json
+{
+  "command": "sherlock --print-found --no-color \"target_username\"",
+  "timeout": 180
+}
+```
+
+### 5. Parse and Present Results
+
+Sherlock outputs found accounts in a simple format. Parse the output and present:
+
+1. **Summary line:** "Found X accounts for username 'Y'"
+2. **Categorized links:** Group by platform type if helpful (social, professional, forums, etc.)
+3. **Output file location:** Sherlock saves results to `<username>.txt` by default
+
+**Example output parsing:**
+```
+[+] Instagram: https://instagram.com/username
+[+] Twitter: https://twitter.com/username
+[+] GitHub: https://github.com/username
+```
+
+Present findings as clickable links when possible.
+
+## Pitfalls
+
+### No Results Found
+If Sherlock finds no accounts, this is often correct — the username may not be registered on checked platforms. Suggest:
+- Checking spelling/variation
+- Trying similar usernames with `?` wildcard: `sherlock "user?name"`
+- The user may have privacy settings or deleted accounts
+
+### Timeout Issues
+Some sites are slow or block automated requests. Use `--timeout 120` to increase wait time, or `--site` to limit scope.
+
+### Tor Configuration
+`--tor` requires Tor daemon running. If user wants anonymity but Tor isn't available, suggest:
+- Installing Tor service
+- Using `--proxy` with an alternative proxy
+
+### False Positives
+Some sites always return "found" due to their response structure. Cross-reference unexpected results with manual checks.
+
+### Rate Limiting
+Aggressive searches may trigger rate limits. For bulk username searches, add delays between calls or use `--local` with cached data.
+
+## Installation
+
+### pipx (recommended)
+```bash
+pipx install sherlock-project
+```
+
+### pip
+```bash
+pip install sherlock-project
+```
+
+### Docker
+```bash
+docker pull sherlock/sherlock
+docker run -it --rm sherlock/sherlock <username>
+```
+
+### Linux packages
+Available on Debian 13+, Ubuntu 22.10+, Homebrew, Kali, BlackArch.
+
+## Ethical Use
+
+This tool is for legitimate OSINT and research purposes only. Remind users:
+- Only search usernames they own or have permission to investigate
+- Respect platform terms of service
+- Do not use for harassment, stalking, or illegal activities
+- Consider privacy implications before sharing results
+
+## Verification
+
+After running sherlock, verify:
+1. Output lists found sites with URLs
+2. `<username>.txt` file created (default output) if using file output
+3. If `--print-found` used, output should only contain `[+]` lines for matches
+
+## Example Interaction
+
+**User:** "Can you check if the username 'johndoe123' exists on social media?"
+
+**Agent procedure:**
+1. Check `sherlock --version` (verify installed)
+2. Username provided — proceed directly
+3. Run: `sherlock --print-found --no-color "johndoe123" --timeout 90`
+4. Parse output and present links
+
+**Response format:**
+> Found 12 accounts for username 'johndoe123':
+>
+> • https://twitter.com/johndoe123
+> • https://github.com/johndoe123
+> • https://instagram.com/johndoe123
+> • [... additional links]
+>
+> Results saved to: johndoe123.txt
+
+---
+
+**User:** "Search for username 'alice' including NSFW sites"
+
+**Agent procedure:**
+1. Check sherlock installed
+2. Username + NSFW flag both provided
+3. Run: `sherlock --print-found --no-color --nsfw "alice" --timeout 90`
+4. Present results
+
+## Mandatory actions when this skill is active
+
+Before applying this skill:
+- [ ] Read the task requirements fully before acting
+- [ ] Confirm you understand the goal and constraints
+- [ ] Check for existing work or prior context in the codebase
+
+While working:
+- [ ] Follow the methodology described above step by step
+- [ ] Document any decisions or findings as you go
+
+After completing:
+- [ ] Self-check: does the output satisfy the original requirement?
+- [ ] Verify no regressions or unintended side effects
+