mindforge-cc 11.5.1 → 11.7.0

package/.agent/skills/scrapling/SKILL.md

@@ -0,0 +1,328 @@
+---
+name: scrapling
+description: Web scraping with Scrapling - HTTP fetching, stealth browser automation, Cloudflare bypass, and spider crawling via CLI and Python.
+version: 1.0.0
+prerequisites:
+  commands: [scrapling, python]
+---
+
+# Scrapling
+
+[Scrapling](https://github.com/D4Vinci/Scrapling) is a web scraping framework with anti-bot bypass, stealth browser automation, and a spider framework. It provides three fetching strategies (HTTP, dynamic JS, stealth/Cloudflare) and a full CLI.
+
+**This skill is for educational and research purposes only.** Users must comply with local/international data scraping laws and respect website Terms of Service.
+
+## When to Use
+
+- Scraping static HTML pages (faster than browser tools)
+- Scraping JS-rendered pages that need a real browser
+- Bypassing Cloudflare Turnstile or bot detection
+- Crawling multiple pages with a spider
+- When the built-in `web_extract` tool does not return the data you need
+
+## Installation
+
+```bash
+pip install "scrapling[all]"
+scrapling install
+```
+
+Minimal install (HTTP only, no browser):
+```bash
+pip install scrapling
+```
+
+With browser automation only:
+```bash
+pip install "scrapling[fetchers]"
+scrapling install
+```
+
+## Quick Reference
+
+| Approach | Class | Use When |
+|----------|-------|----------|
+| HTTP | `Fetcher` / `FetcherSession` | Static pages, APIs, fast bulk requests |
+| Dynamic | `DynamicFetcher` / `DynamicSession` | JS-rendered content, SPAs |
+| Stealth | `StealthyFetcher` / `StealthySession` | Cloudflare, anti-bot protected sites |
+| Spider | `Spider` | Multi-page crawling with link following |
+
+## CLI Usage
+
+### Extract Static Page
+
+```bash
+scrapling extract get 'https://example.com' output.md
+```
+
+With CSS selector and browser impersonation:
+
+```bash
+scrapling extract get 'https://example.com' output.md \
+  --css-selector '.content' \
+  --impersonate 'chrome'
+```
+
+### Extract JS-Rendered Page
+
+```bash
+scrapling extract fetch 'https://example.com' output.md \
+  --css-selector '.dynamic-content' \
+  --disable-resources \
+  --network-idle
+```
+
+### Extract Cloudflare-Protected Page
+
+```bash
+scrapling extract stealthy-fetch 'https://protected-site.com' output.html \
+  --solve-cloudflare \
+  --block-webrtc \
+  --hide-canvas
+```
+
+### POST Request
+
+```bash
+scrapling extract post 'https://example.com/api' output.json \
+  --json '{"query": "search term"}'
+```
+
+### Output Formats
+
+The output format is determined by the file extension:
+- `.html` -- raw HTML
+- `.md` -- converted to Markdown
+- `.txt` -- plain text
+- `.json` / `.jsonl` -- JSON
+
+## Python: HTTP Scraping
+
+### Single Request
+
+```python
+from scrapling.fetchers import Fetcher
+
+page = Fetcher.get('https://quotes.toscrape.com/')
+quotes = page.css('.quote .text::text').getall()
+for q in quotes:
+    print(q)
+```
+
+### Session (Persistent Cookies)
+
+```python
+from scrapling.fetchers import FetcherSession
+
+with FetcherSession(impersonate='chrome') as session:
+    page = session.get('https://example.com/', stealthy_headers=True)
+    links = page.css('a::attr(href)').getall()
+    for link in links[:5]:
+        sub = session.get(link)
+        print(sub.css('h1::text').get())
+```
+
+### POST / PUT / DELETE
+
+```python
+page = Fetcher.post('https://api.example.com/data', json={"key": "value"})
+page = Fetcher.put('https://api.example.com/item/1', data={"name": "updated"})
+page = Fetcher.delete('https://api.example.com/item/1')
+```
+
+### With Proxy
+
+```python
+page = Fetcher.get('https://example.com', proxy='http://user:pass@proxy:8080')
+```
+
+## Python: Dynamic Pages (JS-Rendered)
+
+For pages that require JavaScript execution (SPAs, lazy-loaded content):
+
+```python
+from scrapling.fetchers import DynamicFetcher
+
+page = DynamicFetcher.fetch('https://example.com', headless=True)
+data = page.css('.js-loaded-content::text').getall()
+```
+
+### Wait for Specific Element
+
+```python
+page = DynamicFetcher.fetch(
+    'https://example.com',
+    wait_selector=('.results', 'visible'),
+    network_idle=True,
+)
+```
+
+### Disable Resources for Speed
+
+Blocks fonts, images, media, stylesheets (~25% faster):
+
+```python
+from scrapling.fetchers import DynamicSession
+
+with DynamicSession(headless=True, disable_resources=True, network_idle=True) as session:
+    page = session.fetch('https://example.com')
+    items = page.css('.item::text').getall()
+```
+
+### Custom Page Automation
+
+```python
+from playwright.sync_api import Page
+from scrapling.fetchers import DynamicFetcher
+
+def scroll_and_click(page: Page):
+    page.mouse.wheel(0, 3000)
+    page.wait_for_timeout(1000)
+    page.click('button.load-more')
+    page.wait_for_selector('.extra-results')
+
+page = DynamicFetcher.fetch('https://example.com', page_action=scroll_and_click)
+results = page.css('.extra-results .item::text').getall()
+```
+
+## Python: Stealth Mode (Anti-Bot Bypass)
+
+For Cloudflare-protected or heavily fingerprinted sites:
+
+```python
+from scrapling.fetchers import StealthyFetcher
+
+page = StealthyFetcher.fetch(
+    'https://protected-site.com',
+    headless=True,
+    solve_cloudflare=True,
+    block_webrtc=True,
+    hide_canvas=True,
+)
+content = page.css('.protected-content::text').getall()
+```
+
+### Stealth Session
+
+```python
+from scrapling.fetchers import StealthySession
+
+with StealthySession(headless=True, solve_cloudflare=True) as session:
+    page1 = session.fetch('https://protected-site.com/page1')
+    page2 = session.fetch('https://protected-site.com/page2')
+```
+
+## Element Selection
+
+All fetchers return a `Selector` object with these methods:
+
+### CSS Selectors
+
+```python
+page.css('h1::text').get()              # First h1 text
+page.css('a::attr(href)').getall()      # All link hrefs
+page.css('.quote .text::text').getall() # Nested selection
+```
+
+### XPath
+
+```python
+page.xpath('//div[@class="content"]/text()').getall()
+page.xpath('//a/@href').getall()
+```
+
+### Find Methods
+
+```python
+page.find_all('div', class_='quote')       # By tag + attribute
+page.find_by_text('Read more', tag='a')    # By text content
+page.find_by_regex(r'\$\d+\.\d{2}')       # By regex pattern
+```
+
+### Similar Elements
+
+Find elements with similar structure (useful for product listings, etc.):
+
+```python
+first_product = page.css('.product')[0]
+all_similar = first_product.find_similar()
+```
+
+### Navigation
+
+```python
+el = page.css('.target')[0]
+el.parent                # Parent element
+el.children              # Child elements
+el.next_sibling          # Next sibling
+el.prev_sibling          # Previous sibling
+```
+
+## Python: Spider Framework
+
+For multi-page crawling with link following:
+
+```python
+from scrapling.spiders import Spider, Request, Response
+
+class QuotesSpider(Spider):
+    name = "quotes"
+    start_urls = ["https://quotes.toscrape.com/"]
+    concurrent_requests = 10
+    download_delay = 1
+
+    async def parse(self, response: Response):
+        for quote in response.css('.quote'):
+            yield {
+                "text": quote.css('.text::text').get(),
+                "author": quote.css('.author::text').get(),
+                "tags": quote.css('.tag::text').getall(),
+            }
+
+        next_page = response.css('.next a::attr(href)').get()
+        if next_page:
+            yield response.follow(next_page)
+
+result = QuotesSpider().start()
+print(f"Scraped {len(result.items)} quotes")
+result.items.to_json("quotes.json")
+```
+
+### Multi-Session Spider
+
+Route requests to different fetcher types:
+
+```python
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
+
+class SmartSpider(Spider):
+    name = "smart"
+    start_urls = ["https://example.com/"]
+
+    def configure_sessions(self, manager):
+        manager.add("fast", FetcherSession(impersonate="chrome"))
+        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
+
+    async def parse(self, response: Response):
+        for link in response.css('a::attr(href)').getall():
+            if "protected" in link:
+                yield Request(link, sid="stealth")
+            else:
+                yield Request(link, sid="fast", callback=self.parse)
+```
+
+### Pause/Resume Crawling
+
+```python
+spider = QuotesSpider(crawldir="./crawl_checkpoint")
+spider.start()  # Ctrl+C to pause, re-run to resume from checkpoint
+```
+
+## Pitfalls
+
+- **Browser install required**: run `scrapling install` after pip install -- without it, `DynamicFetcher` and `StealthyFetcher` will fail
+- **Timeouts**: DynamicFetcher/StealthyFetcher timeout is in **milliseconds** (default 30000), Fetcher timeout is in **seconds**
+- **Cloudflare bypass**: `solve_cloudflare=True` adds 5-15 seconds to fetch time -- only enable when needed
+- **Resource usage**: StealthyFetcher runs a real browser -- limit concurrent usage
+- **Legal**: always check robots.txt and website ToS before scraping. This library is for educational and research purposes
+- **Python version**: requires Python 3.10+

package/.agent/skills/sherlock/SKILL.md

@@ -0,0 +1,186 @@
+---
+name: sherlock
+description: OSINT username search across 400+ social networks. Hunt down social media accounts by username.
+version: 1.0.0
+prerequisites:
+  commands: [sherlock]
+---
+
+# Sherlock OSINT Username Search
+
+Hunt down social media accounts by username across 400+ social networks using the [Sherlock Project](https://github.com/sherlock-project/sherlock).
+
+## When to Use
+
+- User asks to find accounts associated with a username
+- User wants to check username availability across platforms
+- User is conducting OSINT or reconnaissance research
+- User asks "where is this username registered?" or similar
+
+## Requirements
+
+- Sherlock CLI installed: `pipx install sherlock-project` or `pip install sherlock-project`
+- Alternatively: Docker available (`docker run -it --rm sherlock/sherlock`)
+- Network access to query social platforms
+
+## Procedure
+
+### 1. Check if Sherlock is Installed
+
+**Before doing anything else**, verify sherlock is available:
+
+```bash
+sherlock --version
+```
+
+If the command fails:
+- Offer to install: `pipx install sherlock-project` (recommended) or `pip install sherlock-project`
+- **Do NOT** try multiple installation methods — pick one and proceed
+- If installation fails, inform the user and stop
+
+### 2. Extract Username
+
+**Extract the username directly from the user's message if clearly stated.**
+
+Examples where you should **NOT** use clarify:
+- "Find accounts for nasa" → username is `nasa`
+- "Search for johndoe123" → username is `johndoe123`
+- "Check if alice exists on social media" → username is `alice`
+- "Look up user bob on social networks" → username is `bob`
+
+**Only use clarify if:**
+- Multiple potential usernames mentioned ("search for alice or bob")
+- Ambiguous phrasing ("search for my username" without specifying)
+- No username mentioned at all ("do an OSINT search")
+
+When extracting, take the **exact** username as stated — preserve case, numbers, underscores, etc.
+
+### 3. Build Command
+
+**Default command** (use this unless user specifically requests otherwise):
+```bash
+sherlock --print-found --no-color "<username>" --timeout 90
+```
+
+**Optional flags** (only add if user explicitly requests):
+- `--nsfw` — Include NSFW sites (only if user asks)
+- `--tor` — Route through Tor (only if user asks for anonymity)
+
+**Do NOT ask about options via clarify** — just run the default search. Users can request specific options if needed.
+
+### 4. Execute Search
+
+Run via the `terminal` tool. The command typically takes 30-120 seconds depending on network conditions and site count.
+
+**Example terminal call:**
+```json
+{
+  "command": "sherlock --print-found --no-color \"target_username\"",
+  "timeout": 180
+}
+```
+
+### 5. Parse and Present Results
+
+Sherlock outputs found accounts in a simple format. Parse the output and present:
+
+1. **Summary line:** "Found X accounts for username 'Y'"
+2. **Categorized links:** Group by platform type if helpful (social, professional, forums, etc.)
+3. **Output file location:** Sherlock saves results to `<username>.txt` by default
+
+**Example output parsing:**
+```
+[+] Instagram: https://instagram.com/username
+[+] Twitter: https://twitter.com/username
+[+] GitHub: https://github.com/username
+```
+
+Present findings as clickable links when possible.
+
+## Pitfalls
+
+### No Results Found
+If Sherlock finds no accounts, this is often correct — the username may not be registered on checked platforms. Suggest:
+- Checking spelling/variation
+- Trying similar usernames with `?` wildcard: `sherlock "user?name"`
+- The user may have privacy settings or deleted accounts
+
+### Timeout Issues
+Some sites are slow or block automated requests. Use `--timeout 120` to increase wait time, or `--site` to limit scope.
+
+### Tor Configuration
+`--tor` requires Tor daemon running. If user wants anonymity but Tor isn't available, suggest:
+- Installing Tor service
+- Using `--proxy` with an alternative proxy
+
+### False Positives
+Some sites always return "found" due to their response structure. Cross-reference unexpected results with manual checks.
+
+### Rate Limiting
+Aggressive searches may trigger rate limits. For bulk username searches, add delays between calls or use `--local` with cached data.
+
+## Installation
+
+### pipx (recommended)
+```bash
+pipx install sherlock-project
+```
+
+### pip
+```bash
+pip install sherlock-project
+```
+
+### Docker
+```bash
+docker pull sherlock/sherlock
+docker run -it --rm sherlock/sherlock <username>
+```
+
+### Linux packages
+Available on Debian 13+, Ubuntu 22.10+, Homebrew, Kali, BlackArch.
+
+## Ethical Use
+
+This tool is for legitimate OSINT and research purposes only. Remind users:
+- Only search usernames they own or have permission to investigate
+- Respect platform terms of service
+- Do not use for harassment, stalking, or illegal activities
+- Consider privacy implications before sharing results
+
+## Verification
+
+After running sherlock, verify:
+1. Output lists found sites with URLs
+2. `<username>.txt` file created (default output) if using file output
+3. If `--print-found` used, output should only contain `[+]` lines for matches
+
+## Example Interaction
+
+**User:** "Can you check if the username 'johndoe123' exists on social media?"
+
+**Agent procedure:**
+1. Check `sherlock --version` (verify installed)
+2. Username provided — proceed directly
+3. Run: `sherlock --print-found --no-color "johndoe123" --timeout 90`
+4. Parse output and present links
+
+**Response format:**
+> Found 12 accounts for username 'johndoe123':
+>
+> • https://twitter.com/johndoe123
+> • https://github.com/johndoe123
+> • https://instagram.com/johndoe123
+> • [... additional links]
+>
+> Results saved to: johndoe123.txt
+
+---
+
+**User:** "Search for username 'alice' including NSFW sites"
+
+**Agent procedure:**
+1. Check sherlock installed
+2. Username + NSFW flag both provided
+3. Run: `sherlock --print-found --no-color --nsfw "alice" --timeout 90`
+4. Present results

package/.agent/skills/simplify-code/SKILL.md

@@ -0,0 +1,168 @@
+---
+name: simplify-code
+description: "Parallel 3-agent cleanup of recent code changes."
+version: 1.0.0
+---
+
+# Simplify Code — Parallel Review & Cleanup
+
+Review your recent code changes with three focused reviewers running in
+parallel, aggregate their findings, and apply the fixes worth applying.
+
+**Core principle:** Three narrow reviewers beat one broad reviewer. Each one
+deeply searches the codebase for a single class of problem — reuse, quality,
+efficiency — without diluting its attention across all three. They run
+concurrently, so you pay the latency of one review, not three.
+
+## When to Use
+
+Trigger this skill when the user says any of:
+
+- "simplify" / "simplify my changes" / "simplify these changes"
+- "review my code" / "review my recent changes" / "clean up my changes"
+- "/simplify" (if they're carrying the Claude Code habit over)
+
+Optional modifiers the user may add — honor them:
+
+- **Focus:** "simplify focus on efficiency" → run only the efficiency reviewer
+  (or weight the aggregation toward it). Recognized focuses: `reuse`,
+  `quality`, `efficiency`.
+- **Dry run:** "simplify but don't change anything" / "just report" → run the
+  three reviewers, present findings, apply NOTHING. Ask before applying.
+- **Scope:** "simplify the last commit" / "simplify staged" / "simplify
+  src/foo.py" → narrow the diff source accordingly (see Phase 1).
+
+Do NOT auto-run this after every edit. It costs three subagents' worth of
+tokens — invoke it only when the user explicitly asks.
+
+## The Process
+
+### Phase 1 — Identify the changes
+
+Capture the diff to review. Pick the source by what the user asked for, in
+this default order:
+
+```bash
+# 1. Default: uncommitted working-tree changes (tracked files)
+git diff
+
+# 2. If that's empty, include staged changes
+git diff HEAD
+
+# 3. Scoped variants the user may request:
+git diff --staged                 # "staged changes"
+git diff HEAD~1                    # "the last commit"
+git diff main...HEAD              # "this branch" / "my PR"
+git diff -- src/foo.py            # specific file(s)
+```
+
+If `git diff` and `git diff HEAD` are both empty and there's no git repo or no
+changes, fall back to the files the user explicitly named or that were
+recently created/edited in this session. If you genuinely can't find any
+changed code, say so and stop — there's nothing to simplify.
+
+Capture the full diff text. Note its size: if it's very large (say >2000
+changed lines), warn the user that three subagents each carrying the full diff
+will be token-heavy, and offer to scope it down (per-directory, per-commit)
+before proceeding.
+
+### Phase 2 — Launch three reviewers in parallel
+
+Use `delegate_task` **batch mode** — pass all three tasks in one `tasks`
+array so they run concurrently. Three is the right fan-out for this pattern;
+it's well within the `delegation.max_concurrent_children` budget on any
+default install.
+
+Give **every** reviewer the **complete diff** (not fragments — cross-file
+issues hide in the gaps) plus the absolute repo path so they can search the
+wider codebase. Each reviewer gets `terminal`, `file`, and `search`
+toolsets (so they can `git`, `read_file`, and `search_files`/grep).
+
+Tell each reviewer to:
+- Search the existing codebase for evidence (don't reason from the diff alone).
+- Report findings as a concrete list: `file:line → problem → suggested fix`.
+- Rank each finding `high` / `medium` / `low` confidence.
+- Skip nits and style-only churn. Only flag things that materially improve
+  the code.
+
+Pass these three goals (drop any the user's focus excludes):
+
+**Reviewer 1 — Code Reuse**
+> Review this diff for code that duplicates functionality already in the
+> codebase. Search utility modules, shared helpers, and adjacent files
+> (use search_files / grep) for existing functions, constants, or patterns
+> the new code could call instead of reimplementing. Flag: new functions
+> that duplicate existing ones; hand-rolled logic that an existing utility
+> already does (manual string/path manipulation, custom env checks, ad-hoc
+> type guards, re-implemented parsing). For each, name the existing thing to
+> use and where it lives.
+
+**Reviewer 2 — Code Quality**
+> Review this diff for quality problems. Look for: redundant state (values
+> that duplicate or could be derived from existing state; caches that don't
+> need to exist); parameter sprawl (new params bolted on where the function
+> should have been restructured); copy-paste-with-variation (near-duplicate
+> blocks that should share an abstraction); leaky abstractions (exposing
+> internals, breaking an existing encapsulation boundary); stringly-typed
+> code (raw strings where a constant/enum/registry already exists — check the
+> canonical registries before flagging). For each, give the concrete refactor.
+
+**Reviewer 3 — Efficiency**
+> Review this diff for efficiency problems. Look for: unnecessary work
+> (redundant computation, repeated file reads, duplicate API calls, N+1
+> access patterns); missed concurrency (independent ops run sequentially);
+> hot-path bloat (heavy/blocking work on startup or per-request paths);
+> TOCTOU anti-patterns (existence pre-checks before an op instead of doing
+> the op and handling the error); memory issues (unbounded growth, missing
+> cleanup, listener/handle leaks); overly broad reads (loading whole files
+> when a slice would do). For each, give the concrete fix and why it's faster
+> or lighter.
+
+### Phase 3 — Aggregate and apply
+
+Wait for all three to return (batch mode returns them together).
+
+1. **Merge** the findings into one list, deduping where reviewers overlap.
+2. **Discard false positives** — you have the most context; you don't have to
+   argue with a reviewer, just drop weak or wrong suggestions silently.
+3. **Resolve conflicts.** Reviewers can disagree (Reviewer 1: "use existing
+   util X"; Reviewer 3: "X is slow, inline it"). Default resolution order:
+   **correctness > the user's stated focus > readability/reuse > micro-perf.**
+   Don't apply a perf "fix" that hurts clarity unless the path is genuinely
+   hot. When two suggestions are mutually exclusive and both defensible, pick
+   the one that touches less code and note the alternative.
+4. **Apply** the surviving fixes directly with `patch` / `write_file` — unless
+   the user asked for a dry run, in which case present the list and ask first.
+5. **Verify** you didn't break anything: run the project's targeted tests for
+   the touched files (not the full suite), and re-run any linter/type check the
+   repo uses. If a fix breaks a test, revert that one fix and report it.
+6. **Summarize** what you changed: a short list of applied fixes grouped by
+   reviewer category, plus any findings you deliberately skipped and why.
+
+## Pitfalls
+
+- **Don't fan out wider than ~3.** More reviewers means more cost and more
+  conflicting suggestions to reconcile, not better coverage. Three categories
+  cover the space.
+- **Give the WHOLE diff to each reviewer.** Splitting the diff across reviewers
+  defeats the design — cross-file duplication and N+1s only show up with the
+  full picture.
+- **Reviewers search, they don't guess.** A reuse finding with no pointer to
+  the existing utility ("there's probably a helper for this") is noise. Require
+  `file:line` evidence; drop findings that lack it.
+- **Apply ≠ rewrite.** This is cleanup of the user's recent changes, not a
+  license to refactor the whole module. Keep edits scoped to what the diff
+  touched plus the minimal surrounding change a fix requires.
+- **Respect project conventions.** If the repo has AGENTS.md / CLAUDE.md /
+  HERMES.md or a linter config, fold those rules into the reviewer prompts so
+  suggestions match house style instead of fighting it.
+- **Large diffs blow context.** If the diff is huge, scope it down before
+  delegating — three subagents each carrying a 5000-line diff is expensive and
+  may truncate.
+
+## Related
+
+If your install has the `subagent-driven-development` skill (optional), it
+covers the complementary case: parallel review *during* implementation, per
+task. This skill is the standalone *after-the-fact* cleanup pass. Use
+`requesting-code-review` for the pre-commit security/quality gate.