npm - better-dsh - Versions diffs - 0.2.5 → 0.3.0 - Mend

better-dsh 0.2.5 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (999) hide show

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/research/research-rss-feeds.md DELETED Viewed

@@ -1,114 +0,0 @@
----
-title: "Rss Feeds — Read RSS, Atom, JSON feeds; discover feeds behind a page"
-sidebar_label: "Rss Feeds"
-description: "Read RSS, Atom, JSON feeds; discover feeds behind a page"
----
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-# Rss Feeds
-Read RSS, Atom, JSON feeds; discover feeds behind a page.
-## Skill metadata
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/research/rss-feeds` |
-| Path | `optional-skills/research/rss-feeds` |
-| Version | `1.0.0` |
-| Author | Teknium (teknium1), Hermes Agent |
-| License | MIT |
-| Platforms | linux, macos, windows |
-| Tags | `RSS`, `Atom`, `Feeds`, `Monitoring`, `Research`, `Blogs`, `Releases` |
-| Related skills | [`reddit-reading`](/docs/user-guide/skills/optional/social-media/social-media-reddit-reading), [`competitor-news-monitor`](/docs/user-guide/skills/bundled/research/research-competitor-news-monitor), [`grounded-citations`](/docs/user-guide/skills/bundled/research/research-grounded-citations), [`youtube-content`](/docs/user-guide/skills/bundled/media/media-youtube-content), [`blogwatcher`](/docs/user-guide/skills/optional/research/research-blogwatcher) |
-## Reference: full SKILL.md
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-# RSS Feeds Skill
-Reads any RSS 2.0, RSS 1.0/RDF, Atom, or JSON Feed URL into a clean, date-sorted list of
-entries, and discovers the feed behind an ordinary page URL (`<link rel="alternate">` or
-the usual `/feed`, `/rss.xml`, `/atom.xml` paths). Standard library only, nothing to
-install. It does not fetch full article bodies — pass an entry's link to `web_extract` for
-that.
-## When to Use
-- "What's new on &lt;blog/site>", "latest releases of &lt;GitHub repo>", "recent posts in
-  &lt;subreddit>", "read this feed", "does this site have an RSS feed".
-- Building a recurring digest with `cronjob_manage` (feeds are cheaper and more stable than
-  scraping the HTML front page every run). For a persistent read/unread database across
-  many feeds install the optional `blogwatcher` skill; this skill is the zero-install read.
-- Anything where a structured list of `title / link / date / author / summary` beats a
-  rendered page: podcasts, changelogs, YouTube channels, newsrooms, forum categories.
-## Prerequisites
-None. Python 3.10+, network access to the feed host.
-## How to Run
-Run through `terminal` with the skill-relative script path:
-```bash
-python3 scripts/feed.py read https://hnrss.org/frontpage --limit 10
-python3 scripts/feed.py read https://simonwillison.net/            # page URL → discovers the feed
-python3 scripts/feed.py read URL --since 2026-09-01 --json          # only newer entries, machine-readable
-python3 scripts/feed.py discover https://example.com/               # list candidate feed URLs
-```
-## Quick Reference
-| Source | Feed URL pattern |
-|---|---|
-| GitHub releases / commits / tags | `https://github.com/OWNER/REPO/releases.atom`, `…/commits/BRANCH.atom`, `…/tags.atom` |
-| Subreddit / Reddit search | `https://www.reddit.com/r/NAME/.rss`, `https://www.reddit.com/search.rss?q=…` (1 req/min anon; see `reddit-reading`) |
-| YouTube channel | `https://www.youtube.com/feeds/videos.xml?channel_id=UC…` |
-| Hacker News | `https://hnrss.org/frontpage`, `https://hnrss.org/newest?q=TERM` |
-| arXiv category | `https://rss.arxiv.org/rss/cs.CL` |
-| Substack / Medium / WordPress / Ghost | `SITE/feed`, `medium.com/feed/@user`, `SITE/rss/` |
-| Podcasts | the show's RSS URL from its hosting page (`discover` finds it) |
-Output fields per entry: `title`, `link`, `published` (UTC ISO 8601), `author`, `summary`
-(HTML stripped, ≤ 2000 chars). Entries are sorted newest-first.
-## Procedure
-① If you only have a site URL, run `read` on it directly; the script discovers the feed
-and reports which URL it used (`discovered_from`). Use `discover` when you want to choose
-between several advertised feeds (comments feed vs posts feed, per-category feeds).
-② Bound the request: `--limit` for "latest N", `--since YYYY-MM-DD` for "since last
-check". For a cron digest persist the last-seen `published` value and pass it as
-`--since` next run.
-③ For full text, hand the entry `link` to `web_extract`; feed summaries are frequently
-truncated or the first paragraph only.
-④ Cite the entry `link`, not the feed URL, when the result feeds a report
-(`grounded-citations`).
-## Pitfalls
-- A 200 response with HTML means the URL is a page, not a feed; the script falls through
-  to discovery automatically, but a site with no `<link rel="alternate">` and none of the
-  common paths reports `no feed found` — check the site's footer or `/sitemap.xml` before
-  concluding there is none.
-- Reddit feeds share Reddit's anonymous throttle (about one request per minute per IP).
-  Chain them through `reddit-reading`, which waits out the window, when you need more
-  than one Reddit call.
-- Dates: RSS `pubDate` is RFC 822 and Atom uses ISO 8601; the script normalises both to
-  UTC. Feeds that omit dates sort to the bottom and are dropped by `--since`.
-- Some feeds are Cloudflare-fronted and 403 non-browser clients; `blocked-page-recovery`
-  handles that class.
-## Verification
-`python3 scripts/feed.py read https://github.com/NousResearch/hermes-agent/releases.atom
---limit 1` prints one entry with a `releases/tag/` link and a `[atom]` format tag;
-`discover https://simonwillison.net/` prints an `/atom/` URL.

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/research/research-scrapling.md DELETED Viewed

@@ -1,351 +0,0 @@
----
-title: "Scrapling — Scrape sites with stealth browsing and Cloudflare bypass"
-sidebar_label: "Scrapling"
-description: "Scrape sites with stealth browsing and Cloudflare bypass"
----
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-# Scrapling
-Scrape sites with stealth browsing and Cloudflare bypass.
-## Skill metadata
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/research/scrapling` |
-| Path | `optional-skills/research\scrapling` |
-| Version | `1.0.0` |
-| Author | FEUAZUR |
-| License | MIT |
-| Platforms | linux, macos, windows |
-| Tags | `Web Scraping`, `Browser`, `Cloudflare`, `Stealth`, `Crawling`, `Spider` |
-| Related skills | [`duckduckgo-search`](/docs/user-guide/skills/optional/research/research-duckduckgo-search), [`domain-intel`](/docs/user-guide/skills/optional/research/research-domain-intel) |
-## Reference: full SKILL.md
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-# Scrapling
-[Scrapling](https://github.com/D4Vinci/Scrapling) is a web scraping framework with anti-bot bypass, stealth browser automation, and a spider framework. It provides three fetching strategies (HTTP, dynamic JS, stealth/Cloudflare) and a full CLI.
-**This skill is for educational and research purposes only.** Users must comply with local/international data scraping laws and respect website Terms of Service.
-## When to Use
-- Scraping static HTML pages (faster than browser tools)
-- Scraping JS-rendered pages that need a real browser
-- Bypassing Cloudflare Turnstile or bot detection
-- Crawling multiple pages with a spider
-- When the built-in `web_extract` tool does not return the data you need
-## Installation
-```bash
-pip install "scrapling[all]"
-scrapling install
-```
-Minimal install (HTTP only, no browser):
-```bash
-pip install scrapling
-```
-With browser automation only:
-```bash
-pip install "scrapling[fetchers]"
-scrapling install
-```
-## Quick Reference
-| Approach | Class | Use When |
-|----------|-------|----------|
-| HTTP | `Fetcher` / `FetcherSession` | Static pages, APIs, fast bulk requests |
-| Dynamic | `DynamicFetcher` / `DynamicSession` | JS-rendered content, SPAs |
-| Stealth | `StealthyFetcher` / `StealthySession` | Cloudflare, anti-bot protected sites |
-| Spider | `Spider` | Multi-page crawling with link following |
-## CLI Usage
-### Extract Static Page
-```bash
-scrapling extract get 'https://example.com' output.md
-```
-With CSS selector and browser impersonation:
-```bash
-scrapling extract get 'https://example.com' output.md \
-  --css-selector '.content' \
-  --impersonate 'chrome'
-```
-### Extract JS-Rendered Page
-```bash
-scrapling extract fetch 'https://example.com' output.md \
-  --css-selector '.dynamic-content' \
-  --disable-resources \
-  --network-idle
-```
-### Extract Cloudflare-Protected Page
-```bash
-scrapling extract stealthy-fetch 'https://protected-site.com' output.html \
-  --solve-cloudflare \
-  --block-webrtc \
-  --hide-canvas
-```
-### POST Request
-```bash
-scrapling extract post 'https://example.com/api' output.json \
-  --json '{"query": "search term"}'
-```
-### Output Formats
-The output format is determined by the file extension:
-- `.html` -- raw HTML
-- `.md` -- converted to Markdown
-- `.txt` -- plain text
-- `.json` / `.jsonl` -- JSON
-## Python: HTTP Scraping
-### Single Request
-```python
-from scrapling.fetchers import Fetcher
-page = Fetcher.get('https://quotes.toscrape.com/')
-quotes = page.css('.quote .text::text').getall()
-for q in quotes:
-    print(q)
-```
-### Session (Persistent Cookies)
-```python
-from scrapling.fetchers import FetcherSession
-with FetcherSession(impersonate='chrome') as session:
-    page = session.get('https://example.com/', stealthy_headers=True)
-    links = page.css('a::attr(href)').getall()
-    for link in links[:5]:
-        sub = session.get(link)
-        print(sub.css('h1::text').get())
-```
-### POST / PUT / DELETE
-```python
-page = Fetcher.post('https://api.example.com/data', json={"key": "value"})
-page = Fetcher.put('https://api.example.com/item/1', data={"name": "updated"})
-page = Fetcher.delete('https://api.example.com/item/1')
-```
-### With Proxy
-```python
-page = Fetcher.get('https://example.com', proxy='http://user:pass@proxy:8080')
-```
-## Python: Dynamic Pages (JS-Rendered)
-For pages that require JavaScript execution (SPAs, lazy-loaded content):
-```python
-from scrapling.fetchers import DynamicFetcher
-page = DynamicFetcher.fetch('https://example.com', headless=True)
-data = page.css('.js-loaded-content::text').getall()
-```
-### Wait for Specific Element
-```python
-page = DynamicFetcher.fetch(
-    'https://example.com',
-    wait_selector=('.results', 'visible'),
-    network_idle=True,
-)
-```
-### Disable Resources for Speed
-Blocks fonts, images, media, stylesheets (~25% faster):
-```python
-from scrapling.fetchers import DynamicSession
-with DynamicSession(headless=True, disable_resources=True, network_idle=True) as session:
-    page = session.fetch('https://example.com')
-    items = page.css('.item::text').getall()
-```
-### Custom Page Automation
-```python
-from playwright.sync_api import Page
-from scrapling.fetchers import DynamicFetcher
-def scroll_and_click(page: Page):
-    page.mouse.wheel(0, 3000)
-    page.wait_for_timeout(1000)
-    page.click('button.load-more')
-    page.wait_for_selector('.extra-results')
-page = DynamicFetcher.fetch('https://example.com', page_action=scroll_and_click)
-results = page.css('.extra-results .item::text').getall()
-```
-## Python: Stealth Mode (Anti-Bot Bypass)
-For Cloudflare-protected or heavily fingerprinted sites:
-```python
-from scrapling.fetchers import StealthyFetcher
-page = StealthyFetcher.fetch(
-    'https://protected-site.com',
-    headless=True,
-    solve_cloudflare=True,
-    block_webrtc=True,
-    hide_canvas=True,
-)
-content = page.css('.protected-content::text').getall()
-```
-### Stealth Session
-```python
-from scrapling.fetchers import StealthySession
-with StealthySession(headless=True, solve_cloudflare=True) as session:
-    page1 = session.fetch('https://protected-site.com/page1')
-    page2 = session.fetch('https://protected-site.com/page2')
-```
-## Element Selection
-All fetchers return a `Selector` object with these methods:
-### CSS Selectors
-```python
-page.css('h1::text').get()              # First h1 text
-page.css('a::attr(href)').getall()      # All link hrefs
-page.css('.quote .text::text').getall() # Nested selection
-```
-### XPath
-```python
-page.xpath('//div[@class="content"]/text()').getall()
-page.xpath('//a/@href').getall()
-```
-### Find Methods
-```python
-page.find_all('div', class_='quote')       # By tag + attribute
-page.find_by_text('Read more', tag='a')    # By text content
-page.find_by_regex(r'\$\d+\.\d{2}')       # By regex pattern
-```
-### Similar Elements
-Find elements with similar structure (useful for product listings, etc.):
-```python
-first_product = page.css('.product')[0]
-all_similar = first_product.find_similar()
-```
-### Navigation
-```python
-el = page.css('.target')[0]
-el.parent                # Parent element
-el.children              # Child elements
-el.next_sibling          # Next sibling
-el.prev_sibling          # Previous sibling
-```
-## Python: Spider Framework
-For multi-page crawling with link following:
-```python
-from scrapling.spiders import Spider, Request, Response
-class QuotesSpider(Spider):
-    name = "quotes"
-    start_urls = ["https://quotes.toscrape.com/"]
-    concurrent_requests = 10
-    download_delay = 1
-    async def parse(self, response: Response):
-        for quote in response.css('.quote'):
-            yield {
-                "text": quote.css('.text::text').get(),
-                "author": quote.css('.author::text').get(),
-                "tags": quote.css('.tag::text').getall(),
-            }
-        next_page = response.css('.next a::attr(href)').get()
-        if next_page:
-            yield response.follow(next_page)
-result = QuotesSpider().start()
-print(f"Scraped {len(result.items)} quotes")
-result.items.to_json("quotes.json")
-```
-### Multi-Session Spider
-Route requests to different fetcher types:
-```python
-from scrapling.fetchers import FetcherSession, AsyncStealthySession
-class SmartSpider(Spider):
-    name = "smart"
-    start_urls = ["https://example.com/"]
-    def configure_sessions(self, manager):
-        manager.add("fast", FetcherSession(impersonate="chrome"))
-        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
-    async def parse(self, response: Response):
-        for link in response.css('a::attr(href)').getall():
-            if "protected" in link:
-                yield Request(link, sid="stealth")
-            else:
-                yield Request(link, sid="fast", callback=self.parse)
-```
-### Pause/Resume Crawling
-```python
-spider = QuotesSpider(crawldir="./crawl_checkpoint")
-spider.start()  # Ctrl+C to pause, re-run to resume from checkpoint
-```
-## Pitfalls
-- **Browser install required**: run `scrapling install` after pip install -- without it, `DynamicFetcher` and `StealthyFetcher` will fail
-- **Timeouts**: DynamicFetcher/StealthyFetcher timeout is in **milliseconds** (default 30000), Fetcher timeout is in **seconds**
-- **Cloudflare bypass**: `solve_cloudflare=True` adds 5-15 seconds to fetch time -- only enable when needed
-- **Resource usage**: StealthyFetcher runs a real browser -- limit concurrent usage
-- **Legal**: always check robots.txt and website ToS before scraping. This library is for educational and research purposes
-- **Python version**: requires Python 3.10+

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/research/research-searxng-search.md DELETED Viewed

@@ -1,212 +0,0 @@
----
-title: "Searxng Search — Free keyless meta-search aggregating 70+ engines"
-sidebar_label: "Searxng Search"
-description: "Free keyless meta-search aggregating 70+ engines"
----
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-# Searxng Search
-Free keyless meta-search aggregating 70+ engines.
-## Skill metadata
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/research/searxng-search` |
-| Path | `optional-skills/research\searxng-search` |
-| Version | `1.0.1` |
-| Author | hermes-agent |
-| License | MIT |
-| Platforms | linux, macos |
-| Tags | `search`, `searxng`, `meta-search`, `self-hosted`, `free`, `fallback` |
-| Related skills | [`duckduckgo-search`](/docs/user-guide/skills/optional/research/research-duckduckgo-search), [`domain-intel`](/docs/user-guide/skills/optional/research/research-domain-intel) |
-## Reference: full SKILL.md
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-# SearXNG Search
-Free meta-search using [SearXNG](https://searxng.org/) — a privacy-respecting, self-hosted search aggregator that queries 70+ search engines simultaneously.
-**No API key required** when using a public instance. Can also be self-hosted for full control. Automatically appears as a fallback when the main web search toolset (`FIRECRAWL_API_KEY`) is not configured.
-## Configuration
-SearXNG requires a `SEARXNG_URL` environment variable pointing to your SearXNG instance:
-```bash
-# Public instances (no setup required)
-SEARXNG_URL=https://searxng.example.com
-# Self-hosted SearXNG
-SEARXNG_URL=http://localhost:8888
-```
-If no instance is configured, this skill is unavailable and the agent falls back to other search options.
-## Detection Flow
-Check what is actually available before choosing an approach:
-```bash
-# Check if SEARXNG_URL is set and the instance is reachable
-curl -s --max-time 5 "${SEARXNG_URL}/search?q=test&format=json" | head -c 200
-```
-Decision tree:
-1. If `SEARXNG_URL` is set and the instance responds, use SearXNG
-2. If `SEARXNG_URL` is unset or unreachable, fall back to other available search tools
-3. If the user wants SearXNG specifically, help them set up an instance or find a public one
-## Method 1: CLI via curl (Preferred)
-Use `curl` via `terminal` to call the SearXNG JSON API. This avoids assuming any particular Python package is installed.
-```bash
-# Text search (JSON output)
-curl -s --max-time 10 \
-  "${SEARXNG_URL}/search?q=python+async+programming&format=json&engines=google,bing&limit=10"
-# With Safesearch off
-curl -s --max-time 10 \
-  "${SEARXNG_URL}/search?q=example&format=json&safesearch=0"
-# Specific categories (general, news, science, etc.)
-curl -s --max-time 10 \
-  "${SEARXNG_URL}/search?q=AI+news&format=json&categories=news"
-```
-### Common CLI Flags
-| Flag | Description | Example |
-|------|-------------|---------|
-| `q` | Query string (URL-encoded) | `q=python+async` |
-| `format` | Output format: `json`, `csv`, `rss` | `format=json` |
-| `engines` | Comma-separated engine names | `engines=google,bing,ddg` |
-| `limit` | Max results per engine (default 10) | `limit=5` |
-| `categories` | Filter by category | `categories=news,science` |
-| `safesearch` | 0=none, 1=moderate, 2=strict | `safesearch=0` |
-| `time_range` | Filter: `day`, `week`, `month`, `year` | `time_range=week` |
-### Parsing JSON Results
-```bash
-# Extract titles and URLs from JSON
-curl -s --max-time 10 "${SEARXNG_URL}/search?q=fastapi&format=json&limit=5" \
-  | python3 -c "
-import json, sys
-data = json.load(sys.stdin)
-for r in data.get('results', []):
-    print(r.get('title',''))
-    print(r.get('url',''))
-    print(r.get('content','')[:200])
-    print()
-"
-```
-Returns per result: `title`, `url`, `content` (snippet), `engine`, `parsed_url`, `img_src`, `thumbnail`, `author`, `published_date`
-## Method 2: Python API via `requests`
-Use the SearXNG REST API directly from Python with the `requests` library:
-```python
-import os, requests, urllib.parse
-base_url = os.environ.get("SEARXNG_URL", "")
-if not base_url:
-    raise RuntimeError("SEARXNG_URL is not set")
-query = "fastapi deployment guide"
-params = {
-    "q": query,
-    "format": "json",
-    "limit": 5,
-    "engines": "google,bing",
-}
-resp = requests.get(f"{base_url}/search", params=params, timeout=10)
-resp.raise_for_status()
-data = resp.json()
-for r in data.get("results", []):
-    print(r["title"])
-    print(r["url"])
-    print(r.get("content", "")[:200])
-    print()
-```
-## Self-Hosting SearXNG
-To run your own SearXNG instance:
-```bash
-# Using Docker
-docker run -d -p 8888:8080 \
-  -v $(pwd)/searxng:/etc/searxng \
-  searxng/searxng:latest
-# Then set
-SEARXNG_URL=http://localhost:8888
-```
-Or install via pip:
-```bash
-pip install searxng
-# Edit /etc/searxng/settings.yml
-searxng-run
-```
-Public SearXNG instances are available at:
-- `https://searxng.example.com` (replace with any public instance)
-## Workflow: Search then Extract
-SearXNG returns titles, URLs, and snippets — not full page content. To get full page content, search first and then extract the most relevant URL with `web_extract`, browser tools, or `curl`.
-```bash
-# Search for relevant pages
-curl -s "${SEARXNG_URL}/search?q=fastapi+deployment&format=json&limit=3"
-# Output: list of results with titles and URLs
-# Then extract the best URL with web_extract
-```
-## Limitations
-- **Instance availability**: If the SearXNG instance is down or unreachable, search fails. Always check `SEARXNG_URL` is set and the instance is reachable.
-- **No content extraction**: SearXNG returns snippets, not full page content. Use `web_extract`, browser tools, or `curl` for full articles.
-- **Rate limiting**: Some public instances limit requests. Self-hosting avoids this.
-- **Engine coverage**: Available engines depend on the SearXNG instance configuration. Some engines may be disabled.
-- **Results freshness**: Meta-search aggregates external engines — result freshness depends on those engines.
-## Troubleshooting
-| Problem | Likely Cause | What To Do |
-|---------|--------------|------------|
-| `SEARXNG_URL` not set | No instance configured | Use a public SearXNG instance or set up your own |
-| Connection refused | Instance not running or wrong URL | Check the URL is correct and the instance is running |
-| Empty results | Instance blocks the query | Try a different instance or self-host |
-| Slow responses | Public instance under load | Self-host or use a less-loaded public instance |
-| `json` format not supported | Old SearXNG version | Try `format=rss` or upgrade SearXNG |
-## Pitfalls
-- **Always set `SEARXNG_URL`**: Without it, the skill cannot function.
-- **URL-encode queries**: Spaces and special characters must be URL-encoded in curl, or use `urllib.parse.quote()` in Python.
-- **Use `format=json`**: The default format may not be machine-readable. Always request JSON explicitly.
-- **Set a timeout**: Always use `--max-time` or `timeout=` to avoid hanging on unreachable instances.
-- **Self-hosting is best**: Public instances may go down, rate-limit, or block. A self-hosted instance is reliable.
-## Instance Discovery
-If `SEARXNG_URL` is not set and the user asks about SearXNG, help them either:
-1. Find a public SearXNG instance (search for "public searxng instance")
-2. Set up their own with Docker or pip
-Public instances are listed at: https://searxng.org/