better-dsh 0.2.5 → 0.3.0

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/research/research-gitnexus-explorer.md

@@ -1,232 +0,0 @@
----
-title: "Gitnexus Explorer — Serve an interactive codebase knowledge graph web UI"
-sidebar_label: "Gitnexus Explorer"
-description: "Serve an interactive codebase knowledge graph web UI"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Gitnexus Explorer
-
-Serve an interactive codebase knowledge graph web UI.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/research/gitnexus-explorer` |
-| Path | `optional-skills/research\gitnexus-explorer` |
-| Version | `1.0.0` |
-| Author | Hermes Agent + Teknium |
-| License | MIT |
-| Platforms | linux, macos, windows |
-| Tags | `gitnexus`, `code-intelligence`, `knowledge-graph`, `visualization` |
-| Related skills | [`hermes-agent`](/docs/user-guide/skills/bundled/autonomous-ai-agents/autonomous-ai-agents-hermes-agent), [`codebase-inspection`](/docs/user-guide/skills/bundled/software-development/software-development-codebase-inspection) |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# GitNexus Explorer
-
-Index any codebase into a knowledge graph and serve an interactive web UI for exploring
-symbols, call chains, clusters, and execution flows. Tunneled via Cloudflare for remote access.
-
-## When to Use
-
-- User wants to visually explore a codebase's architecture
-- User asks for a knowledge graph / dependency graph of a repo
-- User wants to share an interactive codebase explorer with someone
-
-## Prerequisites
-
-- **Node.js** (v18+) — required for GitNexus and the proxy
-- **git** — repo must have a `.git` directory
-- **cloudflared** — for tunneling (auto-installed to ~/.local/bin if missing)
-
-## Size Warning
-
-The web UI renders all nodes in the browser. Repos under ~5,000 files work well. Large
-repos (30k+ nodes) will be sluggish or crash the browser tab. The CLI/MCP tools work
-at any scale — only the web visualization has this limit.
-
-## Steps
-
-### 1. Clone and Build GitNexus (one-time setup)
-
-```bash
-GITNEXUS_DIR="${GITNEXUS_DIR:-$HOME/.local/share/gitnexus}"
-
-if [ ! -d "$GITNEXUS_DIR/gitnexus-web/dist" ]; then
-  git clone https://github.com/abhigyanpatwari/GitNexus.git "$GITNEXUS_DIR"
-  cd "$GITNEXUS_DIR/gitnexus-shared" && npm install && npm run build
-  cd "$GITNEXUS_DIR/gitnexus-web" && npm install
-fi
-```
-
-### 2. Patch the Web UI for Remote Access
-
-The web UI defaults to `localhost:4747` for API calls. Patch it to use same-origin
-so it works through a tunnel/proxy:
-
-**File: `$GITNEXUS_DIR/gitnexus-web/src/config/ui-constants.ts`**
-Change:
-```typescript
-export const DEFAULT_BACKEND_URL = 'http://localhost:4747';
-```
-To:
-```typescript
-export const DEFAULT_BACKEND_URL = typeof window !== 'undefined' && window.location.hostname !== 'localhost' ? window.location.origin : 'http://localhost:4747';
-```
-
-**File: `$GITNEXUS_DIR/gitnexus-web/vite.config.ts`**
-Add `allowedHosts: true` inside the `server: { }` block (only needed if running dev
-mode instead of production build):
-```typescript
-server: {
-    allowedHosts: true,
-    // ... existing config
-},
-```
-
-Then build the production bundle:
-```bash
-cd "$GITNEXUS_DIR/gitnexus-web" && npx vite build
-```
-
-### 3. Index the Target Repo
-
-```bash
-cd /path/to/target-repo
-npx gitnexus analyze --skip-agents-md
-rm -rf .claude/    # remove Claude Code-specific artifacts
-```
-
-Add `--embeddings` for semantic search (slower — minutes instead of seconds).
-
-The index lives in `.gitnexus/` inside the repo (auto-gitignored).
-
-### 4. Create the Proxy Script
-
-Write this to a file (e.g., `$GITNEXUS_DIR/proxy.mjs`). It serves the production
-web UI and proxies `/api/*` to the GitNexus backend — same origin, no CORS issues,
-no sudo, no nginx.
-
-```javascript
-import http from 'node:http';
-import fs from 'node:fs';
-import path from 'node:path';
-
-const API_PORT = parseInt(process.env.API_PORT || '4747');
-const DIST_DIR = process.argv[2] || './dist';
-const PORT = parseInt(process.argv[3] || '8888');
-
-const MIME = {
-  '.html': 'text/html', '.js': 'application/javascript', '.css': 'text/css',
-  '.json': 'application/json', '.png': 'image/png', '.svg': 'image/svg+xml',
-  '.ico': 'image/x-icon', '.woff2': 'font/woff2', '.woff': 'font/woff',
-  '.wasm': 'application/wasm',
-};
-
-function proxyToApi(req, res) {
-  const opts = {
-    hostname: '127.0.0.1', port: API_PORT,
-    path: req.url, method: req.method, headers: req.headers,
-  };
-  const proxy = http.request(opts, (upstream) => {
-    res.writeHead(upstream.statusCode, upstream.headers);
-    upstream.pipe(res, { end: true });
-  });
-  proxy.on('error', () => { res.writeHead(502); res.end('Backend unavailable'); });
-  req.pipe(proxy, { end: true });
-}
-
-function serveStatic(req, res) {
-  let filePath = path.join(DIST_DIR, req.url === '/' ? 'index.html' : req.url.split('?')[0]);
-  if (!fs.existsSync(filePath)) filePath = path.join(DIST_DIR, 'index.html');
-  const ext = path.extname(filePath);
-  const mime = MIME[ext] || 'application/octet-stream';
-  try {
-    const data = fs.readFileSync(filePath);
-    res.writeHead(200, { 'Content-Type': mime, 'Cache-Control': 'public, max-age=3600' });
-    res.end(data);
-  } catch { res.writeHead(404); res.end('Not found'); }
-}
-
-http.createServer((req, res) => {
-  if (req.url.startsWith('/api')) proxyToApi(req, res);
-  else serveStatic(req, res);
-}).listen(PORT, () => console.log(`GitNexus proxy on http://localhost:${PORT}`));
-```
-
-### 5. Start the Services
-
-```bash
-# Terminal 1: GitNexus backend API
-npx gitnexus serve &
-
-# Terminal 2: Proxy (web UI + API on one port)
-node "$GITNEXUS_DIR/proxy.mjs" "$GITNEXUS_DIR/gitnexus-web/dist" 8888 &
-```
-
-Verify: `curl -s http://localhost:8888/api/repos` should return the indexed repo(s).
-
-### 6. Tunnel with Cloudflare (optional — for remote access)
-
-```bash
-# Install cloudflared if needed (no sudo)
-if ! command -v cloudflared &>/dev/null; then
-  mkdir -p ~/.local/bin
-  curl -sL https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 \
-    -o ~/.local/bin/cloudflared
-  chmod +x ~/.local/bin/cloudflared
-  export PATH="$HOME/.local/bin:$PATH"
-fi
-
-# Start tunnel (--config /dev/null avoids conflicts with existing named tunnels)
-cloudflared tunnel --config /dev/null --url http://localhost:8888 --no-autoupdate --protocol http2
-```
-
-The tunnel URL (e.g., `https://random-words.trycloudflare.com`) is printed to stderr.
-Share it — anyone with the link can explore the graph.
-
-### 7. Cleanup
-
-```bash
-# Stop services
-pkill -f "gitnexus serve"
-pkill -f "proxy.mjs"
-pkill -f cloudflared
-
-# Remove index from the target repo
-cd /path/to/target-repo
-npx gitnexus clean
-rm -rf .claude/
-```
-
-## Pitfalls
-
-- **`--config /dev/null` is required for cloudflared** if the user has an existing
-  named tunnel config at `~/.cloudflared/config.yml`. Without it, the catch-all
-  ingress rule in the config returns 404 for all quick tunnel requests.
-
-- **Production build is mandatory for tunneling.** The Vite dev server blocks
-  non-localhost hosts by default (`allowedHosts`). The production build + Node
-  proxy avoids this entirely.
-
-- **The web UI does NOT create `.claude/` or `CLAUDE.md`.** Those are created by
-  `npx gitnexus analyze`. Use `--skip-agents-md` to suppress the markdown files,
-  then `rm -rf .claude/` for the rest. These are Claude Code integrations that
-  hermes-agent users don't need.
-
-- **Browser memory limit.** The web UI loads the entire graph into browser memory.
-  Repos with 5k+ files may be sluggish. 30k+ files will likely crash the tab.
-
-- **Embeddings are optional.** `--embeddings` enables semantic search but takes
-  minutes on large repos. Skip it for quick exploration; add it if you want
-  natural language queries via the AI chat panel.
-
-- **Multiple repos.** `gitnexus serve` serves ALL indexed repos. Index several
-  repos, start serve once, and the web UI lets you switch between them.

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/research/research-osint-investigation.md

@@ -1,295 +0,0 @@
----
-title: "Osint Investigation — Follow the money via public records and sanctions data"
-sidebar_label: "Osint Investigation"
-description: "Follow the money via public records and sanctions data"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Osint Investigation
-
-Follow the money via public records and sanctions data.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/research/osint-investigation` |
-| Path | `optional-skills/research\osint-investigation` |
-| Version | `0.1.0` |
-| Author | Hermes Agent (adapted from ShinMegamiBoson/OpenPlanter, MIT) |
-| License | MIT |
-| Platforms | linux, macos, windows |
-| Tags | `osint`, `investigation`, `public-records`, `sec`, `sanctions`, `corporate-registry`, `property`, `courts`, `due-diligence`, `journalism` |
-| Related skills | [`domain-intel`](/docs/user-guide/skills/optional/research/research-domain-intel), [`arxiv`](/docs/user-guide/skills/bundled/research/research-arxiv) |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# OSINT Investigation — Public Records Cross-Reference
-
-Investigative framework for public-records OSINT: government contracts,
-corporate filings, lobbying, sanctions, offshore leaks, property records,
-court records, web archives, knowledge bases, and global news. Resolve
-entities across heterogeneous sources, build cross-links with explicit
-confidence, run statistical timing tests, and produce structured evidence
-chains.
-
-**Python stdlib only.** Zero install. Works on Linux, macOS, Windows. Most
-sources work with no API key (OpenCorporates has an optional free token
-that raises rate limits).
-
-Adapted from the MIT-licensed ShinMegamiBoson/OpenPlanter project; expanded
-to cover identity / property / litigation / archives / news sources that
-the original didn't address.
-
-## When to use this skill
-
-Use when the user asks for:
-
-- "follow the money" — government contracts, lobbying → legislation, sanctions
-- corporate due diligence — who controls company X, where are they
-  incorporated, who serves on their boards, what filings have they made
-- sanctions screening — is entity X on OFAC SDN, ICIJ offshore leaks
-- pay-to-play investigation — contractors with offshore ties, lobbying
-  clients winning awards
-- property ownership — find recorded deeds/mortgages by name or address
-  (NYC; for other counties point users at the relevant recorder)
-- litigation history — find federal + state court opinions and PACER dockets
-- multi-source entity resolution where naming varies (LLC suffixes, abbreviations)
-- evidence-chain construction with explicit confidence levels
-- "what's been said about X" — international news (GDELT) + Wikipedia
-  narrative + Wayback Machine to recover dead URLs
-
-Do NOT use this skill for:
-
-- general web research → `web_search` / `web_extract`
-- domain/infrastructure OSINT → `domain-intel` skill
-- academic literature → `arxiv` skill
-- social-media profile discovery → `sherlock` skill (optional)
-- US **federal** campaign finance — FEC is intentionally NOT covered here
-  (the API is unreliable for ad-hoc contributor-name queries on the free
-  DEMO_KEY tier). For federal donations, point users at
-  https://www.fec.gov/data/ directly.
-
-## Workflow
-
-The agent runs scripts via the `terminal` tool. `SKILL_DIR` is the directory
-holding this SKILL.md.
-
-### 1. Identify which sources apply
-
-Read the data-source wiki entries to plan the investigation:
-
-```
-ls SKILL_DIR/references/sources/
-
-# Federal financial / regulatory
-cat SKILL_DIR/references/sources/sec-edgar.md       # corporate filings
-cat SKILL_DIR/references/sources/usaspending.md     # federal contracts
-cat SKILL_DIR/references/sources/senate-ld.md       # lobbying
-cat SKILL_DIR/references/sources/ofac-sdn.md        # sanctions
-cat SKILL_DIR/references/sources/icij-offshore.md   # offshore leaks
-
-# Identity / property / litigation / archives / news
-cat SKILL_DIR/references/sources/nyc-acris.md       # NYC property records
-cat SKILL_DIR/references/sources/opencorporates.md  # global corporate registry
-cat SKILL_DIR/references/sources/courtlistener.md   # court records (federal + state)
-cat SKILL_DIR/references/sources/wayback.md         # Wayback Machine archives
-cat SKILL_DIR/references/sources/wikipedia.md       # Wikipedia + Wikidata
-cat SKILL_DIR/references/sources/gdelt.md           # global news monitoring
-```
-
-Each entry follows a 9-section template: summary, access, schema, coverage,
-cross-reference keys, data quality, acquisition, legal, references.
-
-The **cross-reference potential** section maps join keys between sources — read
-those first to pick the right pair.
-
-### 2. Acquire data
-
-Each source has a stdlib-only fetch script in `SKILL_DIR/scripts/`:
-
-**Federal financial / regulatory**
-
-```bash
-# SEC EDGAR filings (corporate disclosures)
-python SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
-    --types 10-K,10-Q --out data/edgar_filings.csv
-
-# USAspending federal contracts
-python SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
-    --fy 2024 --out data/contracts.csv
-
-# Senate LD-1 / LD-2 lobbying disclosures
-python SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
-    --year 2024 --out data/lobbying.csv
-
-# OFAC SDN sanctions list (full snapshot)
-python SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
-
-# ICIJ Offshore Leaks — downloads ~70 MB bulk CSV on first use,
-# then searches it locally. Cached for 30 days under
-# $HERMES_OSINT_CACHE/icij/ (default: ~/.cache/hermes-osint/icij/).
-python SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
-    --out data/icij.csv
-```
-
-**Identity / property / litigation / archives / news**
-
-```bash
-# NYC property records (deeds, mortgages, liens) — ACRIS via Socrata
-python SKILL_DIR/scripts/fetch_nyc_acris.py --name "SMITH, JOHN" \
-    --out data/acris.csv
-python SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" \
-    --out data/acris_addr.csv
-
-# OpenCorporates — 130+ jurisdiction corporate registry
-# (free token required; set OPENCORPORATES_API_TOKEN or pass --token)
-python SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
-    --jurisdiction us_ny --out data/opencorporates.csv
-
-# CourtListener — federal + state court opinions, PACER dockets
-python SKILL_DIR/scripts/fetch_courtlistener.py --query "Smith v. Example Corp" \
-    --type opinions --out data/courts.csv
-
-# Wayback Machine — historical web captures
-python SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
-    --match host --collapse digest --out data/wayback.csv
-
-# Wikipedia + Wikidata — narrative bio + structured facts
-# Set HERMES_OSINT_UA=your-app/1.0 (your@email) to identify yourself
-python SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" \
-    --out data/wp.csv
-
-# GDELT — global news in 100+ languages, ~2015→present
-python SKILL_DIR/scripts/fetch_gdelt.py --query '"Example Corp"' \
-    --timespan 1y --out data/gdelt.csv
-```
-
-All outputs are normalized CSV with a header row. Re-run scripts idempotently.
-
-When a private individual won't be in a source (e.g. SEC EDGAR for a non-public-
-company person, USAspending for someone who isn't a federal contractor, Senate
-LDA for someone who isn't a lobbying client), the script returns 0 rows with a
-clear warning rather than silently writing an empty CSV. EDGAR specifically
-flags when the company-name resolver matched an individual Form 3/4/5 filer
-rather than a corporate registrant.
-
-Rate-limit notes are in each source's wiki entry. Default fetchers sleep
-politely between paginated requests. **API keys raise rate limits** for
-sources that support them (`SEC_USER_AGENT`, `SENATE_LDA_TOKEN`,
-`OPENCORPORATES_API_TOKEN`, `COURTLISTENER_TOKEN`). All scripts surface
-429 responses immediately with the upstream's quota message so the user
-knows to slow down or supply a key.
-
-### 3. Resolve entities across sources
-
-Normalize names and find matches between two CSV files:
-
-```bash
-# Match lobbying clients (Senate LDA) against contract recipients (USAspending)
-python SKILL_DIR/scripts/entity_resolution.py \
-    --left  data/lobbying.csv   --left-name-col  client_name \
-    --right data/contracts.csv  --right-name-col recipient_name \
-    --out data/cross_links.csv
-```
-
-Three matching tiers with explicit confidence:
-
-| Tier | Method | Confidence |
-|------|--------|------------|
-| `exact` | Normalized strings equal after suffix/punctuation strip | high |
-| `fuzzy` | Sorted-token equality (word-bag match) | medium |
-| `token_overlap` | ≥60% token overlap, ≥2 shared tokens, tokens ≥4 chars | low |
-
-Output `cross_links.csv` columns: `match_type, confidence, left_name,
-right_name, left_normalized, right_normalized, left_row, right_row`.
-
-### 4. Statistical timing correlation (optional)
-
-Test whether two time series cluster suspiciously close together — e.g.
-lobbying filings near contract awards — using a permutation test:
-
-```bash
-python SKILL_DIR/scripts/timing_analysis.py \
-    --donations data/lobbying.csv --donation-date-col filing_date \
-        --donation-amount-col income --donation-donor-col client_name \
-        --donation-recipient-col registrant_name \
-    --contracts data/contracts.csv --contract-date-col award_date \
-        --contract-vendor-col recipient_name \
-    --cross-links data/cross_links.csv \
-    --permutations 1000 \
-    --out data/timing.json
-```
-
-The script's column flags are intentionally generic — the original tool was
-written for donations vs awards, but it works for any (event, payee) time
-series joined through cross-links. Null hypothesis: event timing is
-independent of award dates. One-tailed p-value = fraction of permutations
-with mean nearest-award distance ≤ observed. Minimum 3 events per (payer,
-vendor) pair to run the test.
-
-### 5. Build the findings JSON (evidence chain)
-
-```bash
-python SKILL_DIR/scripts/build_findings.py \
-    --cross-links data/cross_links.csv \
-    --timing data/timing.json \
-    --out data/findings.json
-```
-
-Every finding has `id, title, severity, confidence, summary, evidence[], sources[]`.
-Each evidence item points back to a specific row in a source CSV. The user (or a
-follow-up agent) can verify every claim against its source.
-
-## Confidence and evidence discipline
-
-This is the load-bearing rule of the skill. Tell the user:
-
-- Every claim must trace to a record. No naked assertions.
-- Confidence tier travels with the claim. `match_type=fuzzy` is "probable",
-  not "confirmed."
-- Entity resolution produces candidates, NOT conclusions. A `fuzzy` match
-  between "ACME LLC" and "Acme Holdings Group" is a lead, not a fact.
-- Statistical significance ≠ wrongdoing. p < 0.05 means the timing pattern
-  is unlikely under the null. It does not establish corruption.
-- All data sources here are public records. They may still contain
-  inaccuracies, stale info, or redactions (GDPR, sealed records).
-
-## Adding a new data source
-
-Use the template:
-
-```bash
-cp SKILL_DIR/templates/source-template.md \
-    SKILL_DIR/references/sources/<your-source>.md
-```
-
-Fill in all 9 sections. Write a `fetch_<source>.py` script in `scripts/` that
-uses stdlib only and writes a normalized CSV. Update the source list in the
-"When to use" section above.
-
-## Tools and their limits
-
-- `entity_resolution.py` does NOT use external fuzzy libraries (no rapidfuzz,
-  no jellyfish). Token-bag matching is the upper bound here. If you need
-  Levenshtein, transliteration, or phonetic matching, pip-install separately.
-- `timing_analysis.py` uses Python's `random` for permutations. For
-  reproducibility, pass `--seed N`.
-- `fetch_*.py` scripts use `urllib.request` and respect `Retry-After`. Heavy
-  bulk usage may still violate ToS — read each source's legal section first.
-
-## Legal note
-
-All Phase-1 sources are public records. Bulk acquisition is permitted under
-their respective access terms (FOIA, public records law, ICIJ explicit
-publication, OFAC public data). However:
-
-- Some sources rate-limit aggressively. Respect their headers.
-- Some redact registrant info (GDPR on WHOIS, sealed filings).
-- Cross-referencing public records to identify private individuals can have
-  ethical implications. The skill produces evidence chains, not accusations.