mindforge-cc 11.8.2 → 11.9.0

package/.agent/skills/osint-investigation/references/sources/opencorporates.md

@@ -0,0 +1,103 @@
+# OpenCorporates — Global Corporate Registry
+
+## 1. Summary
+
+OpenCorporates aggregates corporate registry data from 130+ jurisdictions
+worldwide (~200M companies). Covers US state-level filings (NY DOS, Delaware
+DOC, California SOS, etc.), UK Companies House, EU registries, and most
+common-law jurisdictions.
+
+## 2. Access Methods
+
+- **REST API:** `https://api.opencorporates.com/v0.4/`
+- **HTML fallback:** `https://opencorporates.com/companies?q=...`
+- **Auth:** API token required (free tier 500 calls/month, paid plans available)
+- **Rate limit:** Token-bound; un-tokened requests return 401
+
+Set `OPENCORPORATES_API_TOKEN` env var. Get a free token at
+https://opencorporates.com/api_accounts/new.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_opencorporates.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `name` | str | Company legal name |
+| `company_number` | str | Registry-assigned number |
+| `jurisdiction_code` | str | e.g. `us_ny`, `us_de`, `gb` |
+| `jurisdiction_name` | str | Human-readable jurisdiction |
+| `incorporation_date` | str | YYYY-MM-DD |
+| `dissolution_date` | str | YYYY-MM-DD (empty if active) |
+| `company_type` | str | Domestic LLC / Foreign Corp / etc. |
+| `status` | str | Active / Inactive / Dissolved |
+| `registered_address` | str | Registered office address |
+| `opencorporates_url` | str | Link to OpenCorporates entity page |
+| `officers_count` | str | Total officers on record |
+| `source` | str | `api`, `html`, or `html-fallback` |
+
+## 4. Coverage
+
+- US: all 50 states + DC at state level (LLCs, corps, LPs)
+- International: UK, EU, Canada, Australia, NZ, many APAC + LATAM jurisdictions
+- ~200M company records cumulative
+- Update frequency varies by jurisdiction (UK CH is near-realtime; some
+  state registries lag months)
+
+## 5. Cross-Reference Potential
+
+- **NYC ACRIS** ↔ `name` (LLC/corp owners of NYC property)
+- **USAspending** ↔ `name` (corporate federal contractors)
+- **SEC EDGAR** ↔ `name` (public companies + their subsidiaries)
+- **ICIJ Offshore** ↔ `name` (international corporate structures)
+
+Join key: normalized company name. Some entries have `previous_names` arrays
+which are not currently exported by the fetch script — query OC directly
+for that.
+
+## 6. Data Quality
+
+- Company-name spellings vary across re-incorporations and renames
+- Officer records are spottier than company records (many jurisdictions
+  don't require officer disclosure)
+- Beneficial-ownership data is generally NOT here — most jurisdictions
+  don't require it. UK Companies House has PSC (people with significant
+  control) but that's not universal.
+- Cross-jurisdictional links (parent / subsidiary) are based on registry
+  filings only; corporate trees are often incomplete
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_opencorporates.py`
+
+```bash
+# Search globally by name
+python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
+    --out data/oc.csv
+
+# Restrict to a jurisdiction
+python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
+    --jurisdiction us_ny --out data/oc_ny.csv
+
+# Set token via env or flag
+OPENCORPORATES_API_TOKEN=xxx python3 SKILL_DIR/scripts/fetch_opencorporates.py \
+    --query "Microsoft" --out data/oc.csv
+```
+
+Without a token the script falls back to scraping the HTML search page.
+The fallback is brittle and only fills in `name`, `jurisdiction_code`,
+`opencorporates_url` — set the token for serious work.
+
+## 8. Legal & Licensing
+
+- OpenCorporates aggregates public records — the underlying facts are
+  public domain
+- OpenCorporates own database is licensed CC-BY-SA-4.0; attribution required
+- API ToS prohibits redistributing the full dataset; per-record reference
+  is fine
+
+## 9. References
+
+- API docs: https://api.opencorporates.com/documentation/API-Reference
+- Jurisdiction codes: https://api.opencorporates.com/v0.4/jurisdictions.json
+- Schema: https://opencorporates.com/info/our_data

package/.agent/skills/osint-investigation/references/sources/sec-edgar.md

@@ -0,0 +1,83 @@
+# SEC EDGAR — Corporate Filings
+
+## 1. Summary
+
+EDGAR (Electronic Data Gathering, Analysis, and Retrieval) is the SEC's system
+for corporate disclosure filings: 10-K (annual), 10-Q (quarterly), 8-K (current
+events), DEF 14A (proxy), Form 4 (insider trading), 13F (institutional holdings).
+
+## 2. Access Methods
+
+- **API:** `https://data.sec.gov/submissions/CIK<10-digit-padded>.json` (no auth)
+- **Filing index:** `https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=...`
+- **Full-text search:** `https://efts.sec.gov/LATEST/search-index?q=...`
+- **Auth:** None — requires `User-Agent` header with contact info per SEC policy
+- **Rate limit:** 10 requests/second per IP (enforced)
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_sec_edgar.py` (filings index):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `cik` | str | Central Index Key (10-digit padded) |
+| `company_name` | str | Registrant name |
+| `form_type` | str | 10-K, 10-Q, 8-K, etc. |
+| `filing_date` | str | YYYY-MM-DD |
+| `accession_number` | str | Filing accession (e.g. 0000320193-24-000123) |
+| `primary_document` | str | Filename of main document |
+| `filing_url` | str | Direct URL to filing index |
+| `reporting_period` | str | Period of report (where applicable) |
+
+## 4. Coverage
+
+- All public US registrants from 1993 → present
+- 1993-2000 has spotty coverage of older filings (paper-to-electronic migration)
+- ~12M filings cumulative
+- Updated within minutes of filing acceptance
+
+## 5. Cross-Reference Potential
+
+- **USAspending** ↔ `company_name` (public companies as federal contractors)
+- **Senate LD** ↔ `company_name` (public companies hire lobbyists)
+- **OFAC SDN** ↔ `company_name` (sanctions screening of public registrants)
+
+Join key: company name OR CIK if you have it. CIK is canonical and stable.
+
+## 6. Data Quality
+
+- Subsidiaries often filed under parent CIK — be careful with name matches
+- Name changes over time (rebrands, acquisitions) — CIK remains constant
+- 10-K Item 1A Risk Factors are free-form text — useful for `web_extract`-style
+  parsing, not structured queries
+- Foreign private issuers file 20-F instead of 10-K
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_sec_edgar.py`
+
+```bash
+# By CIK
+python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
+    --types 10-K,10-Q --out data/edgar_filings.csv
+
+# By company name (resolves to CIK first via name search)
+python3 SKILL_DIR/scripts/fetch_sec_edgar.py --company "APPLE INC" \
+    --types 8-K --since 2024-01-01 --out data/edgar_filings.csv
+```
+
+Set `SEC_USER_AGENT` env var with your contact email (SEC requirement).
+Example: `SEC_USER_AGENT="Research example@example.com"`.
+
+## 8. Legal & Licensing
+
+- Public record under SEC Rule 24b-2 / 17 CFR § 230.401
+- No commercial use restrictions on filing content
+- SEC asks all bulk users to include a `User-Agent` with contact info and to
+  respect 10 req/s — failure to do so can result in IP blocking
+
+## 9. References
+
+- Developer docs: https://www.sec.gov/edgar/sec-api-documentation
+- EDGAR full-text search: https://efts.sec.gov/LATEST/search-index
+- Fair access policy: https://www.sec.gov/os/accessing-edgar-data

package/.agent/skills/osint-investigation/references/sources/senate-ld.md

@@ -0,0 +1,89 @@
+# Senate LD — Lobbying Disclosure (LD-1 / LD-2)
+
+## 1. Summary
+
+The Senate Office of Public Records publishes lobbying disclosures under the
+Lobbying Disclosure Act of 1995 (LDA, as amended by HLOGA 2007). LD-1 is
+registration of a new client-lobbyist relationship; LD-2 is the quarterly
+activity report.
+
+## 2. Access Methods
+
+- **API:** `https://lda.senate.gov/api/v1/` (no auth required for read-only)
+- **Bulk download:** `https://lda.senate.gov/api/v1/filings/?format=csv` (paginated)
+- **Auth:** Token required for >120 req/hour — register at https://lda.senate.gov/api/auth/register/
+- **Rate limit:** 120 req/hour unauthenticated, 1,200 req/hour authenticated
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_senate_ld.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `filing_uuid` | str | Unique filing ID |
+| `filing_type` | str | LD-1, LD-2, LD-203, etc. |
+| `filing_year` | int | Year |
+| `filing_period` | str | Q1/Q2/Q3/Q4 or annual |
+| `registrant_name` | str | Lobbying firm or organization |
+| `registrant_id` | str | Senate-assigned registrant ID |
+| `client_name` | str | Client being represented |
+| `client_id` | str | Senate-assigned client ID |
+| `client_general_description` | str | Client industry / business |
+| `income` | float | LD-2 income from client this quarter (USD) |
+| `expenses` | float | LD-2 expenses (in-house lobbying) |
+| `lobbyists` | str | Semicolon-separated lobbyist names |
+| `issues` | str | Semicolon-separated issue areas |
+| `government_entities` | str | Agencies/chambers contacted |
+| `filing_date` | str | YYYY-MM-DD |
+
+## 4. Coverage
+
+- US federal lobbying only (state lobbying handled by individual state ethics offices)
+- 1999 → present (full electronic coverage from 2008)
+- Quarterly reporting cycle (LD-2)
+- ~1M+ filings cumulative
+
+## 5. Cross-Reference Potential
+
+- **USAspending** ↔ `client_name` (clients lobbying for contracts)
+- **SEC EDGAR** ↔ `client_name` (public companies as lobbying clients)
+- **OFAC SDN** ↔ `client_name` (sanctions screening of lobbying clients)
+
+Join key: normalized client_name. registrant_id and client_id are canonical
+when joining Senate-internal records.
+
+## 6. Data Quality
+
+- Many lobbyist names appear in multiple registrants over time (job changes)
+- `issues` and `government_entities` are free-text — Inconsistent capitalization
+- Foreign agents register under FARA (Department of Justice), NOT here
+- Income/expenses are reported in $10,000 brackets in some older filings
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_senate_ld.py`
+
+```bash
+# By client
+python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
+    --year 2024 --out data/lobbying.csv
+
+# By registrant (lobbying firm)
+python3 SKILL_DIR/scripts/fetch_senate_ld.py --registrant "BIG K STREET LLP" \
+    --year 2024 --out data/lobbying.csv
+```
+
+Set `SENATE_LDA_TOKEN` env var if you have one (or pass `--token`).
+Defaults to anonymous (120 req/hour).
+
+## 8. Legal & Licensing
+
+- Public record under 2 U.S.C. § 1604 (LDA)
+- No commercial use restrictions
+- Reuse is unconditional — see Senate Public Records Office disclaimer
+
+## 9. References
+
+- API docs: https://lda.senate.gov/api/redoc/v1/
+- LDA guidance: https://lobbyingdisclosure.house.gov/ld_guidance.pdf
+- Senate Public Records: https://lda.senate.gov/

package/.agent/skills/osint-investigation/references/sources/usaspending.md

@@ -0,0 +1,97 @@
+# USAspending — Federal Government Contracts and Grants
+
+## 1. Summary
+
+USAspending.gov is the official source of federal spending data. Coverage:
+contracts, grants, loans, direct payments, sub-awards. Required by the DATA Act
+of 2014 — all federal agencies must report to a single schema.
+
+## 2. Access Methods
+
+- **API v2:** `https://api.usaspending.gov/api/v2/` (no auth, no key)
+- **Bulk:** `https://files.usaspending.gov/` (CSV / Parquet by award type)
+- **Auth:** None
+- **Rate limit:** Not strictly enforced, but be polite — keep to <10 req/s
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_usaspending.py` (prime awards):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `award_id` | str | Federal award ID (PIID for contracts, FAIN for grants) |
+| `recipient_name` | str | Awardee legal name |
+| `recipient_uei` | str | Unique Entity Identifier (replaced DUNS in 2022) |
+| `recipient_duns` | str | Legacy DUNS number (historical only) |
+| `recipient_parent_name` | str | Ultimate parent organization |
+| `recipient_state` | str | Recipient state |
+| `awarding_agency` | str | Department / agency name |
+| `awarding_sub_agency` | str | Sub-tier (e.g. DoD → Army) |
+| `award_type` | str | Contract / Grant / Loan / Direct Payment |
+| `award_amount` | float | Current total obligation in USD |
+| `award_date` | str | Action / signed date YYYY-MM-DD |
+| `period_of_performance_start` | str | YYYY-MM-DD |
+| `period_of_performance_end` | str | YYYY-MM-DD |
+| `naics_code` | str | Industry classification |
+| `psc_code` | str | Product / Service Code |
+| `competition_extent` | str | Full / limited / sole-source |
+| `description` | str | Award description (free-text) |
+
+## 4. Coverage
+
+- US federal awards only (state/local not included)
+- FY 2008 → present (full coverage from FY 2017)
+- Updated bi-weekly from agency reporting
+- ~100M+ transaction records cumulative
+
+## 5. Cross-Reference Potential
+
+- **SEC EDGAR** ↔ `recipient_name` (public companies as contractors)
+- **Senate LD** ↔ `recipient_name` (lobbying clients winning contracts)
+- **OFAC SDN** ↔ `recipient_name` (sanctions screening of contractors — must be
+  filtered out by SAM.gov but verify)
+- **ICIJ Offshore** ↔ `recipient_name` (offshore-linked contractors)
+
+Join key: normalized recipient name. UEI is canonical when present.
+
+## 6. Data Quality
+
+- DUNS → UEI transition (April 2022) — old records have DUNS, new records have UEI
+- Some sub-awards aren't reported (FFATA threshold is $30k)
+- Award amount changes over time (mod actions) — fetch script reports current total
+- `competition_extent` field is free-text in older records — `fetch_usaspending.py`
+  normalizes to canonical values
+- Recipient name variations are extensive — "ACME LLC", "Acme L.L.C.", "ACME, INC"
+  all appear. Use `entity_resolution.py`.
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_usaspending.py`
+
+```bash
+# By recipient name
+python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
+    --fy 2024 --out data/contracts.csv
+
+# By awarding agency
+python3 SKILL_DIR/scripts/fetch_usaspending.py --agency "Department of Defense" \
+    --fy 2024 --out data/contracts.csv
+
+# Filter to sole-source only
+python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
+    --fy 2024 --sole-source-only --out data/contracts.csv
+```
+
+## 8. Legal & Licensing
+
+- Public record under the Federal Funding Accountability and Transparency Act
+  (FFATA, 2006) and DATA Act (2014)
+- No commercial use restrictions on the data
+- Personal information of award recipients (e.g. small business owners' addresses
+  in some grants) should be handled per the source agency's privacy notice
+
+## 9. References
+
+- API docs: https://api.usaspending.gov/
+- Data dictionary: https://www.usaspending.gov/data-dictionary
+- Award schema: https://files.usaspending.gov/docs/Data_Dictionary_Crosswalk.xlsx

package/.agent/skills/osint-investigation/references/sources/wayback.md

@@ -0,0 +1,93 @@
+# Wayback Machine — Internet Archive CDX
+
+## 1. Summary
+
+The Internet Archive's Wayback Machine has captured ~900B+ web pages since
+1996. The CDX server API indexes those captures by URL, timestamp, and
+content hash. Free, anonymous, no auth.
+
+## 2. Access Methods
+
+- **CDX server:** `https://web.archive.org/cdx/search/cdx`
+- **Wayback URL:** `https://web.archive.org/web/<timestamp>/<url>`
+- **Save Page Now (write):** `https://web.archive.org/save/<url>` (different API)
+- **Auth:** None
+- **Rate limit:** Generous; be polite (~1 req/s)
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_wayback.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `url` | str | Original URL captured |
+| `timestamp` | str | YYYYMMDDHHMMSS (CDX format) |
+| `wayback_url` | str | Direct replay URL |
+| `mimetype` | str | Content-type at capture |
+| `status` | str | HTTP status (typically 200) |
+| `digest` | str | SHA1 of capture content (collapse-friendly) |
+| `length` | str | Byte length of capture |
+
+## 4. Coverage
+
+- 1996 → present
+- ~900B+ captures across ~700M domains
+- Updated continuously by automated crawls + manual saves
+- Some domains have aggressive coverage (news), others sparse (private)
+
+## 5. Cross-Reference Potential
+
+- **Wikipedia** ↔ Reverse-lookup pages cited as references that have since
+  disappeared
+- **News URLs** ↔ Original article content when present-day URLs 404
+- **Corporate websites** ↔ Historical "About" pages, executive bios that
+  have been scrubbed
+
+The Wayback CDX is most useful as a **content-recovery** layer when other
+sources point to URLs that no longer exist.
+
+## 6. Data Quality
+
+- robots.txt-blocked domains may have spotty or no coverage
+- Captures vary in completeness (HTML may be saved without CSS/JS)
+- Some content is excluded by domain owner request (DMCA, etc.)
+- Coverage of "deep links" (URLs with query strings) is uneven
+- Time resolution is per-capture, not continuous — gaps are common
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_wayback.py`
+
+```bash
+# All captures of a specific URL
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "https://example.com/page" \
+    --out data/wb.csv
+
+# All captures of a host
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
+    --match host --out data/wb.csv
+
+# All captures of a domain + subdomains
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
+    --match domain --out data/wb.csv
+
+# Only unique-content captures within a date window
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
+    --match host --collapse digest \
+    --from-date 2020-01-01 --to-date 2023-12-31 \
+    --out data/wb.csv
+```
+
+## 8. Legal & Licensing
+
+- Internet Archive captures are made under fair-use research provisions
+- Replay URLs are stable references — citing them is encouraged
+- Internet Archive non-profit terms of use govern content
+- Some content is rights-restricted; replay may be blocked even if the
+  CDX entry shows it as captured
+
+## 9. References
+
+- CDX server docs: https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md
+- Wayback API: https://archive.org/help/wayback_api.php
+- Internet Archive: https://archive.org/

package/.agent/skills/osint-investigation/references/sources/wikipedia.md

@@ -0,0 +1,107 @@
+# Wikipedia + Wikidata
+
+## 1. Summary
+
+Wikipedia is the canonical narrative-bio source for notable people, places,
+and organizations. Wikidata is its structured-data counterpart: ~110M
+items, each with claims, dates, identifiers, and cross-references to
+external authorities (VIAF, ISNI, ORCID, GRID, etc.).
+
+Together they're a high-precision entity-resolution layer — the bar for
+inclusion is real, but anything past that bar is well-cross-referenced.
+
+## 2. Access Methods
+
+- **Wikipedia OpenSearch:** `https://en.wikipedia.org/w/api.php?action=opensearch`
+- **Wikipedia REST summary:** `https://en.wikipedia.org/api/rest_v1/page/summary/<title>`
+- **Wikidata Action API:** `https://www.wikidata.org/w/api.php?action=wbgetentities`
+- **Wikidata SPARQL:** `https://query.wikidata.org/sparql` (more powerful but aggressively rate-limited)
+- **Auth:** None, but **a meaningful User-Agent is required**
+
+Set `HERMES_OSINT_UA` to something identifying (e.g. `your-app/1.0 (you@example.com)`).
+Wikimedia returns HTTP 429 to generic UAs.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_wikipedia.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `source` | str | `wikipedia` or `wikipedia+wikidata` |
+| `label` | str | Wikipedia article title |
+| `description` | str | Short Wikidata description |
+| `qid` | str | Wikidata QID (e.g. Q2283 for Microsoft) |
+| `wikipedia_title`, `wikipedia_url` | str | Article identifier + URL |
+| `wikidata_url` | str | Wikidata entity URL |
+| `instance_of` | str | What kind of thing it is (P31) |
+| `country` | str | Country (P17 for orgs/places, P27 for people) |
+| `occupation` | str | P106 |
+| `employer` | str | P108 |
+| `date_of_birth` | str | P569, YYYY-MM-DD |
+| `place_of_birth` | str | P19 |
+| `summary` | str | Wikipedia REST extract (~1000 chars) |
+
+The fetch script uses Wikidata's Action API (NOT SPARQL) for structured
+facts — far more lenient on rate limits.
+
+## 4. Coverage
+
+- Wikipedia EN: ~7M articles
+- Wikidata: ~110M items, ~1.5B statements
+- Updated continuously; abuse filters and bots run constantly
+- High notability bar — most private individuals are not in Wikipedia
+
+## 5. Cross-Reference Potential
+
+- **All sources** ↔ `label` (entity identity resolution)
+- **SEC EDGAR** ↔ `label` (public companies)
+- **CourtListener** ↔ `label` (parties to notable litigation)
+- **Wikidata external identifiers** (not currently in this fetcher's output)
+  link to VIAF, ISNI, ORCID, GRID, GitHub, Twitter, IMDb, ...
+
+Join key: Wikidata QID is canonical. Wikipedia titles are stable for
+most articles but can be renamed.
+
+## 6. Data Quality
+
+- Notability filter — only notable entities (criteria vary by topic)
+- Recency lag — current events take days to weeks to be reflected
+- POV / vandalism — moderated, but edits between sweeps can be bad
+- Living-persons biographies have stricter sourcing requirements
+- Wikidata claims have qualifiers and references — the fetch script
+  doesn't currently export them
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_wikipedia.py`
+
+```bash
+# Look up a notable entity
+python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --out data/wp.csv
+
+# A specific person
+python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" --out data/wp_bg.csv
+
+# Skip the Wikidata enrichment for speed
+python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --no-wikidata \
+    --limit 5 --out data/wp.csv
+```
+
+The OpenSearch is fuzzy — `--limit 5` returns the top 5 Wikipedia article
+matches. Each is enriched with the QID + structured facts unless
+`--no-wikidata` is passed.
+
+## 8. Legal & Licensing
+
+- Wikipedia text: CC-BY-SA-3.0 / GFDL
+- Wikidata claims: CC0 (public domain)
+- API ToS: respect rate limits, identify your agent
+- Commercial use allowed with attribution
+
+## 9. References
+
+- Wikipedia OpenSearch: https://www.mediawiki.org/wiki/API:Opensearch
+- Wikipedia REST: https://en.wikipedia.org/api/rest_v1/
+- Wikidata Action API: https://www.wikidata.org/wiki/Wikidata:Data_access
+- Wikidata SPARQL: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service
+- User-Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy

package/.agent/skills/osint-investigation/scripts/_http.py

@@ -0,0 +1,82 @@
+"""Tiny stdlib HTTP helper used by fetch_*.py scripts.
+
+Provides polite retry + JSON convenience + User-Agent enforcement.
+"""
+from __future__ import annotations
+
+import json
+import os
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+DEFAULT_UA = (
+    "hermes-osint-investigation/0.2 "
+    "(+https://github.com/NousResearch/hermes-agent; "
+    "set HERMES_OSINT_UA env var to identify yourself per "
+    "Wikimedia / SEC fair-use guidance)"
+)
+
+
+def get(
+    url: str,
+    *,
+    params: dict | None = None,
+    headers: dict | None = None,
+    user_agent: str | None = None,
+    max_retries: int = 3,
+    backoff: float = 1.5,
+    timeout: float = 30.0,
+) -> bytes:
+    """GET with retry on 5xx and Retry-After honoring.
+
+    429 (rate-limit) is raised IMMEDIATELY with a clear message — retrying
+    when the upstream says "you're over quota" just wastes time. The caller
+    should slow down or supply real credentials.
+    """
+    if params:
+        sep = "&" if "?" in url else "?"
+        url = f"{url}{sep}{urllib.parse.urlencode(params)}"
+    h = {"User-Agent": user_agent or os.environ.get("HERMES_OSINT_UA", DEFAULT_UA)}
+    if headers:
+        h.update(headers)
+
+    last_err: Exception | None = None
+    for attempt in range(max_retries + 1):
+        req = urllib.request.Request(url, headers=h)
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                return resp.read()
+        except urllib.error.HTTPError as e:
+            if e.code == 429:
+                # Surface immediately. Read the body so the caller sees the
+                # provider's actual message ("OVER_RATE_LIMIT" etc.).
+                try:
+                    body = e.read(2048).decode("utf-8", errors="replace")
+                except Exception:  # noqa: BLE001
+                    body = ""
+                raise RuntimeError(
+                    f"HTTP 429 rate-limited by {urllib.parse.urlsplit(url).netloc}. "
+                    f"Slow down or supply a real API key. Body: {body[:300]}"
+                ) from e
+            if e.code in {500, 502, 503, 504} and attempt < max_retries:
+                retry_after = e.headers.get("Retry-After") if e.headers else None
+                wait = float(retry_after) if (retry_after and retry_after.isdigit()) else backoff ** (attempt + 1)
+                time.sleep(wait)
+                last_err = e
+                continue
+            raise
+        except urllib.error.URLError as e:
+            if attempt < max_retries:
+                time.sleep(backoff ** (attempt + 1))
+                last_err = e
+                continue
+            raise
+    if last_err:
+        raise last_err
+    raise RuntimeError("unreachable")
+
+
+def get_json(url: str, **kwargs) -> dict | list:
+    return json.loads(get(url, **kwargs).decode("utf-8"))