mindforge-cc 11.8.2 → 11.9.0

package/.mindforge/skills/osint-investigation/references/sources/senate-ld.md

@@ -0,0 +1,89 @@
+# Senate LD — Lobbying Disclosure (LD-1 / LD-2)
+
+## 1. Summary
+
+The Senate Office of Public Records publishes lobbying disclosures under the
+Lobbying Disclosure Act of 1995 (LDA, as amended by HLOGA 2007). LD-1 is
+registration of a new client-lobbyist relationship; LD-2 is the quarterly
+activity report.
+
+## 2. Access Methods
+
+- **API:** `https://lda.senate.gov/api/v1/` (no auth required for read-only)
+- **Bulk download:** `https://lda.senate.gov/api/v1/filings/?format=csv` (paginated)
+- **Auth:** Token required for >120 req/hour — register at https://lda.senate.gov/api/auth/register/
+- **Rate limit:** 120 req/hour unauthenticated, 1,200 req/hour authenticated
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_senate_ld.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `filing_uuid` | str | Unique filing ID |
+| `filing_type` | str | LD-1, LD-2, LD-203, etc. |
+| `filing_year` | int | Year |
+| `filing_period` | str | Q1/Q2/Q3/Q4 or annual |
+| `registrant_name` | str | Lobbying firm or organization |
+| `registrant_id` | str | Senate-assigned registrant ID |
+| `client_name` | str | Client being represented |
+| `client_id` | str | Senate-assigned client ID |
+| `client_general_description` | str | Client industry / business |
+| `income` | float | LD-2 income from client this quarter (USD) |
+| `expenses` | float | LD-2 expenses (in-house lobbying) |
+| `lobbyists` | str | Semicolon-separated lobbyist names |
+| `issues` | str | Semicolon-separated issue areas |
+| `government_entities` | str | Agencies/chambers contacted |
+| `filing_date` | str | YYYY-MM-DD |
+
+## 4. Coverage
+
+- US federal lobbying only (state lobbying handled by individual state ethics offices)
+- 1999 → present (full electronic coverage from 2008)
+- Quarterly reporting cycle (LD-2)
+- ~1M+ filings cumulative
+
+## 5. Cross-Reference Potential
+
+- **USAspending** ↔ `client_name` (clients lobbying for contracts)
+- **SEC EDGAR** ↔ `client_name` (public companies as lobbying clients)
+- **OFAC SDN** ↔ `client_name` (sanctions screening of lobbying clients)
+
+Join key: normalized client_name. registrant_id and client_id are canonical
+when joining Senate-internal records.
+
+## 6. Data Quality
+
+- Many lobbyist names appear in multiple registrants over time (job changes)
+- `issues` and `government_entities` are free-text — Inconsistent capitalization
+- Foreign agents register under FARA (Department of Justice), NOT here
+- Income/expenses are reported in $10,000 brackets in some older filings
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_senate_ld.py`
+
+```bash
+# By client
+python3 SKILL_DIR/scripts/fetch_senate_ld.py --client "EXAMPLE CORP" \
+    --year 2024 --out data/lobbying.csv
+
+# By registrant (lobbying firm)
+python3 SKILL_DIR/scripts/fetch_senate_ld.py --registrant "BIG K STREET LLP" \
+    --year 2024 --out data/lobbying.csv
+```
+
+Set `SENATE_LDA_TOKEN` env var if you have one (or pass `--token`).
+Defaults to anonymous (120 req/hour).
+
+## 8. Legal & Licensing
+
+- Public record under 2 U.S.C. § 1604 (LDA)
+- No commercial use restrictions
+- Reuse is unconditional — see Senate Public Records Office disclaimer
+
+## 9. References
+
+- API docs: https://lda.senate.gov/api/redoc/v1/
+- LDA guidance: https://lobbyingdisclosure.house.gov/ld_guidance.pdf
+- Senate Public Records: https://lda.senate.gov/

package/.mindforge/skills/osint-investigation/references/sources/usaspending.md

@@ -0,0 +1,97 @@
+# USAspending — Federal Government Contracts and Grants
+
+## 1. Summary
+
+USAspending.gov is the official source of federal spending data. Coverage:
+contracts, grants, loans, direct payments, sub-awards. Required by the DATA Act
+of 2014 — all federal agencies must report to a single schema.
+
+## 2. Access Methods
+
+- **API v2:** `https://api.usaspending.gov/api/v2/` (no auth, no key)
+- **Bulk:** `https://files.usaspending.gov/` (CSV / Parquet by award type)
+- **Auth:** None
+- **Rate limit:** Not strictly enforced, but be polite — keep to <10 req/s
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_usaspending.py` (prime awards):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `award_id` | str | Federal award ID (PIID for contracts, FAIN for grants) |
+| `recipient_name` | str | Awardee legal name |
+| `recipient_uei` | str | Unique Entity Identifier (replaced DUNS in 2022) |
+| `recipient_duns` | str | Legacy DUNS number (historical only) |
+| `recipient_parent_name` | str | Ultimate parent organization |
+| `recipient_state` | str | Recipient state |
+| `awarding_agency` | str | Department / agency name |
+| `awarding_sub_agency` | str | Sub-tier (e.g. DoD → Army) |
+| `award_type` | str | Contract / Grant / Loan / Direct Payment |
+| `award_amount` | float | Current total obligation in USD |
+| `award_date` | str | Action / signed date YYYY-MM-DD |
+| `period_of_performance_start` | str | YYYY-MM-DD |
+| `period_of_performance_end` | str | YYYY-MM-DD |
+| `naics_code` | str | Industry classification |
+| `psc_code` | str | Product / Service Code |
+| `competition_extent` | str | Full / limited / sole-source |
+| `description` | str | Award description (free-text) |
+
+## 4. Coverage
+
+- US federal awards only (state/local not included)
+- FY 2008 → present (full coverage from FY 2017)
+- Updated bi-weekly from agency reporting
+- ~100M+ transaction records cumulative
+
+## 5. Cross-Reference Potential
+
+- **SEC EDGAR** ↔ `recipient_name` (public companies as contractors)
+- **Senate LD** ↔ `recipient_name` (lobbying clients winning contracts)
+- **OFAC SDN** ↔ `recipient_name` (sanctions screening of contractors — must be
+  filtered out by SAM.gov but verify)
+- **ICIJ Offshore** ↔ `recipient_name` (offshore-linked contractors)
+
+Join key: normalized recipient name. UEI is canonical when present.
+
+## 6. Data Quality
+
+- DUNS → UEI transition (April 2022) — old records have DUNS, new records have UEI
+- Some sub-awards aren't reported (FFATA threshold is $30k)
+- Award amount changes over time (mod actions) — fetch script reports current total
+- `competition_extent` field is free-text in older records — `fetch_usaspending.py`
+  normalizes to canonical values
+- Recipient name variations are extensive — "ACME LLC", "Acme L.L.C.", "ACME, INC"
+  all appear. Use `entity_resolution.py`.
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_usaspending.py`
+
+```bash
+# By recipient name
+python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
+    --fy 2024 --out data/contracts.csv
+
+# By awarding agency
+python3 SKILL_DIR/scripts/fetch_usaspending.py --agency "Department of Defense" \
+    --fy 2024 --out data/contracts.csv
+
+# Filter to sole-source only
+python3 SKILL_DIR/scripts/fetch_usaspending.py --recipient "EXAMPLE CORP" \
+    --fy 2024 --sole-source-only --out data/contracts.csv
+```
+
+## 8. Legal & Licensing
+
+- Public record under the Federal Funding Accountability and Transparency Act
+  (FFATA, 2006) and DATA Act (2014)
+- No commercial use restrictions on the data
+- Personal information of award recipients (e.g. small business owners' addresses
+  in some grants) should be handled per the source agency's privacy notice
+
+## 9. References
+
+- API docs: https://api.usaspending.gov/
+- Data dictionary: https://www.usaspending.gov/data-dictionary
+- Award schema: https://files.usaspending.gov/docs/Data_Dictionary_Crosswalk.xlsx

package/.mindforge/skills/osint-investigation/references/sources/wayback.md

@@ -0,0 +1,93 @@
+# Wayback Machine — Internet Archive CDX
+
+## 1. Summary
+
+The Internet Archive's Wayback Machine has captured ~900B+ web pages since
+1996. The CDX server API indexes those captures by URL, timestamp, and
+content hash. Free, anonymous, no auth.
+
+## 2. Access Methods
+
+- **CDX server:** `https://web.archive.org/cdx/search/cdx`
+- **Wayback URL:** `https://web.archive.org/web/<timestamp>/<url>`
+- **Save Page Now (write):** `https://web.archive.org/save/<url>` (different API)
+- **Auth:** None
+- **Rate limit:** Generous; be polite (~1 req/s)
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_wayback.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `url` | str | Original URL captured |
+| `timestamp` | str | YYYYMMDDHHMMSS (CDX format) |
+| `wayback_url` | str | Direct replay URL |
+| `mimetype` | str | Content-type at capture |
+| `status` | str | HTTP status (typically 200) |
+| `digest` | str | SHA1 of capture content (collapse-friendly) |
+| `length` | str | Byte length of capture |
+
+## 4. Coverage
+
+- 1996 → present
+- ~900B+ captures across ~700M domains
+- Updated continuously by automated crawls + manual saves
+- Some domains have aggressive coverage (news), others sparse (private)
+
+## 5. Cross-Reference Potential
+
+- **Wikipedia** ↔ Reverse-lookup pages cited as references that have since
+  disappeared
+- **News URLs** ↔ Original article content when present-day URLs 404
+- **Corporate websites** ↔ Historical "About" pages, executive bios that
+  have been scrubbed
+
+The Wayback CDX is most useful as a **content-recovery** layer when other
+sources point to URLs that no longer exist.
+
+## 6. Data Quality
+
+- robots.txt-blocked domains may have spotty or no coverage
+- Captures vary in completeness (HTML may be saved without CSS/JS)
+- Some content is excluded by domain owner request (DMCA, etc.)
+- Coverage of "deep links" (URLs with query strings) is uneven
+- Time resolution is per-capture, not continuous — gaps are common
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_wayback.py`
+
+```bash
+# All captures of a specific URL
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "https://example.com/page" \
+    --out data/wb.csv
+
+# All captures of a host
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
+    --match host --out data/wb.csv
+
+# All captures of a domain + subdomains
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
+    --match domain --out data/wb.csv
+
+# Only unique-content captures within a date window
+python3 SKILL_DIR/scripts/fetch_wayback.py --url "example.com" \
+    --match host --collapse digest \
+    --from-date 2020-01-01 --to-date 2023-12-31 \
+    --out data/wb.csv
+```
+
+## 8. Legal & Licensing
+
+- Internet Archive captures are made under fair-use research provisions
+- Replay URLs are stable references — citing them is encouraged
+- Internet Archive non-profit terms of use govern content
+- Some content is rights-restricted; replay may be blocked even if the
+  CDX entry shows it as captured
+
+## 9. References
+
+- CDX server docs: https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md
+- Wayback API: https://archive.org/help/wayback_api.php
+- Internet Archive: https://archive.org/

package/.mindforge/skills/osint-investigation/references/sources/wikipedia.md

@@ -0,0 +1,107 @@
+# Wikipedia + Wikidata
+
+## 1. Summary
+
+Wikipedia is the canonical narrative-bio source for notable people, places,
+and organizations. Wikidata is its structured-data counterpart: ~110M
+items, each with claims, dates, identifiers, and cross-references to
+external authorities (VIAF, ISNI, ORCID, GRID, etc.).
+
+Together they're a high-precision entity-resolution layer — the bar for
+inclusion is real, but anything past that bar is well-cross-referenced.
+
+## 2. Access Methods
+
+- **Wikipedia OpenSearch:** `https://en.wikipedia.org/w/api.php?action=opensearch`
+- **Wikipedia REST summary:** `https://en.wikipedia.org/api/rest_v1/page/summary/<title>`
+- **Wikidata Action API:** `https://www.wikidata.org/w/api.php?action=wbgetentities`
+- **Wikidata SPARQL:** `https://query.wikidata.org/sparql` (more powerful but aggressively rate-limited)
+- **Auth:** None, but **a meaningful User-Agent is required**
+
+Set `HERMES_OSINT_UA` to something identifying (e.g. `your-app/1.0 (you@example.com)`).
+Wikimedia returns HTTP 429 to generic UAs.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_wikipedia.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `source` | str | `wikipedia` or `wikipedia+wikidata` |
+| `label` | str | Wikipedia article title |
+| `description` | str | Short Wikidata description |
+| `qid` | str | Wikidata QID (e.g. Q2283 for Microsoft) |
+| `wikipedia_title`, `wikipedia_url` | str | Article identifier + URL |
+| `wikidata_url` | str | Wikidata entity URL |
+| `instance_of` | str | What kind of thing it is (P31) |
+| `country` | str | Country (P17 for orgs/places, P27 for people) |
+| `occupation` | str | P106 |
+| `employer` | str | P108 |
+| `date_of_birth` | str | P569, YYYY-MM-DD |
+| `place_of_birth` | str | P19 |
+| `summary` | str | Wikipedia REST extract (~1000 chars) |
+
+The fetch script uses Wikidata's Action API (NOT SPARQL) for structured
+facts — far more lenient on rate limits.
+
+## 4. Coverage
+
+- Wikipedia EN: ~7M articles
+- Wikidata: ~110M items, ~1.5B statements
+- Updated continuously; abuse filters and bots run constantly
+- High notability bar — most private individuals are not in Wikipedia
+
+## 5. Cross-Reference Potential
+
+- **All sources** ↔ `label` (entity identity resolution)
+- **SEC EDGAR** ↔ `label` (public companies)
+- **CourtListener** ↔ `label` (parties to notable litigation)
+- **Wikidata external identifiers** (not currently in this fetcher's output)
+  link to VIAF, ISNI, ORCID, GRID, GitHub, Twitter, IMDb, ...
+
+Join key: Wikidata QID is canonical. Wikipedia titles are stable for
+most articles but can be renamed.
+
+## 6. Data Quality
+
+- Notability filter — only notable entities (criteria vary by topic)
+- Recency lag — current events take days to weeks to be reflected
+- POV / vandalism — moderated, but edits between sweeps can be bad
+- Living-persons biographies have stricter sourcing requirements
+- Wikidata claims have qualifiers and references — the fetch script
+  doesn't currently export them
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_wikipedia.py`
+
+```bash
+# Look up a notable entity
+python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --out data/wp.csv
+
+# A specific person
+python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Bill Gates" --out data/wp_bg.csv
+
+# Skip the Wikidata enrichment for speed
+python3 SKILL_DIR/scripts/fetch_wikipedia.py --query "Microsoft" --no-wikidata \
+    --limit 5 --out data/wp.csv
+```
+
+The OpenSearch is fuzzy — `--limit 5` returns the top 5 Wikipedia article
+matches. Each is enriched with the QID + structured facts unless
+`--no-wikidata` is passed.
+
+## 8. Legal & Licensing
+
+- Wikipedia text: CC-BY-SA-3.0 / GFDL
+- Wikidata claims: CC0 (public domain)
+- API ToS: respect rate limits, identify your agent
+- Commercial use allowed with attribution
+
+## 9. References
+
+- Wikipedia OpenSearch: https://www.mediawiki.org/wiki/API:Opensearch
+- Wikipedia REST: https://en.wikipedia.org/api/rest_v1/
+- Wikidata Action API: https://www.wikidata.org/wiki/Wikidata:Data_access
+- Wikidata SPARQL: https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service
+- User-Agent policy: https://meta.wikimedia.org/wiki/User-Agent_policy

package/.mindforge/skills/osint-investigation/scripts/_http.py

@@ -0,0 +1,82 @@
+"""Tiny stdlib HTTP helper used by fetch_*.py scripts.
+
+Provides polite retry + JSON convenience + User-Agent enforcement.
+"""
+from __future__ import annotations
+
+import json
+import os
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+
+DEFAULT_UA = (
+    "hermes-osint-investigation/0.2 "
+    "(+https://github.com/NousResearch/hermes-agent; "
+    "set HERMES_OSINT_UA env var to identify yourself per "
+    "Wikimedia / SEC fair-use guidance)"
+)
+
+
+def get(
+    url: str,
+    *,
+    params: dict | None = None,
+    headers: dict | None = None,
+    user_agent: str | None = None,
+    max_retries: int = 3,
+    backoff: float = 1.5,
+    timeout: float = 30.0,
+) -> bytes:
+    """GET with retry on 5xx and Retry-After honoring.
+
+    429 (rate-limit) is raised IMMEDIATELY with a clear message — retrying
+    when the upstream says "you're over quota" just wastes time. The caller
+    should slow down or supply real credentials.
+    """
+    if params:
+        sep = "&" if "?" in url else "?"
+        url = f"{url}{sep}{urllib.parse.urlencode(params)}"
+    h = {"User-Agent": user_agent or os.environ.get("HERMES_OSINT_UA", DEFAULT_UA)}
+    if headers:
+        h.update(headers)
+
+    last_err: Exception | None = None
+    for attempt in range(max_retries + 1):
+        req = urllib.request.Request(url, headers=h)
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                return resp.read()
+        except urllib.error.HTTPError as e:
+            if e.code == 429:
+                # Surface immediately. Read the body so the caller sees the
+                # provider's actual message ("OVER_RATE_LIMIT" etc.).
+                try:
+                    body = e.read(2048).decode("utf-8", errors="replace")
+                except Exception:  # noqa: BLE001
+                    body = ""
+                raise RuntimeError(
+                    f"HTTP 429 rate-limited by {urllib.parse.urlsplit(url).netloc}. "
+                    f"Slow down or supply a real API key. Body: {body[:300]}"
+                ) from e
+            if e.code in {500, 502, 503, 504} and attempt < max_retries:
+                retry_after = e.headers.get("Retry-After") if e.headers else None
+                wait = float(retry_after) if (retry_after and retry_after.isdigit()) else backoff ** (attempt + 1)
+                time.sleep(wait)
+                last_err = e
+                continue
+            raise
+        except urllib.error.URLError as e:
+            if attempt < max_retries:
+                time.sleep(backoff ** (attempt + 1))
+                last_err = e
+                continue
+            raise
+    if last_err:
+        raise last_err
+    raise RuntimeError("unreachable")
+
+
+def get_json(url: str, **kwargs) -> dict | list:
+    return json.loads(get(url, **kwargs).decode("utf-8"))

package/.mindforge/skills/osint-investigation/scripts/_normalize.py

@@ -0,0 +1,67 @@
+"""Shared entity-name normalization helpers (stdlib-only).
+
+Used by entity_resolution.py and timing_analysis.py.
+"""
+from __future__ import annotations
+
+import re
+
+# Legal suffixes / corporate boilerplate to strip during normalization.
+_SUFFIX_TOKENS = {
+    "INC", "INCORPORATED", "LLC", "LLP", "LP", "LTD", "LIMITED",
+    "CORP", "CORPORATION", "CO", "COMPANY",
+    "GROUP", "GRP", "HOLDINGS", "HOLDING",
+    "PARTNERS", "ASSOCIATES",
+    "INTERNATIONAL", "INTL",
+    "ENTERPRISES", "ENTERPRISE",
+    "SERVICES", "SERVICE", "SVCS",
+    "SOLUTIONS", "MANAGEMENT", "MGMT", "CONSULTING",
+    "TECHNOLOGY", "TECHNOLOGIES", "TECH",
+    "INDUSTRIES", "INDUSTRY",
+    "AMERICA", "AMERICAN",
+    "USA", "US",
+    "PLLC", "PC",
+    "TRUST", "FOUNDATION",
+}
+
+_PUNCT_RE = re.compile(r"[^\w\s]")
+_WS_RE = re.compile(r"\s+")
+
+
+def normalize_name(name: str | None) -> str:
+    """Standard normalization: uppercase, strip suffixes, drop punctuation."""
+    if not name:
+        return ""
+    s = _PUNCT_RE.sub(" ", name.upper())
+    s = _WS_RE.sub(" ", s).strip()
+    tokens = [t for t in s.split() if t and t not in _SUFFIX_TOKENS]
+    return " ".join(tokens)
+
+
+def normalize_aggressive(name: str | None) -> str:
+    """Aggressive normalization: sorted unique tokens (word-bag)."""
+    base = normalize_name(name)
+    if not base:
+        return ""
+    return " ".join(sorted(set(base.split())))
+
+
+def name_tokens(name: str | None, min_len: int = 4) -> set[str]:
+    """Token set used for overlap matching."""
+    base = normalize_name(name)
+    if not base:
+        return set()
+    return {t for t in base.split() if len(t) >= min_len}
+
+
+def token_overlap_ratio(left: str | None, right: str | None) -> tuple[float, int]:
+    """Return (jaccard-like ratio, shared token count) over min-len tokens."""
+    a = name_tokens(left)
+    b = name_tokens(right)
+    if not a or not b:
+        return 0.0, 0
+    shared = a & b
+    if not shared:
+        return 0.0, 0
+    union = a | b
+    return len(shared) / len(union), len(shared)