mindforge-cc 11.8.2 → 11.9.0

package/.mindforge/skills/osint-investigation/references/sources/courtlistener.md

@@ -0,0 +1,98 @@
+# CourtListener — Free Law Project
+
+## 1. Summary
+
+CourtListener (Free Law Project) aggregates court opinions, dockets, oral
+arguments, and judge data. Covers ~10M federal and state court opinions
+back to colonial America, plus PACER docket data from RECAP submissions.
+
+## 2. Access Methods
+
+- **REST API v4:** `https://www.courtlistener.com/api/rest/v4/`
+- **Auth:** Anonymous reads allowed on most endpoints; token raises rate
+  limits and unlocks bulk export
+- **Rate limit:** ~5,000 req/hour unauthenticated for search; higher with token
+
+Set `COURTLISTENER_TOKEN` env var. Get a free token at
+https://www.courtlistener.com/sign-in/ then create an API key.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_courtlistener.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `case_name` | str | Case name |
+| `court` | str | Court name |
+| `court_id` | str | Court ID (e.g. `nysd`, `scotus`, `ca9`) |
+| `date_filed` | str | YYYY-MM-DD |
+| `docket_number` | str | Court docket number |
+| `judge` | str | Judge name(s) |
+| `citation` | str | Reporter citation(s) |
+| `result_type` | str | opinions / dockets / oral / people |
+| `snippet` | str | Search-match snippet (up to 500 chars) |
+| `absolute_url` | str | Direct CourtListener URL |
+
+## 4. Coverage
+
+- Federal: all circuit and district courts, SCOTUS
+- State: all 50 state supreme/appellate courts, many trial courts
+- Opinions: ~10M back to 1600s (colonial), full coverage 1950 → present
+- Dockets via RECAP: ~3M+ from user-submitted PACER PDFs
+- Updated continuously
+
+## 5. Cross-Reference Potential
+
+- **OpenCorporates** ↔ `case_name` (corporate litigation)
+- **SEC EDGAR** ↔ `case_name` (securities class actions)
+- **OFAC SDN** ↔ `case_name` (sanctions-related civil/criminal cases)
+
+Join key: party name from `case_name`. Note: `case_name` often abbreviates
+("Smith v. Jones" rather than full party names) — use the full case URL
+to get all parties.
+
+## 6. Data Quality
+
+- Older opinions (pre-1990) often lack docket numbers and judges
+- State coverage is more uneven than federal
+- PACER docket coverage depends on RECAP user submissions — not exhaustive
+- Sealed documents are excluded
+- Party names in case captions don't always match filing names exactly
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_courtlistener.py`
+
+```bash
+# Search opinions for a party / keyword
+python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
+    --out data/cl.csv
+
+# PACER dockets (best for recent litigation)
+python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
+    --type dockets --out data/cl_dockets.csv
+
+# Restrict to a court
+python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Microsoft" \
+    --court ca9 --out data/cl_9th.csv
+
+# Date range
+python3 SKILL_DIR/scripts/fetch_courtlistener.py --query "Example Corp" \
+    --date-from 2020-01-01 --date-to 2024-12-31 --out data/cl.csv
+```
+
+Pass `--token` or set `COURTLISTENER_TOKEN`.
+
+## 8. Legal & Licensing
+
+- Court opinions are public domain
+- Free Law Project provides the data under CC0 / public domain dedication
+- No commercial use restrictions on opinion text or metadata
+- Some PACER PDFs have copyright on layout (not text) — fair use applies
+
+## 9. References
+
+- API docs: https://www.courtlistener.com/help/api/rest/
+- Court IDs: https://www.courtlistener.com/api/jurisdictions/
+- RECAP archive: https://www.courtlistener.com/recap/
+- Bulk data: https://www.courtlistener.com/help/api/bulk-data/

package/.mindforge/skills/osint-investigation/references/sources/gdelt.md

@@ -0,0 +1,104 @@
+# GDELT — Global News Monitoring
+
+## 1. Summary
+
+GDELT (Global Database of Events, Language, and Tone) monitors world news
+in 100+ languages with full-text indexing. Updated every 15 minutes.
+~2015 → present, ~1B+ articles indexed. Free anonymous access.
+
+GDELT is wider than Google News (more international, more long-tail
+sources) and indexed by tone/sentiment, themes (CAMEO codes), people, and
+organizations.
+
+## 2. Access Methods
+
+- **DOC 2.0 API:** `https://api.gdeltproject.org/api/v2/doc/doc`
+- **Events / GKG 2.0:** `https://api.gdeltproject.org/api/v2/events/events`
+- **Auth:** None
+- **Rate limit:** **1 request per 5 seconds** for the DOC API — strict
+
+The fetch script automatically retries after a 6-second sleep when a
+429 is received.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_gdelt.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `title` | str | Article title |
+| `url` | str | Article URL |
+| `seen_date` | str | When GDELT first saw the article (UTC) |
+| `domain` | str | Publisher domain |
+| `language` | str | Source language |
+| `source_country` | str | 2-letter country code |
+| `tone` | str | GDELT-computed tone score (negative = negative coverage) |
+| `social_image` | str | Open Graph image URL when available |
+
+## 4. Coverage
+
+- Worldwide news in 100+ languages
+- ~2015 → present (Events back to 1979 via a separate stream)
+- Update frequency: 15 minutes
+- Bias: heavily Anglophone in volume but very wide source list overall
+
+## 5. Cross-Reference Potential
+
+- **All sources** ↔ `title` / `url` (news context for any subject)
+- **Wikipedia** ↔ event timeline for notable entities
+- **Wayback Machine** ↔ recover articles whose URLs have died
+- **OFAC SDN** ↔ news context for sanctions designations
+- **SEC EDGAR** ↔ news context for 8-K material events
+
+Join key: entity name appearing in article title or full-text. GDELT also
+extracts named entities into a separate stream (GKG) not exposed by this
+fetcher — query GDELT directly for entity-level filtering.
+
+## 6. Data Quality
+
+- Title extraction is automated and can be wrong (sometimes captures the
+  site name + delimiter + article title; sometimes a generic page title)
+- Sentiment / tone is computed by GDELT, not source-supplied
+- Some domains are oversampled (newswires, aggregators)
+- Source country is inferred from domain registration / TLD — can be
+  wrong for international news sites with country-neutral domains
+- Article URLs can rot — pair with Wayback Machine to preserve content
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_gdelt.py`
+
+```bash
+# Recent news mentioning an entity
+python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Nous Research" \
+    --timespan 6m --out data/gdelt.csv
+
+# Phrase-exact (use double quotes inside single quotes for the shell)
+python3 SKILL_DIR/scripts/fetch_gdelt.py --query '"Dillon Rolnick"' \
+    --timespan 1y --out data/gdelt.csv
+
+# Filter to a country / language
+python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Microsoft" \
+    --source-country US --source-lang English --out data/gdelt.csv
+
+# Date range
+python3 SKILL_DIR/scripts/fetch_gdelt.py --query "Microsoft" \
+    --start 2024-01-01 --end 2024-12-31 --out data/gdelt.csv
+```
+
+GDELT supports its own query operators: phrase quoting, AND/OR/NOT,
+`sourcecountry:US`, `theme:ECON_BANKRUPTCY`, `tone<-5`, etc.
+See https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/ for syntax.
+
+## 8. Legal & Licensing
+
+- GDELT data is provided free for academic and journalistic use
+- Article URLs link out to original publishers — copyright remains with
+  the publisher
+- GDELT is NOT a content archive; it's a metadata index
+
+## 9. References
+
+- DOC 2.0 API: https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/
+- Themes & query syntax: https://blog.gdeltproject.org/gkg-2-0-our-global-knowledge-graph-2-0-amazing-data-at-your-fingertips/
+- Project home: https://www.gdeltproject.org/

package/.mindforge/skills/osint-investigation/references/sources/icij-offshore.md

@@ -0,0 +1,104 @@
+# ICIJ Offshore Leaks Database
+
+## 1. Summary
+
+The International Consortium of Investigative Journalists (ICIJ) publishes a
+combined database of offshore entities from the Panama Papers, Paradise Papers,
+Pandora Papers, Bahamas Leaks, and Offshore Leaks. ~800,000+ offshore entities
+with their officers, intermediaries, and addresses.
+
+## 2. Access Methods
+
+- **Bulk download (primary):** `https://offshoreleaks-data.icij.org/offshoreleaks/csv/full-oldb.LATEST.zip` (~70 MB ZIP, refreshed periodically)
+- **Search UI (human):** `https://offshoreleaks.icij.org/`
+- **Auth:** None
+- **Note:** The previous Open Refine reconciliation endpoint at
+  `/reconcile` now returns 404. ICIJ has removed it. The bulk ZIP is the
+  remaining stable access path. The skill's `fetch_icij_offshore.py` caches
+  the ZIP locally (default `~/.cache/hermes-osint/icij/`, refreshes after
+  30 days) and searches it offline.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_icij_offshore.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `node_id` | int | ICIJ canonical node ID |
+| `name` | str | Entity / officer / intermediary name |
+| `node_type` | str | entity / officer / intermediary / address |
+| `country_codes` | str | Semicolon-separated ISO codes |
+| `countries` | str | Country names |
+| `jurisdiction` | str | Offshore jurisdiction (BVI, Panama, etc.) |
+| `incorporation_date` | str | YYYY-MM-DD |
+| `inactivation_date` | str | YYYY-MM-DD (if struck) |
+| `source` | str | Panama Papers / Paradise Papers / Pandora Papers / etc. |
+| `entity_url` | str | Link to ICIJ page |
+| `connections` | str | Semicolon-separated node IDs of related entities |
+
+## 4. Coverage
+
+- Worldwide offshore entity records
+- Earliest records: 1970s (Bahamas Leaks). Most data 1990–2018.
+- NOT updated in real-time — new leaks added when ICIJ publishes them
+- ~810,000 offshore entities + ~750,000 officers + ~150,000 intermediaries
+
+## 5. Cross-Reference Potential
+
+- **SEC EDGAR** ↔ `name` (public companies with offshore arms)
+- **USAspending** ↔ `name` (federal contractors with offshore structure)
+- **OFAC SDN** ↔ `name` (sanctioned entities using offshore vehicles)
+
+Join key: normalized entity/officer name. `node_id` is canonical for cross-
+referencing within ICIJ. Connections graph traversal is in-script (BFS over
+`connections`).
+
+## 6. Data Quality
+
+- Offshore entity names sometimes appear in multiple leaks with slight variations
+- Officers may be nominees (front persons), not beneficial owners
+- Some entries have minimal info (just a name + jurisdiction)
+- The connections graph is incomplete — some relationships are documented in
+  source materials but not in the structured database
+- Inactive/struck-off entities are still included with `inactivation_date`
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_icij_offshore.py`
+
+```bash
+# Search by entity name (case-insensitive substring across the bulk DB)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
+    --out data/icij.csv
+
+# Search by officer (individual person)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH JOHN" \
+    --out data/icij.csv
+
+# Search by jurisdiction (filter on cached results)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --officer "SMITH" \
+    --jurisdiction "BRITISH VIRGIN ISLANDS" --out data/icij_bvi.csv
+
+# Force a fresh download (default refresh window is 30 days)
+python3 SKILL_DIR/scripts/fetch_icij_offshore.py --entity "EXAMPLE CORP" \
+    --force-refresh --out data/icij.csv
+```
+
+First call downloads the ~70 MB ZIP under `~/.cache/hermes-osint/icij/`
+(or `$HERMES_OSINT_CACHE/icij/`). Subsequent calls reuse the cache for 30 days.
+
+## 8. Legal & Licensing
+
+- Public record as published by ICIJ under explicit publication
+- No copyright on the underlying facts (entity names, jurisdictions)
+- ICIJ asks for attribution if used in derivative reporting
+- **Ethical note**: Presence in this database does NOT imply wrongdoing. Many
+  offshore structures are legal. The database is a research tool, not a list of
+  criminals.
+
+## 9. References
+
+- Database: https://offshoreleaks.icij.org/
+- About the data: https://offshoreleaks.icij.org/pages/about
+- Methodology: https://www.icij.org/investigations/panama-papers/
+- API hints: Open Refine reconciliation endpoint at `https://offshoreleaks.icij.org/reconcile`

package/.mindforge/skills/osint-investigation/references/sources/nyc-acris.md

@@ -0,0 +1,90 @@
+# NYC ACRIS — NYC Real Property Records
+
+## 1. Summary
+
+The Automated City Register Information System (ACRIS) is NYC's index of
+recorded property documents: deeds, mortgages, satisfactions, liens, UCC
+filings. Covers Manhattan, Bronx, Brooklyn, Queens, Staten Island.
+Published as 4 linked Socrata datasets on the NYC Open Data portal.
+
+## 2. Access Methods
+
+- **Socrata API:** `https://data.cityofnewyork.us/resource/636b-3b5g.json` (Parties)
+- **Other datasets:** `bnx9-e6tj` (Master), `8h5j-fqxa` (Legal), `uqqa-hym2` (References)
+- **Auth:** None for read access (Socrata `$app_token` raises rate limits if needed)
+- **Rate limit:** Generous (~1000 req/hour unauthenticated)
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_nyc_acris.py` (Parties joined to Master):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `document_id` | str | ACRIS document ID |
+| `name` | str | Party name as recorded (often "LAST, FIRST" but varies) |
+| `party_type` | str | 1=grantor, 2=grantee, 3=other |
+| `party_role` | str | Human-readable role label |
+| `address_1` | str | Property or party address line 1 |
+| `city`, `state`, `zip`, `country` | str | Address parts |
+| `doc_type` | str | DEED, MTGE (mortgage), SAT (satisfaction), AGMT, etc. |
+| `doc_date`, `recorded_date` | str | YYYY-MM-DD |
+| `borough` | str | Manhattan / Bronx / Brooklyn / Queens / Staten Island |
+| `amount` | str | Document amount (USD, when applicable) |
+| `filing_url` | str | Direct ACRIS DocumentImageView link |
+
+## 4. Coverage
+
+- NYC 5 boroughs only — other counties have their own recorders
+- 1966 → present (older filings exist on microfilm at the County Clerk)
+- Updated nightly
+- ~70M+ party records cumulative
+
+## 5. Cross-Reference Potential
+
+- **SEC EDGAR** ↔ `name` (insider filers with NYC property)
+- **USAspending** ↔ `name` (federal contractors with NYC property)
+- **Senate LDA** ↔ `name` (lobbyists / clients with NYC property)
+- **ICIJ Offshore** ↔ `name` (NYC properties owned via offshore vehicles)
+
+Join key: normalized party name. NYC property records typically store names
+as "LAST, FIRST" or full LLC names — use `entity_resolution.py`.
+
+## 6. Data Quality
+
+- Same person appears with multiple name formats over time
+- LLC and trust ownership obscures beneficial owners
+- Recording lag can be 2-4 weeks after closing
+- Older documents have spottier address data
+- Sealed records (e.g. domestic violence shelters) are excluded by law
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_nyc_acris.py`
+
+```bash
+# By party name
+python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "ROLNICK" --out data/acris.csv
+
+# By address (useful when you know the property but not the names)
+python3 SKILL_DIR/scripts/fetch_nyc_acris.py --address "571 HUDSON" --out data/acris.csv
+
+# Restrict to grantees (buyers / mortgagees)
+python3 SKILL_DIR/scripts/fetch_nyc_acris.py --name "ROLNICK" --party-type 2 \
+    --out data/acris_buyers.csv
+```
+
+The script joins Parties → Master to populate doc_type, dates, borough, and
+amount. Pass `--no-enrich` to skip the join (faster, fewer columns).
+
+## 8. Legal & Licensing
+
+- Public record under NYS Real Property Law and NYC Charter
+- No commercial use restrictions on the data
+- All ACRIS data is public information by statute
+
+## 9. References
+
+- ACRIS portal: https://a836-acris.nyc.gov/CP/
+- NYC Open Data: https://data.cityofnewyork.us/
+- Parties dataset: https://data.cityofnewyork.us/City-Government/ACRIS-Real-Property-Parties/636b-3b5g
+- Document type codes: https://www1.nyc.gov/site/finance/taxes/acris.page

package/.mindforge/skills/osint-investigation/references/sources/ofac-sdn.md

@@ -0,0 +1,92 @@
+# OFAC SDN — Specially Designated Nationals List
+
+## 1. Summary
+
+The Office of Foreign Assets Control (OFAC) publishes the Specially Designated
+Nationals and Blocked Persons List (SDN). US persons are generally prohibited
+from dealing with individuals and entities on this list. Also published:
+non-SDN consolidated lists (BIS Denied Persons, FSE, etc.).
+
+## 2. Access Methods
+
+- **Full XML:** `https://www.treasury.gov/ofac/downloads/sdn.xml`
+- **Delimited:** `https://www.treasury.gov/ofac/downloads/sdn.csv`
+- **Consolidated:** `https://www.treasury.gov/ofac/downloads/consolidated/consolidated.xml`
+- **Auth:** None
+- **Rate limit:** None (static file downloads). Updated continuously.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_ofac_sdn.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `entity_id` | int | OFAC unique ID |
+| `name` | str | Primary name |
+| `entity_type` | str | individual / entity / vessel / aircraft |
+| `program_list` | str | Semicolon-separated sanctions programs (e.g. SDGT;IRAN) |
+| `title` | str | For individuals: title/role |
+| `nationalities` | str | Semicolon-separated country codes |
+| `aka_list` | str | Semicolon-separated "also known as" names |
+| `addresses` | str | Semicolon-separated known addresses |
+| `dob` | str | Date of birth (individuals) |
+| `pob` | str | Place of birth (individuals) |
+| `remarks` | str | OFAC's free-text remarks |
+| `last_updated` | str | YYYY-MM-DD (publication date) |
+
+## 4. Coverage
+
+- Worldwide — all entities sanctioned by US Treasury
+- ~10,000 entries on SDN, ~15,000 on consolidated lists
+- Updated continuously (sometimes daily during active enforcement)
+- Includes AKAs (very common, can be 10+ per entity)
+
+## 5. Cross-Reference Potential
+
+- **SEC EDGAR** ↔ `name` (public companies sanctioned)
+- **USAspending** ↔ `name` (sanctioned entity as federal contractor — should
+  be impossible but verify)
+- **ICIJ Offshore** ↔ `name` (offshore entities also sanctioned)
+
+Join key: normalized name. **CRITICAL**: must match against `aka_list` too.
+Many sanctioned entities are caught only via aliases.
+
+## 6. Data Quality
+
+- Names are transliterated from many scripts — multiple romanizations possible
+- AKAs often differ wildly from primary name
+- Some entries have minimal info (no DOB, no address) for individuals
+- Free-text `remarks` contain critical context — read them
+- "Specially Designated Global Terrorists" (SDGT) and "Cyber-related" (CYBER2)
+  programs add and remove entries frequently
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_ofac_sdn.py`
+
+```bash
+# Full snapshot
+python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --out data/ofac_sdn.csv
+
+# Filter to specific program
+python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --program SDGT --out data/sdn_sdgt.csv
+
+# Entities only (skip individuals, vessels, aircraft)
+python3 SKILL_DIR/scripts/fetch_ofac_sdn.py --entity-type entity --out data/sdn_entities.csv
+```
+
+## 8. Legal & Licensing
+
+- Public record under Executive Order authority and statutory sanctions programs
+- US persons MUST screen against this list — it is enforced
+- No restrictions on the data itself; restrictions are on transactions with
+  the listed entities
+- ZERO penalty for "over-matching" — false positives must be cleared but are not
+  prohibited
+
+## 9. References
+
+- OFAC home: https://ofac.treasury.gov/
+- SDN list: https://ofac.treasury.gov/specially-designated-nationals-and-blocked-persons-list-sdn-human-readable-lists
+- Data formats: https://ofac.treasury.gov/sdn-list/sanctions-list-search-tool
+- Compliance guidance: https://ofac.treasury.gov/recent-actions

package/.mindforge/skills/osint-investigation/references/sources/opencorporates.md

@@ -0,0 +1,103 @@
+# OpenCorporates — Global Corporate Registry
+
+## 1. Summary
+
+OpenCorporates aggregates corporate registry data from 130+ jurisdictions
+worldwide (~200M companies). Covers US state-level filings (NY DOS, Delaware
+DOC, California SOS, etc.), UK Companies House, EU registries, and most
+common-law jurisdictions.
+
+## 2. Access Methods
+
+- **REST API:** `https://api.opencorporates.com/v0.4/`
+- **HTML fallback:** `https://opencorporates.com/companies?q=...`
+- **Auth:** API token required (free tier 500 calls/month, paid plans available)
+- **Rate limit:** Token-bound; un-tokened requests return 401
+
+Set `OPENCORPORATES_API_TOKEN` env var. Get a free token at
+https://opencorporates.com/api_accounts/new.
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_opencorporates.py`:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `name` | str | Company legal name |
+| `company_number` | str | Registry-assigned number |
+| `jurisdiction_code` | str | e.g. `us_ny`, `us_de`, `gb` |
+| `jurisdiction_name` | str | Human-readable jurisdiction |
+| `incorporation_date` | str | YYYY-MM-DD |
+| `dissolution_date` | str | YYYY-MM-DD (empty if active) |
+| `company_type` | str | Domestic LLC / Foreign Corp / etc. |
+| `status` | str | Active / Inactive / Dissolved |
+| `registered_address` | str | Registered office address |
+| `opencorporates_url` | str | Link to OpenCorporates entity page |
+| `officers_count` | str | Total officers on record |
+| `source` | str | `api`, `html`, or `html-fallback` |
+
+## 4. Coverage
+
+- US: all 50 states + DC at state level (LLCs, corps, LPs)
+- International: UK, EU, Canada, Australia, NZ, many APAC + LATAM jurisdictions
+- ~200M company records cumulative
+- Update frequency varies by jurisdiction (UK CH is near-realtime; some
+  state registries lag months)
+
+## 5. Cross-Reference Potential
+
+- **NYC ACRIS** ↔ `name` (LLC/corp owners of NYC property)
+- **USAspending** ↔ `name` (corporate federal contractors)
+- **SEC EDGAR** ↔ `name` (public companies + their subsidiaries)
+- **ICIJ Offshore** ↔ `name` (international corporate structures)
+
+Join key: normalized company name. Some entries have `previous_names` arrays
+which are not currently exported by the fetch script — query OC directly
+for that.
+
+## 6. Data Quality
+
+- Company-name spellings vary across re-incorporations and renames
+- Officer records are spottier than company records (many jurisdictions
+  don't require officer disclosure)
+- Beneficial-ownership data is generally NOT here — most jurisdictions
+  don't require it. UK Companies House has PSC (people with significant
+  control) but that's not universal.
+- Cross-jurisdictional links (parent / subsidiary) are based on registry
+  filings only; corporate trees are often incomplete
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_opencorporates.py`
+
+```bash
+# Search globally by name
+python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
+    --out data/oc.csv
+
+# Restrict to a jurisdiction
+python3 SKILL_DIR/scripts/fetch_opencorporates.py --query "Example Corp" \
+    --jurisdiction us_ny --out data/oc_ny.csv
+
+# Set token via env or flag
+OPENCORPORATES_API_TOKEN=xxx python3 SKILL_DIR/scripts/fetch_opencorporates.py \
+    --query "Microsoft" --out data/oc.csv
+```
+
+Without a token the script falls back to scraping the HTML search page.
+The fallback is brittle and only fills in `name`, `jurisdiction_code`,
+`opencorporates_url` — set the token for serious work.
+
+## 8. Legal & Licensing
+
+- OpenCorporates aggregates public records — the underlying facts are
+  public domain
+- OpenCorporates own database is licensed CC-BY-SA-4.0; attribution required
+- API ToS prohibits redistributing the full dataset; per-record reference
+  is fine
+
+## 9. References
+
+- API docs: https://api.opencorporates.com/documentation/API-Reference
+- Jurisdiction codes: https://api.opencorporates.com/v0.4/jurisdictions.json
+- Schema: https://opencorporates.com/info/our_data

package/.mindforge/skills/osint-investigation/references/sources/sec-edgar.md

@@ -0,0 +1,83 @@
+# SEC EDGAR — Corporate Filings
+
+## 1. Summary
+
+EDGAR (Electronic Data Gathering, Analysis, and Retrieval) is the SEC's system
+for corporate disclosure filings: 10-K (annual), 10-Q (quarterly), 8-K (current
+events), DEF 14A (proxy), Form 4 (insider trading), 13F (institutional holdings).
+
+## 2. Access Methods
+
+- **API:** `https://data.sec.gov/submissions/CIK<10-digit-padded>.json` (no auth)
+- **Filing index:** `https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=...`
+- **Full-text search:** `https://efts.sec.gov/LATEST/search-index?q=...`
+- **Auth:** None — requires `User-Agent` header with contact info per SEC policy
+- **Rate limit:** 10 requests/second per IP (enforced)
+
+## 3. Data Schema
+
+Key fields emitted by `fetch_sec_edgar.py` (filings index):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `cik` | str | Central Index Key (10-digit padded) |
+| `company_name` | str | Registrant name |
+| `form_type` | str | 10-K, 10-Q, 8-K, etc. |
+| `filing_date` | str | YYYY-MM-DD |
+| `accession_number` | str | Filing accession (e.g. 0000320193-24-000123) |
+| `primary_document` | str | Filename of main document |
+| `filing_url` | str | Direct URL to filing index |
+| `reporting_period` | str | Period of report (where applicable) |
+
+## 4. Coverage
+
+- All public US registrants from 1993 → present
+- 1993-2000 has spotty coverage of older filings (paper-to-electronic migration)
+- ~12M filings cumulative
+- Updated within minutes of filing acceptance
+
+## 5. Cross-Reference Potential
+
+- **USAspending** ↔ `company_name` (public companies as federal contractors)
+- **Senate LD** ↔ `company_name` (public companies hire lobbyists)
+- **OFAC SDN** ↔ `company_name` (sanctions screening of public registrants)
+
+Join key: company name OR CIK if you have it. CIK is canonical and stable.
+
+## 6. Data Quality
+
+- Subsidiaries often filed under parent CIK — be careful with name matches
+- Name changes over time (rebrands, acquisitions) — CIK remains constant
+- 10-K Item 1A Risk Factors are free-form text — useful for `web_extract`-style
+  parsing, not structured queries
+- Foreign private issuers file 20-F instead of 10-K
+
+## 7. Acquisition Script
+
+Path: `scripts/fetch_sec_edgar.py`
+
+```bash
+# By CIK
+python3 SKILL_DIR/scripts/fetch_sec_edgar.py --cik 0000320193 \
+    --types 10-K,10-Q --out data/edgar_filings.csv
+
+# By company name (resolves to CIK first via name search)
+python3 SKILL_DIR/scripts/fetch_sec_edgar.py --company "APPLE INC" \
+    --types 8-K --since 2024-01-01 --out data/edgar_filings.csv
+```
+
+Set `SEC_USER_AGENT` env var with your contact email (SEC requirement).
+Example: `SEC_USER_AGENT="Research example@example.com"`.
+
+## 8. Legal & Licensing
+
+- Public record under SEC Rule 24b-2 / 17 CFR § 230.401
+- No commercial use restrictions on filing content
+- SEC asks all bulk users to include a `User-Agent` with contact info and to
+  respect 10 req/s — failure to do so can result in IP blocking
+
+## 9. References
+
+- Developer docs: https://www.sec.gov/edgar/sec-api-documentation
+- EDGAR full-text search: https://efts.sec.gov/LATEST/search-index
+- Fair access policy: https://www.sec.gov/os/accessing-edgar-data