commoner-probe 0.4.0__tar.gz

commoner_probe-0.4.0/CHANGELOG.md

@@ -0,0 +1,118 @@
+# Changelog
+
+## 0.4.0 (2026-06-22)
+
+### Added
+
+- **`commoner-probe mca-csr`** — download MCA CDM CSR company-spend CSV exports by financial year.
+- **`manifest_mca_csr` schema** and `ManifestMcaCsrRecord` / `Corpus.manifest_mca_csr()` for typed access to MCA CSR manifest records.
+- **`commoner-probe mines-dmft`** — acquire Ministry of Mines / Odisha DMFT public disclosure files with source provenance.
+- **`commoner-probe evidence dmft`** — build side-by-side DMFT evidence bundles from executive disclosure and Sansad oversight records.
+- **`docs/ENDPOINTS.md`** — public source-family endpoint reference.
+- **`narcotics_substance` built-in topic** for NDPS, trafficking, and substance-abuse oversight records.
+
+### Changed
+
+- **Relicensed**: AGPL-3.0-or-later → MIT, so `commoner-probe` can be the permissive shared acquisition floor that downstream repos (including the non-AGPL `sansad-semantic-crawler`) depend on without copyleft friction.
+- `commoner_probe.csr.mca` now uses the verified MCA CDM live contract: `GET /csr-data` for the CSRF-bearing form and `POST /cdm/export.php` for CSV export.
+- Public packaging now includes only release-facing docs; local coordination files (`notes/`, `memory/`, `.ai/`, `.beads/`, `.codex/`, `WORKING.md`, `TODO.md`) are ignored and removed from the tracked public tree.
+- `scripts/check_leaks.py` now blocks private coordination paths if they are accidentally staged.
+
+## 0.3.0 (2026-06-06)
+
+### Breaking changes
+
+- **Package renamed**: `sansad-crawler` → `commoner-probe`. Update your `pip install` and imports.
+  - Python: `from sansad_crawler import ...` → `from commoner_probe import ...`
+  - CLI: `sansad-crawl` → `commoner-probe`
+  - Subcommands renamed: `crawl` → `sansad`, `crawl-committees` → `committees`, `extract-atr-linkage` → `atr-linkage`
+- **New subcommand added**: `state-assembly` (NeVA state assembly portals)
+- **Schema field renamed**: `crawled_at` → `probed_at` in all output records
+- **Relicensed**: MIT → AGPL-3.0-or-later
+
+### Added
+
+- **`commoner-probe state-assembly`** — probe NeVA state assembly portals (`{portal}.neva.gov.in`). Writes `questions.jsonl`, `questions_unlisted.jsonl`, `members.jsonl`, `papers_laid.jsonl`. Tested on Gujarat assembly 15.
+- **HTTP hardening** (`commoner_probe/http_client.py`): SSRF guard, robots.txt checking, per-domain rate limiting (1 req/s), exponential backoff (3 retries), optional `requests_cache` (6h TTL). Install via `pip install commoner-probe[cache]`.
+- **Committee composition** (`CommitteeProbe.probe_composition()`): writes `committee_members.jsonl`.
+- **`filter_fn` hook on `TopicProfile`**: callable injected by analytics layer at runtime.
+- **`classifier_config` in `TopicProfile`**: propagated to `_runs.jsonl` for corpus auditability.
+- **JSON schemas for new outputs**: `committee_members`, `state_assembly_question`, `state_assembly_question_unlisted`, `state_assembly_member`, `state_assembly_paper_laid`.
+- **`commoner-probe init-topic`**: write a bundled example topic profile to disk (built-ins: `libraries`, `home_affairs_starred`, `affirmative_action`).
+- **Single-sourced version**: `__version__` reads from `importlib.metadata` with pyproject fallback.
+- **GitHub Actions**: CI (matrix 3.10–3.12, ruff, pytest) and OIDC PyPI release workflow.
+- **`MANIFEST.in`**, **`CONTRIBUTING.md`**, **`CODE_OF_CONDUCT.md`** (Contributor Covenant v2.1).
+
+### Changed
+
+- Base class `BaseCrawler` → `BaseProbe`; `crawl_ls`/`crawl_rs` → `probe_ls`/`probe_rs`; `crawl_composition` → `probe_composition`.
+- User-Agent: `commoner-probe/0.3.0`.
+- HTTP cache env var: `COMMONER_CACHE_DIR` (was `SANSAD_CACHE_DIR`; old name still honoured with deprecation warning).
+
+---
+
+## 0.2.0 (2026-05-21)
+
+### Added
+
+- **`docs/SCHEMAS.md`** — complete field-level reference for every output
+  stream: all four manifest record shapes (LS Q/A, RS Q/A, LS committee,
+  RS committee), `_runs.jsonl`, three `answers.jsonl` kinds,
+  `atr_linkage.jsonl`, and five `entities/*.jsonl` files. Includes
+  controlled vocabularies and join-key documentation.
+
+- **JSON Schemas** — twelve Draft-2020-12 schemas shipped as package data
+  under `sansad_crawler/schemas/`. Exposed via
+  `sansad_crawler.schemas.load(name)` and `schemas.list_all()`.
+
+- **`sansad_crawler/records.py`** — typed dataclasses for every record kind
+  (`ManifestQaRecord`, `ManifestCommitteeReportRecord`, `AnswerQaResponse`,
+  `AnswerAtrResponse`, `AnswerDfgRecommendation`, `AtrLinkageRecord`,
+  `RunRecord`). Each has `from_dict()` that tolerates unknown keys and
+  missing optional fields.
+
+- **`sansad_crawler/corpus.py`** — `Corpus` class with streaming iterators
+  (`manifest_qa`, `manifest_committee_reports`, `answers_qa`, `answers_atr`,
+  `answers_dfg`, `atr_linkages`, `runs`, `entities`), join helpers
+  (`join_qa`, `join_atr_chain`), and an opt-in `to_dataframe(stream)` that
+  requires `pip install sansad-crawler[pandas]`.
+
+- **`sansad-crawl stats`** — new CLI subcommand that prints corpus health:
+  record counts by house/year/ministry/committee/report_type, answers
+  extraction coverage, entity resolution rate, and date ranges. Use
+  `--json` for machine-readable output.
+
+- **`sansad-crawl validate`** — new CLI subcommand that validates every
+  JSONL file in a corpus against its JSON Schema. Requires
+  `pip install sansad-crawler[dev]`. Prints line numbers and JSON pointers
+  on failure; exits 1 on any error.
+
+- **`[dev]` optional-dependency group** — `jsonschema>=4.20` and
+  `pytest>=7`. Install with `pip install sansad-crawler[dev]`.
+
+- **`[pandas]` optional-dependency group** — `pandas>=2.0`. Install with
+  `pip install sansad-crawler[pandas]`.
+
+- **`examples/usage.py`** — demonstration script for the `Corpus` API.
+
+### Changed (non-breaking)
+
+- `sansad_crawler.__init__` now re-exports `Corpus`, `QaPair`, `AtrChain`,
+  all record dataclasses, and the `schemas` module.
+- `run_id` and `crawled_at` in manifest schemas changed from `required` to
+  optional (always present in freshly crawled corpora; may be absent in
+  synthetic or backfilled data).
+
+### Unchanged
+
+Crawler behaviour, extractor logic, and manifest field set are unchanged.
+All corpora produced by v0.1.0 load and validate cleanly under v0.2.0.
+
+---
+
+## 0.1.0 (2026-05-21)
+
+Initial release. Lok Sabha + Rajya Sabha Q/A crawler, standing-committee
+report crawler, regex-based Q/A and ATR extractors, ATR linkage extractor,
+entity resolution, four CLI subcommands (`crawl`, `crawl-committees`,
+`extract-answers`, `extract-atr-linkage`).

commoner_probe-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Commoner LLP
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

commoner_probe-0.4.0/MANIFEST.in

@@ -0,0 +1,13 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+include docs/SCHEMAS.md
+include docs/INTEGRATION_SMOKE.md
+include docs/RATIONALE.md
+include docs/ENDPOINTS.md
+prune .beads
+prune .ai
+prune .claude
+prune .codex
+prune notes
+prune memory

commoner_probe-0.4.0/PKG-INFO

@@ -0,0 +1,531 @@
+Metadata-Version: 2.4
+Name: commoner-probe
+Version: 0.4.0
+Summary: Sousveillance infrastructure for state mandatory-disclosure portals — parliamentary questions, committee reports, budget data, and state assembly records.
+Author: Sreeram Ramasubramanian
+License: MIT
+Project-URL: Homepage, https://github.com/CommonerLLP/commoner-probe
+Project-URL: Source, https://github.com/CommonerLLP/commoner-probe
+Project-URL: Issues, https://github.com/CommonerLLP/commoner-probe/issues
+Project-URL: Documentation, https://github.com/CommonerLLP/commoner-probe/blob/master/README.md
+Project-URL: Changelog, https://github.com/CommonerLLP/commoner-probe/blob/master/CHANGELOG.md
+Keywords: parliament-of-india,lok-sabha,rajya-sabha,parliamentary-questions,standing-committees,state-assembly,neva,public-records,civic-tech,sousveillance,counter-forensics,right-to-information,open-government,data-justice
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Sociology
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pdf
+Requires-Dist: pdfminer.six>=20231228; extra == "pdf"
+Provides-Extra: http
+Requires-Dist: requests>=2.31.0; extra == "http"
+Provides-Extra: pandas
+Requires-Dist: pandas>=2.0; extra == "pandas"
+Provides-Extra: dev
+Requires-Dist: jsonschema>=4.20; extra == "dev"
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Provides-Extra: all
+Requires-Dist: pdfminer.six>=20231228; extra == "all"
+Requires-Dist: requests>=2.31.0; extra == "all"
+Dynamic: license-file
+
+# commoner-probe
+
+Sousveillance infrastructure for the state's mandatory disclosure systems.
+
+A commoner probes the state's own paperwork — parliamentary questions, committee
+reports, state assembly records — and turns it into evidence. `commoner-probe`
+automates the acquisition so you can focus on the analysis.
+
+```bash
+pip install "commoner-probe[all]"
+import commoner_probe as probe   # alias used throughout CommonerLLP toolchain
+```
+
+---
+
+## Why this exists
+
+Parliamentary questions, committee reports, state assembly records, CSR
+exports, and public mining-district disclosures are mandatory or official
+public disclosures. The data exists. The problem
+is that it lives across undocumented portals with inconsistent APIs, no bulk
+export, and PDFs that require extraction to read programmatically.
+
+`commoner-probe` handles the entire acquisition pipeline:
+
+```
+public disclosure portals  →  manifest.jsonl  →  files/PDFs  →  extracted records  →  your analysis
+                               (metadata)        (raw source)      (structured text)
+```
+
+Classification, topic modelling, and dossier generation are intentionally out
+of scope. This library does one thing: acquire public disclosure data into
+provenance-rich, schema-validated JSONL and source files.
+
+---
+
+## Install
+
+```bash
+pip install "commoner-probe[all]"          # requests + PDF extraction
+pip install "commoner-probe[all,dev]"      # + schema validation and tests
+```
+
+---
+
+## Five-minute quickstart
+
+### Step 1 — Write a topic profile
+
+```json
+{
+  "name": "climate",
+  "description": "Climate change and environmental policy",
+  "search_groups": {
+    "climate": ["climate change", "global warming", "net zero"],
+    "air_quality": ["air pollution", "AQI", "particulate matter"]
+  },
+  "lok_sabha_ministries": ["ENVIRONMENT", "POWER", "PETROLEUM"],
+  "rajya_sabha_ministry_likes": ["ENVIRONMENT", "POWER", "PETROLEUM"]
+}
+```
+
+### Step 2 — Probe parliamentary questions
+
+```bash
+commoner-probe sansad \
+  --topic topic.json \
+  --out data/climate \
+  --house both \
+  --from-date 2019-01-01
+```
+
+Writes `data/climate/manifest.jsonl` — one record per question from both houses.
+
+### Step 3 — Probe committee reports
+
+```bash
+commoner-probe committees \
+  --topic topic.json \
+  --out data/climate-committees \
+  --house both
+```
+
+One record per standing committee report (LS and RS DRSCs).
+
+### Step 4 — Extract text from PDFs
+
+```bash
+commoner-probe extract-answers --out data/climate
+commoner-probe extract-answers --out data/climate-committees
+```
+
+Parses downloaded PDFs into `answers.jsonl`: Q/A pairs, committee
+recommendations, and government responses.
+
+### Step 5 — Load in Python
+
+```python
+import commoner_probe as probe
+
+c = probe.Corpus("data/climate")
+
+for r in c.manifest_qa():
+    print(r.date, r.house, r.ministry, r.title)
+
+for pair in c.join_qa():
+    if pair.answers:
+        print(pair.manifest.title)
+        print(pair.answers[0].question_text[:200])
+```
+
+---
+
+## What you can study
+
+### Parliamentary questions (Lok Sabha + Rajya Sabha)
+
+Each record carries who asked (MP name, party, state), which ministry answered,
+question number, type (starred / unstarred), date, session, and the full PDF.
+After `extract-answers` — extracted question and answer text.
+
+**Typical research questions**: ministry responsiveness rates, which MPs ask
+the most questions by topic, how the same policy question evolves across
+sessions, party-level questioning patterns.
+
+```python
+import commoner_probe as probe
+from collections import Counter
+
+c = probe.Corpus("data/climate")
+ministry_counts = Counter(r.ministry for r in c.manifest_qa())
+for ministry, n in ministry_counts.most_common(10):
+    print(f"{ministry}: {n}")
+```
+
+### Standing committee reports (LS + RS DRSCs)
+
+Committee reports come in four shapes:
+
+| `report_type` | What it is |
+|---|---|
+| `demands_for_grants` | Annual budget scrutiny — the committee dissects ministry spending |
+| `bill` | The committee's examination of a pending bill before it passes |
+| `subject` | Own-initiative policy investigation — deepest substantive record |
+| `action_taken` | The government's formal response to the committee's recommendations |
+
+Action Taken Reports (ATRs) are the government's formal written responses to
+committee recommendations. The `atr-linkage` command connects each ATR back
+to the original report, enabling lifecycle analysis:
+*recommendation → government rejection/acceptance → follow-up*.
+
+```python
+import commoner_probe as probe
+
+c = probe.Corpus("data/climate-committees")
+
+for chain in c.join_atr_chain():
+    print(f"Report: {chain.original and chain.original.title}")
+    print(f"  Recommendations: {len(chain.original_observations)}")
+    print(f"  Government responses: {len(chain.atr_answers)}")
+```
+
+### State assembly records (NeVA portals)
+
+From 2020, sub-national governments have been adopting NIC's NeVA (National
+e-Vidhan Application) infrastructure under a centrally sponsored scheme run
+by the Ministry of Parliamentary Affairs. Most state assemblies are onboarding,
+though coverage varies. The `state-assembly` command probes any NeVA portal:
+
+```bash
+commoner-probe state-assembly \
+  --portal gujarat \
+  --state GJ \
+  --out data/gujarat-assembly \
+  --assemblies 15
+```
+
+### MCA CSR company-spend exports
+
+The Ministry of Corporate Affairs CDM CSR data page exposes downloadable CSV
+exports by financial year. These records compare reporting/spending companies
+and project-sector amounts. They do not identify CSR consultants or implementing
+agencies unless MCA publishes that in the source export.
+
+```bash
+commoner-probe mca-csr \
+  --out data/mca-csr \
+  --years 2022-23,2021-22
+```
+
+```python
+import commoner_probe as probe
+
+c = probe.Corpus("data/mca-csr")
+for r in c.manifest_mca_csr():
+    print(r.financial_year, r.status, r.filename)
+```
+
+### Mines DMFT / PMKKKY disclosures
+
+`mines-dmft` acquires raw Ministry of Mines and Odisha DMFT public disclosure
+files. Ministry CSVs are current cumulative snapshots timestamped by the
+source; treat them as snapshots, not fiscal-year series.
+
+```bash
+commoner-probe mines-dmft \
+  --out data/mines-dmft \
+  --sources mines-gov-in,odisha
+```
+
+Pair the executive disclosure snapshots with Sansad oversight records without
+flattening the source families:
+
+```bash
+commoner-probe evidence dmft \
+  --mines-dmft-dir data/mines-dmft \
+  --sansad-dir data/sansad/mines-dmft-pmkkky \
+  --out data/evidence/dmft.json
+```
+
+---
+
+## All commands
+
+### `commoner-probe sansad` — parliamentary questions
+
+```bash
+commoner-probe sansad \
+  --topic topic.json \
+  --out data/climate \
+  --house both \
+  --from-date 2019-01-01 \
+  --to-date 2026-01-01
+```
+
+| Flag | Default | What it does |
+|---|---|---|
+| `--topic` | required | Path to topic profile JSON |
+| `--out` | required | Output corpus directory |
+| `--house` | `both` | `ls`, `rs`, or `both` |
+| `--from-date` | — | Earliest question date (YYYY-MM-DD) |
+| `--to-date` | — | Latest question date |
+| `--qtype` | `both` | `starred`, `unstarred`, or `both` |
+| `--sessions` | `1-267` | Rajya Sabha session range |
+| `--no-download` | off | Skip PDF downloads; metadata only |
+| `--with-entities` | off | Resolve asker names to stable entity IDs |
+| `--max-records N` | — | Stop after N new records per house (smoke-test) |
+| `--max-buckets N` | — | Only run the first N search/ministry combos |
+| `--reset` | off | Wipe existing manifest and start fresh |
+
+### `commoner-probe committees` — standing committee reports
+
+```bash
+commoner-probe committees \
+  --topic topic.json \
+  --out data/committees \
+  --house both \
+  --committees finance,education
+```
+
+| Flag | Default | What it does |
+|---|---|---|
+| `--committees` | all | Comma-separated committee slugs |
+| `--lok-sabha-no` | `18` | LS number for LS reports |
+| `--from-date` / `--to-date` | — | Date range filter |
+| `--no-download` | off | Skip PDF downloads |
+
+**Available LS committees** (16 DRSCs):
+`agriculture`, `chemicals`, `coal`, `communications`, `consumer_affairs`,
+`defence`, `energy`, `external_affairs`, `finance`, `housing`, `labour`,
+`petroleum`, `railways`, `rural_development`, `social_justice`, `water_resources`
+
+**Available RS committees** (8 DRSCs):
+`commerce`, `education`, `health`, `home_affairs`, `industry`, `personnel`,
+`science`, `transport`
+
+### `commoner-probe extract-answers` — PDF text extraction
+
+```bash
+commoner-probe extract-answers --out data/climate
+commoner-probe extract-answers --out data/climate --refresh
+```
+
+Reads `manifest.jsonl` and downloaded PDFs; writes `answers.jsonl` with:
+
+- `qa_response` — (question_text, answer_text) pairs from Q/A PDFs
+- `atr_response` — (recommendation_no, recommendation_text, response_text) triples from ATR PDFs
+- `dfg_recommendation` — numbered observation paragraphs from DFG/Bill/Subject PDFs
+
+Requires `pip install "commoner-probe[pdf]"`.
+
+### `commoner-probe atr-linkage` — ATR → original report
+
+```bash
+commoner-probe atr-linkage --out data/committees
+```
+
+Writes `atr_linkage.jsonl` — each ATR linked back to the report it responds to.
+Safe to re-run (idempotent overwrite).
+
+### `commoner-probe state-assembly` — state legislature records
+
+```bash
+commoner-probe state-assembly \
+  --portal gujarat \
+  --state GJ \
+  --out data/gujarat \
+  --assemblies 15
+```
+
+### `commoner-probe mca-csr` — MCA CSR company-spend exports
+
+```bash
+commoner-probe mca-csr \
+  --out data/mca-csr \
+  --years 2022-23
+```
+
+Downloads CSV exports from the MCA CDM CSR data page and writes one
+`manifest.jsonl` record per financial year. Use `--dry-run` to print manifest
+records without opening a network session.
+
+### `commoner-probe mines-dmft` — Ministry of Mines / DMFT files
+
+```bash
+commoner-probe mines-dmft \
+  --out data/mines-dmft \
+  --sources mines-gov-in,odisha
+```
+
+Downloads raw Ministry of Mines static CSV snapshots and Odisha DMFT public
+JSON/report surfaces. Use `--dry-run` to print manifest records without opening
+network sessions.
+
+### `commoner-probe evidence dmft` — cross-source evidence bundle
+
+```bash
+commoner-probe evidence dmft \
+  --mines-dmft-dir data/mines-dmft \
+  --sansad-dir data/sansad/mines-dmft-pmkkky \
+  --out data/evidence/dmft.json
+```
+
+Builds a JSON bundle with separate `executive_disclosure` and
+`parliamentary_oversight` sections. It does not merge unlike source families
+into one table.
+
+### `commoner-probe stats` — corpus health
+
+```bash
+commoner-probe stats --out data/climate
+commoner-probe stats --out data/climate --json
+```
+
+### `commoner-probe validate` — schema validation
+
+```bash
+commoner-probe validate --out data/climate
+```
+
+Validates every JSONL file against its JSON Schema. Exits 1 on errors.
+Requires `[dev]` extra.
+
+---
+
+## Topic profile
+
+Controls what the probe acquires:
+
+```json
+{
+  "name": "libraries",
+  "description": "Public library infrastructure and policy",
+  "search_groups": {
+    "public_libraries": ["public library", "rural library"],
+    "policy": ["National Mission on Libraries", "RRRLF"]
+  },
+  "lok_sabha_ministries": ["CULTURE", "EDUCATION"],
+  "rajya_sabha_ministry_likes": ["CULTURE", "EDUCATION"]
+}
+```
+
+- `search_groups` — keyword groups for LS full-text search. Each query runs
+  independently; results are union-deduped on `key`.
+- `lok_sabha_ministries` — exact ministry filter for LS (case-sensitive).
+- `rajya_sabha_ministry_likes` — ministry LIKE filter for RS (prefix match).
+
+See `examples/topics/` for working examples.
+
+---
+
+## Output files
+
+| File | Contents |
+|------|----------|
+| `manifest.jsonl` | One record per question or committee report |
+| `_runs.jsonl` | Audit log: scope, topic hash, errors, per-bucket counts |
+| `answers.jsonl` | Extracted Q/A and recommendation/response pairs |
+| `atr_linkage.jsonl` | ATR → original report linkages |
+| source CSV/JSON/HTML files | Raw source files for source-specific probes such as MCA CSR and DMFT |
+| `pdfs/ls/` | Downloaded LS PDFs |
+| `pdfs/rs/` | Downloaded RS PDFs |
+| `probe.log` | Human-readable probe progress log |
+
+For complete field-level documentation see [`docs/SCHEMAS.md`](docs/SCHEMAS.md).
+
+---
+
+## Entity resolution (`--with-entities`)
+
+Pass `--with-entities` to `commoner-probe sansad` to resolve asker names to
+stable `entity_id` values. On first run the entity store is populated from
+the sansad.in MP roster; subsequent runs reuse the local cache.
+
+Resolved entity IDs join across corpora and sessions — useful for studying
+the same MP's questioning behaviour over time or across houses.
+
+---
+
+## Python API
+
+```python
+import commoner_probe as probe
+
+c = probe.Corpus("data/climate")
+
+# Typed iterators
+for r in c.manifest_qa():                 # ManifestQaRecord
+    ...
+for r in c.manifest_committee_reports():  # ManifestCommitteeReportRecord
+    ...
+for r in c.answers_qa():                  # AnswerQaResponse
+    ...
+for r in c.answers_atr():                 # AnswerAtrResponse
+    ...
+for r in c.answers_dfg():                 # AnswerDfgRecommendation
+    ...
+for r in c.atr_linkages():                # AtrLinkageRecord
+    ...
+for r in c.manifest_mca_csr():            # ManifestMcaCsrRecord
+    ...
+for r in c.manifest_mines_dmft():         # ManifestMinesDmftRecord
+    ...
+for r in c.runs():                        # RunRecord
+    ...
+
+# Join helpers
+for pair in c.join_qa():                  # manifest + extracted answers
+    ...
+for chain in c.join_atr_chain():          # ATR + original report + observations
+    ...
+
+# pandas (pip install commoner-probe[pandas])
+df = c.to_dataframe("manifest_committee_reports")
+```
+
+See [`examples/usage.py`](examples/usage.py) for a runnable walkthrough.
+See [`docs/ENDPOINTS.md`](docs/ENDPOINTS.md) for source-family endpoint notes.
+
+---
+
+## License
+
+MIT License — see [`LICENSE`](LICENSE).
+
+`commoner-probe` is sousveillance infrastructure, built for the commons. It is
+released under the permissive MIT license so it can serve as a shared
+acquisition floor that any downstream project — including the other repos in the
+CommonerLLP federation, whatever their own licenses — can build on without
+copyleft friction.
+
+---
+
+## Upcoming
+
+### Floor debates
+
+sansad.in exposes full debate proceedings via `api_ls/debate/text-of-debate`
+(structured JSON, 17th Lok Sabha onwards). Each record covers a single day:
+type of business, member who spoke, and verbatim text. The richest longitudinal
+record of what MPs say on the floor.
+
+### Bills and legislation
+
+`sansad.in/ls/legislation/bills` lists every bill since independence with
+introduction date, debate dates, and status at each stage. Enables tracking
+legislative velocity, committee scrutiny rates, and private member bill outcomes.
+
+### MP profiles and career timelines
+
+Structured biographical data for each member: constituency, state, party, terms
+served, educational background, declared profession. Pairs with the Q/A corpus
+for studies of how MP background predicts parliamentary participation.