indonesia-civic-stack 0.1.0__tar.gz

indonesia_civic_stack-0.1.0/.cursorrules

@@ -0,0 +1,15 @@
+This is indonesia-civic-stack — a Python SDK for Indonesian government data.
+
+Architecture: shared/ (core) + civic_stack/ (11 gov portal wrappers) + proxy/ (CF Worker).
+All async Python 3.11+, Pydantic v2, httpx, FastMCP for MCP servers.
+
+Rules:
+- Use civic_client() from civic_stack.shared.http, never raw httpx.AsyncClient
+- Return CivicStackResponse from civic_stack.shared.schema, never raw dicts
+- On errors: return error_response(), never raise in search()
+- All functions take proxy_url: str | None = None
+- Rate limit with RateLimiter from civic_stack.shared.http
+- Tests use VCR cassettes, never live portal calls
+- Line length 100, ruff formatting
+
+Read AGENTS.md for full patterns and CONTRIBUTING.md for module contract.

indonesia_civic_stack-0.1.0/.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,66 @@
+name: Bug report
+description: A module is returning wrong data, erroring unexpectedly, or behaving incorrectly.
+labels: ["bug"]
+body:
+  - type: dropdown
+    id: module
+    attributes:
+      label: Module
+      description: Which module is affected?
+      options:
+        - bpom
+        - bpjph
+        - ahu
+        - ojk
+        - oss-nib
+        - lpse
+        - kpu
+        - lhkpn
+        - bps
+        - bmkg
+        - simbg
+        - shared (schema / http / mcp)
+        - other
+    validations:
+      required: true
+
+  - type: textarea
+    id: description
+    attributes:
+      label: What happened?
+      description: Describe the bug. Include the query you ran, the response you got, and what you expected.
+      placeholder: |
+        I called `fetch("BPOM MD 123456789012")` and got status=ERROR.
+        Expected status=ACTIVE. The portal shows the product as active.
+    validations:
+      required: true
+
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: Steps to reproduce
+      placeholder: |
+        1. `pip install indonesia-civic-stack`
+        2. `await fetch("BPOM MD 123456789012")`
+        3. Response: {...}
+    validations:
+      required: true
+
+  - type: textarea
+    id: environment
+    attributes:
+      label: Environment
+      placeholder: |
+        - civic-stack version: 0.1.0
+        - Python: 3.11.9
+        - OS: Ubuntu 22.04
+        - proxy_url: yes/no
+    validations:
+      required: false
+
+  - type: checkboxes
+    id: portal_check
+    attributes:
+      label: Portal verification
+      options:
+        - label: I checked the live portal manually and the data is there / the portal is reachable

indonesia_civic_stack-0.1.0/.github/ISSUE_TEMPLATE/degraded_module.yml

@@ -0,0 +1,58 @@
+name: Degraded module report
+description: A module has stopped working because the portal changed. Use this to flag it for community fix.
+labels: ["degraded"]
+body:
+  - type: dropdown
+    id: module
+    attributes:
+      label: Affected module
+      options:
+        - bpom
+        - bpjph
+        - ahu
+        - ojk
+        - oss-nib
+        - lpse
+        - kpu
+        - lhkpn
+        - bps
+        - bmkg
+        - simbg
+    validations:
+      required: true
+
+  - type: textarea
+    id: symptom
+    attributes:
+      label: Symptom
+      description: What is failing? Which test or query broke?
+      placeholder: |
+        fetch("BPOM MD 123456789012") now returns status=ERROR.
+        CI cassette `found.yaml` replays fine but live portal returns 404.
+    validations:
+      required: true
+
+  - type: textarea
+    id: portal_change
+    attributes:
+      label: What changed on the portal?
+      description: If known — new URL, new HTML structure, added JS rendering, etc.
+      placeholder: "Portal moved from PHP to React SPA. Table selectors are now dynamic class names."
+    validations:
+      required: false
+
+  - type: input
+    id: first_seen
+    attributes:
+      label: When did this break?
+      placeholder: "e.g. 2026-03-10 (noticed in weekly CI run)"
+    validations:
+      required: false
+
+  - type: checkboxes
+    id: fix_interest
+    attributes:
+      label: Fix interest
+      options:
+        - label: I am willing to investigate and open a fix PR
+        - label: I can provide updated cassettes / fixtures once the portal change is understood

indonesia_civic_stack-0.1.0/.github/ISSUE_TEMPLATE/module_addition.yml

@@ -0,0 +1,81 @@
+name: New module proposal
+description: Propose adding a new Indonesian government data source as a civic-stack module.
+labels: ["module-proposal"]
+body:
+  - type: input
+    id: module_name
+    attributes:
+      label: Proposed module name
+      placeholder: "e.g. ojk, kpu, lhkpn"
+    validations:
+      required: true
+
+  - type: input
+    id: source_url
+    attributes:
+      label: Source portal URL
+      placeholder: "e.g. https://ojk.go.id"
+    validations:
+      required: true
+
+  - type: input
+    id: operator
+    attributes:
+      label: Portal operator (government agency)
+      placeholder: "e.g. Otoritas Jasa Keuangan (OJK)"
+    validations:
+      required: true
+
+  - type: dropdown
+    id: scrape_method
+    attributes:
+      label: Scrape method
+      options:
+        - httpx + BeautifulSoup (static HTML)
+        - Playwright (JS-rendered)
+        - REST API wrapper
+        - PDF extraction (pdfplumber / Claude Vision)
+        - Mixed
+    validations:
+      required: true
+
+  - type: dropdown
+    id: auth_required
+    attributes:
+      label: Authentication required?
+      options:
+        - "No — public search, no login"
+        - "Partial — public tier available, login for full data"
+        - "Yes — login required"
+    validations:
+      required: true
+
+  - type: textarea
+    id: data_fields
+    attributes:
+      label: Key data fields the module would expose
+      placeholder: |
+        - Institution name
+        - License number
+        - License status (active/revoked)
+        - Regulated products
+    validations:
+      required: true
+
+  - type: textarea
+    id: use_cases
+    attributes:
+      label: What Kah products or use cases does this enable?
+      placeholder: "e.g. legalkah.id financial license verification, AmanKah unlicensed fintech detection"
+    validations:
+      required: true
+
+  - type: checkboxes
+    id: commitment
+    attributes:
+      label: Contributor commitment
+      description: Check all that apply.
+      options:
+        - label: I am willing to implement this module (full contract — fetch, search, MCP, VCR fixtures, README)
+        - label: I can verify the portal works and document rate limits / block behaviour
+        - label: I can maintain this module for at least 6 months after merging

indonesia_civic_stack-0.1.0/.github/copilot-instructions.md

@@ -0,0 +1,23 @@
+# Copilot Instructions
+
+This is a Python SDK for Indonesian government data portals.
+
+## Key rules:
+- All functions are async. Use `async def` and `await`.
+- Always use `civic_client()` from `shared.http` — never raw httpx.
+- Every function must accept `proxy_url: str | None = None` parameter.
+- Return `CivicStackResponse` from `shared.schema` — never raw dicts.
+- On errors, return `error_response()` — never raise in `search()`.
+- Rate limit with `RateLimiter` from `shared.http`.
+- Tests use VCR cassettes — never hit live government portals.
+- Python 3.11+, Pydantic v2, ruff for formatting, line length 100.
+
+## File structure per module:
+- `scraper.py` — fetch() and search() functions
+- `normalizer.py` — raw HTML/JSON → dict transformations
+- `router.py` — FastAPI routes
+- `server.py` — MCP server using CivicStackMCPBase
+- `app.py` — FastAPI app
+- `README.md` — module docs
+
+See `AGENTS.md` for full architecture guide.

indonesia_civic_stack-0.1.0/.github/pull_request_template.md

@@ -0,0 +1,38 @@
+# PR Checklist
+
+## Type
+- [ ] New module
+- [ ] Bug fix (module scraper / normalizer)
+- [ ] Fix for degraded module (reference issue: #___)
+- [ ] Shared layer change (schema / http / mcp)
+- [ ] Docs / tests only
+
+## Module contract (required for new modules — skip for non-module PRs)
+
+- [ ] `fetch(query, *, debug, proxy_url) -> CivicStackResponse` implemented
+- [ ] `search(keyword, filters, *, proxy_url) -> list[CivicStackResponse]` implemented
+- [ ] MCP server class inheriting `CivicStackMCPBase` with `_register_tools()`
+- [ ] At least 3 VCR cassettes or HTML fixtures (`found`, `not_found`, `error`)
+- [ ] `modules/<name>/README.md` filled from `docs/module-spec.md` template
+- [ ] `modules/<name>/LICENSE` file added (MIT or Apache-2.0 per `LICENSES.md`)
+- [ ] `LICENSES.md` updated with new module entry
+
+## Quality
+
+- [ ] `ruff check .` passes
+- [ ] `ruff format --check .` passes
+- [ ] `mypy shared/ modules/<name>/` passes
+- [ ] `pytest tests/<name>/` passes
+- [ ] No live portal calls in any test (VCR `record_mode="none"` or monkeypatched Playwright)
+
+## Description
+
+<!-- What does this PR do? Why is it needed? -->
+
+## Test evidence
+
+<!-- Paste a `pytest -v tests/<name>/` run or link to CI run -->
+
+## Portal notes
+
+<!-- Any quirks discovered, rate limits tested, known block behaviour -->

indonesia_civic_stack-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,99 @@
+name: CI
+
+on:
+  push:
+    branches: [main, "claude/**", "feat/**", "fix/**"]
+  pull_request:
+    branches: [main]
+  schedule:
+    - cron: "0 2 * * 1"
+
+env:
+  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
+
+jobs:
+  lint:
+    name: Lint & type-check
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: pip install -e ".[dev,mcp]"
+      - run: ruff check civic_stack/ tests/
+      - run: ruff format --check civic_stack/ tests/
+      - run: mypy civic_stack/shared/
+
+  test:
+    name: Tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: pip install -e ".[dev]"
+      - name: Run tests (VCR replay, skip browser modules)
+        run: |
+          python -m pytest tests/ -v --tb=short \
+            --ignore=tests/ahu \
+            --ignore=tests/bpjph \
+            --ignore=tests/oss-nib
+        env:
+          VCR_RECORD_MODE: none
+
+  test-browser:
+    name: Tests (browser modules)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: pip install -e ".[dev,browser]"
+      - run: playwright install chromium --with-deps
+      - name: Run browser module tests
+        run: |
+          python -m pytest tests/ahu tests/bpjph tests/oss-nib -v --tb=short
+        env:
+          VCR_RECORD_MODE: none
+        continue-on-error: true  # Browser tests may fail due to geo-blocking
+
+  openapi:
+    name: Export OpenAPI spec
+    runs-on: ubuntu-latest
+    needs: [lint, test]
+    if: github.ref == 'refs/heads/main'
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: pip install -e ".[dev,api]"
+      - run: python scripts/export_openapi.py
+      - uses: actions/upload-artifact@v4
+        with:
+          name: openapi-spec
+          path: docs/openapi.json
+
+  live-portal-check:
+    name: Live portal regression check
+    runs-on: ubuntu-latest
+    if: github.event_name == 'schedule'
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: pip install -e ".[dev]"
+      - name: Run live integration tests
+        run: python scripts/test_module.py --live
+        env:
+          PROXY_URL: ${{ secrets.PROXY_URL }}
+        continue-on-error: true

indonesia_civic_stack-0.1.0/.gitignore

@@ -0,0 +1,5 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+*.egg-info/

indonesia_civic_stack-0.1.0/.mcp.json

@@ -0,0 +1,12 @@
+{
+  "mcpServers": {
+    "civic-stack": {
+      "command": ".venv/bin/python",
+      "args": ["-m", "civic_stack.server"],
+      "env": {
+        "PROXY_URL": "",
+        "BPS_API_KEY": ""
+      }
+    }
+  }
+}

indonesia_civic_stack-0.1.0/AGENTS.md

@@ -0,0 +1,166 @@
+# AGENTS.md — AI Agent Development Guide
+
+> This file helps AI coding agents (Claude Code, Codex, Cursor, etc.) work effectively in this repo.
+
+## Project Overview
+
+**indonesia-civic-stack** is a Python SDK + MCP server + REST API that wraps 11 Indonesian government data portals into a unified interface. Every module returns `CivicStackResponse`.
+
+## Architecture
+
+```
+indonesia-civic-stack/
+├── shared/              # Core shared code (DO NOT break these)
+│   ├── schema.py        # CivicStackResponse — the universal envelope
+│   ├── http.py          # civic_client(), fetch_with_retry(), proxy support
+│   └── mcp.py           # CivicStackMCPBase — base class for MCP servers
+├── modules/             # One directory per government portal
+│   ├── bpom/            # Example module (Food & Drug registry)
+│   │   ├── scraper.py   # fetch() + search() — core logic
+│   │   ├── normalizer.py # Raw HTML/JSON → structured dict
+│   │   ├── router.py    # FastAPI routes
+│   │   ├── server.py    # MCP server (FastMCP)
+│   │   ├── app.py       # FastAPI app entry point
+│   │   ├── Dockerfile
+│   │   └── README.md
+│   └── ... (11 modules total)
+├── proxy/               # CF Worker proxy for geo-blocked portals
+├── tests/               # VCR-based tests (no live portal calls in CI)
+└── app.py               # Unified FastAPI app (all modules)
+```
+
+## Key Patterns
+
+### Every module MUST follow this contract:
+
+```python
+# civic_stack/<name>/scraper.py
+async def fetch(query: str, *, proxy_url: str | None = None) -> CivicStackResponse:
+    """Single-record lookup by ID."""
+
+async def search(keyword: str, *, proxy_url: str | None = None) -> list[CivicStackResponse]:
+    """Multi-result keyword search. Returns [] on not-found, never raises."""
+```
+
+### HTTP client — always use civic_client():
+
+```python
+from civic_stack.shared.http import civic_client, fetch_with_retry
+
+async with civic_client(proxy_url=proxy_url) as client:
+    response = await fetch_with_retry(client, "GET", url, rate_limiter=_limiter)
+```
+
+`civic_client()` auto-reads `PROXY_URL` from environment. Never create raw `httpx.AsyncClient`.
+
+### MCP servers — two valid patterns:
+
+```python
+# Pattern 1: explicit init
+class BpomMCPServer(CivicStackMCPBase):
+    def __init__(self):
+        super().__init__("bpom")
+    def _register_tools(self): ...
+
+# Pattern 2: class attribute
+class BmkgMCPServer(CivicStackMCPBase):
+    module_name = "bmkg"
+    def _register_tools(self): ...
+```
+
+### Error handling — return envelopes, don't raise:
+
+```python
+from civic_stack.shared.schema import error_response, not_found_response
+
+# When portal is unreachable:
+return error_response("bpom", url, detail="Portal returned 404")
+
+# When no results found:
+return not_found_response("bpom", url)
+```
+
+## Critical Rules
+
+1. **Never break `shared/`** — all 11 modules depend on schema.py, http.py, mcp.py
+2. **Never hit live portals in tests** — use VCR cassettes (`tests/<module>/cassettes/`)
+3. **Always return CivicStackResponse** — never return raw dicts or raise on not-found
+4. **Rate limiting is mandatory** — every scraper must use `RateLimiter` from civic_stack.shared.http
+5. **Proxy support is mandatory** — every function takes `proxy_url` kwarg and passes to `civic_client()`
+6. **No data persistence** — modules fetch and return, no databases, no file writes
+
+## Running Tests
+
+```bash
+pytest -v                    # All tests (VCR replay, no live calls)
+pytest tests/bpom/ -v        # Single module
+ruff check .                 # Lint
+mypy shared/                 # Type check
+```
+
+## Adding a New Module
+
+1. Copy `civic_stack/bpom/` as template
+2. Implement `scraper.py` (fetch + search)
+3. Add `normalizer.py`, `router.py`, `server.py`, `app.py`
+4. Record VCR cassettes: `pytest --vcr-record=new_episodes`
+5. Add module to README table and `docker-compose.yml`
+6. See `CONTRIBUTING.md` for full checklist
+
+## Known Gotchas
+
+- **Geo-blocking**: Most .go.id portals block non-Indonesian IPs. Set `PROXY_URL` env var.
+- **Portal URL churn**: Indonesian gov portals change URLs without notice. Check issue tracker.
+- **Browser modules**: bpjph, ahu, oss_nib need Playwright. ahu also needs Camoufox.
+- **BPS needs API key**: Set `BPS_API_KEY` env var (free registration).
+- **LHKPN is degraded**: Portal moved behind auth. Module returns errors.
+- **CF Worker proxy has limits**: Can't proxy to CF-protected origins (403/522).
+
+## Environment Variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `PROXY_URL` | For non-ID deploys | Proxy URL (auto-detects *.workers.dev as rewrite mode) |
+| `PROXY_MODE` | No | Override: `connect` or `rewrite` |
+| `BPS_API_KEY` | For BPS module | Free key from webapi.bps.go.id |
+| `CIVIC_API_KEY` | For REST API auth | Set to enable API key checking |
+| `CIVIC_RATE_LIMIT` | No | Requests per minute (default: 60) |
+
+## Common Tasks (Copy-Paste Recipes)
+
+### "Add a new search parameter to an existing module"
+1. Read `civic_stack/<name>/scraper.py` — find the `search()` function
+2. Add the new parameter to the function signature (with default `None`)
+3. Pass it to `fetch_with_retry()` via `params=` dict
+4. Update `civic_stack/<name>/server.py` — add parameter to MCP tool
+5. Update `civic_stack/<name>/router.py` — add query param to FastAPI endpoint
+6. Record new VCR cassette: `pytest tests/<name>/ --vcr-record=new_episodes`
+
+### "Fix a broken portal URL"
+1. Check the portal manually: `curl -sI "https://portal.go.id/old-path"`
+2. Find the new URL (check portal homepage, inspect network tab)
+3. Update the URL constant at top of `civic_stack/<name>/scraper.py`
+4. Re-record VCR cassettes
+5. Update `civic_stack/<name>/README.md` with the URL change
+6. Check if other modules reference the same portal
+
+### "Add proxy support to a new function"
+1. Add `proxy_url: str | None = None` to function signature
+2. Pass to `civic_client(proxy_url=proxy_url)` — that's it
+3. `civic_client()` auto-reads `PROXY_URL` from env when `proxy_url=None`
+
+### "Debug a failing scraper"
+1. Check if it's a geo-block: `curl -sI "https://portal.go.id" | head -5`
+2. Check if URL changed: compare with `civic_stack/<name>/README.md`
+3. Check portal HTML structure: `curl -s "https://portal.go.id" | python -m bs4`
+4. Run with debug: `await fetch(query, debug=True)` — check `.raw` field in response
+5. Check GitHub issues for known portal changes
+
+## File Conventions
+
+- `scraper.py` — all HTTP logic lives here
+- `normalizer.py` — transforms raw HTML/JSON to structured dicts
+- `router.py` — FastAPI routes (thin, delegates to scraper)
+- `server.py` — MCP server (thin, delegates to scraper)
+- Module constants (URLs, rate limits) go at top of scraper.py
+- Use `MODULE = "module_name"` constant in every scraper

indonesia_civic_stack-0.1.0/CLAUDE.md

@@ -0,0 +1,67 @@
+# CLAUDE.md — Instructions for Claude Code
+
+## MCP Integration
+
+This repo includes `.mcp.json` — Claude Code auto-discovers 40 MCP tools on startup.
+The unified server (`server.py`) registers all tools from all 11 modules in one process.
+
+## Quick Context
+
+This is a Python SDK for Indonesian government data. 11 modules, each wrapping a gov portal.
+Shared layer in `shared/`. Tests use VCR (no live calls). All async, Pydantic v2, Python 3.11+.
+
+## Before Writing Code
+
+1. Read `AGENTS.md` for architecture and patterns
+2. Read `CONTRIBUTING.md` for the module contract
+3. Check the module's `README.md` for portal-specific quirks
+4. Check GitHub issues for known portal changes
+
+## Commands
+
+```bash
+# Test
+pytest -v
+pytest tests/<module>/ -v
+
+# Lint + format
+ruff check . && ruff format --check .
+
+# Type check
+mypy shared/ civic_stack/
+
+# Run unified MCP server (40 tools)
+python server.py
+
+# Run single module MCP server
+python -m civic_stack.bpom.server
+
+# Run single module REST API
+uvicorn modules.bpom.app:app --port 8001
+
+# Run unified REST API
+uvicorn app:app --port 8000
+
+# Build & deploy proxy
+cd proxy && npx wrangler deploy
+
+# Verify MCP tools load
+python -c "import asyncio; from server import mcp; print(asyncio.run(mcp.list_tools()))"
+```
+
+## Do Not
+
+- Create raw `httpx.AsyncClient` — use `civic_client()`
+- Raise exceptions from `search()` — return `error_response()` envelope
+- Hit live portals in tests — record VCR cassettes
+- Modify `shared/schema.py` without checking all 11 modules
+- Remove proxy_url parameter from any function signature
+- Use `print()` — use `logger` from `logging`
+
+## Style
+
+- Line length: 100
+- Async-first: `async def`, `httpx.AsyncClient`
+- Type hints on all public functions
+- Pydantic v2 models
+- `ruff` for linting and formatting