openquery 0.1.0__tar.gz

openquery-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,32 @@
+---
+name: Bug Report
+about: Report a bug or unexpected behavior
+title: ""
+labels: bug
+assignees: ""
+---
+
+## Description
+
+A clear description of the bug.
+
+## Steps to Reproduce
+
+1. Run `openquery ...`
+2. ...
+3. See error
+
+## Expected Behavior
+
+What you expected to happen.
+
+## Actual Behavior
+
+What actually happened. Include error messages or output.
+
+## Environment
+
+- OpenQuery version: (`openquery --version`)
+- Python version:
+- OS:
+- Source queried (if applicable):

openquery-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,25 @@
+---
+name: Feature Request
+about: Suggest a new feature or data source
+title: ""
+labels: enhancement
+assignees: ""
+---
+
+## Description
+
+What feature or data source would you like?
+
+## Use Case
+
+Why is this useful? What problem does it solve?
+
+## Proposed Solution
+
+How should it work? (CLI usage, API response, etc.)
+
+## Additional Context
+
+- Source URL (if requesting a new data source):
+- Country/region:
+- Does it require CAPTCHA or authentication?

openquery-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,17 @@
+## Summary
+
+Brief description of changes.
+
+## Changes
+
+- ...
+
+## Testing
+
+- [ ] Tests pass (`uv run pytest`)
+- [ ] Linting passes (`uv run ruff check src/ tests/`)
+- [ ] New source includes both unit and integration tests (if applicable)
+
+## Related Issues
+
+Closes #

openquery-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,56 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest]
+        python-version: ["3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --all-extras
+
+      - name: Install system dependencies (Ubuntu)
+        if: runner.os == 'Linux'
+        run: sudo apt-get update && sudo apt-get install -y tesseract-ocr
+
+      - name: Install system dependencies (macOS)
+        if: runner.os == 'macOS'
+        run: brew install tesseract
+
+      - name: Install Playwright browsers
+        run: uv run playwright install --with-deps chromium
+
+      - name: Lint
+        run: uv run ruff check src/ tests/
+
+      - name: Test
+        run: uv run pytest --tb=short -q
+
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Build Docker image
+        run: docker build -t openquery:test .

openquery-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,43 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+
+      - name: Build package
+        run: uv build
+
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/project/openquery/
+    steps:
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

openquery-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.venv/
+.env
+.pytest_cache/
+.ruff_cache/
+*.db

openquery-0.1.0/.pre-commit-config.yaml

@@ -0,0 +1,17 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+        args: ["--maxkb=500"]
+      - id: check-merge-conflict
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.9.10
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format

openquery-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

openquery-0.1.0/CHANGELOG.md

@@ -0,0 +1,31 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-03-31
+
+### Added
+
+- Core framework with `BaseSource` plugin architecture
+- `BrowserManager` for Playwright-based scraping with WAF bypass
+- CAPTCHA solving: `OCRSolver` (pytesseract), `TwoCaptchaSolver`, `ChainedSolver`
+- Cache backends: in-memory (cachetools), Redis, SQLite
+- Per-source token-bucket rate limiter
+- Retry with exponential backoff
+- **co.simit** — Colombian traffic fines (SIMIT) via Playwright DOM scraping
+- **co.runt** — Colombian vehicle registry (RUNT) with CAPTCHA and Imperva WAF bypass
+- FastAPI REST API with `/api/v1/query`, `/api/v1/sources`, `/api/v1/health`
+- API key authentication middleware
+- Typer CLI: `openquery query`, `openquery sources`, `openquery serve`
+- Pydantic models for all response types
+- Configuration via environment variables (`OPENQUERY_*`)
+- Docker and docker-compose support with Redis
+- 29 unit tests
+
+[Unreleased]: https://github.com/dacrypt/openquery/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/dacrypt/openquery/releases/tag/v0.1.0

openquery-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,81 @@
+# Contributing to OpenQuery
+
+Thanks for your interest in contributing! This document provides guidelines to make the process smooth.
+
+## Development Setup
+
+```bash
+git clone https://github.com/dacrypt/openquery.git
+cd openquery
+uv sync --all-extras
+playwright install chromium
+```
+
+## Running Tests
+
+```bash
+# Unit tests
+uv run pytest
+
+# With coverage
+uv run pytest --cov=openquery
+
+# Integration tests (hits real external services)
+uv run pytest -m integration
+```
+
+## Code Quality
+
+```bash
+# Lint
+uv run ruff check src/ tests/
+
+# Auto-fix
+uv run ruff check --fix src/ tests/
+```
+
+## Adding a New Data Source
+
+This is the most common type of contribution. To add a new source:
+
+1. **Create the model** in `src/openquery/models/<country>/` with a Pydantic `BaseModel`
+2. **Create the source** in `src/openquery/sources/<country>/` implementing `BaseSource`
+3. **Register it** with the `@register` decorator
+4. **Add tests** in `tests/test_<source>.py`
+5. **Update the README** source table
+
+See [README.md](README.md#adding-a-new-source) for a complete example.
+
+### Source Guidelines
+
+- Use `BrowserManager` for browser automation — don't manage Playwright directly
+- Use `CaptchaSolver` for CAPTCHA handling — don't implement solving inline
+- Include a `SourceMeta` with accurate `rate_limit_rpm` to be respectful to servers
+- Return typed Pydantic models, not raw dicts
+- Add both unit tests (mocked) and integration tests (marked with `@pytest.mark.integration`)
+
+## Pull Requests
+
+1. Fork the repo and create a branch from `main`
+2. Make your changes
+3. Ensure tests pass: `uv run pytest`
+4. Ensure linting passes: `uv run ruff check src/ tests/`
+5. Write a clear PR description explaining what and why
+
+## Reporting Bugs
+
+Open an issue with:
+
+- Steps to reproduce
+- Expected vs actual behavior
+- OpenQuery version (`openquery --version`)
+- Python version and OS
+
+## Suggesting Sources
+
+If you know of a useful public data source, open an issue with:
+
+- Source name and URL
+- What data it provides
+- Whether it requires CAPTCHA or authentication
+- Country/region it covers

openquery-0.1.0/Dockerfile

@@ -0,0 +1,26 @@
+FROM python:3.12-slim
+
+# System dependencies for tesseract OCR and Playwright
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    tesseract-ocr \
+    libtesseract-dev \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install uv
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
+
+WORKDIR /app
+
+# Copy project files
+COPY pyproject.toml .
+COPY src/ src/
+
+# Install dependencies
+RUN uv pip install --system ".[serve]"
+
+# Install Playwright browsers
+RUN playwright install --with-deps chromium
+
+EXPOSE 8000
+
+CMD ["openquery", "serve", "--host", "0.0.0.0", "--port", "8000"]

openquery-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 dacrypt
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

openquery-0.1.0/PKG-INFO

@@ -0,0 +1,285 @@
+Metadata-Version: 2.4
+Name: openquery
+Version: 0.1.0
+Summary: Query public data sources worldwide via scraping and APIs
+Project-URL: Homepage, https://github.com/dacrypt/openquery
+Project-URL: Repository, https://github.com/dacrypt/openquery
+Project-URL: Issues, https://github.com/dacrypt/openquery/issues
+Project-URL: Changelog, https://github.com/dacrypt/openquery/blob/main/CHANGELOG.md
+Author-email: dacrypt <dev@dacrypt.dev>
+License: MIT
+License-File: LICENSE
+Keywords: api,captcha,government,playwright,public-data,scraping,web-scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: cachetools>=5.5
+Requires-Dist: httpx>=0.28
+Requires-Dist: pillow>=11.0
+Requires-Dist: playwright>=1.49
+Requires-Dist: pydantic-settings>=2.7
+Requires-Dist: pydantic>=2.10
+Requires-Dist: pytesseract>=0.3.13
+Requires-Dist: rich>=13.9
+Requires-Dist: typer>=0.15
+Provides-Extra: captcha
+Requires-Dist: 2captcha-python>=1.5; extra == 'captcha'
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest-httpx>=0.35; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.9; extra == 'dev'
+Provides-Extra: redis
+Requires-Dist: redis[hiredis]>=5.2; extra == 'redis'
+Provides-Extra: serve
+Requires-Dist: fastapi>=0.115; extra == 'serve'
+Requires-Dist: uvicorn[standard]>=0.34; extra == 'serve'
+Description-Content-Type: text/markdown
+
+# OpenQuery
+
+[![CI](https://github.com/dacrypt/openquery/actions/workflows/ci.yml/badge.svg)](https://github.com/dacrypt/openquery/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/openquery)](https://pypi.org/project/openquery/)
+[![Python](https://img.shields.io/pypi/pyversions/openquery)](https://pypi.org/project/openquery/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+Query public data sources worldwide through a unified CLI and REST API.
+
+OpenQuery provides a plugin-based framework for scraping government websites, public registries, and open data APIs. It handles the hard parts — browser automation, CAPTCHA solving, WAF bypass, caching, and rate limiting — so you can focus on the data.
+
+## Features
+
+- **Unified interface** — one CLI and one API endpoint for all data sources
+- **Browser automation** — Playwright-based scraping for JavaScript-heavy sites
+- **CAPTCHA solving** — local OCR (pytesseract) with optional paid service fallback
+- **WAF bypass** — browser-context API calls preserve session cookies
+- **Caching** — in-memory, Redis, or SQLite backends with configurable TTL
+- **Rate limiting** — per-source token-bucket to respect server limits
+- **REST API** — FastAPI server with auto-generated OpenAPI docs
+- **Extensible** — add new data sources by implementing a single class
+- **Country-organized** — sources grouped by country code (`co`, `us`, etc.)
+
+## Built-in Sources
+
+| Source | Country | Description | Inputs | CAPTCHA |
+|--------|---------|-------------|--------|---------|
+| `co.simit` | CO | Traffic fines and violations | cedula, placa | No |
+| `co.runt` | CO | National vehicle registry (SOAT, RTM, ownership) | vin, placa | Yes (OCR) |
+
+## Installation
+
+```bash
+pip install openquery
+```
+
+Or with [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv add openquery
+```
+
+### System Dependencies
+
+OpenQuery requires [Tesseract OCR](https://github.com/tesseract-ocr/tesseract) for CAPTCHA solving and Playwright browsers for web scraping:
+
+```bash
+# macOS
+brew install tesseract
+playwright install chromium
+
+# Ubuntu/Debian
+sudo apt-get install tesseract-ocr
+playwright install --with-deps chromium
+```
+
+### Optional Extras
+
+```bash
+pip install "openquery[serve]"    # FastAPI server (fastapi, uvicorn)
+pip install "openquery[redis]"    # Redis cache backend
+pip install "openquery[captcha]"  # 2captcha paid CAPTCHA solving
+```
+
+## Quick Start
+
+### CLI
+
+```bash
+# List available data sources
+openquery sources
+
+# Query Colombian traffic fines by cedula
+openquery query co.simit --cedula 12345678
+
+# Query Colombian vehicle registry by plate
+openquery query co.runt --placa ABC123
+
+# Query by VIN
+openquery query co.runt --vin 5YJ3E1EA1PF000001
+
+# Output raw JSON
+openquery query co.simit --cedula 12345678 --json
+```
+
+### REST API
+
+```bash
+# Start the API server
+openquery serve
+
+# Or with custom host/port
+openquery serve --host 127.0.0.1 --port 3000
+```
+
+Then query via HTTP:
+
+```bash
+curl -X POST http://localhost:8000/api/v1/query \
+  -H "Content-Type: application/json" \
+  -d '{
+    "source": "co.simit",
+    "document_type": "cedula",
+    "document_number": "12345678"
+  }'
+```
+
+**Response:**
+
+```json
+{
+  "ok": true,
+  "source": "co.simit",
+  "queried_at": "2026-03-31T10:30:00Z",
+  "cached": false,
+  "latency_ms": 4523,
+  "data": {
+    "comparendos": 0,
+    "multas": 0,
+    "total_deuda": 0.0,
+    "paz_y_salvo": true
+  }
+}
+```
+
+**API Endpoints:**
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `POST` | `/api/v1/query` | Query a data source |
+| `GET` | `/api/v1/sources` | List available sources |
+| `GET` | `/api/v1/health` | Health check and cache stats |
+| `GET` | `/docs` | Interactive API documentation |
+
+### Docker
+
+```bash
+docker compose up
+```
+
+This starts the API server with Redis caching on port 8000.
+
+## Configuration
+
+All settings use environment variables with the `OPENQUERY_` prefix:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `OPENQUERY_API_KEY` | _(none)_ | API key for server authentication |
+| `OPENQUERY_CACHE_BACKEND` | `memory` | Cache backend: `memory`, `redis`, `sqlite` |
+| `OPENQUERY_CACHE_TTL_DEFAULT` | `3600` | Default cache TTL in seconds |
+| `OPENQUERY_REDIS_URL` | `redis://localhost:6379/0` | Redis connection URL |
+| `OPENQUERY_BROWSER_HEADLESS` | `true` | Run browser in headless mode |
+| `OPENQUERY_BROWSER_TIMEOUT` | `30.0` | Browser operation timeout in seconds |
+| `OPENQUERY_RATE_LIMIT_DEFAULT_RPM` | `10` | Default requests per minute per source |
+| `OPENQUERY_CAPTCHA_SOLVER` | `ocr` | CAPTCHA solver: `ocr`, `2captcha`, `chained` |
+| `OPENQUERY_TWO_CAPTCHA_API_KEY` | _(none)_ | 2captcha.com API key |
+| `OPENQUERY_LOG_LEVEL` | `INFO` | Logging level |
+
+## Adding a New Source
+
+Create a new source by implementing the `BaseSource` class:
+
+```python
+# src/openquery/sources/us/nhtsa.py
+from pydantic import BaseModel
+from openquery.sources import register
+from openquery.sources.base import BaseSource, DocumentType, QueryInput, SourceMeta
+
+
+class NhtsaResult(BaseModel):
+    manufacturer: str = ""
+    model: str = ""
+    year: int = 0
+    recalls: list[dict] = []
+
+
+@register
+class NhtsaSource(BaseSource):
+    def meta(self) -> SourceMeta:
+        return SourceMeta(
+            name="us.nhtsa",
+            display_name="NHTSA Vehicle Safety",
+            description="US vehicle safety recalls and VIN decoding",
+            country="US",
+            url="https://vpic.nhtsa.dot.gov/api/",
+            supported_inputs=[DocumentType.VIN],
+            requires_captcha=False,
+            requires_browser=False,
+            rate_limit_rpm=30,
+        )
+
+    def query(self, input: QueryInput) -> NhtsaResult:
+        import httpx
+        resp = httpx.get(
+            f"https://vpic.nhtsa.dot.gov/api/vehicles/decodevin/{input.document_number}",
+            params={"format": "json"},
+        )
+        data = resp.json()
+        # Parse and return NhtsaResult...
+```
+
+The `@register` decorator automatically makes the source available in the CLI, API, and source listing.
+
+## Architecture
+
+```
+openquery/
+├── core/           # Infrastructure (browser, captcha, cache, rate limiting)
+├── sources/        # Data source plugins, organized by country
+│   ├── base.py     # BaseSource ABC — implement this to add sources
+│   ├── co/         # Colombia (SIMIT, RUNT)
+│   └── us/         # United States (future)
+├── models/         # Pydantic response models, organized by country
+├── server/         # FastAPI REST API
+└── commands/       # Typer CLI commands
+```
+
+## Development
+
+```bash
+git clone https://github.com/dacrypt/openquery.git
+cd openquery
+uv sync --all-extras
+playwright install chromium
+
+# Run tests
+uv run pytest
+
+# Lint
+uv run ruff check src/ tests/
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
+
+## License
+
+[MIT](LICENSE)