apoc-data 0.1.2__tar.gz → 0.2.0__tar.gz

apoc_data-0.2.0/PKG-INFO

@@ -0,0 +1,93 @@
+Metadata-Version: 2.3
+Name: apoc-data
+Version: 0.2.0
+Summary: Data from the Alaska Public Offices Commission.
+Author: Nick Crews
+Author-email: Nick Crews <nicholas.b.crews@gmail.com>
+License: MIT
+Requires-Dist: playwright ; extra == 'scrape'
+Requires-Python: >=3.9
+Provides-Extra: scrape
+Description-Content-Type: text/markdown
+
+# APOC Data
+
+Alaska campaign financial disclosure data from the [Alaska Public Offices Commission](https://aws.state.ak.us/ApocReports/Campaign/).
+
+This scrapes the CSV files from the APOC website once a day and uploads them to
+[this repo's releases](https://github.com/NickCrews/apoc-data/releases).
+
+---
+
+## Download a Recent Scrape
+
+You can download the daily-scraped CSVs from the GitHub releases (this is what most users want).
+
+### From the Web Interface
+
+Browse from [this repo's releases](https://github.com/NickCrews/apoc-data/releases).
+
+### From the CLI
+
+Using [uv](https://docs.astral.sh/uv/)'s `uvx`:
+
+```shell
+uvx apoc-data release download # downloads all files from the latest release to ./downloads/ folder
+uvx apoc-data release download "20260702-125614" --destination mydownloads/ # specify explicitly
+uvx apoc-data asset download debt.csv --destination apoc_debt.csv # download a single file
+uvx apoc-data release list # see what releases are available
+uvx apoc-data asset list --json # see what files are in the latest release, as JSON
+```
+
+Or, you can download these CSVs directly using the direct URLs from the releases page
+using curl, pandas, ibis, whatever!
+
+```bash
+curl -L https://github.com/NickCrews/apoc-data/releases/latest/download/candidate_registration.csv > candidate_registration.csv # get latest
+curl -L https://github.com/NickCrews/apoc-data/releases/download/20240716-025636/candidate_registration.csv > candidate_registration.csv # or a different url pattern for specific releases
+
+# query directly using duckdb
+duckdb -c "SELECT count(*) FROM 'https://github.com/NickCrews/apoc-data/releases/latest/download/candidate_registration.csv'"
+duckdb -c "SELECT count(*) FROM 'https://github.com/NickCrews/apoc-data/releases/download/20240716-025636/candidate_registration.csv'"
+```
+
+### From python
+
+We provide a python API too. `uv add apoc-data` and then
+
+```python
+from apoc_data.releases import asset_download, release_download, release_list
+
+release_list()  # all releases, newest first, as `Release` objects
+release_download(destination="downloads/")  # all files from the latest release
+asset_download("debt.csv", destination="apoc_debt.csv")  # a single file
+```
+
+---
+
+## Scrape Yourself
+
+You can also scrape fresh data directly from the APOC website
+(requires the `scrape` extra for playwright):
+
+```shell
+uvx "apoc-data[scrape]" scrape --directory scraped/
+```
+
+There is also a python API. Read the source code.
+
+---
+
+## Dev Notes
+
+Create venv and install dev deps:
+
+```shell
+uv sync
+```
+
+scrape:
+
+```shell
+uv run apoc-data scrape --directory downloads --no-headless
+```

apoc_data-0.2.0/README.md

@@ -0,0 +1,81 @@
+# APOC Data
+
+Alaska campaign financial disclosure data from the [Alaska Public Offices Commission](https://aws.state.ak.us/ApocReports/Campaign/).
+
+This scrapes the CSV files from the APOC website once a day and uploads them to
+[this repo's releases](https://github.com/NickCrews/apoc-data/releases).
+
+---
+
+## Download a Recent Scrape
+
+You can download the daily-scraped CSVs from the GitHub releases (this is what most users want).
+
+### From the Web Interface
+
+Browse from [this repo's releases](https://github.com/NickCrews/apoc-data/releases).
+
+### From the CLI
+
+Using [uv](https://docs.astral.sh/uv/)'s `uvx`:
+
+```shell
+uvx apoc-data release download # downloads all files from the latest release to ./downloads/ folder
+uvx apoc-data release download "20260702-125614" --destination mydownloads/ # specify explicitly
+uvx apoc-data asset download debt.csv --destination apoc_debt.csv # download a single file
+uvx apoc-data release list # see what releases are available
+uvx apoc-data asset list --json # see what files are in the latest release, as JSON
+```
+
+Or, you can download these CSVs directly using the direct URLs from the releases page
+using curl, pandas, ibis, whatever!
+
+```bash
+curl -L https://github.com/NickCrews/apoc-data/releases/latest/download/candidate_registration.csv > candidate_registration.csv # get latest
+curl -L https://github.com/NickCrews/apoc-data/releases/download/20240716-025636/candidate_registration.csv > candidate_registration.csv # or a different url pattern for specific releases
+
+# query directly using duckdb
+duckdb -c "SELECT count(*) FROM 'https://github.com/NickCrews/apoc-data/releases/latest/download/candidate_registration.csv'"
+duckdb -c "SELECT count(*) FROM 'https://github.com/NickCrews/apoc-data/releases/download/20240716-025636/candidate_registration.csv'"
+```
+
+### From python
+
+We provide a python API too. `uv add apoc-data` and then
+
+```python
+from apoc_data.releases import asset_download, release_download, release_list
+
+release_list()  # all releases, newest first, as `Release` objects
+release_download(destination="downloads/")  # all files from the latest release
+asset_download("debt.csv", destination="apoc_debt.csv")  # a single file
+```
+
+---
+
+## Scrape Yourself
+
+You can also scrape fresh data directly from the APOC website
+(requires the `scrape` extra for playwright):
+
+```shell
+uvx "apoc-data[scrape]" scrape --directory scraped/
+```
+
+There is also a python API. Read the source code.
+
+---
+
+## Dev Notes
+
+Create venv and install dev deps:
+
+```shell
+uv sync
+```
+
+scrape:
+
+```shell
+uv run apoc-data scrape --directory downloads --no-headless
+```

apoc_data-0.2.0/pyproject.toml

@@ -0,0 +1,29 @@
+[project]
+name = "apoc-data"
+version = "0.2.0"
+description = "Data from the Alaska Public Offices Commission."
+authors = [
+    {name = "Nick Crews", email = "nicholas.b.crews@gmail.com"},
+]
+dependencies = []
+requires-python = ">=3.9"
+readme = "README.md"
+license = {text = "MIT"}
+
+[project.scripts]
+apoc-data = "apoc_data.cli:main"
+
+[project.optional-dependencies]
+scrape = [
+    "playwright",
+]
+
+[dependency-groups]
+dev = [
+    "apoc-data[scrape]",
+    "ruff",
+]
+
+[build-system]
+requires = ["uv_build>=0.11,<0.12"]
+build-backend = "uv_build"

apoc_data-0.2.0/src/apoc_data/__main__.py

@@ -0,0 +1,4 @@
+from apoc_data.cli import main
+
+if __name__ == "__main__":
+    main()

apoc_data-0.2.0/src/apoc_data/cli.py

@@ -0,0 +1,261 @@
+"""Single CLI entry point for apoc-data.
+
+Subcommands:
+
+- ``release list|get|download``: inspect the available releases and download
+  all files in one (what most end users want; no extra dependencies).
+- ``asset list|download``: inspect and fetch individual files within a release.
+- ``scrape``: scrape the data from the APOC website using playwright
+  (requires installing the ``scrape`` extra, e.g. ``apoc-data[scrape]``).
+
+Usage:
+
+```shell
+uvx apoc-data release download
+uvx apoc-data release list --json
+uvx apoc-data asset list --release 20240716-025636
+uvx apoc-data asset download debt.csv --destination apoc_debt.csv
+uvx "apoc-data[scrape]" scrape --directory scraped/
+```
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+from pathlib import Path
+
+from apoc_data.releases import (
+    Asset,
+    Release,
+    asset_download,
+    asset_list,
+    release_download,
+    release_get,
+    release_list,
+)
+
+
+def _add_json_flag(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--json",
+        action="store_true",
+        help="Output JSON instead of human-readable text",
+    )
+
+
+def _add_release_flag(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--release",
+        type=str,
+        default="latest",
+        help='A release tag, or "latest" (the default)',
+    )
+
+
+def _human_size(size: int) -> str:
+    n = float(size)
+    for unit in ("B", "KB", "MB", "GB"):
+        if n < 1024 or unit == "GB":
+            return f"{n:.0f} {unit}" if unit == "B" else f"{n:.1f} {unit}"
+        n /= 1024
+    raise AssertionError("unreachable")
+
+
+def _print_release(release: Release) -> None:
+    print(f"tag:          {release.tag}")
+    print(f"name:         {release.name}")
+    print(f"url:          {release.url}")
+    print(f"published_at: {release.published_at.isoformat()}")
+    print("assets:")
+    for asset in release.assets:
+        print(f"  {asset.name} ({_human_size(asset.size)})")
+
+
+def _print_assets(assets: list[Asset]) -> None:
+    if not assets:
+        print("<no files>")
+        return
+    width = max(len(a.name) for a in assets)
+    for asset in assets:
+        print(f"{asset.name:<{width}}  {_human_size(asset.size):>9}  {asset.url}")
+
+
+def _run_release_list(args: argparse.Namespace) -> None:
+    releases = release_list()
+    if args.json:
+        print(json.dumps([r.to_dict() for r in releases], indent=2))
+        return
+    for release in releases:
+        n_assets = len(release.assets)
+        print(f"{release.tag}  {release.published_at.isoformat()}  {n_assets} assets")
+
+
+def _run_release_get(args: argparse.Namespace) -> None:
+    release = release_get(args.release)
+    if args.json:
+        print(json.dumps(release.to_dict(), indent=2))
+    else:
+        _print_release(release)
+
+
+def _run_asset_list(args: argparse.Namespace) -> None:
+    assets = asset_list(args.release)
+    if args.json:
+        print(json.dumps([a.to_dict() for a in assets], indent=2))
+    else:
+        _print_assets(assets)
+
+
+def _run_release_download(args: argparse.Namespace) -> None:
+    paths = release_download(args.release, destination=args.destination)
+    if args.json:
+        print(json.dumps([str(p) for p in paths], indent=2))
+    else:
+        for path in paths:
+            print(path)
+
+
+def _run_asset_download(args: argparse.Namespace) -> None:
+    path = asset_download(
+        args.filename, release=args.release, destination=args.destination
+    )
+    if args.json:
+        print(json.dumps(str(path)))
+    else:
+        print(path)
+
+
+def _run_scrape(args: argparse.Namespace) -> None:
+    try:
+        from apoc_data.scrape import scrape_all
+        from apoc_data.scrape._scraper import DEFAULT_DIRECTORY
+    except ImportError as e:
+        raise SystemExit(
+            "The scrape command requires extra dependencies. "
+            'Install them with the "scrape" extra, e.g. '
+            '`pip install "apoc-data[scrape]"` or `uvx "apoc-data[scrape]" scrape`.'
+            f"\n(import failed: {e})"
+        ) from e
+
+    directory = Path(args.directory or DEFAULT_DIRECTORY).absolute()
+    if directory.is_file():
+        raise ValueError("The directory can't be a file")
+    logging.basicConfig(level=logging.INFO)
+    scrape_all(directory, headless=args.headless)
+
+
+def main(argv: list[str] | None = None) -> None:
+    parser = argparse.ArgumentParser(
+        prog="apoc-data",
+        description="Data from the Alaska Public Offices Commission",
+    )
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    release_parser = subparsers.add_parser(
+        "release",
+        help="Inspect the available releases",
+    )
+    release_subparsers = release_parser.add_subparsers(
+        dest="release_command", required=True
+    )
+
+    release_list_parser = release_subparsers.add_parser(
+        "list",
+        help="List all releases, newest first",
+    )
+    _add_json_flag(release_list_parser)
+    release_list_parser.set_defaults(func=_run_release_list)
+
+    release_get_parser = release_subparsers.add_parser(
+        "get",
+        help="Show a single release",
+    )
+    release_get_parser.add_argument(
+        "release",
+        nargs="?",
+        default="latest",
+        help='A release tag, or "latest" (the default)',
+    )
+    _add_json_flag(release_get_parser)
+    release_get_parser.set_defaults(func=_run_release_get)
+
+    release_download_parser = release_subparsers.add_parser(
+        "download",
+        help="Download all files in a release to a folder",
+        description="Download all CSVs of APOC data from "
+        "https://github.com/NickCrews/apoc-data/releases",
+    )
+    release_download_parser.add_argument(
+        "release",
+        nargs="?",
+        default="latest",
+        help='A release tag, or "latest" (the default)',
+    )
+    release_download_parser.add_argument(
+        "--destination",
+        type=str,
+        default="downloads/",
+        help="The folder to save the files under (default: downloads/)",
+    )
+    _add_json_flag(release_download_parser)
+    release_download_parser.set_defaults(func=_run_release_download)
+
+    asset_parser = subparsers.add_parser(
+        "asset",
+        help="Inspect and download the files within a release",
+    )
+    asset_subparsers = asset_parser.add_subparsers(dest="asset_command", required=True)
+
+    asset_list_parser = asset_subparsers.add_parser(
+        "list",
+        help="List the files in a release",
+    )
+    _add_release_flag(asset_list_parser)
+    _add_json_flag(asset_list_parser)
+    asset_list_parser.set_defaults(func=_run_asset_list)
+
+    asset_download_parser = asset_subparsers.add_parser(
+        "download",
+        help="Download file(s) from a release",
+    )
+    asset_download_parser.add_argument(
+        "filename",
+        help="The name of the file to download, e.g. debt.csv",
+    )
+    _add_release_flag(asset_download_parser)
+    asset_download_parser.add_argument(
+        "--destination",
+        type=str,
+        default=None,
+        help="The file path to save to (default: the filename in the current directory)",
+    )
+    _add_json_flag(asset_download_parser)
+    asset_download_parser.set_defaults(func=_run_asset_download)
+
+    scrape_parser = subparsers.add_parser(
+        "scrape",
+        help='Scrape the data from the APOC website (requires the "scrape" extra)',
+        description="Scrape .CSVs from https://aws.state.ak.us/ApocReports/Campaign/",
+    )
+    scrape_parser.add_argument(
+        "--directory",
+        type=str,
+        default=None,
+        help="The directory to save the data to (default: scraped/)",
+    )
+    scrape_parser.add_argument(
+        "--headless",
+        default=True,
+        action=argparse.BooleanOptionalAction,
+        help="Run the browser in headless mode",
+    )
+    scrape_parser.set_defaults(func=_run_scrape)
+
+    args = parser.parse_args(argv)
+    args.func(args)
+
+
+if __name__ == "__main__":
+    main()

apoc_data-0.2.0/src/apoc_data/releases/__init__.py

@@ -0,0 +1,17 @@
+from apoc_data.releases._gh import Asset as Asset
+from apoc_data.releases._gh import Release as Release
+from apoc_data.releases._gh import asset_download as asset_download
+from apoc_data.releases._gh import asset_list as asset_list
+from apoc_data.releases._gh import release_download as release_download
+from apoc_data.releases._gh import release_get as release_get
+from apoc_data.releases._gh import release_list as release_list
+
+__all__ = [
+    "Asset",
+    "Release",
+    "asset_download",
+    "asset_list",
+    "release_download",
+    "release_get",
+    "release_list",
+]

apoc_data-0.2.0/src/apoc_data/releases/_gh.py

@@ -0,0 +1,213 @@
+"""Access the APOC data published at https://github.com/NickCrews/apoc-data/releases."""
+
+from __future__ import annotations
+
+import dataclasses
+import json
+import os
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+from urllib.error import HTTPError
+from urllib.request import Request, urlopen
+
+_REPO = "NickCrews/apoc-data"
+_API_ROOT = f"https://api.github.com/repos/{_REPO}"
+
+
+@dataclasses.dataclass(frozen=True)
+class Asset:
+    """A single downloadable file in a release."""
+
+    name: str
+    """The filename, e.g. ``candidate_registration.csv``."""
+    url: str
+    """The direct download URL."""
+    size: int
+    """The size in bytes."""
+    updated_at: datetime
+    """When the asset was last updated."""
+
+    @classmethod
+    def _from_api(cls, raw: dict[str, Any]) -> Asset:
+        return cls(
+            name=raw["name"],
+            url=raw["browser_download_url"],
+            size=raw["size"],
+            updated_at=_parse_timestamp(raw["updated_at"]),
+        )
+
+    def to_dict(self) -> dict[str, Any]:
+        """Convert to a JSON-serializable dict."""
+        return {
+            "name": self.name,
+            "url": self.url,
+            "size": self.size,
+            "updated_at": self.updated_at.isoformat(),
+        }
+
+    def download(self, destination: str | Path) -> Path:
+        """Download this asset to the given file path and return it."""
+        destination = Path(destination)
+        destination.parent.mkdir(parents=True, exist_ok=True)
+        destination.write_bytes(_get(self.url))
+        return destination
+
+
+@dataclasses.dataclass(frozen=True)
+class Release:
+    """A release of the APOC data on GitHub."""
+
+    tag: str
+    """The git tag, e.g. ``20240716-025636``."""
+    name: str
+    """The human-readable release title."""
+    url: str
+    """The web page for the release."""
+    published_at: datetime
+    """When the release was published."""
+    assets: tuple[Asset, ...]
+    """The downloadable files in this release."""
+
+    @classmethod
+    def _from_api(cls, raw: dict[str, Any]) -> Release:
+        return cls(
+            tag=raw["tag_name"],
+            name=raw["name"] or raw["tag_name"],
+            url=raw["html_url"],
+            published_at=_parse_timestamp(raw["published_at"]),
+            assets=tuple(Asset._from_api(a) for a in raw["assets"]),
+        )
+
+    def to_dict(self) -> dict[str, Any]:
+        """Convert to a JSON-serializable dict."""
+        return {
+            "tag": self.tag,
+            "name": self.name,
+            "url": self.url,
+            "published_at": self.published_at.isoformat(),
+            "assets": [a.to_dict() for a in self.assets],
+        }
+
+    def asset(self, filename: str) -> Asset:
+        """Get the asset with the given filename, or raise ValueError."""
+        for asset in self.assets:
+            if asset.name == filename:
+                return asset
+        available = ", ".join(sorted(a.name for a in self.assets)) or "<no files>"
+        raise ValueError(
+            f"Release {self.tag} does not have a file named {filename}. "
+            f"Available files: {available}"
+        )
+
+
+def release_list() -> list[Release]:
+    """List all releases of the APOC data, newest first."""
+    raw = json.loads(_get(f"{_API_ROOT}/releases"))
+    return [Release._from_api(r) for r in raw]
+
+
+def release_get(release: str = "latest") -> Release:
+    """Get a single release by tag, or the latest release.
+
+    Parameters
+    ----------
+    release :
+        A tag such as ``20240716-025636``, or ``latest`` for the most recent release.
+    """
+    if release == "latest":
+        url = f"{_API_ROOT}/releases/latest"
+    else:
+        url = f"{_API_ROOT}/releases/tags/{release}"
+    try:
+        raw = json.loads(_get(url))
+    except HTTPError as e:
+        if e.code == 404:
+            raise ValueError(
+                f"No release found for {release!r}. "
+                f"Available releases: {_available_releases_hint()}"
+            ) from e
+        raise
+    return Release._from_api(raw)
+
+
+def asset_list(release: str = "latest") -> list[Asset]:
+    """List the downloadable files in a release.
+
+    Parameters
+    ----------
+    release :
+        A tag such as ``20240716-025636``, or ``latest`` for the most recent release.
+    """
+    return list(release_get(release).assets)
+
+
+def release_download(
+    release: str = "latest",
+    *,
+    destination: str | Path = "downloads/",
+) -> list[Path]:
+    """Download all files in a release to a folder and return the downloaded paths.
+
+    Parameters
+    ----------
+    release :
+        A tag such as ``20240716-025636``, or ``latest`` for the most recent release.
+    destination :
+        The folder to save the files under.
+    """
+    destination = Path(destination)
+    rel = release_get(release)
+    return [asset.download(destination / asset.name) for asset in rel.assets]
+
+
+def asset_download(
+    filename: str,
+    *,
+    release: str = "latest",
+    destination: str | Path | None = None,
+) -> Path:
+    """Download a single file from a release and return the downloaded path.
+
+    Parameters
+    ----------
+    filename :
+        The name of the file to download, e.g. ``debt.csv``.
+    release :
+        A tag such as ``20240716-025636``, or ``latest`` for the most recent release.
+    destination :
+        The file path to save to.
+        Default is None, which saves to ``filename`` in the current directory.
+    """
+    asset = release_get(release).asset(filename)
+    if destination is None:
+        destination = Path(asset.name)
+    return asset.download(destination)
+
+
+def _available_releases_hint() -> str:
+    try:
+        tags = [r.tag for r in release_list()]
+    except Exception:
+        return "<unable to fetch releases>"
+    return ", ".join(["latest", *tags]) or "<no releases>"
+
+
+def _parse_timestamp(raw: str) -> datetime:
+    # GitHub timestamps look like "2024-07-16T02:56:36Z".
+    # datetime.fromisoformat can't parse the trailing "Z" until python 3.11.
+    return datetime.strptime(raw, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=timezone.utc)
+
+
+def _get(url: str) -> bytes:
+    # I'm getting hit by rate limits when using streamlit cloud, I assume because
+    # the IP address is shared. So I'm trying to use a personal access token to
+    # authenticate.
+    headers = {"Accept": "application/vnd.github.v3+json"}
+    try:
+        pat = os.environ["GITHUB_PAT"]
+        headers["Authorization"] = f"token {pat}"
+    except KeyError:
+        pass
+    with urlopen(Request(url, headers=headers)) as response:
+        return response.read()

{apoc_data-0.1.2 → apoc_data-0.2.0}/src/apoc_data/scrape/_filters.py

@@ -26,6 +26,8 @@ class YearEnum(str, Enum):
     _2024 = "2024"
     _2025 = "2025"
     _2026 = "2026"
+    _2027 = "2027"
+    _2028 = "2028"
 
     @classmethod
     def _missing_(cls, value):

{apoc_data-0.1.2 → apoc_data-0.2.0}/src/apoc_data/scrape/_scraper.py

@@ -13,10 +13,19 @@ from __future__ import annotations
 import asyncio
 import csv
 import logging
+import subprocess
+import sys
 import tempfile
 from contextlib import asynccontextmanager
 from pathlib import Path
-from typing import TYPE_CHECKING, AsyncIterable, ClassVar, Coroutine, Iterable, Protocol
+from typing import (
+    TYPE_CHECKING,
+    AsyncGenerator,
+    ClassVar,
+    Coroutine,
+    Iterable,
+    Protocol,
+)
 
 from playwright.async_api import BrowserContext, async_playwright, expect
 
@@ -30,9 +39,24 @@ _logger = logging.getLogger(__name__)
 DEFAULT_DIRECTORY = "scraped/"
 
 
+def _ensure_chromium_installed(executable_path: str) -> None:
+    """Install the playwright chromium browser if it isn't already.
+
+    This saves users from needing to run `playwright install chromium`
+    manually before their first scrape.
+    """
+    if Path(executable_path).exists():
+        return
+    _logger.info("Chromium not found. Running 'playwright install chromium'...")
+    subprocess.run(
+        [sys.executable, "-m", "playwright", "install", "chromium"], check=True
+    )
+
+
 @asynccontextmanager
-async def make_browser_async(headless: bool = True) -> AsyncIterable[BrowserContext]:
+async def make_browser_async(headless: bool = True) -> AsyncGenerator[BrowserContext]:
     async with async_playwright() as p:
+        _ensure_chromium_installed(p.chromium.executable_path)
         browser = await p.chromium.launch(
             headless=headless,
             # This sometimes avoids race conditions?
@@ -79,7 +103,7 @@ async def _run_scrape_flow(page: Page, url: str, filters: ScrapeFilters) -> Down
 
 
 class PScraper(Protocol):
-    def __call__(self, browser_context: BrowserContext) -> None:
+    async def __call__(self, browser_context: BrowserContext) -> None:
         """Given a browser context, scrape the data.
 
         The destination path, the chosen filters, etc all should be known
@@ -309,7 +333,7 @@ def scrape_all(
         If not provided, a temporary one will be created.
     """
     directory = Path(directory)
-    classes: list[_ScraperBase] = [
+    classes: list[type[_ScraperBase]] = [
         CampaignFormScraper,
         IncomeScraper,
         CandidateRegistrationScraper,

apoc_data-0.1.2/PKG-INFO

@@ -1,62 +0,0 @@
-Metadata-Version: 2.1
-Name: apoc-data
-Version: 0.1.2
-Summary: Data from the Alaska Public Offices Commission.
-Author-Email: Nick Crews <nicholas.b.crews@gmail.com>
-License: MIT
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-
-# APOC Data
-
-Data from the [Alaska Public Offices Commission](https://aws.state.ak.us/ApocReports/Campaign/).
-
-This scrapes the CSV files from the APOC website once a day and uploads them to
-[this repo's releases](https://github.com/NickCrews/apoc-data/releases).
-
-## Manual
-
-Browse from [this repo's releases](https://github.com/NickCrews/apoc-data/releases).
-
-## Python
-
-`pip install apoc-data` and then
-
-```python
-from apoc_data.download import download
-
-download(
-    release="latest",
-    filename="debt.csv",
-    destination="apoc_debt.csv",
-)
-```
-
-## Shell
-
-You can download these CSVs using the direct URLs from the releases page
-using curl, pandas, ibis, whatever!
-
-```bash
-curl -L https://github.com/NickCrews/apoc-data/releases/download/20240716-025636/candidate_registration.csv > candidate_registration.csv
-```
-
-or we have a tiny python script that makes this a little nicer, eg get the latest
-release, choose the download directory, etc. Read the script for more info.
-
-```bash
-curl -s https://raw.githubusercontent.com/NickCrews/apoc-data/main/src/apoc_data/download.py | python - --release latest
-```
-
-## Dev Notes
-
-```shell
-pdm install
-playwright install chromium
-```
-
-scrape:
-
-```shell
-python -m apoc_data.scrape --directory downloads --no-headless
-```

apoc_data-0.1.2/README.md

@@ -1,53 +0,0 @@
-# APOC Data
-
-Data from the [Alaska Public Offices Commission](https://aws.state.ak.us/ApocReports/Campaign/).
-
-This scrapes the CSV files from the APOC website once a day and uploads them to
-[this repo's releases](https://github.com/NickCrews/apoc-data/releases).
-
-## Manual
-
-Browse from [this repo's releases](https://github.com/NickCrews/apoc-data/releases).
-
-## Python
-
-`pip install apoc-data` and then
-
-```python
-from apoc_data.download import download
-
-download(
-    release="latest",
-    filename="debt.csv",
-    destination="apoc_debt.csv",
-)
-```
-
-## Shell
-
-You can download these CSVs using the direct URLs from the releases page
-using curl, pandas, ibis, whatever!
-
-```bash
-curl -L https://github.com/NickCrews/apoc-data/releases/download/20240716-025636/candidate_registration.csv > candidate_registration.csv
-```
-
-or we have a tiny python script that makes this a little nicer, eg get the latest
-release, choose the download directory, etc. Read the script for more info.
-
-```bash
-curl -s https://raw.githubusercontent.com/NickCrews/apoc-data/main/src/apoc_data/download.py | python - --release latest
-```
-
-## Dev Notes
-
-```shell
-pdm install
-playwright install chromium
-```
-
-scrape:
-
-```shell
-python -m apoc_data.scrape --directory downloads --no-headless
-```

apoc_data-0.1.2/pyproject.toml

@@ -1,28 +0,0 @@
-[project]
-name = "apoc-data"
-version = "0.1.2"
-description = "Data from the Alaska Public Offices Commission."
-authors = [
-    { name = "Nick Crews", email = "nicholas.b.crews@gmail.com" },
-]
-dependencies = []
-requires-python = ">=3.9"
-readme = "README.md"
-
-[project.license]
-text = "MIT"
-
-[tool.pdm]
-distribution = true
-
-[tool.pdm.dev-dependencies]
-dev = [
-    "playwright",
-    "ruff",
-]
-
-[build-system]
-requires = [
-    "pdm-backend",
-]
-build-backend = "pdm.backend"

apoc_data-0.1.2/src/apoc_data/download.py

@@ -1,111 +0,0 @@
-"""Download CSV(s) of APOC data from https://github.com/NickCrews/apoc-data/releases
-
-A no-install way to use this script is to download it from github and pipe to curl:
-
-```shell
-curl -s https://raw.githubusercontent.com/NickCrews/apoc-data/main/src/apoc_data/download.py | python - --release latest
-```
-"""
-
-import argparse
-import json
-import os
-from pathlib import Path
-from urllib.request import Request, urlopen
-
-
-def download(
-    *,
-    release: str = "latest",
-    filename: str | None = None,
-    destination: str | Path = "downloads/",
-) -> None:
-    """Download CSV(s) of APOC data from https://github.com/NickCrews/apoc-data/releases.
-
-    Parameters
-    ----------
-    release : str, optional
-        The name of the release to download.
-        Default is None, which means latest release
-    filename : str, optional
-        The name of the file to download.
-        Default is None, which downloads all files.
-    destination : str or Path, optional
-        Where to save the file(s).
-        If this looks like a file (the final path segment contains a `.`),
-        then we can only download a single file, and it will be saved to that location.
-        Otherwise, the file(s) will be saved underneath there.
-    """
-    destination = Path(destination)
-    release, assets = _get_release_info(release)
-    if filename is not None:
-        if filename not in assets:
-            raise ValueError(f"Release {release} does not have a file named {filename}")
-        if not _is_file(destination):
-            destination = destination / filename
-        _download_asset(assets[filename], destination)
-    else:
-        if _is_file(destination):
-            raise ValueError("Can't download all files to a single file")
-        for name, url in assets.items():
-            _download_asset(url, destination / name)
-
-
-def _is_file(destination: Path) -> bool:
-    return "." in destination.name
-
-
-def _get_release_info(release: str) -> tuple[str, dict[str, str]]:
-    url = f"https://api.github.com/repos/NickCrews/apoc-data/releases/{release}"
-    info = json.loads(_get(url))
-    assets = {asset["name"]: asset["browser_download_url"] for asset in info["assets"]}
-    return info["tag_name"], assets
-
-
-def _download_asset(url: str, destination: Path) -> None:
-    destination.parent.mkdir(parents=True, exist_ok=True)
-    with open(destination, "wb") as file:
-        file.write(_get(url))
-
-
-def cli():
-    parser = argparse.ArgumentParser(
-        description="Download data from the Alaska Public Offices Commission"
-    )
-    parser.add_argument(
-        "--release",
-        type=str,
-        default="latest",
-        help="The name of the release to download",
-    )
-    parser.add_argument(
-        "--filename",
-        type=str,
-        help="The name of the file to download",
-    )
-    parser.add_argument(
-        "--destination",
-        type=str,
-        default="downloads/",
-        help="Where to save the file(s)",
-    )
-    args = parser.parse_args()
-    download(release=args.release, filename=args.filename, destination=args.destination)
-
-
-def _get(url: str) -> str:
-    # I'm getting hit by rate limits when using streamlit cloud, I assume because
-    # the IP address is shared. So I'm trying to use a personal access token to
-    # authenticate.
-    headers = {"Accept": "application/vnd.github.v3+json"}
-    try:
-        pat = os.environ["GITHUB_PAT"]
-        headers["Authorization"] = f"toasdasken {pat}"
-    except KeyError:
-        pass
-    with urlopen(Request(url, headers=headers)) as response:
-        return response.read()
-
-
-if __name__ == "__main__":
-    cli()

apoc_data-0.1.2/src/apoc_data/scrape/__main__.py

@@ -1,47 +0,0 @@
-import argparse
-import logging
-from pathlib import Path
-
-from apoc_data.scrape import scrape_all
-from apoc_data.scrape._scraper import DEFAULT_DIRECTORY
-
-
-def main():
-    parser = argparse.ArgumentParser(
-        description="Download data from the Alaska Public Offices Commission"
-    )
-    parser.add_argument(
-        "--directory",
-        type=str,
-        default=DEFAULT_DIRECTORY,
-        help="The directory to save the data to",
-    )
-    parser.add_argument(
-        "--headless",
-        "--no-headless",
-        dest="headless",
-        default=True,
-        action=_BooleanAction,
-        help="Run the browser in headless mode",
-    )
-    args = parser.parse_args()
-    directory = Path(args.directory).absolute()
-    if directory.is_file():
-        raise ValueError("The directory can't be a file")
-    logging.basicConfig(level=logging.INFO)
-    scrape_all(directory, headless=args.headless)
-
-
-# from https://thisdataguy.com/2017/07/03/no-options-with-argparse-and-python/
-class _BooleanAction(argparse.Action):
-    def __init__(self, option_strings, dest, nargs=None, **kwargs):
-        super(_BooleanAction, self).__init__(option_strings, dest, nargs=0, **kwargs)
-
-    def __call__(self, parser, namespace, values, option_string=None):
-        setattr(
-            namespace, self.dest, False if option_string.startswith("--no") else True
-        )
-
-
-if __name__ == "__main__":
-    main()