bdcdata 0.0.11__tar.gz → 2.0.0__tar.gz

bdcdata-2.0.0/.github/workflows/ci.yml

@@ -0,0 +1,50 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install ruff mypy pandas-stubs types-requests
+      - run: ruff check .
+      - run: ruff format --check .
+      - run: pip install -e .
+      - run: mypy
+
+  test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+        include:
+          - os: macos-latest
+            python-version: "3.12"
+          - os: windows-latest
+            python-version: "3.12"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          # hatch-vcs needs tags to compute a version
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[mobile,dotenv,progress]" pytest responses
+      # The offline suite must pass with no credentials configured. Credentials
+      # are deliberately absent here -- that is the point of the test.
+      - run: pytest -m "not integration" -v

bdcdata-2.0.0/.github/workflows/publish.yml

@@ -0,0 +1,37 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      # Required for PyPI Trusted Publishing -- no API token stored in the repo.
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

bdcdata-2.0.0/.gitignore

@@ -0,0 +1,24 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.venv/
+venv/
+
+# Tooling
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Credentials -- never commit these
+.env
+
+# Default bdcdata download cache (cwd-relative, see bdcdata.set_cache)
+bdc_cache/
+
+# OS
+.DS_Store

bdcdata-2.0.0/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/).
+
+## [2.0.0] — unreleased
+
+A ground-up rewrite. The submodule layout (`bdcdata.availability.fixed()`) is
+unchanged, but arguments, credential handling, and return types all changed.
+See [docs/migrating-from-v1.md](docs/migrating-from-v1.md).
+
+### Added
+
+- **Challenge data.** `challenges.fabric()`, `challenges.fixed()`,
+  `challenges.mobile()`, `challenges.verification()`, `challenges.audit()`, and
+  `challenges.get()` for categories added after this release. These were
+  import-only stubs in 1.x.
+- **Funding data.** `funding.programs()`, `funding.projects()`,
+  `funding.unserved_unfunded()`, `funding.funded_locations()`,
+  `funding.projects_in_geography()`, `funding.readmes()`, and
+  `funding.download_readme()`. Also stubs in 1.x.
+- **Mobile availability.** `availability.mobile()` reads the H3 coverage
+  attribute table without requiring geopandas, via the optional
+  `bdcdata[mobile]` extra.
+- **Served/unserved.** `availability.served_unserved()`, covering the export
+  added in the 2026-08-11 specification revision.
+- **Summary tables.** `availability.provider_list()`,
+  `availability.provider_summary()`, and
+  `availability.summary_by_geography()`.
+- **A browsable catalog.** `bdcdata.catalog` wraps `listAsOfDates`,
+  `listAvailabilityData`, `listChallengeData`, `listFundingData`,
+  `listReadmeFiles`, and `listGeographyData`.
+- **Reference tables.** `bdcdata.lookups.states()`,
+  `lookups.technologies()`, and `lookups.challenge_categories()`.
+- **Friendly identifiers.** `state=` accepts FIPS codes, USPS abbreviations,
+  and full names; `technology=` accepts codes, slugs, descriptions, aliases
+  (`"dsl"`, `"fttp"`, `"lte"`), and groups (`"wired"`, `"satellite"`,
+  `"terrestrial"`, `"wireless"`).
+- **Typed exceptions.** `BdcError` and its subclasses, replacing bare
+  `Exception`. Every message says what to do next.
+- **Credential helpers.** `set_credentials()`, `load_dotenv()`,
+  `have_credentials()`, `check_credentials()`.
+- **Cache controls.** `set_cache()`, `cache_info()`, `clear_cache()`.
+- Type hints throughout, with a `py.typed` marker.
+
+### Changed
+
+- **`import bdcdata` has no side effects.** No network call, no `.env` read, no
+  logging configuration. Credentials resolve on first request.
+- **Logging no longer hijacks the root logger.** A `NullHandler` on the
+  `bdcdata` logger, nothing more. No `basicConfig()`, no `StreamHandler`, no
+  `bdc.log` file.
+- **Base URL is now `https://bdc.fcc.gov`**, matching the current FCC
+  specification. Override with `set_base_url()`.
+- **`states=` is now `state=`** (singular), and accepts far more spellings.
+- **`release` defaults to `"latest"`** instead of a hardcoded date, and
+  resolves separately for availability and challenge data.
+- **Identifier columns are strings**: `location_id`, `block_geoid`, `frn`,
+  `h3_res8_id`, `h3_res9_id`, `state_fips`.
+- **Cache is off by default**, lives in `./bdc_cache`, is keyed by request URL
+  rather than filename, and is configured globally via `set_cache()`.
+- **`release` and `state_fips` columns are added** to downloaded frames, so
+  multi-state and multi-release pulls stay separable.
+- Validation happens before any network request.
+- Multi-file downloads concatenate once instead of inside the loop.
+
+### Fixed
+
+- **`business_residential_code` dtype hint.** 1.x misspelled it
+  (`business_residental_code`), so the hint silently never applied.
+- **`location_id` overflow.** 1.x typed it `UInt32` (max 4,294,967,295) while
+  the FCC specification defines it as `String{13}`. Now a string.
+- **Challenge downloads were unreachable.** 1.x hardcoded `availability` as the
+  `data_type` path segment, so no challenge file could be fetched.
+- **Archive member selection.** 1.x read `zip.filelist[0]` blindly; sidecar
+  entries (`__MACOSX`, readmes) are now skipped and CSVs preferred.
+
+### Removed
+
+- **`fabric` module.** Its two functions did nothing — one printed a message,
+  the other read a path nothing wrote. Fabric is licensed data on a different
+  access path, so it is out of scope rather than present and broken.
+- `bdcdata.echo()` and the `main()` entry point.
+- Module-level `apiKey`, `username`, `session`, and `metadata` globals.
+- The `requests-ratelimiter` dependency, replaced by a built-in limiter that
+  also handles retries and `Retry-After`.
+
+### Dependencies
+
+- Core: `requests`, `pandas>=2`, `pyarrow`.
+- Optional: `bdcdata[mobile]` (pyogrio), `bdcdata[dotenv]` (python-dotenv),
+  `bdcdata[progress]` (tqdm).
+- Requires Python 3.10+.
+
+---
+
+## [1.x]
+
+See the [pre-2.0 history](https://github.com/npappin/bdcdata/commits/main).
+Supported fixed availability by state and technology; challenge, funding, and
+fabric modules were stubs.

{bdcdata-0.0.11 → bdcdata-2.0.0}/LICENSE.txt

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2025, W. Nick Pappin
+Copyright (c) 2026 W. Nick Pappin
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

bdcdata-2.0.0/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.5
+Name: bdcdata
+Version: 2.0.0
+Summary: Work with FCC Broadband Data Collection (BDC) data in pandas.
+Project-URL: Homepage, https://github.com/npappin/bdcdata
+Project-URL: Documentation, https://github.com/npappin/bdcdata/tree/main/docs
+Project-URL: Issues, https://github.com/npappin/bdcdata/issues
+Author-email: "W. Nick Pappin" <nick.pappin@wsu.edu>
+Maintainer-email: "W. Nick Pappin" <npappin@gmail.com>
+License-Expression: MIT
+License-File: LICENSE.txt
+Keywords: bdc,broadband,fcc,national broadband map,pandas
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: pandas>=2
+Requires-Dist: pyarrow>=12
+Requires-Dist: python-dotenv
+Requires-Dist: requests>=2.28
+Provides-Extra: all
+Requires-Dist: pyogrio>=0.7; extra == 'all'
+Requires-Dist: tqdm>=4.60; extra == 'all'
+Provides-Extra: mobile
+Requires-Dist: pyogrio>=0.7; extra == 'mobile'
+Provides-Extra: progress
+Requires-Dist: tqdm>=4.60; extra == 'progress'
+Description-Content-Type: text/markdown
+
+# bdcdata
+
+Work with FCC **Broadband Data Collection** (BDC) data in pandas.
+
+`bdcdata` handles the parts of the National Broadband Map that are tedious to
+get right — authentication, the 10-calls-per-minute rate limit, finding the
+right file among thousands, unzipping it, and applying the column types from
+the FCC's published specification — and hands you a DataFrame.
+
+```python
+import bdcdata
+
+bdcdata.set_credentials(username="you@example.com", token="...")
+
+df = bdcdata.availability.fixed(state="WA", technology="fiber")
+```
+
+You can write `state="WA"`, `state="Washington"`, or `state=53`, and
+`technology="fiber"`, `technology="fttp"`, or `technology=50`. They all mean
+the same thing.
+
+## Install
+
+```bash
+pip install bdcdata
+```
+
+Reading the mobile H3 coverage files means opening a shapefile, which needs an
+extra:
+
+```bash
+pip install 'bdcdata[mobile]'
+```
+
+## Credentials
+
+Every BDC endpoint requires an FCC username and API token, including the
+metadata endpoints.
+
+1. Log in at <https://broadbandmap.fcc.gov/login> with your FCC User
+   Registration account.
+2. Click your username in the top right, then **Manage API Access**.
+3. Click **Generate**, accept the terms, and copy the token.
+
+Your username is the email address on the account. Then pick whichever of
+these suits you:
+
+```python
+bdcdata.set_credentials(username="you@example.com", token="...")
+```
+
+```bash
+export BDC_USERNAME=you@example.com
+export BDC_API_KEY=...
+```
+
+```python
+# .env file in your working directory, with BDC_USERNAME and BDC_API_KEY
+bdcdata.load_dotenv()
+```
+
+Check them with `bdcdata.check_credentials()`.
+
+## Documentation
+
+See the [`docs/`](docs/) directory.
+
+## Upgrading from 1.x
+
+**2.0 is a rewrite and the API changed.** See
+[docs/migrating-from-v1.md](docs/migrating-from-v1.md).
+
+## License
+
+MIT

bdcdata-2.0.0/README.md

@@ -0,0 +1,75 @@
+# bdcdata
+
+Work with FCC **Broadband Data Collection** (BDC) data in pandas.
+
+`bdcdata` handles the parts of the National Broadband Map that are tedious to
+get right — authentication, the 10-calls-per-minute rate limit, finding the
+right file among thousands, unzipping it, and applying the column types from
+the FCC's published specification — and hands you a DataFrame.
+
+```python
+import bdcdata
+
+bdcdata.set_credentials(username="you@example.com", token="...")
+
+df = bdcdata.availability.fixed(state="WA", technology="fiber")
+```
+
+You can write `state="WA"`, `state="Washington"`, or `state=53`, and
+`technology="fiber"`, `technology="fttp"`, or `technology=50`. They all mean
+the same thing.
+
+## Install
+
+```bash
+pip install bdcdata
+```
+
+Reading the mobile H3 coverage files means opening a shapefile, which needs an
+extra:
+
+```bash
+pip install 'bdcdata[mobile]'
+```
+
+## Credentials
+
+Every BDC endpoint requires an FCC username and API token, including the
+metadata endpoints.
+
+1. Log in at <https://broadbandmap.fcc.gov/login> with your FCC User
+   Registration account.
+2. Click your username in the top right, then **Manage API Access**.
+3. Click **Generate**, accept the terms, and copy the token.
+
+Your username is the email address on the account. Then pick whichever of
+these suits you:
+
+```python
+bdcdata.set_credentials(username="you@example.com", token="...")
+```
+
+```bash
+export BDC_USERNAME=you@example.com
+export BDC_API_KEY=...
+```
+
+```python
+# .env file in your working directory, with BDC_USERNAME and BDC_API_KEY
+bdcdata.load_dotenv()
+```
+
+Check them with `bdcdata.check_credentials()`.
+
+## Documentation
+
+See the [`docs/`](docs/) directory.
+
+## Upgrading from 1.x
+
+**2.0 is a rewrite and the API changed.** See
+[docs/migrating-from-v1.md](docs/migrating-from-v1.md).
+
+## License
+
+MIT

bdcdata-2.0.0/docs/availability.md

@@ -0,0 +1,201 @@
+# Availability
+
+Who reports offering broadband service, and where.
+
+```python
+import bdcdata
+
+df = bdcdata.availability.fixed(state="WA", technology="fiber")
+```
+
+## `fixed()` — service by location
+
+The core dataset. One row per location, per provider, per technology: what each
+provider reports offering at each Broadband Serviceable Location.
+
+```python
+bdcdata.availability.fixed(state, technology="all", release="latest")
+```
+
+| Column | Type | Notes |
+|---|---|---|
+| `frn` | string | 10-digit FCC Registration Number, leading zeros kept |
+| `provider_id` | int | Join to `provider_list()` |
+| `brand_name` | string | The name consumers see |
+| `location_id` | string | Fabric location ID — **string**, see below |
+| `technology` | int | See `lookups.technologies("fixed")` |
+| `max_advertised_download_speed` | int | Mbps |
+| `max_advertised_upload_speed` | int | Mbps |
+| `low_latency` | bool | ≤100 ms round trip at the 95th percentile |
+| `business_residential_code` | string | `B`, `R`, or `X` (both) |
+| `state_usps` | string | Two-letter abbreviation |
+| `block_geoid` | string | 15-digit census block, leading zeros kept |
+| `h3_res8_id` | string | H3 resolution-8 cell |
+| `release` | string | Added by bdcdata |
+| `state_fips` | string | Added by bdcdata |
+
+`location_id` and `block_geoid` are strings on purpose. A `block_geoid` like
+`011010106033002` loses its meaning as a number, and a 13-digit `location_id`
+exceeds what a 32-bit integer can hold. Join on them as text.
+
+### Examples
+
+```python
+# One state, one technology
+bdcdata.availability.fixed(state="WA", technology="fiber")
+
+# Several states
+bdcdata.availability.fixed(state=["WA", "OR", "ID"], technology="fiber")
+
+# All wired technologies (copper, cable, fiber)
+bdcdata.availability.fixed(state="WA", technology="wired")
+
+# Everything for one state — large
+bdcdata.availability.fixed(state="WA", technology="all")
+
+# A past release
+bdcdata.availability.fixed(state="WA", technology="fiber", release="2023-12-31")
+
+# Compare two releases in one frame
+bdcdata.availability.fixed(state="WA", technology="fiber", release=["2023-12-31", "2024-06-30"])
+```
+
+The `release` column is what makes the last one useful:
+
+```python
+df.groupby("release")["location_id"].nunique()
+```
+
+### Technology groups
+
+Instead of listing codes, you can name a group:
+
+| Group | Codes | Meaning |
+|---|---|---|
+| `"all"` | every fixed code | |
+| `"wired"` | 10, 40, 50 | Copper, cable, fiber — the FCC's own definition |
+| `"terrestrial"` | everything but 60, 61 | All non-satellite |
+| `"satellite"` | 60, 61 | Geostationary and non-geostationary |
+| `"wireless"` | 70, 71, 72 | Fixed wireless |
+
+`"wired"` and `"terrestrial"` match the definitions the FCC uses in the
+served/unserved file, so the groupings line up across datasets.
+
+## `served_unserved()` — the 100/20 question
+
+For **every** location in the Fabric — not just those with service — whether
+any provider reported at least 100 Mbps down / 20 Mbps up.
+
+```python
+df = bdcdata.availability.served_unserved(state="WA")
+```
+
+| Column | Type |
+|---|---|
+| `location_id` | string |
+| `block_geoid` | string |
+| `h3_res8_id` | string |
+| `any_dl100_ul20` | bool |
+| `wired_dl100_ul20` | bool |
+| `terrestrial_dl100_ul20` | bool |
+
+The three flags are real booleans, so they aggregate directly:
+
+```python
+# Unserved locations by census block
+unserved = df[~df["any_dl100_ul20"]]
+unserved.groupby("block_geoid").size().sort_values(ascending=False)
+
+# Share served, statewide
+df["any_dl100_ul20"].mean()
+```
+
+This export was added in the August 2026 revision of the download
+specification, so it may not exist for older releases. If it doesn't, you get
+an empty DataFrame and a warning saying why.
+
+## `mobile()` — mobile coverage by H3 cell
+
+```python
+df = bdcdata.availability.mobile(state="WA", technology="5g")
+```
+
+Needs `pip install 'bdcdata[mobile]'`.
+
+The FCC publishes these as GIS files (shapefile or GeoPackage), not CSV.
+bdcdata reads the **attribute table only** and skips the geometry, so you get
+one row per H3 resolution-9 cell without geopandas in your dependency tree.
+
+| Column | Type | Notes |
+|---|---|---|
+| `technology` | int | 300 = 3G, 400 = 4G LTE, 500 = 5G-NR |
+| `mindown` | float | Minimum modeled download, Mbps |
+| `minup` | float | Minimum modeled upload, Mbps |
+| `environmnt` | int | 0 = outdoor stationary only, 1 = also in-vehicle |
+| `h3_res9_id` | string | H3 resolution-9 cell |
+
+`environmnt` is spelled that way in the FCC's files. It's kept as published
+rather than silently corrected, so what you see matches the specification.
+
+To map it, join `h3_res9_id` to hexagon geometry with the
+[h3](https://pypi.org/project/h3/) package:
+
+```python
+import h3
+
+df["boundary"] = df["h3_res9_id"].map(lambda cell: h3.cell_to_boundary(cell))
+```
+
+**Mobile technology codes are a separate namespace from fixed ones.** Code `0`
+means "Other" for fixed and "Mobile Voice" for mobile. Passing `"fiber"` to
+`mobile()` is an error, not a silent empty result.
+
+## Summary tables
+
+Smaller aggregates, when you don't need location-level detail.
+
+```python
+# Providers that submitted data — join target for provider_id
+bdcdata.availability.provider_list()
+
+# Per-provider totals
+bdcdata.availability.provider_summary(kind="fixed")  # location and unit counts
+bdcdata.availability.provider_summary(kind="mobile")  # covered area in sq km
+
+# Coverage percentages by geography, across all providers
+bdcdata.availability.summary_by_geography(kind="fixed", geography="place")
+bdcdata.availability.summary_by_geography(kind="fixed", geography="other")
+```
+
+`geography="place"` is census places; `geography="other"` is everything else
+(state, county, congressional district, tribal area, CBSA). The FCC split these
+into separate exports in June 2024.
+
+## Size
+
+An availability pull can be very large. A single state's fiber file is tens of
+megabytes; `state="all", technology="all"` is many gigabytes and will likely
+exhaust memory.
+
+bdcdata sums the catalog's `record_count` before downloading and warns you when
+a request is about to load millions of rows. To see what you're asking for
+first:
+
+```python
+files = bdcdata.catalog.availability_files(
+    release="latest", category="State", subcategory="Location Coverage"
+)
+files["record_count"].sum()
+```
+
+If you need everything, loop a state at a time and write each to Parquet rather
+than holding it all in memory:
+
+```python
+for state in bdcdata.lookups.states()["usps"]:
+    df = bdcdata.availability.fixed(state=state, technology="fiber")
+    df.to_parquet(f"fiber_{state}.parquet")
+```
+
+With `bdcdata.set_cache(True)`, a loop like that is resumable — re-running
+skips anything already downloaded.