community-streamflow-service 0.1.0__tar.gz

community_streamflow_service-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,45 @@
+---
+name: Bug report
+about: Report a defect in CSFS
+title: ""
+labels: bug
+assignees: ""
+---
+
+**Describe the bug**
+
+A clear and concise description of what went wrong.
+
+**To reproduce**
+
+```bash
+# Exact command(s) or minimal Python snippet
+csfs fetch -p usgs --lookback 24
+```
+
+**Expected behavior**
+
+What you expected to happen.
+
+**Actual behavior**
+
+What actually happened. Include the full traceback or error output if there is one.
+
+**Environment**
+
+- CSFS version (`pip show csfs` or `csfs --version`):
+- Python version:
+- OS:
+- Install method (PyPI / source):
+
+**Provider-related?**
+
+If the bug involves a specific connector, name the provider slug (e.g. `usgs`,
+`france_hubeau`). Note that live-provider commands can fail because the
+*upstream* agency API is down — please re-run once before filing, and say
+whether the failure is reproducible.
+
+**Additional context**
+
+Anything else that helps (config file excerpt with secrets removed, database
+size, scheduler tier, etc.).

community_streamflow_service-0.1.0/.github/ISSUE_TEMPLATE/provider_request.md

@@ -0,0 +1,39 @@
+---
+name: Provider request
+about: Propose a new streamflow agency, network, or dataset for CSFS
+title: "Provider request: "
+labels: provider, enhancement
+assignees: ""
+---
+
+**Provider**
+
+- Agency / network / dataset name:
+- Country or region covered:
+- Website:
+
+**Data access**
+
+- API or download URL (if known):
+- Access model: open / registration / API key / restricted / no API (scraping or manual download)
+- Data format(s): JSON / CSV / WaterML / NetCDF / other
+- Realtime, recent, or archive-only?
+- Approximate number of discharge stations:
+- License / terms of use (link if possible):
+
+**Why it matters**
+
+What gap does this provider fill (region, density, record length, realtime
+latency)?
+
+**Checklist for whoever implements it**
+
+A new connector must satisfy the roster-integrity guards
+(`tests/test_connector_integrity.py`):
+
+- [ ] Connector module in `src/csfs/connectors/` registered with `@register("slug")`
+- [ ] Entry in `inventory/providers.yaml` (only `status: implemented` once the connector is registered)
+- [ ] Exactly one cron tier in `PROVIDER_TIERS` (`src/csfs/scheduler/cron.py`)
+- [ ] Hermetic test in `tests/connectors/test_<slug>.py` (mocked HTTP; no live calls)
+
+See [CONTRIBUTING.md](../../CONTRIBUTING.md) for the full walkthrough.

community_streamflow_service-0.1.0/.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,35 @@
+## Summary
+
+What does this PR change, and why?
+
+## Type of change
+
+- [ ] New provider connector
+- [ ] Bug fix
+- [ ] Core / store / scheduler change
+- [ ] Documentation
+- [ ] Other (describe)
+
+## Checklist
+
+- [ ] `ruff check src/ tests/` passes
+- [ ] `mypy src/csfs/` passes
+- [ ] `pytest -m "not network"` passes (the suite is hermetic — no test may
+      reach a real upstream unless marked `@pytest.mark.network`)
+
+### For new connectors (roster-integrity guards)
+
+`tests/test_connector_integrity.py` will fail CI unless all of these hold:
+
+- [ ] Class metadata complete: `slug` (matches the registry key),
+      `display_name`, `base_url`, `country_codes`
+- [ ] Slug added to **exactly one** tier in `PROVIDER_TIERS`
+      (`src/csfs/scheduler/cron.py`)
+- [ ] Entry added to `inventory/providers.yaml` with an honest `status`
+- [ ] Hermetic test `tests/connectors/test_<slug>.py` with mocked HTTP
+      (units normalized to m³/s, timestamps to UTC, error paths covered)
+
+## Notes for the reviewer
+
+Anything that needs special attention (upstream quirks, rate limits,
+licensing of the source data, etc.).

community_streamflow_service-0.1.0/.github/workflows/acquire.yml

@@ -0,0 +1,144 @@
+name: Acquire
+
+# Scheduled streamflow acquisition on GitHub Actions.
+#
+# The canonical DuckDB store is kept as an asset on a single "db-snapshot"
+# GitHub Release. Each run downloads it, fetches one tier, re-uploads it, then
+# checks health for that tier and alerts (via a GitHub issue) on failure.
+#
+# NOTE: only the daily and weekly tiers run on schedule here. The realtime
+# (15-min) and hourly tiers would re-upload the ~700 MB DB dozens of times a
+# day — impractical on Actions; run those on a long-lived host instead. You can
+# still trigger any tier manually via "Run workflow".
+
+on:
+  schedule:
+    - cron: "30 2 * * *"   # daily  02:30 UTC
+    - cron: "0 3 * * 0"    # weekly Sun 03:00 UTC
+  workflow_dispatch:
+    inputs:
+      tier:
+        description: "Tier to fetch"
+        type: choice
+        default: daily
+        options: [realtime, hourly, daily, weekly]
+
+# One acquisition at a time — concurrent runs would race the shared DB asset.
+concurrency:
+  group: csfs-acquire
+  cancel-in-progress: false
+
+permissions:
+  contents: write   # upload the DB asset to the release
+  issues: write     # open/update the health-alert issue
+
+env:
+  DB_TAG: db-snapshot
+  DB_FILE: csfs.duckdb
+
+jobs:
+  acquire:
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install
+        run: pip install -e ".[api]"
+
+      - name: Select tier
+        id: tier
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            echo "tier=${{ inputs.tier }}" >> "$GITHUB_OUTPUT"
+          elif [ "${{ github.event.schedule }}" = "0 3 * * 0" ]; then
+            echo "tier=weekly" >> "$GITHUB_OUTPUT"
+          else
+            echo "tier=daily" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Restore DB snapshot
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          if gh release download "$DB_TAG" --pattern "$DB_FILE" --clobber 2>/dev/null; then
+            echo "Restored $DB_FILE ($(du -h "$DB_FILE" | cut -f1))"
+          else
+            echo "No existing snapshot — starting a fresh DB."
+          fi
+
+      - name: Fetch tier ${{ steps.tier.outputs.tier }}
+        # Bound each cycle: some connectors have thousands of stations (e.g.
+        # Brazil ANA ~17.6k), and an unbounded run iterates them all every time,
+        # taking far too long. MAX_STATIONS caps stations per connector per
+        # cycle. (Today the runner takes the first N; rotating coverage across
+        # runs is a future improvement.)
+        env:
+          MAX_STATIONS: "500"
+          CONCURRENCY: "12"
+        run: |
+          csfs fetch --tier "${{ steps.tier.outputs.tier }}" \
+            --max-stations "$MAX_STATIONS" --concurrency "$CONCURRENCY"
+
+      - name: Save DB snapshot
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          gh release view "$DB_TAG" >/dev/null 2>&1 \
+            || gh release create "$DB_TAG" --title "DB snapshot" \
+                 --notes "Rolling canonical CSFS DuckDB store (updated by Acquire)."
+          gh release upload "$DB_TAG" "$DB_FILE" --clobber
+
+      - name: Health report
+        id: health
+        run: |
+          csfs health --tier "${{ steps.tier.outputs.tier }}" | tee health.txt
+          csfs health --tier "${{ steps.tier.outputs.tier }}" --json > health.json
+          # Alert only on acquisition runs that errored/degraded this cycle.
+          if csfs health --tier "${{ steps.tier.outputs.tier }}" \
+               --degraded-only --fail-on error,degraded; then
+            echo "degraded=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "degraded=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Upload health artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: health-${{ steps.tier.outputs.tier }}
+          path: |
+            health.txt
+            health.json
+
+      - name: Alert on degraded connectors
+        if: steps.health.outputs.degraded == 'true'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          TIER: ${{ steps.tier.outputs.tier }}
+        run: |
+          title="CSFS health alert: ${TIER} tier"
+          body="One or more connectors in the **${TIER}** tier errored or degraded in the latest run ($(date -u +%Y-%m-%dT%H:%MZ)).
+
+          \`\`\`
+          $(cat health.txt)
+          \`\`\`
+
+          Run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
+          existing=$(gh issue list --state open --search "$title in:title" --json number --jq '.[0].number')
+          if [ -n "$existing" ]; then
+            gh issue comment "$existing" --body "$body"
+          else
+            gh issue create --title "$title" --label "health" --body "$body" \
+              || gh issue create --title "$title" --body "$body"
+          fi
+
+      # The job is NOT failed when connectors degrade: in a real tier some
+      # connector is almost always degraded (dead host, awaiting data, etc.), so
+      # failing here would make every run red and train people to ignore it.
+      # A successful acquisition that saved the snapshot is a success; degraded
+      # connectors are surfaced via the alert issue and the health artifact.
+      # The job still fails naturally if fetch or the snapshot upload errors.

community_streamflow_service-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,37 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install -e ".[dev,api]"
+
+      - name: Lint
+        run: ruff check src/ tests/
+
+      - name: Type check
+        run: mypy src/csfs/
+
+      - name: Test
+        # Tests are hermetic: tests/conftest.py blocks real network access, so
+        # connectors must mock their HTTP. `-m "not network"` skips any test
+        # explicitly opted into reaching a real upstream.
+        run: pytest tests/ -v --tb=short -m "not network"

community_streamflow_service-0.1.0/.github/workflows/docs.yml

@@ -0,0 +1,47 @@
+name: Docs
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "docs/**"
+      - "mkdocs.yml"
+      - "src/csfs/**"
+      - ".github/workflows/docs.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+# Allow one concurrent deployment; don't cancel an in-progress run.
+concurrency:
+  group: pages
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install docs dependencies
+        run: pip install -e ".[docs]"
+      - name: Build site
+        run: mkdocs build --strict
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: site
+
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

community_streamflow_service-0.1.0/.gitignore

@@ -0,0 +1,42 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.eggs/
+
+*.duckdb
+*.duckdb.wal
+*.parquet
+data/datasets/
+
+# Local data artifacts (station dumps, maps, outreach notes) kept out of git
+data/all_stations.json
+data/all_stations_full.json
+data/coverage_stats.json
+data/provider_summary.json
+data/station_map.html
+data/data_request_emails.md
+
+.env
+.env.*
+*.secret
+
+.venv/
+venv/
+
+# mkdocs build output
+site/
+
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+*.DS_Store
+.idea/
+.vscode/
+*.swp
+
+.claude/

community_streamflow_service-0.1.0/CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] — 2026-06-11
+
+Initial release.
+
+### Added
+
+- 86 provider connectors covering live agency APIs (USGS NWIS, UK EA,
+  Hub'Eau, Environment Canada, PEGELONLINE, and many more), offline archive
+  datasets (GRDC, Caravan, GSIM, EStreams, LamaH, CAMELS variants), and
+  model/aggregator sources (GloFAS, GEOGLOWS, WMO WHOS views).
+- Canonical `Station` / `Observation` / `TimeSeriesChunk` data model
+  (pydantic) with discharge normalized to m³/s and UTC timestamps.
+- DuckDB persistence layer (`DuckDBStore`) with station upserts,
+  observation appends, acquisition history, and connector-health queries.
+- Scheduler with four cron tiers (realtime / hourly / daily / weekly),
+  per-tier lookback windows, and a long-lived daemon mode.
+- `csfs` CLI: `fetch`, `daemon`, `providers`, `stations`, `status`,
+  `health`, `download-data`, `serve`.
+- FastAPI read layer for querying stations and observations over HTTP.
+- Global provider inventory (`inventory/providers.yaml`): 104 cataloged
+  sources labeled by readiness status (implemented / research / fallback /
+  manual).
+- Roster-integrity guard tests: every registered connector must have test
+  coverage, belong to exactly one cron tier, and be documented in the
+  inventory; `status: implemented` entries must have a registered connector.
+- Hermetic test suite: an autouse DNS guard blocks real network access;
+  live tests opt in via `@pytest.mark.network`.
+- Scheduled acquisition workflow on GitHub Actions with a release-hosted
+  DuckDB snapshot and health alerting.
+
+[0.1.0]: https://github.com/DarriEy/CSFS/releases/tag/v0.1.0

community_streamflow_service-0.1.0/CITATION.cff

@@ -0,0 +1,26 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+title: "CSFS: Community Streamflow Service"
+type: software
+authors:
+  - family-names: Eythorsson
+    given-names: Darri
+    email: dae5@hi.is
+    orcid: "https://orcid.org/0000-0003-3653-3936"
+    affiliation: "University of Calgary"
+repository-code: "https://github.com/DarriEy/CSFS"
+license: GPL-3.0-or-later
+version: 0.1.0
+date-released: 2026-06-11
+keywords:
+  - hydrology
+  - streamflow
+  - discharge
+  - open-data
+  - time-series
+abstract: >-
+  CSFS is a live acquisition service that connects to streamflow data
+  providers worldwide, harmonizes their observations into a canonical
+  station/observation schema, and maintains a near-real-time DuckDB
+  store with scheduler tiers, health monitoring, a CLI, and a FastAPI
+  read layer.

community_streamflow_service-0.1.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,85 @@
+
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at dae5@hi.is. All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations
+

community_streamflow_service-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,92 @@
+# Contributing to CSFS
+
+Thank you for considering a contribution! The most valuable contribution to
+CSFS is a new provider connector — there are dozens of national and regional
+hydrological agencies whose data is still out of reach.
+
+## Development setup
+
+```bash
+git clone https://github.com/DarriEy/CSFS.git
+cd CSFS
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev,api]"
+```
+
+Run the checks the CI runs:
+
+```bash
+ruff check src/ tests/
+mypy src/csfs/
+pytest -m "not network"
+```
+
+## Hermetic test policy
+
+The test suite must pass **offline**. An autouse fixture in
+`tests/conftest.py` blocks DNS resolution for any non-local host, so an
+unmocked HTTP call fails fast instead of hanging. Consequences:
+
+- Connector tests must mock their HTTP with [respx](https://lundberg.github.io/respx/)
+  (or equivalent) — never hit the real upstream.
+- A test that intentionally reaches a real API must be marked
+  `@pytest.mark.network`. CI deselects these via `-m "not network"`.
+
+## Adding a connector
+
+1. Create `src/csfs/connectors/your_provider.py`.
+2. Subclass `BaseConnector` and implement `fetch_stations()` and
+   `fetch_observations()`. Normalize discharge to **m³/s** and timestamps to
+   **UTC**; set the class metadata (`slug`, `display_name`, `base_url`,
+   `country_codes`).
+3. Decorate the class with `@register("your_slug")` — the slug must equal the
+   class's `slug` attribute.
+4. Add a provider entry to `inventory/providers.yaml` with
+   `status: implemented`.
+5. Add the slug to exactly one cron tier in
+   `src/csfs/scheduler/cron.py` (`PROVIDER_TIERS`): `realtime`, `hourly`,
+   `daily`, or `weekly`, matching the provider's update cadence.
+6. Write `tests/connectors/test_your_slug.py` with mocked HTTP covering
+   station discovery and observation parsing (units, timestamps, quality
+   flags, error paths).
+
+## Roster-integrity requirements
+
+`tests/test_connector_integrity.py` mechanically enforces the roster's
+honesty. Your PR will fail CI unless **all** of the following hold:
+
+- **Test coverage is mandatory.** Every registered connector must either have
+  its own `tests/connectors/test_<slug>.py` or be referenced by slug inside an
+  existing connector test (for variant/subclass connectors exercised by a
+  shared parent test).
+- **Exactly one cron tier.** Every registered slug must appear in exactly one
+  tier of `PROVIDER_TIERS` — an orphaned connector would never be acquired; a
+  duplicated one would be acquired twice.
+- **No ghost tiers.** Every slug listed in a tier must correspond to a
+  registered connector (catches typos and deleted connectors).
+- **Complete metadata.** Each connector class must expose a `slug` matching
+  its registry key, a non-empty `display_name`, a non-empty `base_url`, and a
+  `country_codes` list of ISO 3166-1 alpha-2 codes (or the `"global"`
+  sentinel for global datasets).
+- **Documented in the inventory.** Every registered connector needs an entry
+  in `inventory/providers.yaml`.
+- **No overclaiming.** An inventory entry may only carry
+  `status: implemented` if its slug is actually registered. Anything
+  documented but not built must be labeled `planned`, `research`, `manual`,
+  or `fallback`.
+- **Valid committed config.** Provider keys in the repo-root `csfs.yaml` must
+  reference registered connectors.
+
+## Style
+
+- `ruff` (lint + import order) and `mypy` must pass; both run in CI.
+- Line length 120; target Python 3.11+.
+- Start new source files with the SPDX header used across the codebase.
+
+## Reporting issues
+
+Use the issue templates: **Bug report** for defects, **Provider request** to
+propose a new streamflow agency or network.
+
+By contributing you agree that your contributions are licensed under
+GPL-3.0-or-later.