python-eia 0.2.0__py3-none-any.whl → 0.3.0__py3-none-any.whl

eia/.agents/skills/eia/SKILL.md

@@ -0,0 +1,172 @@
+---
+name: eia
+description: Query U.S. energy data (EIA API v2). Use when the user asks about U.S. electricity, petroleum, natural gas, or coal data from the EIA.
+version: 2.0.0
+---
+
+# EIA Data Assistant
+
+You have access to the `python-eia` library and CLI for querying the U.S. Energy Information Administration (EIA) API v2.
+
+## When to use what
+
+- **Python scripts** (default): reproducible, composable, saveable. Use for any data work the user will want to keep or iterate on.
+- **CLI**: quick one-shot lookups, exploration, sanity checks. Use when the user wants a fast answer they won't need again.
+- **If unsure**: ask the user whether they want a script or a quick CLI check.
+
+## Built-in Catalog (OFFLINE — no API calls)
+
+The library ships with a YAML catalog containing full API schema for curated routes: columns, frequencies, periods, and **all facet values**. Always check the catalog first to avoid unnecessary API calls.
+
+### Cataloged Routes
+
+| Route | Description | Frequency |
+|-------|-------------|-----------|
+| `electricity/rto/fuel-type-data` | Real-time grid generation by fuel type | hourly |
+| `electricity/rto/region-data` | Real-time grid demand/generation by region | hourly |
+| `electricity/rto/interchange-data` | Real-time interchange between regions | hourly |
+| `electricity/retail-sales` | Retail electricity sales by state/sector | monthly |
+| `petroleum/pri/spt` | Spot petroleum prices (crude, gasoline, etc.) | daily |
+| `natural-gas/pri/sum` | Natural gas prices summary | monthly |
+| `natural-gas/move/expc` | US natural gas exports by country | monthly |
+| `natural-gas/move/impc` | US natural gas imports by country | monthly |
+| `total-energy/data` | Total energy overview (production, consumption, etc.) | monthly |
+
+For routes **not** in the catalog, use `eia routes` (CLI) to discover and `eia meta` to inspect.
+
+## Python Library (default)
+
+```python
+from eia import EIAClient
+
+client = EIAClient()  # reads config file, then EIA_API_KEY env var
+
+# --- Catalog access (offline, no API calls) ---
+
+# Get endpoint with cached schema — no API metadata call
+data = client.get_data_endpoint("electricity/rto/fuel-type-data")
+
+# Inspect metadata (all cached for cataloged routes)
+data.facets                    # FacetContainer with attribute access
+data.frequencies               # List[FrequencyInfo]
+data.data_columns              # Dict[str, DataColumnInfo]
+data.start_period              # "2019-01-01T00"
+data.end_period                # "2026-03-04T07"
+
+# Facet values — cached, no API call
+respondents = data.facets.respondent.get_values()
+# [FacetValue(id='CISO', name='California ISO'), ...]
+
+fuel_types = data.facets.fueltype.get_values()
+# [FacetValue(id='SUN', name='Solar'), ...]
+
+# Or access catalog directly
+from eia.catalog import get_route, list_routes
+route = get_route("electricity/rto/fuel-type-data")
+route.data_columns   # (DataColumn(id='value', units='megawatthours', ...),)
+route.frequencies     # (Frequency(id='hourly', ...), Frequency(id='local-hourly', ...))
+route.facets[0].values  # {'CISO': 'California ISO', 'PJM': 'PJM Interconnection LLC', ...}
+
+# --- Fetch data (hits API) ---
+
+df = data.get(
+    data_columns=["value"],
+    facets={"respondent": "CISO"},
+    frequency="hourly",
+    start="2024-01-01",
+    end="2024-01-31",
+    sort=[{"column": "period", "direction": "desc"}],
+)
+
+# Multiple facet values
+df = data.get(
+    data_columns=["revenue", "sales"],
+    facets={"stateid": "CA", "sectorid": ["RES", "COM"]},
+    frequency="monthly",
+    start="2024-01-01",
+    end="2024-12-31",
+)
+
+# --- Route tree navigation (for discovery — hits API) ---
+route = client.route("electricity/rto/fuel-type-data")
+route.routes          # Dict of child routes (if branch node)
+route.data            # Data object (if leaf node)
+```
+
+### Facet conventions
+
+- **Common facets**: `respondent` (grid operator), `fueltype`, `stateid`, `sectorid`, `series`
+- **Multiple values**: pass a list — `facets={"sectorid": ["RES", "COM"]}`
+- **Prefer catalog** for facet discovery: `get_route().facets[i].values` has all valid values offline
+
+### Key conventions
+
+- The `period` column is auto-converted to datetime (UTC for non-local frequencies)
+- The `value` column is auto-converted to numeric
+- Pagination is automatic by default (fetches all pages)
+- API page limit is 5000 rows per request
+- Custom exception: `EIAError` (includes HTTP status code and API error code)
+
+## CLI Reference (quick lookups)
+
+### Catalog (offline)
+
+```bash
+eia catalog routes                              # List all cataloged routes
+eia catalog show electricity/rto/fuel-type-data # Full details (columns, frequencies, facet values)
+eia catalog recipes                             # Pre-configured query recipes
+eia catalog recipe lng-exports-europe           # Show a specific recipe
+eia catalog refresh --apply                     # Refresh schema from API
+```
+
+### Explore (hits API)
+
+```bash
+eia routes                                      # Top-level routes
+eia routes electricity/rto                      # Navigate deeper
+eia meta electricity/rto/fuel-type-data         # Endpoint metadata
+eia facets electricity/rto/fuel-type-data respondent  # Facet values
+```
+
+### Fetch data
+
+```bash
+eia get electricity/rto/fuel-type-data \
+  --start 2024-06-01 --end 2024-06-08 \
+  --frequency hourly --facet respondent=CISO --data value
+
+# Multiple facet values (repeat --facet)
+eia get electricity/retail-sales \
+  --start 2024-01-01 --end 2024-12-31 \
+  --facet stateid=CA --facet sectorid=RES --facet sectorid=COM \
+  --data revenue --data sales
+
+# Export
+eia get petroleum/pri/spt --start 2024-01-01 --end 2024-06-01 \
+  --format csv --output prices.csv
+```
+
+### Exec (ad-hoc pandas)
+
+```bash
+eia exec electricity/rto/fuel-type-data \
+  --start 2024-06-01 --end 2024-06-08 \
+  --frequency hourly --facet respondent=CISO --data value \
+  -x "df.groupby('fueltype')['value'].mean()"
+```
+
+### Output options
+
+```
+--format table|csv|json   (default: table)
+--output file.csv         (write to file instead of stdout)
+```
+
+## Configuration
+
+```bash
+eia config set api-key YOUR_KEY    # Store API key
+eia config get api-key             # Verify
+```
+
+Config file: `~/.config/eia/config.toml`. API key resolution: config file > `EIA_API_KEY` env var.

eia/__init__.py

@@ -5,6 +5,8 @@ A Python client for interacting with the U.S. Energy Information Administration
 """
 
 from .client import EIAClient, EIAError
+from .cache import CacheConfig
+from . import catalog
 
 __version__ = "0.1.0"
-__all__ = ["EIAClient", "EIAError"]
+__all__ = ["EIAClient", "EIAError", "CacheConfig", "catalog"]

eia/cache.py

@@ -0,0 +1,399 @@
+"""Local parquet cache for EIA API time-series data.
+
+Caches query results as parquet files, fetching only missing date ranges
+on subsequent requests. Historical energy data is immutable once
+published (~48h), so caching is safe and enabled by default.
+
+Storage layout::
+
+    {cache_dir}/
+    └── electricity/rto/fuel-type-data/
+        ├── hourly/
+        │   ├── respondent=CISO/
+        │   │   ├── data.parquet
+        │   │   └── meta.json
+        │   └── respondent=PJM.fueltype=SUN,WND/
+        │       ├── data.parquet
+        │       └── meta.json
+        └── monthly/
+            └── _all_/
+                ├── data.parquet
+                └── meta.json
+
+Unlike ENTSO-E, EIA stores DataFrames in long format (facet columns +
+value column) rather than wide format, because multiple rows per period
+are common (e.g. one row per fuel type per respondent).
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import shutil
+import tempfile
+from dataclasses import dataclass, field
+from datetime import datetime
+from pathlib import Path
+
+import pandas as pd
+
+logger = logging.getLogger("eia")
+
+# Default cache location — respects XDG_CACHE_HOME
+_DEFAULT_CACHE_DIR = Path(
+    os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache")
+) / "eia"
+
+# Data older than this (hours) is considered final and won't be re-fetched
+_DEFAULT_RECENT_TTL_HOURS = 48
+
+
+def _facets_key(facets: dict | None) -> str:
+    """Build a deterministic partition string from facet filters.
+
+    Examples:
+        None  →  "_all_"
+        {"respondent": "CISO"}  →  "respondent=CISO"
+        {"respondent": "PJM", "fueltype": ["SUN", "WND"]}
+            →  "fueltype=SUN,WND.respondent=PJM"
+    """
+    if not facets:
+        return "_all_"
+    parts = []
+    for k in sorted(facets.keys()):
+        v = facets[k]
+        if isinstance(v, list):
+            v_str = ",".join(sorted(str(x) for x in v))
+        else:
+            v_str = str(v)
+        parts.append(f"{k}={v_str}")
+    return ".".join(parts)
+
+
+@dataclass
+class CacheConfig:
+    """Cache configuration."""
+
+    enabled: bool = True
+    cache_dir: Path = field(default_factory=lambda: _DEFAULT_CACHE_DIR)
+    recent_ttl_hours: int = _DEFAULT_RECENT_TTL_HOURS
+
+    def __post_init__(self) -> None:
+        self.cache_dir = Path(self.cache_dir)
+
+
+@dataclass(frozen=True)
+class DateRange:
+    """A contiguous date range [start, end] inclusive."""
+
+    start: pd.Timestamp
+    end: pd.Timestamp
+
+
+class CacheStore:
+    """Read, write, and merge parquet files for cached EIA data."""
+
+    def __init__(self, config: CacheConfig):
+        self.config = config
+
+    # -- Path resolution -------------------------------------------------------
+
+    def _parquet_path(self, route: str, frequency: str, facets_key: str) -> Path:
+        """Data file: {cache_dir}/{route}/{frequency}/{facets_key}/data.parquet"""
+        return self.config.cache_dir / route / frequency / facets_key / "data.parquet"
+
+    def _meta_path(self, route: str, frequency: str, facets_key: str) -> Path:
+        """Metadata file: {cache_dir}/{route}/{frequency}/{facets_key}/meta.json"""
+        return self.config.cache_dir / route / frequency / facets_key / "meta.json"
+
+    # -- Data Read / Write -----------------------------------------------------
+
+    def read(
+        self,
+        route: str,
+        frequency: str,
+        facets_key: str,
+        start: pd.Timestamp,
+        end: pd.Timestamp,
+    ) -> pd.DataFrame:
+        """Read cached data for a date range.
+
+        Returns DataFrame with ``period`` as DatetimeIndex.
+        Returns empty DataFrame on cache miss.
+        """
+        path = self._parquet_path(route, frequency, facets_key)
+        if not path.exists():
+            return pd.DataFrame()
+
+        try:
+            df = pd.read_parquet(path)
+        except Exception as exc:
+            logger.warning("Corrupted cache file %s: %s — removing.", path, exc)
+            path.unlink(missing_ok=True)
+            return pd.DataFrame()
+
+        if df.empty or not isinstance(df.index, pd.DatetimeIndex):
+            return pd.DataFrame()
+
+        return self._slice(df, start, end)
+
+    def _slice(
+        self, df: pd.DataFrame, start: pd.Timestamp, end: pd.Timestamp
+    ) -> pd.DataFrame:
+        """Slice a DataFrame by [start, end], handling timezone alignment."""
+        if df.index.tz is not None:
+            if start.tz is None:
+                start = start.tz_localize(df.index.tz)
+            if end.tz is None:
+                end = end.tz_localize(df.index.tz)
+        elif start.tz is not None:
+            start = start.tz_localize(None)
+        if end.tz is not None and df.index.tz is None:
+            end = end.tz_localize(None)
+
+        # When end is a date-level timestamp (midnight), extend to end of day
+        if end.hour == 0 and end.minute == 0 and end.second == 0:
+            end = end + pd.Timedelta(days=1) - pd.Timedelta(seconds=1)
+
+        return df[start:end]
+
+    def write(
+        self,
+        route: str,
+        frequency: str,
+        facets_key: str,
+        df: pd.DataFrame,
+    ) -> None:
+        """Merge new data with existing cache and persist.
+
+        *df* should have ``period`` as DatetimeIndex. New data is merged
+        with existing, deduplicating on the index. Rows from the new data
+        take precedence for overlapping timestamps.
+        """
+        if df.empty:
+            return
+
+        path = self._parquet_path(route, frequency, facets_key)
+        path.parent.mkdir(parents=True, exist_ok=True)
+
+        # Read existing and merge
+        existing = pd.DataFrame()
+        if path.exists():
+            try:
+                existing = pd.read_parquet(path)
+            except Exception:
+                logger.warning("Corrupted cache at %s — overwriting.", path)
+
+        if not existing.empty:
+            # For long-format data, concat + deduplicate
+            merged = pd.concat([existing, df])
+            # Drop duplicates: keep last (new data wins)
+            # Use all columns for deduplication since index alone isn't unique
+            # (multiple rows per period with different facet values)
+            merged = merged[~merged.index.duplicated(keep="last")]
+            merged = merged.sort_index()
+        else:
+            merged = df.sort_index()
+
+        _atomic_write_parquet(path, merged)
+
+    def write_meta(
+        self,
+        route: str,
+        frequency: str,
+        facets_key: str,
+        meta: dict,
+    ) -> None:
+        """Write metadata for a partition."""
+        meta = {**meta, "cached_at": datetime.now().isoformat()}
+        path = self._meta_path(route, frequency, facets_key)
+        _atomic_write_json(path, meta)
+
+    def read_meta(
+        self,
+        route: str,
+        frequency: str,
+        facets_key: str,
+    ) -> dict | None:
+        """Read cached metadata for a partition."""
+        path = self._meta_path(route, frequency, facets_key)
+        if not path.exists():
+            return None
+        try:
+            return json.loads(path.read_text(encoding="utf-8"))
+        except (json.JSONDecodeError, OSError):
+            return None
+
+    # -- Gap detection ---------------------------------------------------------
+
+    def find_gaps(
+        self,
+        cached_df: pd.DataFrame,
+        start: pd.Timestamp,
+        end: pd.Timestamp,
+        *,
+        recent_ttl_hours: int | None = None,
+    ) -> list[DateRange]:
+        """Find date ranges not covered by cached data.
+
+        Also marks data within ``recent_ttl_hours`` of now as a gap
+        (needs re-fetch since it may have been updated).
+        """
+        ttl = recent_ttl_hours if recent_ttl_hours is not None else self.config.recent_ttl_hours
+        now = pd.Timestamp.now(tz="UTC")
+        cutoff = now - pd.Timedelta(hours=ttl)
+
+        if cached_df.empty:
+            return [DateRange(start, end)]
+
+        # Normalize to UTC for comparison
+        idx = cached_df.index
+        if idx.tz is None:
+            idx = idx.tz_localize("UTC")
+        else:
+            idx = idx.tz_convert("UTC")
+
+        start_utc = start.tz_localize("UTC") if start.tz is None else start.tz_convert("UTC")
+        end_utc = end.tz_localize("UTC") if end.tz is None else end.tz_convert("UTC")
+
+        cached_start = idx.min()
+        cached_end = idx.max()
+
+        gaps: list[DateRange] = []
+
+        # Gap before cached data
+        if start_utc < cached_start:
+            gap_end = min(cached_start - pd.Timedelta(hours=1), end_utc)
+            if gap_end >= start_utc:
+                gaps.append(DateRange(start, _to_tz_aware(gap_end, start)))
+
+        # Gap after cached data
+        if end_utc > cached_end:
+            gap_start = max(cached_end + pd.Timedelta(hours=1), start_utc)
+            if gap_start <= end_utc:
+                gaps.append(DateRange(_to_tz_aware(gap_start, end), end))
+
+        # Recent data that may still change
+        if cached_end > cutoff and end_utc > cutoff:
+            recent_start = max(cutoff, start_utc)
+            if recent_start <= end_utc:
+                gaps.append(DateRange(_to_tz_aware(recent_start, end), end))
+
+        return _merge_overlapping(gaps)
+
+    # -- Maintenance -----------------------------------------------------------
+
+    def clear(
+        self,
+        route: str | None = None,
+        frequency: str | None = None,
+    ) -> int:
+        """Remove cached files. Returns number of files removed.
+
+        - No args: clear everything
+        - route only: clear all data for that route
+        - route + frequency: clear one frequency partition
+        """
+        count = 0
+
+        if route and frequency:
+            target = self.config.cache_dir / route / frequency
+        elif route:
+            target = self.config.cache_dir / route
+        else:
+            target = self.config.cache_dir
+
+        if target.exists():
+            count = sum(1 for f in target.rglob("*") if f.is_file())
+            shutil.rmtree(target)
+
+        return count
+
+    def status(self) -> dict:
+        """Return cache statistics."""
+        cache_dir = self.config.cache_dir
+        if not cache_dir.exists():
+            return {"path": str(cache_dir), "files": 0, "size_mb": 0.0, "routes": {}}
+
+        all_files = [f for f in cache_dir.rglob("*") if f.is_file()]
+        total_size = sum(f.stat().st_size for f in all_files)
+
+        # Per-route breakdown (first path component)
+        routes: dict[str, int] = {}
+        for f in all_files:
+            try:
+                rel = f.relative_to(cache_dir)
+                if len(rel.parts) > 1:
+                    r = rel.parts[0]
+                    routes[r] = routes.get(r, 0) + 1
+            except ValueError:
+                pass
+
+        return {
+            "path": str(cache_dir),
+            "files": len(all_files),
+            "size_mb": round(total_size / (1024 * 1024), 2),
+            "routes": routes,
+        }
+
+
+# -- Helpers -------------------------------------------------------------------
+
+
+def _to_tz_aware(ts: pd.Timestamp, reference: pd.Timestamp) -> pd.Timestamp:
+    """Convert a UTC timestamp to match the reference timestamp's timezone."""
+    if reference.tz is not None:
+        return ts.tz_convert(reference.tz) if ts.tz is not None else ts.tz_localize(reference.tz)
+    return ts.tz_localize(None) if ts.tz is not None else ts
+
+
+def _merge_overlapping(gaps: list[DateRange]) -> list[DateRange]:
+    """Merge overlapping or adjacent date ranges."""
+    if not gaps:
+        return []
+
+    sorted_gaps = sorted(gaps, key=lambda g: g.start)
+    merged = [sorted_gaps[0]]
+
+    for gap in sorted_gaps[1:]:
+        prev = merged[-1]
+        if gap.start <= prev.end + pd.Timedelta(days=1):
+            merged[-1] = DateRange(prev.start, max(prev.end, gap.end))
+        else:
+            merged.append(gap)
+
+    return merged
+
+
+def _atomic_write_json(path: Path, data: dict) -> None:
+    """Write JSON atomically via temp file + rename."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp_path = None
+    try:
+        fd, tmp_path = tempfile.mkstemp(suffix=".json", dir=path.parent)
+        os.close(fd)
+        Path(tmp_path).write_text(
+            json.dumps(data, indent=2, ensure_ascii=False, default=str),
+            encoding="utf-8",
+        )
+        Path(tmp_path).rename(path)
+    except OSError as exc:
+        logger.warning("Failed to write %s: %s", path, exc)
+        if tmp_path:
+            Path(tmp_path).unlink(missing_ok=True)
+
+
+def _atomic_write_parquet(path: Path, df: pd.DataFrame) -> None:
+    """Write parquet atomically via temp file + rename."""
+    path.parent.mkdir(parents=True, exist_ok=True)
+    tmp_path = None
+    try:
+        fd, tmp_path = tempfile.mkstemp(suffix=".parquet", dir=path.parent)
+        os.close(fd)
+        df.to_parquet(tmp_path)
+        Path(tmp_path).rename(path)
+    except OSError as exc:
+        logger.warning("Failed to write cache %s: %s — continuing without cache.", path, exc)
+        if tmp_path:
+            Path(tmp_path).unlink(missing_ok=True)