retaildata 0.1.0__tar.gz

retaildata-0.1.0/PKG-INFO

@@ -0,0 +1,123 @@
+Metadata-Version: 2.4
+Name: retaildata
+Version: 0.1.0
+Summary: Secure CLI + Python API to download and manage real-world retail benchmark datasets.
+Author-email: Gwang-Jin Kim <gwang.jin.kim.phd@gmail.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: argon2-cffi>=25.1.0
+Requires-Dist: cryptography>=46.0.5
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: kaggle>=1.7.4.5
+Requires-Dist: keyring>=25.7.0
+Requires-Dist: platformdirs>=4.4.0
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: rich>=14.3.2
+Requires-Dist: tqdm>=4.67.3
+Requires-Dist: typer>=0.23.0
+Requires-Dist: polars>=0.20.0
+Requires-Dist: huggingface_hub>=0.20.0
+Requires-Dist: ucimlrepo
+Requires-Dist: openml
+Requires-Dist: numpy>=1.20.0
+Provides-Extra: torch
+Requires-Dist: torch>=2.0.0; extra == "torch"
+Provides-Extra: tf
+Requires-Dist: tensorflow>=2.10.0; extra == "tf"
+Provides-Extra: jax
+Requires-Dist: jax>=0.4.0; extra == "jax"
+Requires-Dist: jaxlib>=0.4.0; extra == "jax"
+Provides-Extra: dlt
+Requires-Dist: dlt[duckdb]>=0.3.0; extra == "dlt"
+Provides-Extra: all
+Requires-Dist: torch>=2.0.0; extra == "all"
+Requires-Dist: tensorflow>=2.10.0; extra == "all"
+Requires-Dist: jax>=0.4.0; extra == "all"
+Requires-Dist: jaxlib>=0.4.0; extra == "all"
+Requires-Dist: dlt[duckdb]>=0.3.0; extra == "all"
+
+# RetailData
+
+A unified interface for fetching and preparing retail datasets for benchmarking and analysis.
+
+## Features
+
+- **Unified API**: Fetch datasets from various providers (HTTP, Kaggle, Hugging Face, UCI, OpenML) with a single command.
+- **Secure Credentials**: Integrated support for Kaggle and Hugging Face API keys.
+- **Data Benchmark Pack**: Curated retail datasets (Favorita, Rossmann, Instacart, M5, Olist, and more).
+- **Processing Pipeline**: Automatic conversion to high-performance Parquet optimized for Polars.
+- **Cache Management**: Programmatic disk usage tracking and clearing.
+
+## Installation
+
+```bash
+pip install retaildata
+```
+
+Or using `uv` (recommended for development):
+```bash
+uv pip install -e .
+```
+
+## Quick Start
+
+### CLI
+
+1. **List available datasets**:
+   ```bash
+   retaildata list
+   ```
+
+2. **Download a dataset**:
+   ```bash
+   retaildata get test_http
+   ```
+
+3. **Download with Preparation (Parquet)**:
+   ```bash
+   retaildata get online_retail_ii --prepare
+   ```
+
+4. **Manage Credentials (e.g. Kaggle)**:
+   ```bash
+   retaildata auth set kaggle --file ~/.kaggle/kaggle.json
+   ```
+
+5. **Clean Up**:
+   ```bash
+   retaildata rm test_http
+   retaildata purge --all
+   ```
+
+### Python API
+
+```python
+import retaildata.api as rd
+import polars as pl
+from pathlib import Path
+
+# Download and prepare dataset
+rd.api.download("online_retail_ii", prepare=True)
+
+# Load efficiently with Polars
+df = pl.scan_parquet("~/.local/share/retaildata/prepared/online_retail_ii/*.parquet").collect()
+print(df.head())
+```
+
+## Supported Datasets
+
+- `online_retail_ii`: UK-based online retail transactions.
+- `olist`: Brazilian e-commerce dataset.
+- `m5`: Walmart time-series forecasting.
+- `store_sales`: Corporación Favorita (Ecuador) store sales.
+- `rossmann`: Rossmann store sales benchmarks.
+- `instacart`: Online grocery basket analysis.
+- `online_retail_uci`: Classical transactions dataset (UCI).
+- `credit_approval_openml`: Financial benchmarking (OpenML).
+
+See `retaildata list` for the full registry.
+
+## License
+
+This package is licensed under the MIT License. Individual datasets may have their own licenses.

retaildata-0.1.0/README.md

@@ -0,0 +1,84 @@
+# RetailData
+
+A unified interface for fetching and preparing retail datasets for benchmarking and analysis.
+
+## Features
+
+- **Unified API**: Fetch datasets from various providers (HTTP, Kaggle, Hugging Face, UCI, OpenML) with a single command.
+- **Secure Credentials**: Integrated support for Kaggle and Hugging Face API keys.
+- **Data Benchmark Pack**: Curated retail datasets (Favorita, Rossmann, Instacart, M5, Olist, and more).
+- **Processing Pipeline**: Automatic conversion to high-performance Parquet optimized for Polars.
+- **Cache Management**: Programmatic disk usage tracking and clearing.
+
+## Installation
+
+```bash
+pip install retaildata
+```
+
+Or using `uv` (recommended for development):
+```bash
+uv pip install -e .
+```
+
+## Quick Start
+
+### CLI
+
+1. **List available datasets**:
+   ```bash
+   retaildata list
+   ```
+
+2. **Download a dataset**:
+   ```bash
+   retaildata get test_http
+   ```
+
+3. **Download with Preparation (Parquet)**:
+   ```bash
+   retaildata get online_retail_ii --prepare
+   ```
+
+4. **Manage Credentials (e.g. Kaggle)**:
+   ```bash
+   retaildata auth set kaggle --file ~/.kaggle/kaggle.json
+   ```
+
+5. **Clean Up**:
+   ```bash
+   retaildata rm test_http
+   retaildata purge --all
+   ```
+
+### Python API
+
+```python
+import retaildata.api as rd
+import polars as pl
+from pathlib import Path
+
+# Download and prepare dataset
+rd.api.download("online_retail_ii", prepare=True)
+
+# Load efficiently with Polars
+df = pl.scan_parquet("~/.local/share/retaildata/prepared/online_retail_ii/*.parquet").collect()
+print(df.head())
+```
+
+## Supported Datasets
+
+- `online_retail_ii`: UK-based online retail transactions.
+- `olist`: Brazilian e-commerce dataset.
+- `m5`: Walmart time-series forecasting.
+- `store_sales`: Corporación Favorita (Ecuador) store sales.
+- `rossmann`: Rossmann store sales benchmarks.
+- `instacart`: Online grocery basket analysis.
+- `online_retail_uci`: Classical transactions dataset (UCI).
+- `credit_approval_openml`: Financial benchmarking (OpenML).
+
+See `retaildata list` for the full registry.
+
+## License
+
+This package is licensed under the MIT License. Individual datasets may have their own licenses.

retaildata-0.1.0/pyproject.toml

@@ -0,0 +1,46 @@
+[project]
+name = "retaildata"
+version = "0.1.0"
+description = "Secure CLI + Python API to download and manage real-world retail benchmark datasets."
+authors = [
+  {name = "Gwang-Jin Kim", email = "gwang.jin.kim.phd@gmail.com"}
+]
+readme = "README.md"
+requires-python = ">=3.9"
+dependencies = [
+    "argon2-cffi>=25.1.0",
+    "cryptography>=46.0.5",
+    "httpx>=0.28.1",
+    "kaggle>=1.7.4.5",
+    "keyring>=25.7.0",
+    "platformdirs>=4.4.0",
+    "pydantic>=2.12.5",
+    "pydantic-settings>=2.0.0",
+    "rich>=14.3.2",
+    "tqdm>=4.67.3",
+    "typer>=0.23.0",
+    "polars>=0.20.0",
+    "huggingface_hub>=0.20.0",
+    "ucimlrepo",
+    "openml",
+    "numpy>=1.20.0",
+]
+
+[project.optional-dependencies]
+torch = ["torch>=2.0.0"]
+tf = ["tensorflow>=2.10.0"]
+jax = ["jax>=0.4.0", "jaxlib>=0.4.0"]
+dlt = ["dlt[duckdb]>=0.3.0"]
+all = ["torch>=2.0.0", "tensorflow>=2.10.0", "jax>=0.4.0", "jaxlib>=0.4.0", "dlt[duckdb]>=0.3.0"]
+
+[dependency-groups]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-mock>=3.10.0",
+]
+
+[project.scripts]
+retaildata = "retaildata.main:main"
+
+[tool.setuptools]
+packages = ["retaildata"]

retaildata-0.1.0/retaildata/api.py

@@ -0,0 +1,202 @@
+from pathlib import Path
+from typing import Optional, List, Any, Dict
+from retaildata.datasets.registry import Registry, Dataset
+from retaildata.providers.http import HTTPProvider
+from retaildata.providers.kaggle import KaggleProvider
+from retaildata.config import settings
+from rich import print as rprint
+
+class RetailDataAPI:
+    def list_datasets(self) -> List[Dataset]:
+        """Lists all available datasets."""
+        return Registry.list_all()
+
+    def get_dataset(self, dataset_id: str) -> Optional[Dataset]:
+        """Gets a dataset by ID."""
+        return Registry.get(dataset_id)
+
+    def download(
+        self, 
+        dataset_id: str, 
+        data_dir: Optional[Path] = None, 
+        prepare: bool = False, 
+        lazy: bool = False,
+        sample_fraction: Optional[float] = None,
+        stratify_col: Optional[str] = None,
+        split_fraction: Optional[float] = None,
+        **kwargs
+    ) -> Optional[Dict[str, Any]]:
+        """
+        Downloads a dataset.
+        
+        Args:
+            dataset_id: The ID of the dataset to download.
+            data_dir: Optional directory to download to. Defaults to settings.final_data_dir.
+            prepare: If True, convert the dataset to Parquet after download.
+            sample_fraction: Optional fraction for sampling (0.0 to 1.0).
+            stratify_col: Optional column name for stratified sampling.
+            split_fraction: Optional fraction for train/test splitting (e.g. 0.8).
+            **kwargs: Additional provider-specific arguments.
+        """
+        dataset = self.get_dataset(dataset_id)
+        if not dataset:
+            raise ValueError(f"Dataset '{dataset_id}' not found in registry.")
+        
+        target_dir = data_dir or settings.final_data_dir
+        # Structure: <data_dir>/raw/<dataset_id>/...
+        download_path = target_dir / "raw" / dataset.id
+        meta_dir = target_dir / "meta"
+        
+        rprint(f"[bold blue]RetailData[/bold blue]: Downloading {dataset.id} to {download_path}")
+
+        if dataset.provider == "http":
+            provider = HTTPProvider()
+            provider.download(dataset, download_path, meta_dir=meta_dir, **kwargs)
+        elif dataset.provider == "kaggle":
+            provider = KaggleProvider()
+            provider.download(dataset, download_path, meta_dir=meta_dir, **kwargs)
+        elif dataset.provider == "hf":
+            from retaildata.providers.hf import HFProvider
+            provider = HFProvider()
+            provider.download(dataset, download_path, meta_dir=meta_dir, **kwargs)
+        elif dataset.provider == "uci":
+            from retaildata.providers.uci import UCIProvider
+            provider = UCIProvider()
+            provider.download(dataset, download_path, meta_dir=meta_dir, **kwargs)
+        elif dataset.provider == "openml":
+            from retaildata.providers.openml import OpenMLProvider
+            provider = OpenMLProvider()
+            provider.download(dataset, download_path, meta_dir=meta_dir, **kwargs)
+        elif dataset.provider == "dlt":
+            if dataset.id == "retail_express":
+                from retaildata.providers.retail_express import RetailExpressProvider
+                provider = RetailExpressProvider()
+            else:
+                from retaildata.providers.dlt import DLTProvider
+                provider = DLTProvider()
+            provider.download(dataset, download_path, meta_dir=meta_dir, **kwargs)
+        else:
+            raise NotImplementedError(f"Provider '{dataset.provider}' not yet supported.")
+            
+        rprint(f"[green]Successfully processed dataset '{dataset.id}'[/green]")
+        
+        if prepare:
+            from retaildata.processing.manager import manager as processing_manager
+            rprint(f"[bold blue]RetailData[/bold blue]: Preparing {dataset.id} (converting to Parquet)...")
+            processing_manager.process_dataset(
+                dataset.id, 
+                data_dir=target_dir,
+                sample_fraction=sample_fraction,
+                stratify_col=stratify_col,
+                split_fraction=split_fraction
+            )
+            return self.load(dataset.id, data_dir=target_dir, lazy=lazy)
+        
+        return None
+
+    def load(
+        self, 
+        dataset_id: str, 
+        data_dir: Optional[Path] = None, 
+        lazy: bool = False,
+        standardized: bool = False
+    ) -> Dict[str, Any]:
+        """
+        Loads prepared Parquet files for a dataset.
+        
+        Args:
+            dataset_id: The ID of the dataset to load.
+            data_dir: Optional directory.
+            lazy: If True, returns Polars LazyFrames.
+            standardized: If True, uses the dataset's standard_mapping to rename keys (e.g. 'sales', 'calendar').
+        
+        Returns:
+            A dictionary mapping keys to Polars DataFrames (or LazyFrames).
+        """
+        import polars as pl
+        base_dir = data_dir or settings.final_data_dir
+        prepared_dir = base_dir / "prepared" / dataset_id
+        
+        if not prepared_dir.exists():
+            raise FileNotFoundError(f"Prepared data for '{dataset_id}' not found at {prepared_dir}")
+            
+        data = {}
+        for file_path in prepared_dir.glob("*.parquet"):
+            key = file_path.stem
+            if lazy:
+                data[key] = pl.scan_parquet(file_path)
+            else:
+                data[key] = pl.read_parquet(file_path)
+        
+        # Check for DuckDB files (M7: dlt integration)
+        for file_path in prepared_dir.glob("*.duckdb"):
+            import duckdb
+            con = duckdb.connect(str(file_path))
+            tables = con.execute("SELECT table_name FROM information_schema.tables WHERE table_schema = 'main'").fetchall()
+            for (table_name,) in tables:
+                if table_name.startswith("_dlt"):
+                    continue
+                if lazy:
+                    data[table_name] = pl.read_database(f"SELECT * FROM {table_name}", connection=con).lazy()
+                else:
+                    data[table_name] = pl.read_database(f"SELECT * FROM {table_name}", connection=con)
+            con.close()
+        
+        if standardized:
+            dataset = self.get_dataset(dataset_id)
+            if dataset and dataset.standard_mapping:
+                mapped_data = {}
+                for std_key, actual_key in dataset.standard_mapping.items():
+                    # Handle direct match
+                    if actual_key in data:
+                        mapped_data[std_key] = data[actual_key]
+                    # Handle train/test suffixes from processing_manager
+                    elif f"{actual_key}_train" in data:
+                        mapped_data[f"{std_key}_train"] = data[f"{actual_key}_train"]
+                        mapped_data[f"{std_key}_test"] = data[f"{actual_key}_test"]
+                return mapped_data
+
+        return data
+
+    def split_temporal(
+        self, 
+        dataset_id: str, 
+        date_col: str, 
+        split_date: str, 
+        table_key: Optional[str] = None,
+        data_dir: Optional[Path] = None
+    ) -> Dict[str, Any]:
+        """
+        Splits a specific table in the dataset into train and test sets based on a date.
+        """
+        import polars as pl
+        data = self.load(dataset_id, data_dir=data_dir)
+        
+        # Identify which table to split
+        if table_key:
+            key = table_key
+        else:
+            # Try 'sales' from standard mapping, or fallback to first table
+            dataset = self.get_dataset(dataset_id)
+            if dataset and dataset.standard_mapping:
+                key = dataset.standard_mapping.get("sales", list(data.keys())[0])
+            else:
+                key = list(data.keys())[0]
+
+        df = data[key]
+        if isinstance(df, pl.LazyFrame):
+            df = df.collect()
+            
+        # Ensure date_col is datetime/date
+        if df[date_col].dtype not in [pl.Datetime, pl.Date]:
+            df = df.with_columns(pl.col(date_col).str.to_datetime())
+
+        # Convert split_date string to datetime for comparison
+        split_dt = pl.lit(split_date).str.to_datetime()
+        
+        train = df.filter(pl.col(date_col) < split_dt)
+        test = df.filter(pl.col(date_col) >= split_dt)
+        
+        return {"train": train, "test": test}
+
+api = RetailDataAPI()

retaildata-0.1.0/retaildata/cache/manager.py

@@ -0,0 +1,71 @@
+import shutil
+from pathlib import Path
+from typing import List, Dict, Optional
+import json
+from retaildata.config import settings
+from retaildata.datasets.registry import Registry
+
+class CacheManager:
+    def __init__(self):
+        self.data_dir = settings.final_data_dir
+
+    def _get_path(self, dataset_id: str, subdir: str) -> Path:
+        return self.data_dir / subdir / dataset_id
+
+    def is_downloaded(self, dataset_id: str) -> bool:
+        """Check if a dataset is downloaded (metadata exists)."""
+        meta_path = self._get_path(dataset_id, "meta") / "metadata.json"
+        return meta_path.exists()
+
+    def get_size(self, dataset_id: str) -> int:
+        """Calculate total size of a dataset in bytes."""
+        total_size = 0
+        for subdir in ["raw", "prepared", "meta"]:
+            path = self._get_path(dataset_id, subdir)
+            if path.exists():
+                for p in path.rglob("*"):
+                    if p.is_file():
+                        total_size += p.stat().st_size
+        return total_size
+
+    def list_downloaded(self) -> Dict[str, Dict[str, any]]:
+        """List all downloaded datasets with details."""
+        downloaded = {}
+        meta_dir = self.data_dir / "meta"
+        if not meta_dir.exists():
+            return {}
+
+        for ds_dir in meta_dir.iterdir():
+            if ds_dir.is_dir():
+                dataset_id = ds_dir.name
+                if Registry.get(dataset_id): # Only track known datasets
+                    size = self.get_size(dataset_id)
+                    downloaded[dataset_id] = {
+                        "size": size,
+                        "path": str(self._get_path(dataset_id, "raw"))
+                    }
+        return downloaded
+
+    def delete_dataset(self, dataset_id: str) -> bool:
+        """Delete a dataset's files."""
+        # Check if it was downloaded first? Or just force delete.
+        # We delete from raw, prepared, and meta.
+        deleted = False
+        for subdir in ["raw", "prepared", "meta"]:
+            path = self._get_path(dataset_id, subdir)
+            if path.exists():
+                shutil.rmtree(path)
+                deleted = True
+        return deleted
+
+    def purge_all(self):
+        """Delete all data in the data directory."""
+        if self.data_dir.exists():
+            # We want to keep the root data dir but empty it, or at least the subdirs we manage.
+            # Safety: only delete known subdirs
+            for subdir in ["raw", "prepared", "meta"]:
+                path = self.data_dir / subdir
+                if path.exists():
+                    shutil.rmtree(path)
+
+manager = CacheManager()