silver-data 0.1.0__tar.gz

silver_data-0.1.0/CHANGELOG.md

@@ -0,0 +1,34 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2024-08-04
+
+### Added
+- Initial release of silver-data
+- Dataset loading from CSV, JSON, JSONL, and pandas DataFrames
+- Dataset inspection with column statistics and metadata
+- Dataset validation with error and warning reporting
+- Deterministic dataset fingerprinting for caching and reproducibility
+- Train/validation/test splitting with customizable ratios
+- Immutable dataset design for safe data handling
+- Full type hints for better IDE support
+- Comprehensive test suite with >90% coverage
+- Support for Python 3.8-3.12
+
+### Features
+- `Dataset.from_records()` - Create datasets from Python dictionaries
+- `Dataset.from_csv()` - Load datasets from CSV files
+- `Dataset.from_json()` - Parse JSON data into datasets
+- `Dataset.from_jsonl()` - Handle JSONL format for streaming data
+- `Dataset.from_pandas()` - Convert pandas DataFrames to Silver datasets
+- `Dataset.inspect()` - Get detailed dataset statistics
+- `Dataset.validate()` - Check for data quality issues
+- `Dataset.fingerprint()` - Generate deterministic dataset identifiers
+- `Dataset.split()` - Split datasets for ML workflows
+- `Dataset.to_pandas()` - Convert back to pandas for compatibility
+
+## [Unreleased]

silver_data-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,114 @@
+# Contributing to silver-data
+
+Thank you for your interest in contributing to silver-data! This document provides guidelines and instructions for contributing to the project.
+
+## Development Setup
+
+### Prerequisites
+- Python 3.8 or higher
+- Git
+- Virtual environment (recommended)
+
+### Setting Up Development Environment
+
+1. **Clone the repository**
+   ```bash
+   git clone https://github.com/adfgdartec/silver-data.git
+   cd silver-data
+   ```
+
+2. **Create a virtual environment**
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+3. **Install development dependencies**
+   ```bash
+   pip install -e ".[dev]"
+   ```
+
+4. **Run tests**
+   ```bash
+   pytest
+   ```
+
+5. **Run tests with coverage**
+   ```bash
+   pytest --cov=silver_data --cov-report=html
+   ```
+
+## Code Style
+
+We use the following tools to maintain code quality:
+
+- **flake8** for linting
+- **mypy** for type checking
+- **pytest** for testing
+
+Run all quality checks:
+```bash
+flake8 src/ tests/
+mypy src/
+pytest
+```
+
+## Making Changes
+
+1. **Create a branch**
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. **Make your changes**
+   - Write clear, descriptive commit messages
+   - Add tests for new functionality
+   - Update documentation as needed
+
+3. **Run tests**
+   ```bash
+   pytest
+   ```
+
+4. **Submit a pull request**
+   - Describe your changes clearly
+   - Reference any related issues
+   - Ensure all tests pass
+
+## Testing
+
+We aim for high test coverage. When adding new features:
+
+- Write unit tests for new functions
+- Test edge cases and error conditions
+- Ensure existing tests still pass
+
+### Test Structure
+```
+tests/
+├── __init__.py
+└── test_data.py
+```
+
+## Documentation
+
+- Update docstrings for any modified functions
+- Add examples for new features
+- Update README.md if user-facing changes are made
+
+## Release Process
+
+Releases are managed by maintainers:
+
+1. Update version in `pyproject.toml`
+2. Update `CHANGELOG.md`
+3. Create a GitHub release
+4. Package will be automatically published to PyPI
+
+## Questions?
+
+Feel free to open an issue for questions or discussions about contributions.
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the Apache-2.0 License.

silver_data-0.1.0/LICENSE

@@ -0,0 +1,12 @@
+Apache License
+Version 2.0, January 2004
+
+Copyright 2026 Silver Contributors
+
+Licensed under the Apache License, Version 2.0. You may obtain a copy of the
+License at https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.

silver_data-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include CONTRIBUTING.md
+recursive-include src *.py
+recursive-include tests *.py

silver_data-0.1.0/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.4
+Name: silver-data
+Version: 0.1.0
+Summary: Inspectable, deterministic dataset contracts and loaders for Silver.
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/adfgdartec/silver-data
+Project-URL: Repository, https://github.com/adfgdartec/silver-data
+Project-URL: Issues, https://github.com/adfgdartec/silver-data/issues
+Keywords: machine-learning,datasets,csv,data-validation,python
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.0.0; extra == "pandas"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pandas>=1.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: build>=0.10.0; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Dynamic: license-file
+
+# silver-data
+
+[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](tests/)
+[![Code Style](https://img.shields.io/badge/code%20style-flake8-blue.svg)](https://flake8.pycqa.org/)
+
+Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.
+
+The base install uses only the Python standard library for records, JSON, JSONL,
+and CSV. Add pandas only when you need DataFrame conversion:
+
+```bash
+pip install 'silver-data[pandas]'
+```
+
+## Installation
+
+```bash
+pip install silver-data
+```
+
+## Quick Start
+
+```python
+from silver_data import Dataset
+
+# Load from CSV file
+dataset = Dataset.from_csv("my_data", "path/to/data.csv")
+
+# Optional: load from pandas
+import pandas as pd
+df = pd.read_csv("path/to/data.csv")
+dataset = Dataset.from_pandas("my_data", df)
+
+# Inspect dataset
+report = dataset.inspect()
+print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
+for col in report.columns:
+    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")
+
+# Validate dataset
+validation = dataset.validate()
+if not validation.valid:
+    print("Errors:", validation.errors)
+if validation.warnings:
+    print("Warnings:", validation.warnings)
+
+# Split dataset for ML workflows
+train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
+print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")
+```
+
+## Features
+
+- **Multiple Data Sources**: Load from CSV, JSON, JSONL, and pandas DataFrames
+- **Dataset Inspection**: Get detailed column statistics and metadata
+- **Data Validation**: Automatic detection of missing values, inconsistent columns, and data quality issues
+- **Deterministic Fingerprinting**: Generate unique identifiers for datasets to ensure reproducibility
+- **Smart Splitting**: Train/validation/test splitting with customizable ratios
+- **Immutable Design**: Safe data handling with copy-on-write semantics
+- **Type Safety**: Full type hints for better IDE support and fewer bugs
+
+## Use Cases
+
+### ML Pipeline Integration
+
+```python
+from silver_data import Dataset
+import pandas as pd
+
+# Load and validate training data
+df = pd.read_csv("train.csv")
+dataset = Dataset.from_pandas("training", df)
+
+# Ensure data quality before training
+validation = dataset.validate()
+if not validation.valid:
+    raise ValueError(f"Dataset validation failed: {validation.errors}")
+
+# Split for cross-validation
+train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)
+
+# Use fingerprints for caching
+cache_key = dataset.fingerprint()
+print(f"Dataset fingerprint: {cache_key}")
+```
+
+### Data Quality Monitoring
+
+```python
+from silver_data import Dataset
+
+# Monitor data drift over time
+dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
+dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")
+
+if dataset_v1.fingerprint() != dataset_v2.fingerprint():
+    print("Dataset has changed - retrain models")
+
+# Check for new data quality issues
+report_v2 = dataset_v2.inspect()
+for col in report_v2.columns:
+    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
+        print(f"Warning: {col.name} has high missing rate: {col.missing}")
+```
+
+### Experiment Reproducibility
+
+```python
+from silver_data import Dataset
+
+# Ensure exact same data across experiments
+dataset = Dataset.from_csv("experiment", "data.csv")
+experiment_id = f"exp_{dataset.fingerprint()}"
+
+# Log for reproducibility
+print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")
+```
+
+## Advanced Usage
+
+### Custom Data Loading
+
+```python
+from silver_data import Dataset
+import json
+
+# Load from custom JSON format
+with open("custom_data.json") as f:
+    data = json.load(f)
+dataset = Dataset.from_json("custom", data)
+
+# Load from streaming JSONL
+with open("streaming_data.jsonl") as f:
+    dataset = Dataset.from_jsonl("streaming", f.read())
+```
+
+### Data Type Analysis
+
+```python
+from silver_data import Dataset
+
+dataset = Dataset.from_csv("analysis", "mixed_data.csv")
+report = dataset.inspect()
+
+# Analyze column types
+string_cols = [c.name for c in report.columns if c.value_type == "string"]
+numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
+mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]
+
+print(f"String columns: {string_cols}")
+print(f"Numeric columns: {numeric_cols}")
+print(f"Mixed type columns: {mixed_cols}")
+```
+
+## Requirements
+
+- Python 3.8+
+- pandas 1.0+
+
+## Development
+
+```bash
+# Install development dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run tests with coverage
+pytest --cov=silver_data --cov-report=html
+
+# Run linting
+flake8 src/ tests/
+mypy src/
+```
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+Apache-2.0 - see [LICENSE](LICENSE) file for details.
+
+## Related Packages
+
+- [silver-run](https://github.com/adfgdartec/silver-run) - Training lifecycle management
+- [silver-diagnostics](https://github.com/adfgdartec/silver-diagnostics) - ML diagnostics
+- [silver-adapters](https://github.com/adfgdartec/silver-adapters) - Framework adapters

silver_data-0.1.0/README.md

@@ -0,0 +1,191 @@
+# silver-data
+
+[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](tests/)
+[![Code Style](https://img.shields.io/badge/code%20style-flake8-blue.svg)](https://flake8.pycqa.org/)
+
+Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.
+
+The base install uses only the Python standard library for records, JSON, JSONL,
+and CSV. Add pandas only when you need DataFrame conversion:
+
+```bash
+pip install 'silver-data[pandas]'
+```
+
+## Installation
+
+```bash
+pip install silver-data
+```
+
+## Quick Start
+
+```python
+from silver_data import Dataset
+
+# Load from CSV file
+dataset = Dataset.from_csv("my_data", "path/to/data.csv")
+
+# Optional: load from pandas
+import pandas as pd
+df = pd.read_csv("path/to/data.csv")
+dataset = Dataset.from_pandas("my_data", df)
+
+# Inspect dataset
+report = dataset.inspect()
+print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
+for col in report.columns:
+    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")
+
+# Validate dataset
+validation = dataset.validate()
+if not validation.valid:
+    print("Errors:", validation.errors)
+if validation.warnings:
+    print("Warnings:", validation.warnings)
+
+# Split dataset for ML workflows
+train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
+print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")
+```
+
+## Features
+
+- **Multiple Data Sources**: Load from CSV, JSON, JSONL, and pandas DataFrames
+- **Dataset Inspection**: Get detailed column statistics and metadata
+- **Data Validation**: Automatic detection of missing values, inconsistent columns, and data quality issues
+- **Deterministic Fingerprinting**: Generate unique identifiers for datasets to ensure reproducibility
+- **Smart Splitting**: Train/validation/test splitting with customizable ratios
+- **Immutable Design**: Safe data handling with copy-on-write semantics
+- **Type Safety**: Full type hints for better IDE support and fewer bugs
+
+## Use Cases
+
+### ML Pipeline Integration
+
+```python
+from silver_data import Dataset
+import pandas as pd
+
+# Load and validate training data
+df = pd.read_csv("train.csv")
+dataset = Dataset.from_pandas("training", df)
+
+# Ensure data quality before training
+validation = dataset.validate()
+if not validation.valid:
+    raise ValueError(f"Dataset validation failed: {validation.errors}")
+
+# Split for cross-validation
+train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)
+
+# Use fingerprints for caching
+cache_key = dataset.fingerprint()
+print(f"Dataset fingerprint: {cache_key}")
+```
+
+### Data Quality Monitoring
+
+```python
+from silver_data import Dataset
+
+# Monitor data drift over time
+dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
+dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")
+
+if dataset_v1.fingerprint() != dataset_v2.fingerprint():
+    print("Dataset has changed - retrain models")
+
+# Check for new data quality issues
+report_v2 = dataset_v2.inspect()
+for col in report_v2.columns:
+    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
+        print(f"Warning: {col.name} has high missing rate: {col.missing}")
+```
+
+### Experiment Reproducibility
+
+```python
+from silver_data import Dataset
+
+# Ensure exact same data across experiments
+dataset = Dataset.from_csv("experiment", "data.csv")
+experiment_id = f"exp_{dataset.fingerprint()}"
+
+# Log for reproducibility
+print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")
+```
+
+## Advanced Usage
+
+### Custom Data Loading
+
+```python
+from silver_data import Dataset
+import json
+
+# Load from custom JSON format
+with open("custom_data.json") as f:
+    data = json.load(f)
+dataset = Dataset.from_json("custom", data)
+
+# Load from streaming JSONL
+with open("streaming_data.jsonl") as f:
+    dataset = Dataset.from_jsonl("streaming", f.read())
+```
+
+### Data Type Analysis
+
+```python
+from silver_data import Dataset
+
+dataset = Dataset.from_csv("analysis", "mixed_data.csv")
+report = dataset.inspect()
+
+# Analyze column types
+string_cols = [c.name for c in report.columns if c.value_type == "string"]
+numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
+mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]
+
+print(f"String columns: {string_cols}")
+print(f"Numeric columns: {numeric_cols}")
+print(f"Mixed type columns: {mixed_cols}")
+```
+
+## Requirements
+
+- Python 3.8+
+- pandas 1.0+
+
+## Development
+
+```bash
+# Install development dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run tests with coverage
+pytest --cov=silver_data --cov-report=html
+
+# Run linting
+flake8 src/ tests/
+mypy src/
+```
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+Apache-2.0 - see [LICENSE](LICENSE) file for details.
+
+## Related Packages
+
+- [silver-run](https://github.com/adfgdartec/silver-run) - Training lifecycle management
+- [silver-diagnostics](https://github.com/adfgdartec/silver-diagnostics) - ML diagnostics
+- [silver-adapters](https://github.com/adfgdartec/silver-adapters) - Framework adapters

silver_data-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "silver-data"
+version = "0.1.0"
+description = "Inspectable, deterministic dataset contracts and loaders for Silver."
+readme = "README.md"
+requires-python = ">=3.8"
+license = "Apache-2.0"
+authors = []
+keywords = ["machine-learning", "datasets", "csv", "data-validation", "python"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = []
+
+[project.optional-dependencies]
+pandas = ["pandas>=1.0.0"]
+dev = [
+    "pytest>=7.0.0",
+    "pandas>=1.0",
+    "pytest-cov>=4.0.0",
+    "flake8>=6.0.0",
+    "mypy>=1.0.0",
+    "build>=0.10.0",
+    "twine>=4.0.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/adfgdartec/silver-data"
+Repository = "https://github.com/adfgdartec/silver-data"
+Issues = "https://github.com/adfgdartec/silver-data/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-dir]
+"" = "src"

silver_data-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

silver_data-0.1.0/src/silver_data/__init__.py

@@ -0,0 +1,19 @@
+from .dataset import Dataset
+from .models import (
+    DataRecord,
+    DatasetColumn,
+    DatasetReport,
+    DatasetSplit,
+    DatasetValidation,
+    Primitive,
+)
+
+__all__ = [
+    "Dataset",
+    "DatasetColumn",
+    "DatasetReport",
+    "DatasetValidation",
+    "DatasetSplit",
+    "Primitive",
+    "DataRecord",
+]

silver_data-0.1.0/src/silver_data/dataset.py

@@ -0,0 +1,160 @@
+import csv
+import hashlib
+import json
+from typing import Any, List, Union
+
+from .models import (
+    DataRecord,
+    DatasetColumn,
+    DatasetReport,
+    DatasetSplit,
+    DatasetValidation,
+)
+
+
+class Dataset:
+    def __init__(self, name: str, rows: List[DataRecord]):
+        if not name.strip():
+            raise ValueError("Dataset name is required")
+        self.name = name
+        self._rows = [dict(row) for row in rows]
+
+    @classmethod
+    def from_records(cls, name: str, rows: List[DataRecord]) -> "Dataset":
+        if len(rows) == 0:
+            raise ValueError("Dataset must contain at least one row")
+        return cls(name, [dict(row) for row in rows])
+
+    @classmethod
+    def from_json(
+        cls, name: str, value: Union[List[DataRecord], DataRecord]
+    ) -> "Dataset":
+        if isinstance(value, dict):
+            return cls.from_records(name, [value])
+        return cls.from_records(name, value)
+
+    @classmethod
+    def from_jsonl(cls, name: str, value: str) -> "Dataset":
+        lines = [line.strip() for line in value.strip().splitlines() if line.strip()]
+        records = []
+        for line in lines:
+            record = json.loads(line)
+            if not isinstance(record, dict) or isinstance(record, list):
+                raise ValueError("JSONL rows must be objects")
+            records.append(record)
+        return cls.from_records(name, records)
+
+    @classmethod
+    def from_csv(cls, name: str, path: str) -> "Dataset":
+        try:
+            import pandas as pd
+        except ImportError:
+            pd = None
+        if pd is not None:
+            return cls.from_records(name, pd.read_csv(path).to_dict(orient="records"))
+        with open(path, newline="", encoding="utf-8") as stream:
+            return cls.from_records(name, list(csv.DictReader(stream)))
+
+    @classmethod
+    def from_pandas(cls, name: str, df: Any) -> "Dataset":
+        if not hasattr(df, "to_dict"):
+            raise TypeError("from_pandas requires an object with to_dict")
+        return cls.from_records(name, df.to_dict(orient="records"))
+
+    def records(self) -> List[DataRecord]:
+        return [dict(row) for row in self._rows]
+
+    def columns(self) -> List[str]:
+        all_columns = set()
+        for row in self._rows:
+            all_columns.update(row.keys())
+        return sorted(all_columns)
+
+    def inspect(self) -> DatasetReport:
+        columns_report = []
+        for name in self.columns():
+            values = [row.get(name) for row in self._rows]
+            present = [value for value in values if value is not None]
+            if present:
+                types = {type(value).__name__ for value in present}
+                if len(types) == 1:
+                    type_name = type(present[0]).__name__
+                    value_type = {
+                        "str": "string",
+                        "int": "number",
+                        "float": "number",
+                        "bool": "boolean",
+                    }.get(type_name, "mixed")
+                else:
+                    value_type = "mixed"
+            else:
+                value_type = "null"
+            columns_report.append(
+                DatasetColumn(
+                    name=name,
+                    value_type=value_type,
+                    missing=len(values) - len(present),
+                    unique=len({json.dumps(value, sort_keys=True) for value in values}),
+                )
+            )
+        return DatasetReport(
+            name=self.name,
+            rows=len(self._rows),
+            columns=columns_report,
+            fingerprint=self.fingerprint(),
+        )
+
+    def validate(self) -> DatasetValidation:
+        errors = []
+        warnings = []
+        column_count = len(self.columns())
+        inconsistent_rows = [
+            index
+            for index, row in enumerate(self._rows)
+            if len(row.keys()) != column_count
+        ]
+        if inconsistent_rows:
+            warnings.append(
+                f"Rows do not all contain the same columns: {inconsistent_rows[:5]}"
+            )
+        for column in self.inspect().columns:
+            if column.missing > 0:
+                warnings.append(f"{column.name} has {column.missing} missing values")
+        return DatasetValidation(
+            valid=len(errors) == 0, errors=errors, warnings=warnings
+        )
+
+    def fingerprint(self) -> str:
+        data = json.dumps({"name": self.name, "rows": self._rows}, sort_keys=True)
+        return hashlib.md5(data.encode()).hexdigest()[:8]
+
+    def split(
+        self, train: float = 0.8, validation: float = 0.1, test: float = 0.1
+    ) -> DatasetSplit:
+        if train < 0 or validation < 0 or test < 0:
+            raise ValueError("Dataset split ratios must be non-negative")
+        if abs(train + validation + test - 1.0) > 1e-9:
+            raise ValueError("Dataset split ratios must sum to 1")
+        train_end = int(len(self._rows) * train)
+        validation_end = train_end + int(len(self._rows) * validation)
+        return DatasetSplit(
+            train=Dataset._from_rows(f"{self.name}/train", self._rows[:train_end]),
+            validation=Dataset._from_rows(
+                f"{self.name}/validation", self._rows[train_end:validation_end]
+            ),
+            test=Dataset._from_rows(f"{self.name}/test", self._rows[validation_end:]),
+        )
+
+    @classmethod
+    def _from_rows(cls, name: str, rows: List[DataRecord]) -> "Dataset":
+        return cls(name, [dict(row) for row in rows])
+
+    def to_pandas(self) -> Any:
+        try:
+            import pandas as pd
+        except ImportError as error:
+            raise ImportError(
+                "Dataset.to_pandas() requires the optional 'pandas' dependency; "
+                "install silver-data[pandas]"
+            ) from error
+        return pd.DataFrame(self._rows, columns=self.columns())

silver_data-0.1.0/src/silver_data/models.py

@@ -0,0 +1,35 @@
+from dataclasses import dataclass
+from typing import Dict, List, Literal, Union
+
+Primitive = Union[str, int, float, bool, None]
+DataRecord = Dict[str, Primitive]
+
+
+@dataclass(frozen=True)
+class DatasetColumn:
+    name: str
+    value_type: Literal["string", "number", "boolean", "null", "mixed"]
+    missing: int
+    unique: int
+
+
+@dataclass(frozen=True)
+class DatasetReport:
+    name: str
+    rows: int
+    columns: List[DatasetColumn]
+    fingerprint: str
+
+
+@dataclass(frozen=True)
+class DatasetValidation:
+    valid: bool
+    errors: List[str]
+    warnings: List[str]
+
+
+@dataclass(frozen=True)
+class DatasetSplit:
+    train: "Dataset"
+    validation: "Dataset"
+    test: "Dataset"

silver_data-0.1.0/src/silver_data.egg-info/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.4
+Name: silver-data
+Version: 0.1.0
+Summary: Inspectable, deterministic dataset contracts and loaders for Silver.
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/adfgdartec/silver-data
+Project-URL: Repository, https://github.com/adfgdartec/silver-data
+Project-URL: Issues, https://github.com/adfgdartec/silver-data/issues
+Keywords: machine-learning,datasets,csv,data-validation,python
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.0.0; extra == "pandas"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pandas>=1.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: build>=0.10.0; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Dynamic: license-file
+
+# silver-data
+
+[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](tests/)
+[![Code Style](https://img.shields.io/badge/code%20style-flake8-blue.svg)](https://flake8.pycqa.org/)
+
+Inspectable, deterministic dataset contracts and loaders for Silver. A Python package designed for ML researchers who need reliable dataset handling with built-in validation and reproducibility features.
+
+The base install uses only the Python standard library for records, JSON, JSONL,
+and CSV. Add pandas only when you need DataFrame conversion:
+
+```bash
+pip install 'silver-data[pandas]'
+```
+
+## Installation
+
+```bash
+pip install silver-data
+```
+
+## Quick Start
+
+```python
+from silver_data import Dataset
+
+# Load from CSV file
+dataset = Dataset.from_csv("my_data", "path/to/data.csv")
+
+# Optional: load from pandas
+import pandas as pd
+df = pd.read_csv("path/to/data.csv")
+dataset = Dataset.from_pandas("my_data", df)
+
+# Inspect dataset
+report = dataset.inspect()
+print(f"Rows: {report.rows}, Columns: {len(report.columns)}")
+for col in report.columns:
+    print(f"  {col.name}: {col.value_type} ({col.unique} unique, {col.missing} missing)")
+
+# Validate dataset
+validation = dataset.validate()
+if not validation.valid:
+    print("Errors:", validation.errors)
+if validation.warnings:
+    print("Warnings:", validation.warnings)
+
+# Split dataset for ML workflows
+train, val, test = dataset.split(train=0.8, validation=0.1, test=0.1)
+print(f"Train: {len(train.records())}, Val: {len(val.records())}, Test: {len(test.records())}")
+```
+
+## Features
+
+- **Multiple Data Sources**: Load from CSV, JSON, JSONL, and pandas DataFrames
+- **Dataset Inspection**: Get detailed column statistics and metadata
+- **Data Validation**: Automatic detection of missing values, inconsistent columns, and data quality issues
+- **Deterministic Fingerprinting**: Generate unique identifiers for datasets to ensure reproducibility
+- **Smart Splitting**: Train/validation/test splitting with customizable ratios
+- **Immutable Design**: Safe data handling with copy-on-write semantics
+- **Type Safety**: Full type hints for better IDE support and fewer bugs
+
+## Use Cases
+
+### ML Pipeline Integration
+
+```python
+from silver_data import Dataset
+import pandas as pd
+
+# Load and validate training data
+df = pd.read_csv("train.csv")
+dataset = Dataset.from_pandas("training", df)
+
+# Ensure data quality before training
+validation = dataset.validate()
+if not validation.valid:
+    raise ValueError(f"Dataset validation failed: {validation.errors}")
+
+# Split for cross-validation
+train_split, val_split, test_split = dataset.split(train=0.7, validation=0.15, test=0.15)
+
+# Use fingerprints for caching
+cache_key = dataset.fingerprint()
+print(f"Dataset fingerprint: {cache_key}")
+```
+
+### Data Quality Monitoring
+
+```python
+from silver_data import Dataset
+
+# Monitor data drift over time
+dataset_v1 = Dataset.from_csv("data_v1", "data_2024_01.csv")
+dataset_v2 = Dataset.from_csv("data_v2", "data_2024_02.csv")
+
+if dataset_v1.fingerprint() != dataset_v2.fingerprint():
+    print("Dataset has changed - retrain models")
+
+# Check for new data quality issues
+report_v2 = dataset_v2.inspect()
+for col in report_v2.columns:
+    if col.missing > len(dataset_v2.records()) * 0.1:  # More than 10% missing
+        print(f"Warning: {col.name} has high missing rate: {col.missing}")
+```
+
+### Experiment Reproducibility
+
+```python
+from silver_data import Dataset
+
+# Ensure exact same data across experiments
+dataset = Dataset.from_csv("experiment", "data.csv")
+experiment_id = f"exp_{dataset.fingerprint()}"
+
+# Log for reproducibility
+print(f"Running experiment {experiment_id} with dataset fingerprint {dataset.fingerprint()}")
+```
+
+## Advanced Usage
+
+### Custom Data Loading
+
+```python
+from silver_data import Dataset
+import json
+
+# Load from custom JSON format
+with open("custom_data.json") as f:
+    data = json.load(f)
+dataset = Dataset.from_json("custom", data)
+
+# Load from streaming JSONL
+with open("streaming_data.jsonl") as f:
+    dataset = Dataset.from_jsonl("streaming", f.read())
+```
+
+### Data Type Analysis
+
+```python
+from silver_data import Dataset
+
+dataset = Dataset.from_csv("analysis", "mixed_data.csv")
+report = dataset.inspect()
+
+# Analyze column types
+string_cols = [c.name for c in report.columns if c.value_type == "string"]
+numeric_cols = [c.name for c in report.columns if c.value_type == "number"]
+mixed_cols = [c.name for c in report.columns if c.value_type == "mixed"]
+
+print(f"String columns: {string_cols}")
+print(f"Numeric columns: {numeric_cols}")
+print(f"Mixed type columns: {mixed_cols}")
+```
+
+## Requirements
+
+- Python 3.8+
+- pandas 1.0+
+
+## Development
+
+```bash
+# Install development dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run tests with coverage
+pytest --cov=silver_data --cov-report=html
+
+# Run linting
+flake8 src/ tests/
+mypy src/
+```
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+Apache-2.0 - see [LICENSE](LICENSE) file for details.
+
+## Related Packages
+
+- [silver-run](https://github.com/adfgdartec/silver-run) - Training lifecycle management
+- [silver-diagnostics](https://github.com/adfgdartec/silver-diagnostics) - ML diagnostics
+- [silver-adapters](https://github.com/adfgdartec/silver-adapters) - Framework adapters

silver_data-0.1.0/src/silver_data.egg-info/SOURCES.txt

@@ -0,0 +1,16 @@
+CHANGELOG.md
+CONTRIBUTING.md
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+src/silver_data/__init__.py
+src/silver_data/dataset.py
+src/silver_data/models.py
+src/silver_data.egg-info/PKG-INFO
+src/silver_data.egg-info/SOURCES.txt
+src/silver_data.egg-info/dependency_links.txt
+src/silver_data.egg-info/requires.txt
+src/silver_data.egg-info/top_level.txt
+tests/__init__.py
+tests/test_data.py

silver_data-0.1.0/src/silver_data.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

silver_data-0.1.0/src/silver_data.egg-info/requires.txt

@@ -0,0 +1,12 @@
+
+[dev]
+pytest>=7.0.0
+pandas>=1.0
+pytest-cov>=4.0.0
+flake8>=6.0.0
+mypy>=1.0.0
+build>=0.10.0
+twine>=4.0.0
+
+[pandas]
+pandas>=1.0.0

silver_data-0.1.0/src/silver_data.egg-info/top_level.txt

@@ -0,0 +1 @@
+silver_data

silver_data-0.1.0/tests/__init__.py

@@ -0,0 +1 @@
+# Tests for silver-data package

silver_data-0.1.0/tests/test_data.py

@@ -0,0 +1,194 @@
+import pytest
+import pandas as pd
+from silver_data import (
+    Dataset,
+    DatasetColumn,
+    DatasetReport,
+    DatasetValidation,
+    DatasetSplit,
+)
+
+
+class TestDatasetCreation:
+    def test_from_records(self):
+        records = [
+            {"name": "Alice", "age": 30, "city": "NYC"},
+            {"name": "Bob", "age": 25, "city": "LA"},
+        ]
+        dataset = Dataset.from_records("test", records)
+        assert dataset.name == "test"
+        assert len(dataset.records()) == 2
+
+    def test_from_records_empty_raises_error(self):
+        with pytest.raises(ValueError, match="Dataset must contain at least one row"):
+            Dataset.from_records("test", [])
+
+    def test_from_records_empty_name_raises_error(self):
+        with pytest.raises(ValueError, match="Dataset name is required"):
+            Dataset.from_records("", [{"a": 1}])
+
+    def test_from_json_single_object(self):
+        dataset = Dataset.from_json("test", {"name": "Alice", "age": 30})
+        assert len(dataset.records()) == 1
+        assert dataset.records()[0] == {"name": "Alice", "age": 30}
+
+    def test_from_json_array(self):
+        dataset = Dataset.from_json("test", [{"name": "Alice"}, {"name": "Bob"}])
+        assert len(dataset.records()) == 2
+
+    def test_from_jsonl(self):
+        jsonl = '{"name": "Alice", "age": 30}\n{"name": "Bob", "age": 25}'
+        dataset = Dataset.from_jsonl("test", jsonl)
+        assert len(dataset.records()) == 2
+
+    def test_from_jsonl_invalid_raises_error(self):
+        with pytest.raises(ValueError, match="JSONL rows must be objects"):
+            Dataset.from_jsonl("test", '["not", "an", "object"]')
+
+    def test_from_pandas(self):
+        df = pd.DataFrame({"name": ["Alice", "Bob"], "age": [30, 25]})
+        dataset = Dataset.from_pandas("test", df)
+        assert len(dataset.records()) == 2
+        assert dataset.records()[0]["name"] == "Alice"
+
+
+class TestDatasetInspection:
+    def test_columns(self):
+        records = [
+            {"name": "Alice", "age": 30},
+            {"name": "Bob", "age": 25, "city": "LA"},
+        ]
+        dataset = Dataset.from_records("test", records)
+        columns = dataset.columns()
+        assert "age" in columns
+        assert "name" in columns
+        assert "city" in columns
+
+    def test_inspect(self):
+        records = [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]
+        dataset = Dataset.from_records("test", records)
+        report = dataset.inspect()
+
+        assert isinstance(report, DatasetReport)
+        assert report.name == "test"
+        assert report.rows == 2
+        assert len(report.columns) == 2
+        assert report.fingerprint == dataset.fingerprint()
+
+    def test_inspect_column_types(self):
+        records = [
+            {"name": "Alice", "age": 30, "active": True},
+            {"name": "Bob", "age": 25, "active": False},
+        ]
+        dataset = Dataset.from_records("test", records)
+        report = dataset.inspect()
+
+        col_dict = {col.name: col for col in report.columns}
+        assert col_dict["name"].value_type == "string"
+        assert col_dict["age"].value_type == "number"
+        assert col_dict["active"].value_type == "boolean"
+
+    def test_inspect_missing_values(self):
+        records = [{"name": "Alice", "age": 30}, {"name": None, "age": 25}]
+        dataset = Dataset.from_records("test", records)
+        report = dataset.inspect()
+
+        col_dict = {col.name: col for col in report.columns}
+        assert col_dict["name"].missing == 1
+
+
+class TestDatasetValidation:
+    def test_validate_valid_dataset(self):
+        records = [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]
+        dataset = Dataset.from_records("test", records)
+        validation = dataset.validate()
+
+        assert validation.valid == True
+        assert len(validation.errors) == 0
+
+    def test_validate_inconsistent_columns(self):
+        records = [
+            {"name": "Alice", "age": 30},
+            {"name": "Bob", "age": 25, "city": "LA"},
+        ]
+        dataset = Dataset.from_records("test", records)
+        validation = dataset.validate()
+
+        assert validation.valid == True  # warnings don't make it invalid
+        assert len(validation.warnings) > 0
+        assert "do not all contain the same columns" in validation.warnings[0]
+
+    def test_validate_missing_values(self):
+        records = [{"name": "Alice", "age": 30}, {"name": None, "age": 25}]
+        dataset = Dataset.from_records("test", records)
+        validation = dataset.validate()
+
+        assert len(validation.warnings) > 0
+        assert "missing values" in validation.warnings[0]
+
+
+class TestDatasetFingerprint:
+    def test_fingerprint_deterministic(self):
+        records = [{"name": "Alice", "age": 30}]
+        dataset1 = Dataset.from_records("test", records)
+        dataset2 = Dataset.from_records("test", records)
+
+        assert dataset1.fingerprint() == dataset2.fingerprint()
+
+    def test_fingerprint_different_data(self):
+        dataset1 = Dataset.from_records("test", [{"name": "Alice"}])
+        dataset2 = Dataset.from_records("test", [{"name": "Bob"}])
+
+        assert dataset1.fingerprint() != dataset2.fingerprint()
+
+
+class TestDatasetSplit:
+    def test_split_default_ratios(self):
+        records = [{"value": i} for i in range(100)]
+        dataset = Dataset.from_records("test", records)
+        split = dataset.split()
+
+        assert isinstance(split, DatasetSplit)
+        assert len(split.train.records()) == 80
+        assert len(split.validation.records()) == 10
+        assert len(split.test.records()) == 10
+
+    def test_split_custom_ratios(self):
+        records = [{"value": i} for i in range(100)]
+        dataset = Dataset.from_records("test", records)
+        split = dataset.split(train=0.6, validation=0.2, test=0.2)
+
+        assert len(split.train.records()) == 60
+        assert len(split.validation.records()) == 20
+        assert len(split.test.records()) == 20
+
+    def test_split_invalid_ratios_raises_error(self):
+        dataset = Dataset.from_records("test", [{"value": 1}])
+
+        with pytest.raises(ValueError, match="must sum to 1"):
+            dataset.split(train=0.5, validation=0.3, test=0.1)
+
+        with pytest.raises(ValueError, match="must be non-negative"):
+            dataset.split(train=-0.1)
+
+
+class TestDatasetToPandas:
+    def test_to_pandas(self):
+        records = [{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}]
+        dataset = Dataset.from_records("test", records)
+        df = dataset.to_pandas()
+
+        assert isinstance(df, pd.DataFrame)
+        assert len(df) == 2
+        assert list(df.columns) == ["age", "name"]  # alphabetical order
+
+
+class TestDatasetImmutability:
+    def test_records_returns_copy(self):
+        records = [{"name": "Alice"}]
+        dataset = Dataset.from_records("test", records)
+
+        returned_records = dataset.records()
+        returned_records[0]["name"] = "Bob"
+
+        assert dataset.records()[0]["name"] == "Alice"