fsspeckit 0.3.1__tar.gz

fsspeckit-0.3.1/.factory/docs/2025-10-17-pyarrow-type-promotion-rules-for-unify_schemas.md

@@ -0,0 +1,53 @@
+# PyArrow `unify_schemas` Type Promotion Analysis
+
+## Overview
+When using `pyarrow.unify_schemas` with `promote_options="permissive"`, PyArrow attempts to promote incompatible types to a common denominator. However, not all type combinations can be unified.
+
+## Supported Type Promotions
+
+### Numeric Types (Within same category)
+- **Integer promotion**: int8 → int16 → int32 → int64
+- **Unsigned integer promotion**: uint8 → uint16 → uint32 → uint64  
+- **Float promotion**: float32 → float64
+- **Mixed numeric**: Integer types can be promoted to float types when necessary
+  - int8/int16/int32/int64 → float32/float64
+  - uint8/uint16/uint32/uint64 → float32/float64
+
+### Temporal Types
+- **Date promotion**: date32 → date64
+- **Time promotion**: time32 → time64
+- **Timestamp promotion**: Different timestamp precisions can be unified to higher precision
+
+### Null Type Compatibility
+- **Null with any type**: null type can be promoted to any other type
+
+## Unsupported Type Promotions (Incompatible)
+
+### Numeric vs String Types
+- **String ↔ Numeric**: Cannot promote between string types and any numeric types (int, uint, float)
+- This is the most common source of `ArrowInvalid` errors
+
+### Binary vs Non-Binary Types
+- **Binary ↔ Non-binary**: Binary types cannot be promoted to non-binary types and vice versa
+
+### Complex Nested Types
+- **Struct ↔ Struct**: Only if field names and types are compatible
+- **List ↔ List**: Only if element types are compatible
+- **Struct ↔ List**: Never compatible
+
+### Logical vs Physical Types
+- **Dictionary ↔ Plain types**: Generally incompatible unless explicitly cast
+
+## Error Scenarios
+The function raises `ArrowInvalid` when:
+1. Numeric and string types are mixed
+2. Binary and non-binary types are mixed  
+3. Struct fields have incompatible types
+4. List elements have incompatible types
+5. Any other fundamentally incompatible type combinations
+
+## Best Practices
+1. Ensure compatible types across schemas for the same field names
+2. Use explicit casting before unification for known incompatible types
+3. Consider null types as universal promotable targets
+4. Test with sample schemas before full dataset processing

fsspeckit-0.3.1/.github/workflows/publish-to-pypi.yml

@@ -0,0 +1,64 @@
+name: Publish Python 🐍 distribution to PyPI on version bump
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - "pyproject.toml"
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        run: |
+          pip install uv
+
+      - name: Install dependencies
+        run: |
+          uv sync --no-dev
+
+      - name: Get current version from pyproject.toml
+        id: get_version
+        run: |
+          uv run python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])" > version.txt
+          echo "version=$(cat version.txt)" >> $GITHUB_OUTPUT
+
+      - name: Get latest version from PyPI
+        id: get_latest
+        run: |
+          PKG_NAME=$(uv run python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['name'])")
+          LATEST_VERSION=$(curl -s https://pypi.org/pypi/$PKG_NAME/json | uv run python -c "import sys, json; data = json.load(sys.stdin); print(data['info']['version']) if 'info' in data else print('0.0.0')" 2>/dev/null || echo "0.0.0")
+          echo "latest=$LATEST_VERSION" >> $GITHUB_OUTPUT
+
+      - name: Compare versions and set publish flag
+        id: check_version
+        run: |
+          SHOULD_PUBLISH=$(uv run python -c "
+          from packaging.version import parse
+          import os
+          current = parse('${{ steps.get_version.outputs.version }}')
+          latest = parse('${{ steps.get_latest.outputs.latest }}')
+          print('true' if current > latest else 'false')
+          ")
+          echo "publish=$SHOULD_PUBLISH" >> $GITHUB_OUTPUT
+
+      - name: Build package
+        if: steps.check_version.outputs.publish == 'true'
+        run: |
+          uv build --sdist --wheel
+
+      - name: Publish to PyPI
+        if: steps.check_version.outputs.publish == 'true'
+        env:
+          PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+        run: |
+          uv publish --token $PYPI_TOKEN

fsspeckit-0.3.1/.gitignore

@@ -0,0 +1,9 @@
+.coverage
+dev/
+**/__pycache__/
+documentation_report.md
+documentation_instructions.md
+docs/site/
+coverage.xml
+site/**
+site/

fsspeckit-0.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 FlowerPower Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fsspeckit-0.3.1/PKG-INFO

@@ -0,0 +1,330 @@
+Metadata-Version: 2.4
+Name: fsspeckit
+Version: 0.3.1
+Summary: Enhanced utilities and extensions for fsspec, storage_options and obstore with multi-format I/O support.
+Project-URL: Homepage, https://github.com/legout/fsspeckit
+Project-URL: Documentation, https://legout.github.io/fsspeckit
+Project-URL: Repository, https://github.com/legout/fsspeckit.git
+Project-URL: Issues, https://github.com/legout/fsspeckit/issues
+Author-email: legout <ligno.blades@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: azure,cloud-storage,csv,data-io,filesystem,fsspec,gcs,json,object-storage,obstore,parquet,s3
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Filesystems
+Requires-Python: >=3.11
+Requires-Dist: fsspec>=2025.1.0
+Requires-Dist: joblib>=1.5.0
+Requires-Dist: loguru>=0.7.0
+Requires-Dist: msgspec>=0.18.0
+Requires-Dist: obstore>=0.8.2
+Requires-Dist: orjson>=3.11.0
+Requires-Dist: pandas>=2.2.0
+Requires-Dist: polars>=1.30.0
+Requires-Dist: pyarrow>=20.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: rich>=14.0.0
+Requires-Dist: sqlglot>=27.16.3
+Provides-Extra: aws
+Requires-Dist: boto3>=1.26.0; extra == 'aws'
+Requires-Dist: s3fs>=2025.1.0; extra == 'aws'
+Provides-Extra: azure
+Requires-Dist: adlfs>=2024.12.0; extra == 'azure'
+Provides-Extra: gcp
+Requires-Dist: gcsfs>=2025.1.0; extra == 'gcp'
+Description-Content-Type: text/markdown
+
+# fsspeckit
+
+Enhanced utilities and extensions for fsspec filesystems with multi-format I/O support.
+
+## Overview
+
+`fsspeckit` is a comprehensive toolkit that extends [fsspec](https://filesystem-spec.readthedocs.io/) with:
+
+- **Multi-cloud storage configuration** - Easy setup for AWS S3, Google Cloud Storage, Azure Storage, GitHub, and GitLab
+- **Enhanced caching** - Improved caching filesystem with monitoring and path preservation  
+- **Extended I/O operations** - Read/write operations for JSON, CSV, Parquet with Polars/PyArrow integration
+- **Utility functions** - Type conversion, parallel processing, and data transformation helpers
+
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/legout/fsspeckit)
+
+## Installation
+
+```bash
+# Basic installation
+pip install fsspeckit
+
+# Specific cloud providers
+pip install "fsspeckit[aws]"     # AWS S3 support
+pip install "fsspeckit[gcp]"     # Google Cloud Storage
+pip install "fsspeckit[azure]"   # Azure Storage
+
+# Multiple cloud providers
+pip install "fsspeckit[aws,gcp,azure]"
+```
+
+## Quick Start
+
+### Basic Filesystem Operations
+
+```python
+from fsspeckit import filesystem
+
+# Local filesystem
+fs = filesystem("file")
+files = fs.ls("/path/to/data")
+
+# S3 with caching
+fs = filesystem("s3://my-bucket/", cached=True)
+data = fs.cat("data/file.txt")
+```
+
+### Storage Configuration
+
+```python
+from fsspeckit.storage_options import AwsStorageOptions
+
+# Configure S3 access
+options = AwsStorageOptions(
+    region="us-west-2",
+    access_key_id="YOUR_KEY",
+    secret_access_key="YOUR_SECRET"
+)
+
+fs = filesystem("s3", storage_options=options, cached=True)
+```
+
+### Environment-based Configuration
+
+```python
+from fsspeckit.storage_options import AwsStorageOptions
+
+# Load from environment variables
+options = AwsStorageOptions.from_env()
+fs = filesystem("s3", storage_options=options)
+```
+
+### Multiple Cloud Providers
+
+```python
+from fsspeckit.storage_options import (
+    AwsStorageOptions, 
+    GcsStorageOptions,
+    GitHubStorageOptions
+)
+
+# AWS S3
+s3_fs = filesystem("s3", storage_options=AwsStorageOptions.from_env())
+
+# Google Cloud Storage  
+gcs_fs = filesystem("gs", storage_options=GcsStorageOptions.from_env())
+
+# GitHub repository
+github_fs = filesystem("github", storage_options=GitHubStorageOptions(
+    org="microsoft",
+    repo="vscode", 
+    token="ghp_xxxx"
+))
+```
+
+## Storage Options
+
+### AWS S3
+
+```python
+from fsspeckit.storage_options import AwsStorageOptions
+
+# Basic credentials
+options = AwsStorageOptions(
+    access_key_id="AKIAXXXXXXXX",
+    secret_access_key="SECRET",
+    region="us-east-1"
+)
+
+# From AWS profile
+options = AwsStorageOptions.create(profile="dev")
+
+# S3-compatible service (MinIO)
+options = AwsStorageOptions(
+    endpoint_url="http://localhost:9000",
+    access_key_id="minioadmin",
+    secret_access_key="minioadmin",
+    allow_http=True
+)
+```
+
+### Google Cloud Storage
+
+```python
+from fsspeckit.storage_options import GcsStorageOptions
+
+# Service account
+options = GcsStorageOptions(
+    token="path/to/service-account.json",
+    project="my-project-123"
+)
+
+# From environment
+options = GcsStorageOptions.from_env()
+```
+
+### Azure Storage
+
+```python
+from fsspeckit.storage_options import AzureStorageOptions
+
+# Account key
+options = AzureStorageOptions(
+    protocol="az",
+    account_name="mystorageacct",
+    account_key="key123..."
+)
+
+# Connection string
+options = AzureStorageOptions(
+    protocol="az",
+    connection_string="DefaultEndpoints..."
+)
+```
+
+### GitHub
+
+```python
+from fsspeckit.storage_options import GitHubStorageOptions
+
+# Public repository
+options = GitHubStorageOptions(
+    org="microsoft",
+    repo="vscode",
+    ref="main"
+)
+
+# Private repository
+options = GitHubStorageOptions(
+    org="myorg",
+    repo="private-repo",
+    token="ghp_xxxx",
+    ref="develop"
+)
+```
+
+### GitLab
+
+```python
+from fsspeckit.storage_options import GitLabStorageOptions
+
+# Public project
+options = GitLabStorageOptions(
+    project_name="group/project",
+    ref="main"
+)
+
+# Private project with token
+options = GitLabStorageOptions(
+    project_id=12345,
+    token="glpat_xxxx",
+    ref="develop"
+)
+```
+
+## Enhanced Caching
+
+```python
+from fsspeckit import filesystem
+
+# Enable caching with monitoring
+fs = filesystem(
+    "s3://my-bucket/",
+    cached=True,
+    cache_storage="/tmp/my_cache",
+    verbose=True
+)
+
+# Cache preserves directory structure
+data = fs.cat("deep/nested/path/file.txt")
+# Cached at: /tmp/my_cache/deep/nested/path/file.txt
+```
+
+## Utilities
+
+### Parallel Processing
+
+```python
+from fsspeckit.helpers import run_parallel
+
+# Run function in parallel
+def process_file(path, multiplier=1):
+    return len(path) * multiplier
+
+results = run_parallel(
+    process_file,
+    ["/path1", "/path2", "/path3"],
+    multiplier=2,
+    n_jobs=4,
+    verbose=True
+)
+```
+
+### Type Conversion
+
+```python
+from fsspeckit.helpers import dict_to_dataframe, to_pyarrow_table
+
+# Convert dict to DataFrame
+data = {"col1": [1, 2, 3], "col2": [4, 5, 6]}
+df = dict_to_dataframe(data)
+
+# Convert to PyArrow table
+table = to_pyarrow_table(df)
+```
+
+### Logging
+
+```python
+from fsspeckit.helpers import setup_logging
+
+# Configure logging
+setup_logging(level="DEBUG", format_string="{time} | {level} | {message}")
+```
+
+## Dependencies
+
+### Core Dependencies
+- `fsspec>=2023.1.0` - Filesystem interface
+- `msgspec>=0.18.0` - Serialization
+- `pyyaml>=6.0` - YAML support
+- `requests>=2.25.0` - HTTP requests
+- `loguru>=0.7.0` - Logging
+
+### Optional Dependencies
+- `orjson>=3.8.0` - Fast JSON processing
+- `polars>=0.19.0` - Fast DataFrames
+- `pyarrow>=10.0.0` - Columnar data
+- `pandas>=1.5.0` - Data analysis
+- `joblib>=1.3.0` - Parallel processing
+- `rich>=13.0.0` - Progress bars
+
+### Cloud Provider Dependencies
+- `boto3>=1.26.0`, `s3fs>=2023.1.0` - AWS S3
+- `gcsfs>=2023.1.0` - Google Cloud Storage  
+- `adlfs>=2023.1.0` - Azure Storage
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+