alcove 0.1.1__tar.gz

alcove-0.1.1/.env.example

@@ -0,0 +1,10 @@
+# S3-compatible storage credentials for Backblaze B2
+S3_ACCESS_KEY=your_application_key_id
+S3_SECRET_KEY=your_application_key
+S3_BUCKET_NAME=your_bucket_name
+S3_ENDPOINT_URL=your_endpoint_url
+
+TEST_ACCESS_KEY=
+TEST_SECRET_KEY=
+TEST_ENDPOINT_URL=
+TEST_BUCKET_NAME=

alcove-0.1.1/.github/workflows/ci.yml

@@ -0,0 +1,63 @@
+name: CI
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    services:
+      minio:
+        image: lazybit/minio
+        ports:
+          - 9000:9000
+        env:
+          MINIO_ACCESS_KEY: justtesting
+          MINIO_SECRET_KEY: justtesting
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.12
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+      - name: Install dependencies
+        run: make .venv
+
+      - name: Set up testing minio bucket
+        run: |
+          .venv/bin/python3 - <<'EOF'
+          from minio import Minio
+
+          minio = Minio(
+              'localhost:9000',
+              access_key='justtesting',
+              secret_key='justtesting',
+              secure=False
+          )
+
+          minio.make_bucket('test')
+          print(f'{minio.list_buckets()}')
+          EOF
+
+      - name: Run tests
+        run: make test
+        env:
+          B2_APPLICATION_KEY: justtesting
+          B2_APPLICATION_KEY_ID: justtesting
+          B2_BUCKET_NAME: test
+          B2_ENDPOINT_URL: http://localhost:9000/

alcove-0.1.1/.gitignore

@@ -0,0 +1,165 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#pdm.lock
+#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
+#   in version control.
+#   https://pdm.fming.dev/latest/usage/project/#working-with-version-control
+.pdm.toml
+.pdm-python
+.pdm-build/
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+.DS_Store
+.aider*

alcove-0.1.1/.python-version

@@ -0,0 +1 @@
+3.12

alcove-0.1.1/.vscode/settings.json

@@ -0,0 +1,11 @@
+{
+  "[python]": {
+    "editor.formatOnSave": true,
+    "editor.defaultFormatter": "charliermarsh.ruff",
+  },
+  "python.analysis.typeCheckingMode": "basic",
+  "python.analysis.diagnosticSeverityOverrides": {
+    "reportMissingTypeStubs": "none"
+  },
+  "files.trimTrailingWhitespace": true
+}

alcove-0.1.1/CLAUDE.md

@@ -0,0 +1,4 @@
+- After each change, run `make test-strict` to check project health (requires Docker)
+- Install packages with `uv add <pkg>` or `uv add --dev <pkg>`
+- Test Github CI runs cleanly with `make act`
+- Avoid modifying the global Python environment or relying on globally installed packages

alcove-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Lars Yencken
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

alcove-0.1.1/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE
+include README.md
+include pyproject.toml
+recursive-include src/alcove/schemas *.json

alcove-0.1.1/Makefile

@@ -0,0 +1,69 @@
+.PHONY: help unittest format lint typecheck test test-strict act act-dry
+
+# Default target
+help:
+	@echo "Available targets:"
+	@echo "  make unittest       - Run unittests with pytest"
+	@echo "  make format         - Reformat using ruff"
+	@echo "  make lint           - Lint using ruff"
+	@echo "  make typecheck      - Typecheck with pyright"
+	@echo "  make test           - Run lint, typecheck, and unittest sequentially"
+	@echo "  make test-strict    - Run tests requiring MinIO container (will fail if not available)"
+	@echo "  make act            - Run GitHub Actions locally with act"
+
+# Check if .venv exists and is up to date
+.venv: pyproject.toml
+	@echo "==> Installing packages"
+	@uv sync
+	@touch $@
+
+# Run unittests with pytest
+unittest: .venv
+	@echo "==> Running unit tests"
+	@uv run pytest --sw
+
+# Reformat using rye
+format: .venv
+	@echo "==> Formatting all files"
+	@uv run ruff format
+	@uv run ruff check --fix
+
+# Lint using rye
+lint: .venv
+	@echo "==> Linting all files"
+	@uv run ruff check
+
+# Typecheck with pyright
+typecheck: .venv
+	@echo "==> Typechecking"
+	@uv run pyright
+
+# Run lint, typecheck, and unittest sequentially
+test: lint typecheck unittest
+
+# Run tests in strict mode - MinIO container is required
+test-strict: lint typecheck
+	@echo "==> Detecting Docker context"
+	$(eval DOCKER_HOST := $(shell docker context inspect --format '{{.Endpoints.docker.Host}}'))
+	@echo "Using Docker context: $(DOCKER_HOST)"
+	@echo "==> Running unit tests with strict MinIO requirements"
+	@DOCKER_HOST=$(DOCKER_HOST) REQUIRE_MINIO=1 uv run pytest --sw
+
+clean:
+	rm -rf data/* metadata/*
+
+act:
+	@echo "==> Detecting Docker context"
+	$(eval DOCKER_HOST := $(shell docker context inspect --format '{{.Endpoints.docker.Host}}'))
+	@echo "Using Docker context: $(DOCKER_HOST)"
+	@echo "==> Running GitHub Actions workflow locally with act"
+	@DOCKER_HOST=$(DOCKER_HOST) act
+
+build:
+	@echo "==> Building package"
+	@uv build
+	@uv run twine check dist/*
+
+publish: build
+	@echo "==> Publishing to PyPI"
+	@uv run twine upload dist/*

alcove-0.1.1/PKG-INFO

@@ -0,0 +1,211 @@
+Metadata-Version: 2.4
+Name: alcove
+Version: 0.1.1
+Summary: An opinionated small-scale ETL framework for managing data files and directories in a content-addressable way.
+Project-URL: Homepage, https://github.com/larsyencken/alcove
+Project-URL: Repository, https://github.com/larsyencken/alcove
+Project-URL: Bug Tracker, https://github.com/larsyencken/alcove/issues
+Author-email: Lars Yencken <lars@yencken.org>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: content-addressable,data-engineering,data-lake,etl
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.12
+Requires-Dist: boto3>=1.34.145
+Requires-Dist: duckdb>=1.0.0
+Requires-Dist: jsonschema>=4.23.0
+Requires-Dist: polars>=1.4.0
+Requires-Dist: python-dotenv>=1.0.1
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: rich>=13.7.1
+Requires-Dist: ruff>=0.5.5
+Description-Content-Type: text/markdown
+
+# alcove
+
+[![CI](https://github.com/larsyencken/alcove/actions/workflows/ci.yml/badge.svg)](https://github.com/larsyencken/alcove/actions/workflows/ci.yml)
+[![Python 3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/)
+
+_A personal ETL and data lake._
+
+Status: in alpha, changing often
+
+## Overview
+
+Alcove is an opinionated small-scale ETL framework for managing data files and directories in a content-addressable way.
+
+## Core principles
+
+- **A reusable framework.** Alcove provides a structured way of managing data files, scripts and their interdependencies that can be used across multiple projects.
+- **First class metadata.** Every data file has an accompanying metadata sidecar that can be used to store provenance, licensing and other information.
+- **Content addressed.** An `alcove` DAG is a Merkle tree of checksums that includes data, metadata and scripts, used to lazily rebuild only what is out of date.
+- **Data versioning.** Every step in the DAG has a URI that includes a version, which can be an ISO date or `latest`, to encourage a reproducible workflow that still allows for change.
+- **SQL support.** Alcove is a Python framework, but allows you to write steps in SQL which will be executed by DuckDB.
+- **Parquet interchange.** All derived tables are generated as Parquet, which makes reuse easier.
+
+## Usage
+
+### Install the package
+
+Start by installing the alcove package, either globally, or into an existing Python project.
+
+`pip install git+https://github.com/larsyencken/alcove`
+
+### Initialise an alcove
+
+Enter the folder where you want to store your data and metadata, and run:
+
+`alcove init`
+
+This will create a `alcove.yaml` file, which will serve as the catalogue of all the data in your alcove.
+
+### Configure object storage
+
+You will need to configure your S3-compatible storage credentials in a `.env` file, in the same directory as your `alcove.yaml` file. Define:
+
+```
+S3_ACCESS_KEY=your_application_key_id
+S3_SECRET_KEY=your_application_key
+S3_BUCKET_NAME=your_bucket_name
+S3_ENDPOINT_URL=your_endpoint_url
+```
+
+Now your alcove is ready to use.
+
+### Adding a file or folder
+
+From within your alcove folder, run `alcove snapshot path/to/your/file_or_folder dataset_name` to add a file to your alcove. See the earlier overview for choosing a dataset name.
+
+```
+alcove snapshot ~/Downloads/countries.csv countries/latest
+```
+
+This will upload the file to your S3-compatible storage, and create a metadata file at `data/<dataset_name>.meta.yaml` directory for you to complete.
+
+The metadata format has some minimum fields, but is meant for you to extend as needed for your own purposes. Best practice would be to retain the provenance and licence information of any data you add to your alcove, especially if it originates from a third party.
+
+### Creating a new table
+
+To create a new table, use the `alcove new-table <table-path> [dep1 [dep2 [...]]` command. This command will create a placeholder executable script that generates an example data file of the given type based on the file extension (.parquet or .sql).
+
+For example, to create a new table with a Parquet placeholder script:
+
+```
+alcove new-table path/to/your/table
+```
+
+This will create a placeholder script that generates an example Parquet file with the following content:
+
+```
+#!/usr/bin/env python3
+import sys
+import polars as pl
+
+data = {
+    "a": [1, 1, 3],
+    "b": [2, 3, 5],
+    "c": [3, 4, 6]
+}
+
+df = pl.DataFrame(data)
+
+output_file = sys.argv[-1]
+df.write_parquet(output_file)
+```
+
+For example, to create a new table with a SQL placeholder script:
+
+```
+alcove new-table path/to/your/table.sql
+```
+
+This will create a placeholder script that generates an example SQL file with the following content:
+
+```
+-- SQL script to create a table
+CREATE TABLE example_table AS
+SELECT
+    1 AS a,
+    2 AS b,
+    3 AS c
+```
+
+The command also supports the `--edit` option to open the metadata file for the table in your editor:
+
+```
+alcove new-table path/to/your/table --edit
+```
+
+### Executing SQL step definitions
+
+If a `.sql` step definition is detected, it will be executed using DuckDB with an in-memory database. The SQL file can use `{variable}` to interpolate template variables. The following template variables are available:
+
+- `{output_file}`: The path to the output file.
+- `{dependency}`: The path of each dependency, simplified to a semantic name.
+
+### Building your alcove
+
+Run `alcove run` to fetch any data that's out of date, and build any derived tables.
+
+## Development
+
+### Testing with MinIO
+
+For testing with S3-compatible storage, this project uses automatically managed containers:
+
+```bash
+# Regular testing - continues even if MinIO isn't available
+make test
+
+# Strict testing - fails if MinIO container isn't available
+make test-strict
+```
+
+The test environment provides two modes:
+1. **Default mode**: Attempts to use MinIO container but won't fail if unavailable
+2. **Strict mode**: Requires MinIO container, fails if Docker isn't running
+
+### Docker Context Support
+
+The testing framework automatically detects your current Docker context and uses it for container operations. This ensures tests work properly with:
+- Docker Desktop
+- Colima
+- OrbStack
+- Remote Docker contexts
+
+### MinIO Configuration
+
+With Docker, these credentials are automatically used:
+- Access Key: minioadmin
+- Secret Key: minioadmin
+- Bucket: test-bucket
+- Endpoint: http://localhost:9000
+
+Containers are automatically managed and kept running between test runs for performance.
+MinIO's health is verified before tests run to ensure proper S3 compatibility.
+
+## Bugs
+
+Please report any issues at: https://github.com/larsyencken/alcove/issues
+
+## Changelog
+
+- `0.1.1` (2025-04-25)
+  - Renamed project from "shelf" to "alcove"
+  - Added automated Docker container management for testing with MinIO
+  - Added two testing modes: standard and strict (requires Docker)
+  - Enhanced Docker context support for different environments (Docker Desktop, Colima, OrbStack)
+  - Improved S3-compatible storage testing reliability
+  - Fixed test fixtures to use consistent credentials
+
+- `0.1.0` (Initial release)
+  - Initialise a repo with `alcove.yaml`
+  - `alcove snapshot` and `alcove run` with file and directory support
+  - Only fetch things that are out of date
+  - `alcove list` to see what datasets are available
+  - `alcove audit` to ensure your alcove is coherent and correct
+  - `alcove db` to enter an interactive DuckDB shell with all your data