shackleton 1.1.0__tar.gz → 2.0.1__tar.gz

shackleton-2.0.1/.github/workflows/release.yml

@@ -0,0 +1,27 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+jobs:
+  gate:
+    # compatibility_test.yml here also requires the OS matrix to pass
+    uses: ./.github/workflows/test.yml
+  release:
+    needs: gate
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+    - uses: actions/checkout@v7
+      with:
+        fetch-depth: 0
+    - uses: astral-sh/setup-uv@v10.3.0
+    - run: uv build && uv publish
+      env:
+        UV_PUBLISH_TOKEN: ${{ secrets.PYPI_TOKEN }}
+    - run: gh release create "$GITHUB_REF_NAME" dist/* --verify-tag --notes "$(git tag -l --format='%(contents)' "$GITHUB_REF_NAME")"
+      env:
+        GH_TOKEN: ${{ github.token }}

{shackleton-1.1.0 → shackleton-2.0.1}/.github/workflows/test.yml

@@ -7,13 +7,14 @@ on:
   push:
     branches:
       - main
+  workflow_call:
 
 jobs:
   test:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v4
-    - uses: astral-sh/setup-uv@v5
+    - uses: actions/checkout@v7
+    - uses: astral-sh/setup-uv@v10.3.0
       with:
         python-version: '3.12'
         enable-cache: true

{shackleton-1.1.0 → shackleton-2.0.1}/.gitignore

@@ -68,19 +68,12 @@ instance/
 # Scrapy stuff:
 .scrapy
 
-# Sphinx documentation
-docs/_build/
+# Generated coverage HTML (report.md is committed, HTML is not)
+coverage/html/
 
 # PyBuilder
 target/
 
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
 # pyenv
 .python-version
 
@@ -113,9 +106,6 @@ venv.bak/
 # Rope project settings
 .ropeproject
 
-# mkdocs documentation
-/site
-
 # mypy
 .mypy_cache/
 .dmypy.json

{shackleton-1.1.0 → shackleton-2.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: shackleton
-Version: 1.1.0
+Version: 2.0.1
 Summary: Polars-based persistent tabular store backed by Parquet or Arrow IPC files
 Project-URL: Homepage, https://github.com/endremborza/shackleton
 Author-email: Endre Márk Borza <endremborza@gmail.com>
@@ -13,7 +13,7 @@ License: Copyright 2023 Endre Márk Borza
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 License-File: LICENSE
 Requires-Python: >=3.12
-Requires-Dist: polars
+Requires-Dist: polars>=2
 Description-Content-Type: text/markdown
 
 # shackleton
@@ -22,7 +22,7 @@ Description-Content-Type: text/markdown
 
 Persistent tabular store backed by Parquet or Arrow IPC, built on [polars](https://pola.rs).
 
-- **Partitioned writes** — by column value or xxhash bucket
+- **Partitioned writes** — by column value or hash bucket
 - **Sorted merge** — maintain sorted order on extend via `id_col`
 - **Lazy reads** — returns `pl.LazyFrame` so polars can push down predicates
 - **Schema evolution** — diagonal concat handles mismatched columns transparently
@@ -91,7 +91,7 @@ df_eu = repo.get_partition_df({"region": "eu"})
 ## Hash partitioning
 
 When partition values have unbounded cardinality, hash into fixed buckets instead.
-Uses polars' native xxhash — vectorised, GIL-free.
+Uses polars' `Series.hash` — vectorised, GIL-free. That hash is not stable across polars versions while bucket assignment lives on disk, so before its first bucket-addressed operation (a write, `get_partition_*`, `purge_partition`) a repo samples stored rows and raises `HashDriftError` if they no longer hash to their bucket. `repo.rehash()` moves every row into its current bucket.
 
 ```python
 from shackleton import HashPartitioner
@@ -110,10 +110,28 @@ df = repo.get_full_df()
 
 ---
 
+## Deduplication with `dedup_cols`
+
+When `dedup_cols` is set, each `extend` drops duplicate rows on those columns
+after merging with existing data. With `id_col` the merge is sorted and the
+existing (first) row wins; without `id_col` the surviving occurrence is arbitrary.
+Not supported with `max_records > 0` (raises at construction time).
+
+```python
+repo = TableRepo(Path("/tmp/events"), dedup_cols=["event_id"])
+
+repo.extend(pl.DataFrame({"event_id": [1, 2, 3], "val": [10, 20, 30]}))
+# Re-ingest with overlapping IDs — duplicates collapse to one row per event_id
+repo.extend(pl.DataFrame({"event_id": [2, 4], "val": [99, 40]}))
+
+repo.get_full_df().shape  # (4, 2) — event_id 2 kept once
+```
+
+---
+
 ## Sorted merge with `id_col`
 
-When `id_col` is set, each extend keeps the file sorted by that column and
-`replace_records` does an upsert: matching IDs are updated, new IDs appended.
+When `id_col` is set, each extend keeps the file sorted by it (when `max_records == 0`) and `replace_records` does an upsert: matching IDs are updated, new IDs appended. `id_col` is a column or a list of columns, sorted in that order. On equal keys stored rows come before incoming ones, and within one `replace_records` batch the last row for a key wins.
 
 ```python
 repo = TableRepo(Path("/tmp/prices"), id_col="ticker")
@@ -198,6 +216,41 @@ df = repo.get_full_df()
 
 ---
 
+## Partition-level overwrite
+
+`replace_partition` atomically clears and rewrites only the partitions covered
+by the DataFrame. `purge_partition` deletes matching files without rewriting.
+
+```python
+repo = TableRepo(Path("/tmp/events"), partition_cols=["date"])
+repo.extend(pl.DataFrame({
+    "date": ["2024-01-01", "2024-01-01", "2024-01-02"],
+    "val":  [1, 2, 3],
+}))
+
+# Replace just the 2024-01-01 partition with corrected data
+repo.replace_partition(pl.DataFrame({
+    "date": ["2024-01-01"],
+    "val":  [99],
+}))
+
+# Or wipe it without rewriting
+repo.purge_partition({"date": "2024-01-01"})
+```
+
+---
+
+## Compression
+
+Control the codec and, for Parquet, the compression level.
+
+```python
+repo = TableRepo(Path("/tmp/data"), compression="zstd", compression_level=3)
+repo_ipc = TableRepo(Path("/tmp/fast"), ipc=True, compression="lz4")
+```
+
+---
+
 ## Arrow IPC format
 
 Slightly faster reads for local workloads that don't need Parquet compatibility.

{shackleton-1.1.0 → shackleton-2.0.1}/README.md

@@ -4,7 +4,7 @@
 
 Persistent tabular store backed by Parquet or Arrow IPC, built on [polars](https://pola.rs).
 
-- **Partitioned writes** — by column value or xxhash bucket
+- **Partitioned writes** — by column value or hash bucket
 - **Sorted merge** — maintain sorted order on extend via `id_col`
 - **Lazy reads** — returns `pl.LazyFrame` so polars can push down predicates
 - **Schema evolution** — diagonal concat handles mismatched columns transparently
@@ -73,7 +73,7 @@ df_eu = repo.get_partition_df({"region": "eu"})
 ## Hash partitioning
 
 When partition values have unbounded cardinality, hash into fixed buckets instead.
-Uses polars' native xxhash — vectorised, GIL-free.
+Uses polars' `Series.hash` — vectorised, GIL-free. That hash is not stable across polars versions while bucket assignment lives on disk, so before its first bucket-addressed operation (a write, `get_partition_*`, `purge_partition`) a repo samples stored rows and raises `HashDriftError` if they no longer hash to their bucket. `repo.rehash()` moves every row into its current bucket.
 
 ```python
 from shackleton import HashPartitioner
@@ -92,10 +92,28 @@ df = repo.get_full_df()
 
 ---
 
+## Deduplication with `dedup_cols`
+
+When `dedup_cols` is set, each `extend` drops duplicate rows on those columns
+after merging with existing data. With `id_col` the merge is sorted and the
+existing (first) row wins; without `id_col` the surviving occurrence is arbitrary.
+Not supported with `max_records > 0` (raises at construction time).
+
+```python
+repo = TableRepo(Path("/tmp/events"), dedup_cols=["event_id"])
+
+repo.extend(pl.DataFrame({"event_id": [1, 2, 3], "val": [10, 20, 30]}))
+# Re-ingest with overlapping IDs — duplicates collapse to one row per event_id
+repo.extend(pl.DataFrame({"event_id": [2, 4], "val": [99, 40]}))
+
+repo.get_full_df().shape  # (4, 2) — event_id 2 kept once
+```
+
+---
+
 ## Sorted merge with `id_col`
 
-When `id_col` is set, each extend keeps the file sorted by that column and
-`replace_records` does an upsert: matching IDs are updated, new IDs appended.
+When `id_col` is set, each extend keeps the file sorted by it (when `max_records == 0`) and `replace_records` does an upsert: matching IDs are updated, new IDs appended. `id_col` is a column or a list of columns, sorted in that order. On equal keys stored rows come before incoming ones, and within one `replace_records` batch the last row for a key wins.
 
 ```python
 repo = TableRepo(Path("/tmp/prices"), id_col="ticker")
@@ -180,6 +198,41 @@ df = repo.get_full_df()
 
 ---
 
+## Partition-level overwrite
+
+`replace_partition` atomically clears and rewrites only the partitions covered
+by the DataFrame. `purge_partition` deletes matching files without rewriting.
+
+```python
+repo = TableRepo(Path("/tmp/events"), partition_cols=["date"])
+repo.extend(pl.DataFrame({
+    "date": ["2024-01-01", "2024-01-01", "2024-01-02"],
+    "val":  [1, 2, 3],
+}))
+
+# Replace just the 2024-01-01 partition with corrected data
+repo.replace_partition(pl.DataFrame({
+    "date": ["2024-01-01"],
+    "val":  [99],
+}))
+
+# Or wipe it without rewriting
+repo.purge_partition({"date": "2024-01-01"})
+```
+
+---
+
+## Compression
+
+Control the codec and, for Parquet, the compression level.
+
+```python
+repo = TableRepo(Path("/tmp/data"), compression="zstd", compression_level=3)
+repo_ipc = TableRepo(Path("/tmp/fast"), ipc=True, compression="lz4")
+```
+
+---
+
 ## Arrow IPC format
 
 Slightly faster reads for local workloads that don't need Parquet compatibility.

shackleton-2.0.1/coverage/report.md

@@ -0,0 +1,6 @@
+| Name                           |    Stmts |     Miss |    Cover |
+|------------------------------- | -------: | -------: | -------: |
+| shackleton/\_\_init\_\_.py     |        3 |        0 |     100% |
+| shackleton/core.py             |      289 |        0 |     100% |
+| shackleton/tests/test\_core.py |      349 |        0 |     100% |
+| **TOTAL**                      |  **641** |    **0** | **100%** |

shackleton-2.0.1/docs/release_notes.md

@@ -0,0 +1,29 @@
+## v2.0.1
+
+- Fixed composite-key merges to be order-exact: stored rows now reliably come before incoming ones on equal keys, where polars' streaming engine could previously reorder ties.
+- Fixed `replace_records` to work against a partition with no stored data yet.
+- Writes are now atomic: data is staged to a temp file and moved into place with `os.replace`, so a crash mid-merge can no longer leave a partially-written file.
+- Fixed partitions to be flagged sorted on read only when they are a single file, since appended parts are unsorted and were previously mis-flagged.
+
+---
+
+## v2.0.0
+
+- **Breaking:** requires polars ≥2; existing `HashPartitioner` stores built under older polars versions will raise `HashDriftError` on their next bucket-addressed operation until migrated.
+- Detects hash bucket drift before writes, `get_partition_*` and `purge_partition`, and adds `TableRepo.rehash()` to rebuild a store under the installed polars' hashing; full reads are unaffected and skip the check.
+- `id_col` now accepts a list of columns for composite-key merges and dedup.
+- Merges are now order-exact: on equal keys stored rows precede incoming ones, so `extend` with `dedup_cols` keeps the stored row and `replace_records` keeps the batch's last row instead of an arbitrary one.
+- Writes are now staged to a temp file and atomically renamed into place, and merges with existing data stream instead of collecting fully in memory.
+- `paths` are now returned sorted, so reads are deterministic.
+
+---
+
+## v1.2.0
+
+deduped
+
+---
+
+## v1.1.0
+
+realign, simplify

{shackleton-1.1.0 → shackleton-2.0.1}/pyproject.toml

@@ -6,7 +6,7 @@ authors = [{name = "Endre Márk Borza", email = "endremborza@gmail.com"}]
 license = {file = "LICENSE"}
 readme = "README.md"
 requires-python = ">=3.12"
-dependencies = ["polars"]
+dependencies = ["polars>=2"]
 
 [project.urls]
 Homepage = "https://github.com/endremborza/shackleton"

shackleton-2.0.1/shackleton/__init__.py

@@ -0,0 +1,13 @@
+"""Polars-based persistent tabular store backed by Parquet or Arrow IPC files."""
+
+from .core import (  # noqa: F401
+    DfBatchWriter,
+    HashDriftError,
+    HashPartitioner,
+    RecordWriter,
+    TableRepo,
+)
+
+TableShack = TableRepo  # backward-compat alias
+
+__version__ = "2.0.1"