xarray_sql 0.2.1__tar.gz → 0.2.2__tar.gz

{xarray_sql-0.2.1 → xarray_sql-0.2.2}/.gitignore

@@ -12,3 +12,5 @@ test_data
 .chainlink
 .claude
 CHANGELOG.md
+*.ipynb
+/site

{xarray_sql-0.2.1 → xarray_sql-0.2.2}/Cargo.lock

@@ -3375,7 +3375,7 @@ checksum = "ea2f10b9bb0928dfb1b42b65e1f9e36f7f54dbdf08457afefb38afcdec4fa2bb"
 
 [[package]]
 name = "xarray_sql"
-version = "0.2.1"
+version = "0.2.2"
 dependencies = [
  "arrow",
  "async-stream",

{xarray_sql-0.2.1 → xarray_sql-0.2.2}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "xarray_sql"
-version = "0.2.1"
+version = "0.2.2"
 authors = ["Alex Merose"]
 edition = "2021"
 exclude = [

{xarray_sql-0.2.1 → xarray_sql-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: xarray_sql
-Version: 0.2.1
+Version: 0.2.2
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Science/Research
 Classifier: Intended Audience :: Developers
@@ -19,9 +19,18 @@ Classifier: Topic :: Database :: Front-Ends
 Requires-Dist: dask>=2024.8.0
 Requires-Dist: datafusion==52.0.0
 Requires-Dist: xarray>=2024.7.0
+Requires-Dist: xarray-sql[docs] ; extra == 'dev'
+Requires-Dist: pre-commit ; extra == 'dev'
+Requires-Dist: pytest ; extra == 'dev'
+Requires-Dist: watchfiles ; extra == 'dev'
+Requires-Dist: zensical ; extra == 'docs'
+Requires-Dist: mkdocstrings[python] ; extra == 'docs'
+Requires-Dist: cftime ; extra == 'test'
 Requires-Dist: pytest ; extra == 'test'
 Requires-Dist: xarray[io] ; extra == 'test'
 Requires-Dist: gcsfs ; extra == 'test'
+Provides-Extra: dev
+Provides-Extra: docs
 Provides-Extra: test
 License-File: LICENSE
 Summary: Querry Xarray with SQL.
@@ -128,11 +137,11 @@ That's it!
 _2025 update_: This library now implements a Dask-like `from_map` interface in
 pure DataFusion and PyArrow, but works with the same principle!
 
-_2026 update_: Instead of `from_map()`, we make factory functions from blocks of
-Xarray datasets that return RecordBatchReaders. These feed into a Rust-based
-DataFusion `TableProvider`. Every chunk is uses the Arrow in memory format to
-translate between Python and Rust. Even still, the core of what makes this idea
-work is the core `pivot()` operation from where this project began!
+_2026 update_: Instead of `from_map()`, we create a way to translate Xarray chunks
+into Arrow RecordBatches. We pass a Python callback into a DataFusion `TableProvider`
+that lets the DB engine translate the underlying Dataset arrays into DataFusion partitions.
+Ultimately, the initial insight of the `pivot()` function -- that any ndarray can be
+translated into a 2D table -- underlies this performant query mechanism. 
 
 ## Why does this work?
 
@@ -150,11 +159,6 @@ early users – "tire kickers", if you will. We'd love your input to shape the d
 project! Please, give this a try and [file issues](https://github.com/alxmrs/xarray-sql/issues) as
 you see fit. Check out our [contributing guide](CONTRIBUTING.md), too 😉.
 
-I can say that for now, the library is oriented towards making whole scans of
-Xarray Datasets. Common filter optimizations (even basic ones like an `.sel()` on
-core dimensions, let alone predicate push downs) are not fully implemented yet.
-However, these operations and more are on our roadmap.
-
 ## What would a deeper integration look like?
 
 I have a few ideas so far. One approach involves applying operations directly on
@@ -169,18 +173,21 @@ and BigQuery. More thoughts on this
 in [#4](https://github.com/alxmrs/xarray-sql/issues/4).
 
 _2025 update_: Something like this is being built across a few projects! The ones I know about are:
+
 - [CartoDB's Raquet](https://github.com/CartoDB/raquet)
 - The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr)
 
-As of writing, this project is [amid integrating](https://github.com/alxmrs/xarray-sql/pull/69) a
-rust-based DataFusion backend provided by arrow-zarr.
+_2026 update_: A colleague and I are experimenting with native Zarr RDBMS engines. Check out:
+
+- [Zarr-Datafusion](https://lib.rs/crates/zarr-datafusion)
+- [DuckDB-Zarr](https://github.com/hobbes-bot/duckdb-zarr)
 
 ## Roadmap
 
 - [x] ~Lazy evaluation via the pyarrow Dataset interface [#93](https://github.com/alxmrs/xarray-sql/issues/93).~ _Implemented in [#100](https://github.com/alxmrs/xarray-sql/pull/100)_
-- [ ] Support proper parallelism via proper partition handling on the rust/datafusion side. [#106](https://github.com/alxmrs/xarray-sql/issues/106)
-- [ ] Support core datafusion optimizations to scan less data, like [104](https://github.com/alxmrs/xarray-sql/issues/104), ...
-- [ ] Translate a single Zarr to a collection of tables via DataFusion's catalog interface [#85](https://github.com/alxmrs/xarray-sql/issues/85).
+- [x] Support proper parallelism via proper partition handling on the rust/datafusion side. [#106](https://github.com/alxmrs/xarray-sql/issues/106)
+- [x] Support core datafusion optimizations to scan less data, like [104](https://github.com/alxmrs/xarray-sql/issues/104), ...
+- [ ] Translate a single Zarr to a collection of tables [#85](https://github.com/alxmrs/xarray-sql/issues/85).
 - [ ] Distributed beyond a single node through the DataFusion integration with Ray Datasets [#68](https://github.com/alxmrs/xarray-sql/issues/68) or Apache Ballista [#98](https://github.com/alxmrs/xarray-sql/issues/98).
 - [ ] Demo: calculate Sea Surface Temperature from 1940 - Present in SQL [#36](https://github.com/alxmrs/xarray-sql/issues/36).
 - [ ] Provide an option to integrate DataFusion directly to Zarr via Rust [#4](https://github.com/alxmrs/xarray-sql/issues/4).
@@ -200,6 +207,8 @@ I want to give a special thanks to the following folks and institutions:
   and DataFusion-specific collaboration.
 - The gracious volunteer data science students at [UCSD's DS3](https://www.ds3atucsd.com/) org,
   who are working to make this library better.
+- Andrew Huang for the sense of taste he brings to the project and consummate code
+  changes.
 
 
 ## License

{xarray_sql-0.2.1 → xarray_sql-0.2.2}/README.md

@@ -94,11 +94,11 @@ That's it!
 _2025 update_: This library now implements a Dask-like `from_map` interface in
 pure DataFusion and PyArrow, but works with the same principle!
 
-_2026 update_: Instead of `from_map()`, we make factory functions from blocks of
-Xarray datasets that return RecordBatchReaders. These feed into a Rust-based
-DataFusion `TableProvider`. Every chunk is uses the Arrow in memory format to
-translate between Python and Rust. Even still, the core of what makes this idea
-work is the core `pivot()` operation from where this project began!
+_2026 update_: Instead of `from_map()`, we create a way to translate Xarray chunks
+into Arrow RecordBatches. We pass a Python callback into a DataFusion `TableProvider`
+that lets the DB engine translate the underlying Dataset arrays into DataFusion partitions.
+Ultimately, the initial insight of the `pivot()` function -- that any ndarray can be
+translated into a 2D table -- underlies this performant query mechanism. 
 
 ## Why does this work?
 
@@ -116,11 +116,6 @@ early users – "tire kickers", if you will. We'd love your input to shape the d
 project! Please, give this a try and [file issues](https://github.com/alxmrs/xarray-sql/issues) as
 you see fit. Check out our [contributing guide](CONTRIBUTING.md), too 😉.
 
-I can say that for now, the library is oriented towards making whole scans of
-Xarray Datasets. Common filter optimizations (even basic ones like an `.sel()` on
-core dimensions, let alone predicate push downs) are not fully implemented yet.
-However, these operations and more are on our roadmap.
-
 ## What would a deeper integration look like?
 
 I have a few ideas so far. One approach involves applying operations directly on
@@ -135,18 +130,21 @@ and BigQuery. More thoughts on this
 in [#4](https://github.com/alxmrs/xarray-sql/issues/4).
 
 _2025 update_: Something like this is being built across a few projects! The ones I know about are:
+
 - [CartoDB's Raquet](https://github.com/CartoDB/raquet)
 - The DataFusion community's [arrow-zarr](https://github.com/datafusion-contrib/arrow-zarr)
 
-As of writing, this project is [amid integrating](https://github.com/alxmrs/xarray-sql/pull/69) a
-rust-based DataFusion backend provided by arrow-zarr.
+_2026 update_: A colleague and I are experimenting with native Zarr RDBMS engines. Check out:
+
+- [Zarr-Datafusion](https://lib.rs/crates/zarr-datafusion)
+- [DuckDB-Zarr](https://github.com/hobbes-bot/duckdb-zarr)
 
 ## Roadmap
 
 - [x] ~Lazy evaluation via the pyarrow Dataset interface [#93](https://github.com/alxmrs/xarray-sql/issues/93).~ _Implemented in [#100](https://github.com/alxmrs/xarray-sql/pull/100)_
-- [ ] Support proper parallelism via proper partition handling on the rust/datafusion side. [#106](https://github.com/alxmrs/xarray-sql/issues/106)
-- [ ] Support core datafusion optimizations to scan less data, like [104](https://github.com/alxmrs/xarray-sql/issues/104), ...
-- [ ] Translate a single Zarr to a collection of tables via DataFusion's catalog interface [#85](https://github.com/alxmrs/xarray-sql/issues/85).
+- [x] Support proper parallelism via proper partition handling on the rust/datafusion side. [#106](https://github.com/alxmrs/xarray-sql/issues/106)
+- [x] Support core datafusion optimizations to scan less data, like [104](https://github.com/alxmrs/xarray-sql/issues/104), ...
+- [ ] Translate a single Zarr to a collection of tables [#85](https://github.com/alxmrs/xarray-sql/issues/85).
 - [ ] Distributed beyond a single node through the DataFusion integration with Ray Datasets [#68](https://github.com/alxmrs/xarray-sql/issues/68) or Apache Ballista [#98](https://github.com/alxmrs/xarray-sql/issues/98).
 - [ ] Demo: calculate Sea Surface Temperature from 1940 - Present in SQL [#36](https://github.com/alxmrs/xarray-sql/issues/36).
 - [ ] Provide an option to integrate DataFusion directly to Zarr via Rust [#4](https://github.com/alxmrs/xarray-sql/issues/4).
@@ -166,6 +164,8 @@ I want to give a special thanks to the following folks and institutions:
   and DataFusion-specific collaboration.
 - The gracious volunteer data science students at [UCSD's DS3](https://www.ds3atucsd.com/) org,
   who are working to make this library better.
+- Andrew Huang for the sense of taste he brings to the project and consummate code
+  changes.
 
 
 ## License

xarray_sql-0.2.2/docs/assets/logo.svg

@@ -0,0 +1,104 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   version="1.1"
+   x="0px"
+   y="0px"
+   viewBox="50 115 420 395"
+   xml:space="preserve"
+   id="svg6"
+   sodipodi:docname="logo
+   inkscape:version="1.3.2 (091e20e, 2023-11-25)"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:svg="http://www.w3.org/2000/svg"><defs
+   id="defs6" /><sodipodi:namedview
+   id="namedview6"
+   pagecolor="#ffffff"
+   bordercolor="#000000"
+   borderopacity="0.25"
+   inkscape:showpageshadow="2"
+   inkscape:pageopacity="0.0"
+   inkscape:pagecheckerboard="0"
+   inkscape:deskcolor="#d1d1d1"
+   showguides="false"
+   inkscape:zoom="0.59746835"
+   inkscape:cx="76.154661"
+   inkscape:cy="63.601695"
+   inkscape:window-width="1320"
+   inkscape:window-height="905"
+   inkscape:window-x="181"
+   inkscape:window-y="34"
+   inkscape:window-maximized="0"
+   inkscape:current-layer="g6" />
+<style
+   type="text/css"
+   id="style1">
+	.st0{fill:#216C89;}
+	.st1{fill:#4993AA;}
+	.st2{fill:#0F4565;}
+	.st3{fill:#6BE8E8;}
+	.st4{fill:#9DEEF4;}
+	.st5{fill:#4ACFDD;}
+	.st6{fill:#E38017;}
+	.st7{fill:#16AFB5;}
+</style>
+<g
+   id="g6">
+	<!-- BOTTOM DATABASE (dark) -->
+	<path
+   class="st2"
+   d="m 65,362.92813 c 0,0 0,96.42499 0,96.42499 0,17.9075 70,27.55 112,27.55 42,0 112,-9.6425 112,-27.55 v -96.42499 c 0,17.90749 -70,27.54999 -112,27.54999 -42,0 -112,-9.6425 -112,-27.54999 z"
+   id="path1"
+   style="stroke-width:0.82991" />
+	<path
+   class="st0"
+   d="m 177,390.47812 c 42,0 112,-9.6425 112,-27.54999 v 96.42499 c 0,17.9075 -70,27.55 -112,27.55 z"
+   opacity="0.4"
+   id="path2"
+   style="stroke-width:0.82991" />
+
+
+
+	<!-- TOP DATABASE (teal) -->
+	<path
+   class="st5"
+   d="m 65,276.83438 c 0,0 0,75.76249 0,75.76249 0,17.90751 70,27.55001 112,27.55001 42,0 112,-9.6425 112,-27.55001 v -75.76249 c 0,17.90749 -70,27.54999 -112,27.54999 -42,0 -112,-9.6425 -112,-27.54999 z"
+   id="path3"
+   style="stroke-width:0.82991" />
+	<path
+   class="st3"
+   d="m 177,304.38437 c 42,0 112,-9.6425 112,-27.54999 v 75.76249 c 0,17.90751 -70,27.55001 -112,27.55001 z"
+   opacity="0.4"
+   id="path4"
+   style="stroke-width:0.82991" />
+	<path
+   class="st5"
+   d="m 66.576,190.40937 c 0,0 0,75.7625 0,75.7625 0,17.90751 70,27.55 112,27.55 42,0 112,-9.64249 112,-27.55 v -75.7625 c 0,17.9075 -70,27.55 -112,27.55 -42,0 -112,-9.6425 -112,-27.55 z"
+   id="path3-5"
+   style="fill:#4acfdd;stroke-width:0.82991" /><path
+   class="st3"
+   d="m 178.576,217.95937 c 42,0 112,-9.6425 112,-27.55 v 75.7625 c 0,17.90751 -70,27.55 -112,27.55 z"
+   opacity="0.4"
+   id="path4-7"
+   style="fill:#6be8e8;stroke-width:0.82991" /><ellipse
+   class="st4"
+   cx="178.576"
+   cy="183.52188"
+   rx="112"
+   ry="27.549999"
+   id="ellipse4-5"
+   style="fill:#9deef4;stroke-width:0.82991" />
+
+
+	<!-- DIAGONAL BARS -->
+	<polygon
+   class="st6"
+   points="377.48,412.74 308.66,482.2 308.66,346.56 377.48,277.09"
+   id="polygon5" />
+	<polygon
+   class="st7"
+   points="457.07,412.74 388.25,482.2 388.25,346.56 457.07,277.09"
+   id="polygon6" />
+</g>
+</svg>

xarray_sql-0.2.2/docs/contributing.md

@@ -0,0 +1 @@
+--8<-- "CONTRIBUTING.md"

xarray_sql-0.2.2/docs/examples.md

@@ -0,0 +1,23 @@
+# Examples
+
+```python
+import xarray as xr
+import xarray_sql as xql
+
+ds = xr.tutorial.open_dataset('air_temperature')
+
+ctx = xql.XarrayContext()
+ctx.from_dataset('air', ds, chunks=dict(time=24))
+
+result = ctx.sql('''
+  SELECT
+    "lat", "lon", AVG("air") as air_avg
+  FROM
+    "air"
+  GROUP BY
+   "lat", "lon"
+''')
+
+df = result.to_pandas()
+df.head()
+```

xarray_sql-0.2.2/docs/index.md

@@ -0,0 +1 @@
+--8<-- "README.md"

xarray_sql-0.2.2/docs/reference/xarray_sql.md

@@ -0,0 +1,8 @@
+# xarray-sql
+
+::: xarray_sql
+    options:
+      show_root_heading: true
+      show_source: false
+      members: true
+      show_submodules: true

{xarray_sql-0.2.1 → xarray_sql-0.2.2}/pyproject.toml

@@ -37,10 +37,21 @@ dependencies = [
 
 [project.optional-dependencies]
 test = [
+  "cftime",
   "pytest",
   "xarray[io]",
   "gcsfs",
 ]
+docs = [
+    "zensical",
+    "mkdocstrings[python]",
+]
+dev = [
+    "xarray_sql[docs]",
+    "pre-commit",
+    "pytest",
+    "watchfiles",
+]
 
 [project.urls]
 Homepage = "https://github.com/alxmrs/xarray-sql"
@@ -51,7 +62,7 @@ features = ["pyo3/extension-module"]
 module-name = "xarray_sql._native"
 
 [tool.setuptools.packages.find]
-exclude = ["demo", "perf_tests"]
+exclude = ["demo", "perf_tests", "tests", "tests.*"]
 
 [tool.pyink]
 line-length = 80
@@ -85,6 +96,7 @@ ignore_missing_imports = true
 [dependency-groups]
 dev = [
     "xarray_sql[test]",
+    "xarray_sql[docs]",
     "py-spy>=0.4.0",
     "pyink>=24.10.1",
     "maturin>=1.9.1",
@@ -93,3 +105,6 @@ dev = [
 [tool.uv]
 # Rebuild package when any rust files change
 cache-keys = [{file = "pyproject.toml"}, {file = "rust/Cargo.toml"}, {file = "**/*.rs"}]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

xarray_sql-0.2.2/tests/conftest.py

@@ -0,0 +1,144 @@
+import pytest
+
+import numpy as np
+import pandas as pd
+import xarray as xr
+
+
+def rand_wx(start: str, end: str) -> xr.Dataset:
+  np.random.seed(42)
+  lat = np.linspace(-90, 90, num=720)
+  lon = np.linspace(-180, 180, num=1440)
+  time = pd.date_range(start, end, freq="h")
+  level = np.array([1000, 500], dtype=np.int32)
+  reference_time = pd.Timestamp(start)
+  temperature = 15 + 8 * np.random.randn(720, 1440, len(time), len(level))
+  precipitation = 10 * np.random.rand(720, 1440, len(time), len(level))
+  return xr.Dataset(
+      data_vars=dict(
+          temperature=(["lat", "lon", "time", "level"], temperature),
+          precipitation=(["lat", "lon", "time", "level"], precipitation),
+      ),
+      coords=dict(
+          lat=lat,
+          lon=lon,
+          time=time,
+          level=level,
+          reference_time=reference_time,
+      ),
+      attrs=dict(description="Random weather."),
+  )
+
+
+def create_large_dataset(time_steps=1000, lat_points=100, lon_points=100):
+  """Create a large xarray dataset for memory testing."""
+  np.random.seed(42)
+
+  time = pd.date_range("2020-01-01", periods=time_steps, freq="h")
+  lat = np.linspace(-90, 90, lat_points)
+  lon = np.linspace(-180, 180, lon_points)
+
+  temp_data = np.random.rand(time_steps, lat_points, lon_points) * 40 - 10
+  precip_data = np.random.rand(time_steps, lat_points, lon_points) * 100
+
+  return xr.Dataset(
+      {
+          "temperature": (["time", "lat", "lon"], temp_data),
+          "precipitation": (["time", "lat", "lon"], precip_data),
+      },
+      coords={"time": time, "lat": lat, "lon": lon},
+  )
+
+
+@pytest.fixture
+def air():
+  ds = xr.tutorial.open_dataset("air_temperature")
+  chunks = {"time": 240}
+  return ds.chunk(chunks)
+
+
+@pytest.fixture
+def air_small(air):
+  return air.isel(time=slice(0, 12), lat=slice(0, 11), lon=slice(0, 10)).chunk(
+      {"time": 240}
+  )
+
+
+@pytest.fixture
+def randwx():
+  return rand_wx("1995-01-13T00", "1995-01-13T01")
+
+
+@pytest.fixture
+def large_ds():
+  return create_large_dataset().chunk({"time": 25})
+
+
+@pytest.fixture
+def air_dataset_small():
+  ds = xr.tutorial.open_dataset("air_temperature").chunk({"time": 240})
+  return ds.isel(time=slice(0, 12), lat=slice(0, 11), lon=slice(0, 10))
+
+
+@pytest.fixture
+def air_dataset_large():
+  return xr.tutorial.open_dataset("air_temperature").chunk({"time": 240})
+
+
+@pytest.fixture
+def rasm_ds():
+  """rasm uses cftime.DatetimeNoLeap (noleap / 365_day) for time."""
+  return xr.tutorial.open_dataset("rasm")
+
+
+@pytest.fixture
+def weather_dataset():
+  ds = rand_wx("2023-01-01T00", "2023-01-01T12")
+  return ds.isel(time=slice(0, 6), lat=slice(0, 10), lon=slice(0, 10)).chunk(
+      {"time": 3}
+  )
+
+
+@pytest.fixture
+def synthetic_dataset():
+  return create_large_dataset(
+      time_steps=50, lat_points=20, lon_points=20
+  ).chunk({"time": 25})
+
+
+@pytest.fixture
+def station_dataset():
+  return xr.Dataset(
+      {
+          "station_id": (["station"], [1, 2, 3, 4, 5]),
+          "elevation": (["station"], [100, 250, 500, 750, 1000]),
+          "name": (
+              ["station"],
+              ["Station_A", "Station_B", "Station_C", "Station_D", "Station_E"],
+          ),
+      }
+  ).chunk({"station": 5})
+
+
+@pytest.fixture
+def air_and_stations():
+  air = (
+      xr.tutorial.open_dataset("air_temperature")
+      .isel(time=slice(0, 12), lat=slice(0, 5), lon=slice(0, 8))
+      .chunk({"time": 6})
+  )
+  stations = xr.Dataset(
+      {
+          "station_id": (["station"], [101, 102, 103]),
+          "lat": (
+              ["station"],
+              [air.lat.values[0], air.lat.values[2], air.lat.values[4]],
+          ),
+          "lon": (
+              ["station"],
+              [air.lon.values[1], air.lon.values[3], air.lon.values[5]],
+          ),
+          "elevation": (["station"], [100, 250, 500]),
+      }
+  ).chunk({"station": 3})
+  return air, stations