ducklakepy 0.1.0.dev0__tar.gz → 0.1.0.dev1__tar.gz

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ducklakepy
-Version: 0.1.0.dev0
+Version: 0.1.0.dev1
 Summary: A Python implementation of the DuckLake specification
 Project-URL: Homepage, https://dejii.github.io/ducklake-python/
 Project-URL: Documentation, https://dejii.github.io/ducklake-python/
@@ -34,11 +34,10 @@ lakehouse format published at [ducklake.select](https://ducklake.select).
 
 DuckLake keeps a table's metadata in a SQL database and its rows in Parquet files.
 ducklakepy speaks to both directly, so a Python program can create tables, append Arrow
-data, read at any snapshot, update and delete rows and evolve schemas without a DuckDB
-process. A
-lake written by ducklakepy is readable by DuckDB's `ducklake` extension, and the other way
-round, so the two can share one lake. If you know PyIceberg, ducklakepy plays the same
-role for DuckLake.
+data, read at any snapshot, update and delete rows and evolve schemas with no query engine
+in the process. A lake written by ducklakepy is readable by DuckDB's `ducklake` extension,
+and the other way round, so the two can share one lake. If you know PyIceberg, ducklakepy
+plays the same role for DuckLake ([the calls side by side](https://dejii.github.io/ducklake-python/docs/getting-started/from-pyiceberg/)).
 
 ```bash
 pip install ducklakepy[postgres]   # with the Postgres catalog driver
@@ -97,9 +96,9 @@ To run the site locally, `make docs` (Node 22.12 or newer; there is a `.nvmrc`)
 
 | | |
 |---|---|
-| [Getting started](https://dejii.github.io/ducklake-python/docs/getting-started/) | Install, then a quickstart: a lake on SQLite, written and read back from Python and from DuckDB |
+| [Getting started](https://dejii.github.io/ducklake-python/docs/getting-started/) | Install, a quickstart on SQLite, moving to Postgres and S3, coming from PyIceberg, and troubleshooting |
 | [Concepts](https://dejii.github.io/ducklake-python/docs/concepts/) | How DuckLake works: snapshots, tables and schemas, where rows live, the ducklakepy object model, a glossary |
-| [Guides](https://dejii.github.io/ducklake-python/docs/guides/catalogs/) | One task per page: connect to a catalog, create tables, write, read, time travel, change feeds, update, delete, evolve a schema, transactions, partition, sort, flush, maintain, register existing files, object storage, encryption, distributed reads, Spark, use with DuckDB |
+| [Guides](https://dejii.github.io/ducklake-python/docs/guides/) | One task per page: connect to a catalog, create tables, write, read, time travel, change feeds, update, delete, evolve a schema, transactions, partition, sort, flush, maintain, register existing files, object storage, encryption, use with DuckDB, distributed reads, Spark |
 | [Reference](https://dejii.github.io/ducklake-python/docs/reference/support/) | Feature support by table layout, types, the expression DSL, configuration, write options, errors, catalog backends |
 | [API reference](https://dejii.github.io/ducklake-python/docs/api/) | Every public module, generated from the docstrings |
 | [Internals](https://dejii.github.io/ducklake-python/docs/internals/) | How it works underneath the API, and why |

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/README.md

@@ -5,11 +5,10 @@ lakehouse format published at [ducklake.select](https://ducklake.select).
 
 DuckLake keeps a table's metadata in a SQL database and its rows in Parquet files.
 ducklakepy speaks to both directly, so a Python program can create tables, append Arrow
-data, read at any snapshot, update and delete rows and evolve schemas without a DuckDB
-process. A
-lake written by ducklakepy is readable by DuckDB's `ducklake` extension, and the other way
-round, so the two can share one lake. If you know PyIceberg, ducklakepy plays the same
-role for DuckLake.
+data, read at any snapshot, update and delete rows and evolve schemas with no query engine
+in the process. A lake written by ducklakepy is readable by DuckDB's `ducklake` extension,
+and the other way round, so the two can share one lake. If you know PyIceberg, ducklakepy
+plays the same role for DuckLake ([the calls side by side](https://dejii.github.io/ducklake-python/docs/getting-started/from-pyiceberg/)).
 
 ```bash
 pip install ducklakepy[postgres]   # with the Postgres catalog driver
@@ -68,9 +67,9 @@ To run the site locally, `make docs` (Node 22.12 or newer; there is a `.nvmrc`)
 
 | | |
 |---|---|
-| [Getting started](https://dejii.github.io/ducklake-python/docs/getting-started/) | Install, then a quickstart: a lake on SQLite, written and read back from Python and from DuckDB |
+| [Getting started](https://dejii.github.io/ducklake-python/docs/getting-started/) | Install, a quickstart on SQLite, moving to Postgres and S3, coming from PyIceberg, and troubleshooting |
 | [Concepts](https://dejii.github.io/ducklake-python/docs/concepts/) | How DuckLake works: snapshots, tables and schemas, where rows live, the ducklakepy object model, a glossary |
-| [Guides](https://dejii.github.io/ducklake-python/docs/guides/catalogs/) | One task per page: connect to a catalog, create tables, write, read, time travel, change feeds, update, delete, evolve a schema, transactions, partition, sort, flush, maintain, register existing files, object storage, encryption, distributed reads, Spark, use with DuckDB |
+| [Guides](https://dejii.github.io/ducklake-python/docs/guides/) | One task per page: connect to a catalog, create tables, write, read, time travel, change feeds, update, delete, evolve a schema, transactions, partition, sort, flush, maintain, register existing files, object storage, encryption, use with DuckDB, distributed reads, Spark |
 | [Reference](https://dejii.github.io/ducklake-python/docs/reference/support/) | Feature support by table layout, types, the expression DSL, configuration, write options, errors, catalog backends |
 | [API reference](https://dejii.github.io/ducklake-python/docs/api/) | Every public module, generated from the docstrings |
 | [Internals](https://dejii.github.io/ducklake-python/docs/internals/) | How it works underneath the API, and why |

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ducklakepy"
-version = "0.1.0.dev0"
+version = "0.1.0.dev1"
 description = "A Python implementation of the DuckLake specification"
 readme = "README.md"
 requires-python = ">=3.11"

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/src/ducklakepy/_arrow.py

@@ -19,13 +19,14 @@ own and each become a Parquet column of their own.
 from __future__ import annotations
 
 import uuid as _uuid
+from datetime import UTC, datetime
 from decimal import Decimal
 from typing import TYPE_CHECKING, Any
 
 import pyarrow as pa
 import pyarrow.compute as pc
 
-from ducklakepy._defaults import assign_default, string_literal_value
+from ducklakepy._defaults import assign_default, expression_default_values, string_literal_value
 from ducklakepy._interval import INTERVAL, decoded_interval
 from ducklakepy._lists import as_offset_list, is_parquet_list
 from ducklakepy._logical_values import (
@@ -42,7 +43,7 @@ from ducklakepy.constants import PARQUET_FIELD_ID_KEY
 from ducklakepy.exceptions import DuckLakeError, SchemaMismatchError
 from ducklakepy.schema import Schema, check_unique_column_names, fold_identifier
 from ducklakepy.stats import as_infinite, infinite_text
-from ducklakepy.stats._temporal import far_temporal
+from ducklakepy.stats._temporal import END_OF_DAY, far_temporal
 from ducklakepy.types import (
     BlobType,
     BooleanType,
@@ -239,9 +240,13 @@ def python_values(column: pa.Array | pa.ChunkedArray, field_type: DuckLakeType)
     An infinite DATE or TIMESTAMP is :class:`~ducklakepy.stats.Infinite` and
     one beyond ``date``'s years
     :class:`~ducklakepy.stats._temporal.FarTemporal`, whose text is what the
-    row stores."""
+    row stores. A TIME of ``24:00:00`` is
+    :data:`~ducklakepy.stats._temporal.END_OF_DAY`, which pyarrow would hand
+    back as ``00:00:00``."""
     if isinstance(field_type, (TimestampNsType, TimestampTzNsType, TimeNsType)):
         column = column.cast(pa.int64())
+    if isinstance(field_type, TimeType):
+        return _times(column)
     try:
         values: list[Any] = column.to_pylist()
     except (OverflowError, ValueError):
@@ -252,6 +257,14 @@ def python_values(column: pa.Array | pa.ChunkedArray, field_type: DuckLakeType)
     return values
 
 
+def _times(column: pa.Array | pa.ChunkedArray) -> list[Any]:
+    values: list[Any] = column.to_pylist()
+    ends = pc.equal(column.cast(pa.time64("us")).cast(pa.int64()), END_OF_DAY)
+    for position in pc.indices_nonzero(pc.fill_null(ends, False)).to_pylist():
+        values[position] = END_OF_DAY
+    return values
+
+
 def _temporal_python_value(field_type: DuckLakeType, arrow_type: pa.DataType, raw: int | None) -> Any:
     if raw is None:
         return None
@@ -262,12 +275,14 @@ def _temporal_python_value(field_type: DuckLakeType, arrow_type: pa.DataType, ra
     return pa.scalar(raw, type=arrow_type).as_py()
 
 
-def align_to_schema(arrow_table: pa.Table, schema: Schema) -> pa.Table:
+def align_to_schema(arrow_table: pa.Table, schema: Schema, *, moment: datetime | None = None) -> pa.Table:
     """Reorder, fill and cast ``arrow_table`` so it matches ``schema`` exactly.
 
     Unknown columns are an error; a missing column takes its ``default_value``
     (or NULL when it has none and is nullable), unless that default is an
-    expression, which only a SQL engine can evaluate; values are cast to the
+    expression only a SQL engine can evaluate (``now()`` and the few others
+    :func:`~ducklakepy._defaults.expression_default_values` names are taken
+    at ``moment``, the transaction's time, or now); values are cast to the
     column's canonical Arrow type with pyarrow's safe cast, so an int64 that
     doesn't fit an int32 column is an error rather than a silent wrap.
 
@@ -296,7 +311,7 @@ def align_to_schema(arrow_table: pa.Table, schema: Schema) -> pa.Table:
     for column in schema.fields:
         batch_name = batch_by_fold.get(fold_identifier(column.name))
         if batch_name is None:
-            arrays.append(_filled(column, n, f"column {column.name!r}"))
+            arrays.append(_filled(column, n, f"column {column.name!r}", moment))
             continue
         source = _fitted(arrow_table.column(batch_name), column, column.name)
         if column.required and source.null_count:
@@ -393,7 +408,7 @@ def _fitted_fields(array: pa.Array, field_type: StructType, path: str) -> list[p
         if folded in positions:
             raise SchemaMismatchError(
                 f"column {path!r}: the batch's fields {spelled[folded]!r} and {name!r} name the same struct "
-                "field (identifiers fold ASCII case, as DuckDB's do)"
+                "field (field names compare case-insensitively)"
             )
         positions[folded] = position
         spelled[folded] = name
@@ -441,11 +456,12 @@ def _omitted(child: Column, n: int, path: str) -> pa.Array:
     return arrow_array_for(child, [None] * n)
 
 
-def _filled(column: Column, n: int, where: str) -> pa.Array:
+def _filled(column: Column, n: int, where: str, moment: datetime | None) -> pa.Array:
     """The ``n`` values a column or struct field the batch does not carry
     takes: its ``default_value``, or NULL when it has none and is nullable.
     An expression default is evaluated only when it is one SQL string
-    literal, which is how the spec stores a literal whose text is ``NULL``."""
+    literal, which is how the spec stores a literal whose text is ``NULL``,
+    or one of the zero-argument functions ducklakepy evaluates at ``moment``."""
     default = column.default
     if default is None:
         if column.required:
@@ -453,6 +469,15 @@ def _filled(column: Column, n: int, where: str) -> pa.Array:
         fill = None
     else:
         text = string_literal_value(default.value) if default.is_expression else default.value
+        evaluated = (
+            None
+            if text is not None
+            else expression_default_values(column.field_type, default.value, n, moment or datetime.now(UTC))
+        )
+        if evaluated is not None:
+            array = arrow_array_for(column, evaluated)
+            check_logical_values(column, array, where=where)
+            return array
         if text is None:
             # Only a SQL engine can evaluate one of these; filling the column
             # with its text would write "CAST(uuid() AS VARCHAR)" as the value.

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/src/ducklakepy/_defaults.py

@@ -41,7 +41,9 @@ blob                    DuckDB's own blob text: an ASCII character is its
                         *Not* the spec's hex bounds encoding, which stays
                         with statistics.
 uuid                    the standard UUID spelling
-date / time             ISO 8601
+date                    ISO 8601
+time                    ISO 8601, up to ``24:00:00``, which is
+                        ``END_OF_DAY``
 time_ns                 ISO 8601, as nanoseconds since midnight up to
                         ``24:00:00``
 timestamp / _s / _ms    ISO 8601, rounded to the type's own precision
@@ -72,7 +74,7 @@ from ducklakepy.stats import (
     render_blob_text,
     split_utc_offset,
 )
-from ducklakepy.stats._temporal import parse_time_ns_text
+from ducklakepy.stats._temporal import SUBSECOND_DIGITS, parse_time_ns_text
 from ducklakepy.types import (
     BlobType,
     BooleanType,
@@ -376,10 +378,49 @@ def _round_to_unit(value: int, unit: int) -> int:
     return magnitude * unit * (-1 if value < 0 else 1)
 
 
+# The zero-argument expression defaults ducklakepy evaluates, as the text
+# DuckDB stores for each: a function's name lower case, a keyword in the case
+# it was written (measured against DuckDB 2.0.0), so the text is compared
+# case-insensitively and nothing else is parsed. Anything else needs a SQL
+# engine.
+_NOW = frozenset({"now()", "current_timestamp", "transaction_timestamp()", "get_current_timestamp()"})
+_TODAY = frozenset({"current_date", "today()"})
+_RANDOM_UUID = frozenset({"gen_random_uuid()", "uuid()", "uuidv4()"})
+_ZONED = (TimestampTzType, TimestampTzNsType)
+_NAIVE = (TimestampType, TimestampSType, TimestampMsType, TimestampNsType)
+
+
+def expression_default_values(field_type: DuckLakeType, expression: str, n: int, moment: datetime) -> list[Any] | None:
+    """The ``n`` values an insert that omits a column gets from its default
+    ``expression``, or None for an expression ducklakepy does not evaluate.
+
+    ``now()`` and ``current_timestamp`` are the transaction's ``moment``, one
+    value for every row, as DuckDB takes its transaction's start; a timestamp
+    without a zone takes its UTC wall time and ``current_date`` its UTC date,
+    which is what a DuckDB session set to UTC writes. ``gen_random_uuid()``
+    is a fresh UUID for each row. Each fills only the type it yields."""
+    text = expression.strip().lower()
+    utc = moment.astimezone(UTC)
+    if text in _NOW and isinstance(field_type, _ZONED):
+        return [utc] * n
+    if text in _NOW and isinstance(field_type, _NAIVE):
+        digits = SUBSECOND_DIGITS[type(field_type)]
+        wall = utc.replace(tzinfo=None)
+        if digits < 6:
+            wall = wall.replace(microsecond=wall.microsecond - wall.microsecond % 10 ** (6 - digits))
+        return [wall] * n
+    if text in _TODAY and isinstance(field_type, DateType):
+        return [utc.date()] * n
+    if text in _RANDOM_UUID and isinstance(field_type, UuidType):
+        return [_uuid.uuid4().bytes for _ in range(n)]
+    return None
+
+
 __all__ = [
     "INT_RANGES",
     "assign_default",
     "encode_default",
+    "expression_default_values",
     "integer_digits",
     "literal_default",
     "render_default",

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/src/ducklakepy/_hive.py

@@ -164,8 +164,7 @@ def hive_column_value(column_type: DuckLakeType, text: str | None, *, where: str
     """One path segment as a value of ``column_type``, or None for
     :data:`HIVE_NULL`. The conversion is DuckDB's VARCHAR cast
     (:func:`~ducklakepy._defaults.assign_default`), and a text the type
-    cannot take raises: ``Unable to cast 'p1' (from hive partition column
-    'PART_KEY') to: 'INTEGER'``.
+    cannot take raises :class:`~ducklakepy.exceptions.SchemaMismatchError`.
 
     ``infinity`` and ``-infinity`` are read first, because they are a
     spelling a DATE or TIMESTAMP path really carries -- DuckDB writes such a
@@ -188,8 +187,8 @@ def hive_key_value(key: PathPartitionKey, text: str | None, *, where: str) -> An
     None for :data:`HIVE_NULL`.
 
     Identity keys take the column's own type; every other transform's key is
-    the integer DuckDB reads it as, so ``year=abc`` is refused with ``Unable
-    to cast 'abc' (from hive partition column 'YEAR') to: 'BIGINT'``.
+    an integer, so ``year=abc`` raises
+    :class:`~ducklakepy.exceptions.SchemaMismatchError`.
     Whether a ``bucket`` key's integer is a bucket the specification has is
     :func:`is_valid_key_value`'s question, asked later and answered
     differently.

ducklakepy-0.1.0.dev1/src/ducklakepy/_inlined_lists.py

@@ -0,0 +1,113 @@
+"""A column-wide read of DuckDB's inlined text for a list of numbers, booleans
+or plain strings.
+
+:mod:`ducklakepy._inlined_text` reads DuckDB's VARCHAR rendering of a nested
+value one character at a time, which is exact and slow: a list of 100,000
+integers takes about a second. Most lists DuckDB inlines are flat lists of
+such leaves, and their text is simple enough for Arrow's string kernels to
+split. DuckDB quotes a leaf whose text is empty, has a space at either end,
+reads ``null`` in any case or holds one of ``" ' ( ) , : = [ ] { }``, so in a
+column whose texts hold none of those characters (nor a backslash), every
+comma separates two items and every item is a bare leaf.
+
+:func:`fast_list_column` reads such a column, and answers None for anything
+else: another element type, a text with one of those characters, or a leaf
+not spelled the way DuckDB spells that type. The caller then reads the column
+with the exact parser, so the values never depend on which path read them.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import pyarrow as pa
+import pyarrow.compute as pc
+
+from ducklakepy._arrow import arrow_type_for
+from ducklakepy.types import (
+    BooleanType,
+    DecimalType,
+    Float32Type,
+    Float64Type,
+    Int8Type,
+    Int16Type,
+    Int32Type,
+    Int64Type,
+    ListType,
+    UInt8Type,
+    UInt16Type,
+    UInt32Type,
+    UInt64Type,
+    VarcharType,
+)
+
+if TYPE_CHECKING:
+    from collections.abc import Sequence
+
+    from ducklakepy.types import DuckLakeType
+
+# What DuckDB's ``StringUtil::CharacterIsSpace`` takes for a space.
+_SPACES = " \t\n\v\f\r"
+
+# A character that makes a text more than bare leaves and commas.
+_STRUCTURE = r"""['"\\()\[\]{}:=]"""
+
+_INTEGER = r"^-?[0-9]+$"
+# Each leaf as DuckDB's VARCHAR cast spells the type, which Arrow's cast
+# reads as the exact parser does. Arrow reads more than this (``0x10`` as 16,
+# for one), so anything else goes to the exact parser.
+_LEAF_SPELLINGS: dict[type[DuckLakeType], str] = {
+    Int8Type: _INTEGER,
+    Int16Type: _INTEGER,
+    Int32Type: _INTEGER,
+    Int64Type: _INTEGER,
+    UInt8Type: _INTEGER,
+    UInt16Type: _INTEGER,
+    UInt32Type: _INTEGER,
+    UInt64Type: _INTEGER,
+    Float32Type: r"^-?([0-9]+(\.[0-9]*)?([eE][-+]?[0-9]+)?|inf|nan)$",
+    Float64Type: r"^-?([0-9]+(\.[0-9]*)?([eE][-+]?[0-9]+)?|inf|nan)$",
+    DecimalType: r"^-?[0-9]+(\.[0-9]+)?$",
+    BooleanType: r"^(true|false)$",
+    VarcharType: r"",
+}
+
+
+def fast_list_column(field_type: DuckLakeType, raw: Sequence[Any]) -> pa.Array | None:
+    """``raw``, the inlined texts of a ``field_type`` column, as an array of its
+    Arrow type, or None when the column is not one this reads."""
+    if not isinstance(field_type, ListType) or any(text is not None and not isinstance(text, str) for text in raw):
+        return None
+    element = field_type.element.field_type
+    spelling = _LEAF_SPELLINGS.get(type(element))
+    if spelling is None:
+        return None
+    texts = pc.utf8_trim(pa.array(raw, pa.string()), _SPACES)
+    present = texts.drop_null()
+    if not pc.all(pc.and_(pc.starts_with(present, "["), pc.ends_with(present, "]")), min_count=0).as_py():
+        return None
+    inner = pc.fill_null(pc.utf8_slice_codeunits(texts, 1, -1), "")
+    if pc.any(pc.match_substring_regex(inner, _STRUCTURE)).as_py():
+        return None
+    split = pc.split_pattern(inner, ",")
+    # ``[]`` and ``[ ]`` are empty lists, where splitting finds one empty item.
+    empty = pc.equal(pc.utf8_trim(inner, _SPACES), "")
+    items = pc.utf8_trim(split.flatten(), _SPACES)
+    items = items.filter(pc.invert(pc.take(empty, pc.list_parent_indices(split))))
+    items = pc.if_else(pc.equal(pc.ascii_lower(items), "null"), pa.scalar(None, pa.string()), items)
+    leaves = items.drop_null()
+    if spelling and not pc.all(pc.match_substring_regex(leaves, spelling), min_count=0).as_py():
+        return None
+    try:
+        values = items.cast(arrow_type_for(element))
+    except (pa.ArrowInvalid, pa.ArrowNotImplementedError):
+        return None  # past the type's range or scale, which the exact parser reports or rounds
+    offsets = [0]
+    for count, is_empty in zip(pc.list_value_length(split).to_pylist(), empty.to_pylist(), strict=True):
+        offsets.append(offsets[-1] + (0 if is_empty else count))
+    return pa.ListArray.from_arrays(
+        pa.array(offsets, pa.int32()), values, type=arrow_type_for(field_type), mask=texts.is_null()
+    )
+
+
+__all__ = ["fast_list_column"]

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/src/ducklakepy/_inlined_text.py

@@ -38,7 +38,7 @@ from it.
 from __future__ import annotations
 
 import uuid as _uuid
-from datetime import UTC, date, datetime, time
+from datetime import time
 from decimal import Decimal
 from typing import TYPE_CHECKING, Any
 
@@ -46,6 +46,7 @@ import pyarrow as pa
 
 from ducklakepy._arrow import arrow_type_for, project, widen, with_field_ids
 from ducklakepy._defaults import assign_default
+from ducklakepy._inlined_lists import fast_list_column
 from ducklakepy._interval import INTERVAL
 from ducklakepy._interval_text import IntervalTextError, rendered_interval_parts
 from ducklakepy._json_list import json_list
@@ -53,7 +54,7 @@ from ducklakepy._variant import inlined_value
 from ducklakepy.exceptions import DuckLakeError, SchemaMismatchError
 from ducklakepy.schema import NESTED_TYPES, Schema
 from ducklakepy.stats import infinite_text, parse_blob_text
-from ducklakepy.stats._temporal import far_temporal_text
+from ducklakepy.stats._temporal import far_temporal_text, utc_time_of_day
 from ducklakepy.types import (
     BlobType,
     DuckLakeType,
@@ -112,15 +113,17 @@ def rendered_column(
     the column.
     """
     field_type = source.field_type
-    values: list[Any]
     try:
-        if isinstance(field_type, VariantType):
-            values = [inlined_value(_stored_bytes(decode(field_type, value))) for value in raw]
-        elif isinstance(field_type, IntervalType):
-            values = [_interval(decode(field_type, value)) for value in raw]
-        else:
-            values = [None if text is None else _cast(_stored_text(text), field_type) for text in raw]
-        stored = pa.array(values, type=arrow_type_for(field_type))
+        stored = fast_list_column(field_type, raw)
+        if stored is None:
+            values: list[Any]
+            if isinstance(field_type, VariantType):
+                values = [inlined_value(_stored_bytes(decode(field_type, value))) for value in raw]
+            elif isinstance(field_type, IntervalType):
+                values = [_interval(decode(field_type, value)) for value in raw]
+            else:
+                values = [None if text is None else _cast(_stored_text(text), field_type) for text in raw]
+            stored = pa.array(values, type=arrow_type_for(field_type))
     except (DuckLakeError, ValueError, ArithmeticError, pa.ArrowException) as error:
         raise DuckLakeError(f"cannot read column {source.name!r} of inlined table {table_name!r}: {error}") from None
     if not isinstance(field_type, NESTED_TYPES):
@@ -447,7 +450,7 @@ def _leaf(field_type: DuckLakeType, text: str) -> Any:
     if isinstance(field_type, (Int128Type, UInt128Type)):
         return Decimal(int(text.strip()))
     if isinstance(field_type, TimeTzType):
-        return _utc_time_of_day(time.fromisoformat(text.strip()))
+        return utc_time_of_day(time.fromisoformat(text.strip()))
     if isinstance(field_type, NullType):
         raise ValueError(f"a null column holds {text!r}")
     if (infinite := infinite_text(field_type, text)) is not None:
@@ -460,11 +463,4 @@ def _leaf(field_type: DuckLakeType, text: str) -> Any:
         raise ValueError(str(error)) from None
 
 
-def _utc_time_of_day(value: time) -> time:
-    """The UTC time of day a ``timetz`` names, as a file holds it."""
-    if value.tzinfo is None:
-        return value
-    return datetime.combine(date(2000, 1, 1), value).astimezone(UTC).time()
-
-
 __all__ = ["is_rendered", "rendered_column"]

ducklakepy-0.1.0.dev1/src/ducklakepy/_interval.py

@@ -0,0 +1,100 @@
+"""The Arrow shape a DuckLake interval column is stored in and read as.
+
+Parquet holds an interval in the twelve bytes of its INTERVAL converted type:
+months, days and milliseconds, each a little-endian uint32, which is what a
+file's column reads back as (:data:`STORED_INTERVAL`). Arrow has a type for
+the same three parts, :data:`INTERVAL`, and that is what a reader of the
+column gets: milliseconds scale to nanoseconds exactly, so nothing is lost,
+and it is the type DuckDB's own Arrow output uses for an interval, so the
+values of both implementations compare equal.
+
+A file written here holds the same twelve bytes (:func:`stored_interval`),
+and :mod:`ducklakepy._parquet_types` marks the column with the INTERVAL
+converted type, which pyarrow does not write.
+"""
+
+from __future__ import annotations
+
+import struct
+
+import numpy as np
+import pyarrow as pa
+
+from ducklakepy.exceptions import DuckLakeError, SchemaMismatchError
+
+STORED_INTERVAL = pa.binary(12)
+"""What a Parquet INTERVAL column reads back as."""
+
+INTERVAL = pa.month_day_nano_interval()
+"""The type the column is handed back as: months, days and nanoseconds."""
+
+_PARTS = struct.Struct("<III")
+"""The twelve bytes, as Parquet's INTERVAL spells them."""
+
+_MILLISECOND_NANOS = 1_000_000
+_DAY_MILLISECONDS = 86_400_000
+_PARTS_DTYPE = np.dtype([("months", "<i4"), ("days", "<i4"), ("nanos", "<i8")])
+"""How Arrow lays out one month_day_nano value."""
+_STORED_DTYPE = np.dtype([("months", "<u4"), ("days", "<u4"), ("milliseconds", "<u4")])
+
+
+def decoded_interval(column: pa.Array | pa.ChunkedArray, where: str) -> pa.Array:
+    """``column``, twelve bytes a row, as an :data:`INTERVAL` array.
+
+    ``where`` names the column in a refusal. Parquet counts all three parts
+    unsigned where Arrow counts months and days signed, so a months or days
+    past Arrow's range is refused rather than handed back as a negative:
+    DuckDB writes no such value (its own months and days are signed 32-bit,
+    and it refuses a negative interval into Parquet outright), so a value
+    that wide comes from a writer whose intent is not knowable here.
+    """
+    if column.type != STORED_INTERVAL:
+        raise SchemaMismatchError(
+            f"{where} is an interval stored as {column.type}, and Parquet holds an interval in the "
+            f"{STORED_INTERVAL} of its INTERVAL converted type"
+        )
+    parts = [None if raw is None else _months_days_nanos(raw) for raw in column.to_pylist()]
+    try:
+        return pa.array(parts, type=INTERVAL)
+    except (pa.ArrowInvalid, OverflowError) as error:
+        raise DuckLakeError(f"{where} holds an interval Arrow's {INTERVAL} cannot express: {error}") from error
+
+
+def stored_interval(column: pa.Array | pa.ChunkedArray, where: str) -> pa.Array:
+    """``column``, an :data:`INTERVAL` array, as the twelve bytes a Parquet
+    INTERVAL holds, the way DuckDB 2.0.0 writes its own (measured): the time
+    is cut to whole milliseconds, its whole days are carried into the days,
+    and the months are kept apart. ``25 hours 1500 microseconds`` is stored as
+    1 day and 3,600,001 milliseconds.
+
+    ``where`` names the column in a refusal. A negative part is refused, as
+    DuckDB refuses it ("Parquet files do not support negative intervals"), and
+    so are days that pass a signed 32-bit count once the time's days are
+    carried in, which DuckDB writes but reads back negative.
+    """
+    array = column.combine_chunks() if isinstance(column, pa.ChunkedArray) else column
+    if len(array) == 0:
+        return pa.array([], STORED_INTERVAL)
+    parts = np.frombuffer(array.buffers()[1], _PARTS_DTYPE, count=array.offset + len(array))[array.offset :]
+    valid = np.ones(len(array), bool) if array.null_count == 0 else array.is_valid().to_numpy(zero_copy_only=False)
+    months, days, nanos = (parts[name][valid].astype(np.int64) for name in ("months", "days", "nanos"))
+    if (months < 0).any() or (days < 0).any() or (nanos < 0).any():
+        raise SchemaMismatchError(f"{where} holds a negative interval, which a Parquet INTERVAL cannot store")
+    carried, milliseconds = np.divmod(nanos // _MILLISECOND_NANOS, _DAY_MILLISECONDS)
+    days = days + carried
+    if (days > 2**31 - 1).any():
+        raise SchemaMismatchError(
+            f"{where} holds an interval whose days pass 2147483647 once its time's whole days are carried in"
+        )
+    stored = np.zeros(len(array), _STORED_DTYPE)
+    stored["months"][valid], stored["days"][valid], stored["milliseconds"][valid] = months, days, milliseconds
+    validity = None if array.null_count == 0 else array.is_valid().buffers()[1]
+    return pa.Array.from_buffers(STORED_INTERVAL, len(array), [validity, pa.py_buffer(stored.tobytes())])
+
+
+def _months_days_nanos(raw: bytes) -> tuple[int, int, int]:
+    months, days, milliseconds = _PARTS.unpack(raw)
+    return months, days, milliseconds * _MILLISECOND_NANOS
+
+
+__all__ = ["INTERVAL", "STORED_INTERVAL", "decoded_interval", "stored_interval"]

{ducklakepy-0.1.0.dev0 → ducklakepy-0.1.0.dev1}/src/ducklakepy/_parquet_sink.py

@@ -2,9 +2,10 @@
 measured without reading it back.
 
 A plaintext file streams to storage. One whose bytes have to change once
-pyarrow has written them, by encryption (:mod:`ducklakepy.encryption`) or by
+pyarrow has written them, by encryption (:mod:`ducklakepy.encryption`), by
 the ``TIME`` flags pyarrow cannot set per column
-(:mod:`ducklakepy._time_flags`), is built in memory and only its finished
+(:mod:`ducklakepy._time_flags`) or by the annotations it does not write
+(:mod:`ducklakepy._parquet_types`), is built in memory and only its finished
 bytes reach storage.
 """
 
@@ -15,6 +16,7 @@ from typing import TYPE_CHECKING
 
 import pyarrow as pa
 
+from ducklakepy._parquet_types import Annotations, annotated, footer_length
 from ducklakepy._time_flags import flagged
 from ducklakepy.encryption import encrypt_parquet, key_bytes
 
@@ -77,6 +79,7 @@ def _close_and_measure(
     to: _Destination,
     encryption_key: str | None = None,
     time_patch: Mapping[int, bool] | None = None,
+    annotations: Annotations | None = None,
 ) -> _Measured:
     """Close a file and measure it from what the writer already knows, so
     nothing is read back: ``ducklake_data_file`` wants the file's size, its
@@ -93,7 +96,9 @@ def _close_and_measure(
 
     A buffered file is finished here instead: the ``TIME`` flags
     ``time_patch`` names are set (:func:`~ducklakepy._time_flags.flagged`),
-    which moves no byte's offset, then the Parquet is encrypted
+    which moves no byte's offset, and the ``annotations`` added
+    (:func:`~ducklakepy._parquet_types.annotated`), which lengthen the footer
+    and so have its size measured from the patched file. Then the Parquet is encrypted
     (:func:`~ducklakepy.encryption.encrypt_parquet`) when ``encryption_key``
     is given, and the result written to ``to``. An encrypted file's size and
     footer size are the ciphertext's, and DuckDB refuses a file whose stored
@@ -116,11 +121,13 @@ def _close_and_measure(
     data = buffered.to_pybytes()
     if time_patch:
         data = flagged(data, time_patch)
+    if annotations:
+        data = annotated(data, annotations)
     if encryption_key is not None:
         encrypted = encrypt_parquet(data, key_bytes(encryption_key))
         data, footer_size = encrypted.data, encrypted.footer_size
     else:
-        footer_size = _footer_size(metadata)
+        footer_size = footer_length(data) if annotations else _footer_size(metadata)
     written = _open_output(to.fs, to.fs_path)
     try:
         written.write(data)