pypersistable 1.0.0__tar.gz

pypersistable-1.0.0/.github/workflows/release.yml

@@ -0,0 +1,19 @@
+name: release
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: astral-sh/setup-uv@v6
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

pypersistable-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nick Young
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pypersistable-1.0.0/PKG-INFO

@@ -0,0 +1,214 @@
+Metadata-Version: 2.4
+Name: pypersistable
+Version: 1.0.0
+Summary: Core Object Model
+Author-email: Nick Young <nick.young.github@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2026 Nick Young
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: homepage, https://github.com/nickyoung-github/persistable
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: anyschema
+Requires-Dist: duckdb
+Requires-Dist: fsspec
+Requires-Dist: narwhals
+Requires-Dist: orjson
+Requires-Dist: pyarrow
+Requires-Dist: pydantic>2.0
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Dynamic: license-file
+
+# pypersistable: Define and persist python objects with minimal boilerplate
+
+Any object that can be handled by `pydantic.TypeAdaptor` will work (pydantic models, dataclasses etc.)
+Objects can be persisted as JSON or parquet (with some caveats).
+
+
+## Defining persistable objects
+
+### `TypeRegistry`
+Persistable types must be registered with `TypeRegistry`. This can be done explicitly or using entry points.
+While we could have avoided a registry, the alternative is persisting implementation-specific info,
+such as the module and class names, making code refactoring very difficult.
+You can optionally supply a `root` parameter to group types, e.g., you might specify a root of `Animals`
+for classes `Dog`, `Cat`, `Bunyip` etc. See [DiscriminatedUnion](#discriminatedunion) for another use of root.
+
+### `Persistable`
+`Persistable` is the abstract base class for persistable objects. All it does it provide an `object_id`
+for serialisation and ways to form that id for retrieval. There are two species of `Persistable`:
+
+- `Unique` - As the name suggests, this is the base for types where all fields form the id. E.g.,
+```python
+class Address(Unique):
+    street_number: str  # Could be 221B, for example
+    street_name: str
+    town: str
+    postcode: str
+    country: Country
+```
+- `Identifiable` - A base class for objects where a subset of fields form their id. Such fields
+are declared using `IdField`. E.g.,
+```python
+class Customer(Identifiable):
+    name: IdField[str]
+    address: Address
+```
+
+### `Reference`
+In classes which have user-defined types as attributes, do we want those nested objects to be persisted
+by value in the parent object's payload or saved separately, with the parent persisting a reference?
+The answer is very much, "it depends". If you definitely want the nested payload to be embedded, that is
+easy: simply declare your type like `Customer` above. However, you could instead re-write it as:
+
+```python
+class Customer(Identifiable):
+    name: IdField[str]
+    address: Reference[Address]
+```
+
+This means that if we had multiple customers (e.g. family members) living at the same address,
+when persisting those customers, the `Address` would be persisted once and shared, with a reference being
+saved in the `Customer` payload.
+
+You can force references to be serialised by value via the `by_reference` paramter on `ObjectStore.write`.
+`ObjectStore.read` will always check for persisted references.
+
+### `DiscriminatedUnion`
+Polymorphism in persisted classes tends to work quite badly and should be avoided. Persistable objects
+often also get used in contexts where we need to know a schema, such as web services.
+`fastapi` based services, using pydantic models is a good example.
+
+A better approach is to declare the valid types using a python `Union`. pydantic can try and infer the
+actual type on deserialisation but it can be preferable to indicate that explicitly.
+`DiscriminatedUnion` does this for you by:
+- Including a type specifier in the JSON schema 
+- Adding that type specifier (which uses `TypeRegistry.registered_name`) to the serialisation payload 
+- Extracting and removing the type specifier from the deserialisation payload and selecting the appropriate type
+
+`DiscriminatedUnion` can be combined with `__subclasses__` with one major caveat:
+you need to ensure that all subclasses of your type have been imported or you will get inconsistent results.
+`TypeRegistry.types_for_root` can help you here, e.g.,
+
+```python
+Animals = DiscriminatedUnion[TypeRegistry.types_for_root("Animals")]
+```
+
+Stack Overflow and its ilk are littered with mediocre implementations of this pattern, so if you want
+to use `DiscriminatedUnion` standalone, you can supply your own `Discriminator` and use it without
+`TypeRegistry`:
+
+```python
+Animals = DiscriminatedUnion[
+    Animal.__subclasses__(),
+    Discriminator(
+        field="_type",
+        discriminator=lambda t: t.__qualname__
+    )
+]
+```
+
+## Storing persistable objects
+
+`ObjectStore` is an abstract base class for implementing an object store, such as a database.
+A file system-based store `FSStore` is provided. Objects are written into directories, formed
+from the store's root, the `TypeRegistry` root and registered name.
+Each write creates a new file, whose name is the timestamp of the write, which corresponds to
+`MetaData.entry_time` in the persisted record.
+
+Reading uses `duckdb` to read the stored parquet/json files. Currently only querying by an object's id
+is supported, but this approach also allows for querying by attributes of the object in future.
+
+A nightly job to compact the timestamped files into a single file will improve read performance.
+
+`FSStore` uses [fsspec](https://pypi.org/project/fsspec/), which supports a wide range of file systems.
+For parquet schema generation it uses [anyschema](https://pypi.org/project/anyschema/), extended with
+some handling for dataclasses and references.
+
+> [Note]
+> Parquet cannot handle union types natively. Currently they are only support via a `Reference`, e.g.:
+
+```python
+class Foo:
+    animal: Reference[DiscriminatedUnion[Dog, Cat, Wildebeest]]
+```
+
+### Writing
+
+```python
+store = FSStore(r"file:///users/hades/ObjectStore", format=FSStoreFormat.Parquet)
+best_boy = Dog(name="Cerberus")
+store.write([best_boy])
+```
+
+Will result in:
+
+```
+/users/hades/ObjectStore                               # FSStore root
+    /Animals                                           # TypeRegistry root
+        /Dog                                           # TypeRegistry registered_name
+            /2026-11-08T13:01:55.249821+00:00.parquet  # File name
+```
+
+If you save an object which contains references, multiple objects may be created:
+
+```python
+blofeld = Customer(
+    name="Ernst Stavro Blofeld",
+    address=Address(
+        street_numer=1,
+        street_name="The Hollowed Out Volcano",
+        town="",
+        postcode="HV1 1ZZ",
+        country=Country.Krakatoa
+    )
+)
+store.write([blofeld])
+```
+
+This would create:
+
+```
+/users/hades/ObjectStore                           # FSStore root
+    /Customer                                      # TypeRegistry registered_name
+        /2026-11-08T13:01:55.249821+00:00.parquet  # File name
+    /Address                                       # TypeRegistry registered_name
+        /2026-11-08T13:01:55.249821+00:00.parquet  # File name
+```
+
+#### Reading
+Objects are read by creating an id with `make_object_id`:
+
+```python
+best_boy_id = Dog.make_object_id(name="Cerberus")
+loaded = store.read([best_boy_id])
+best_boy = loaded[best_boy_id]
+```

pypersistable-1.0.0/README.md

@@ -0,0 +1,165 @@
+# pypersistable: Define and persist python objects with minimal boilerplate
+
+Any object that can be handled by `pydantic.TypeAdaptor` will work (pydantic models, dataclasses etc.)
+Objects can be persisted as JSON or parquet (with some caveats).
+
+
+## Defining persistable objects
+
+### `TypeRegistry`
+Persistable types must be registered with `TypeRegistry`. This can be done explicitly or using entry points.
+While we could have avoided a registry, the alternative is persisting implementation-specific info,
+such as the module and class names, making code refactoring very difficult.
+You can optionally supply a `root` parameter to group types, e.g., you might specify a root of `Animals`
+for classes `Dog`, `Cat`, `Bunyip` etc. See [DiscriminatedUnion](#discriminatedunion) for another use of root.
+
+### `Persistable`
+`Persistable` is the abstract base class for persistable objects. All it does it provide an `object_id`
+for serialisation and ways to form that id for retrieval. There are two species of `Persistable`:
+
+- `Unique` - As the name suggests, this is the base for types where all fields form the id. E.g.,
+```python
+class Address(Unique):
+    street_number: str  # Could be 221B, for example
+    street_name: str
+    town: str
+    postcode: str
+    country: Country
+```
+- `Identifiable` - A base class for objects where a subset of fields form their id. Such fields
+are declared using `IdField`. E.g.,
+```python
+class Customer(Identifiable):
+    name: IdField[str]
+    address: Address
+```
+
+### `Reference`
+In classes which have user-defined types as attributes, do we want those nested objects to be persisted
+by value in the parent object's payload or saved separately, with the parent persisting a reference?
+The answer is very much, "it depends". If you definitely want the nested payload to be embedded, that is
+easy: simply declare your type like `Customer` above. However, you could instead re-write it as:
+
+```python
+class Customer(Identifiable):
+    name: IdField[str]
+    address: Reference[Address]
+```
+
+This means that if we had multiple customers (e.g. family members) living at the same address,
+when persisting those customers, the `Address` would be persisted once and shared, with a reference being
+saved in the `Customer` payload.
+
+You can force references to be serialised by value via the `by_reference` paramter on `ObjectStore.write`.
+`ObjectStore.read` will always check for persisted references.
+
+### `DiscriminatedUnion`
+Polymorphism in persisted classes tends to work quite badly and should be avoided. Persistable objects
+often also get used in contexts where we need to know a schema, such as web services.
+`fastapi` based services, using pydantic models is a good example.
+
+A better approach is to declare the valid types using a python `Union`. pydantic can try and infer the
+actual type on deserialisation but it can be preferable to indicate that explicitly.
+`DiscriminatedUnion` does this for you by:
+- Including a type specifier in the JSON schema 
+- Adding that type specifier (which uses `TypeRegistry.registered_name`) to the serialisation payload 
+- Extracting and removing the type specifier from the deserialisation payload and selecting the appropriate type
+
+`DiscriminatedUnion` can be combined with `__subclasses__` with one major caveat:
+you need to ensure that all subclasses of your type have been imported or you will get inconsistent results.
+`TypeRegistry.types_for_root` can help you here, e.g.,
+
+```python
+Animals = DiscriminatedUnion[TypeRegistry.types_for_root("Animals")]
+```
+
+Stack Overflow and its ilk are littered with mediocre implementations of this pattern, so if you want
+to use `DiscriminatedUnion` standalone, you can supply your own `Discriminator` and use it without
+`TypeRegistry`:
+
+```python
+Animals = DiscriminatedUnion[
+    Animal.__subclasses__(),
+    Discriminator(
+        field="_type",
+        discriminator=lambda t: t.__qualname__
+    )
+]
+```
+
+## Storing persistable objects
+
+`ObjectStore` is an abstract base class for implementing an object store, such as a database.
+A file system-based store `FSStore` is provided. Objects are written into directories, formed
+from the store's root, the `TypeRegistry` root and registered name.
+Each write creates a new file, whose name is the timestamp of the write, which corresponds to
+`MetaData.entry_time` in the persisted record.
+
+Reading uses `duckdb` to read the stored parquet/json files. Currently only querying by an object's id
+is supported, but this approach also allows for querying by attributes of the object in future.
+
+A nightly job to compact the timestamped files into a single file will improve read performance.
+
+`FSStore` uses [fsspec](https://pypi.org/project/fsspec/), which supports a wide range of file systems.
+For parquet schema generation it uses [anyschema](https://pypi.org/project/anyschema/), extended with
+some handling for dataclasses and references.
+
+> [Note]
+> Parquet cannot handle union types natively. Currently they are only support via a `Reference`, e.g.:
+
+```python
+class Foo:
+    animal: Reference[DiscriminatedUnion[Dog, Cat, Wildebeest]]
+```
+
+### Writing
+
+```python
+store = FSStore(r"file:///users/hades/ObjectStore", format=FSStoreFormat.Parquet)
+best_boy = Dog(name="Cerberus")
+store.write([best_boy])
+```
+
+Will result in:
+
+```
+/users/hades/ObjectStore                               # FSStore root
+    /Animals                                           # TypeRegistry root
+        /Dog                                           # TypeRegistry registered_name
+            /2026-11-08T13:01:55.249821+00:00.parquet  # File name
+```
+
+If you save an object which contains references, multiple objects may be created:
+
+```python
+blofeld = Customer(
+    name="Ernst Stavro Blofeld",
+    address=Address(
+        street_numer=1,
+        street_name="The Hollowed Out Volcano",
+        town="",
+        postcode="HV1 1ZZ",
+        country=Country.Krakatoa
+    )
+)
+store.write([blofeld])
+```
+
+This would create:
+
+```
+/users/hades/ObjectStore                           # FSStore root
+    /Customer                                      # TypeRegistry registered_name
+        /2026-11-08T13:01:55.249821+00:00.parquet  # File name
+    /Address                                       # TypeRegistry registered_name
+        /2026-11-08T13:01:55.249821+00:00.parquet  # File name
+```
+
+#### Reading
+Objects are read by creating an id with `make_object_id`:
+
+```python
+best_boy_id = Dog.make_object_id(name="Cerberus")
+loaded = store.read([best_boy_id])
+best_boy = loaded[best_boy_id]
+```

pypersistable-1.0.0/examples/deal_model.py

@@ -0,0 +1,78 @@
+import datetime as dt
+from dataclasses import dataclass
+from enum import StrEnum, auto
+from pathlib import Path
+
+from pypersistable import (
+    Discriminated,
+    Identifiable,
+    IdField,
+    Reference,
+    TypeRegistry,
+    Unique,
+)
+
+__registry = TypeRegistry()
+
+
+@dataclass(kw_only=True)
+class Counterparty(Identifiable):
+    name: IdField[str]
+    location: str
+    contact_name: str
+
+__registry.register("Counterparty", Counterparty)
+
+
+@dataclass(kw_only=True)
+class CustomerBook(Identifiable):
+    counterparty: IdField[Counterparty]
+    sales_coverage: tuple[str, ...]
+    
+__registry.register("CustomerBook", CustomerBook)
+
+    
+    
+@dataclass(kw_only=True)
+class TradingBook(Identifiable):
+    name: IdField[str]
+    owner: str
+    
+__registry.register("TradingBook", TradingBook)
+
+    
+@dataclass(kw_only=True)
+class Option(Unique):
+    ticker: str
+    strike: float
+    maturity: dt.datetime
+    
+__registry.register("Option", Option, Path("Instrument"))
+
+Instrument = Discriminated[Path("Instrument")]
+
+
+class TradeAction(StrEnum):
+    Buy = auto()
+    Sell = auto()
+    Pay = auto()
+    Receive = auto()
+    
+ 
+@dataclass(kw_only=True)
+class Position(Unique):
+    quantity: float
+    action: TradeAction
+    instrument: Reference[Instrument]
+
+__registry.register("Position", Position)
+
+
+@dataclass(kw_only=True)
+class Deal(Identifiable):
+    deal_id: IdField[str]
+    book_from: Reference[TradingBook]
+    book_to: Reference[TradingBook | CustomerBook]
+    position: list[Position]
+    
+__registry.register("Deal", Deal)

pypersistable-1.0.0/pyproject.toml

@@ -0,0 +1,48 @@
+
+
+[project]
+name = "pypersistable"
+description = "Core Object Model"
+authors = [
+    {name = "Nick Young", email = "nick.young.github@gmail.com"}
+]
+readme = "README.md"
+license = {file = "LICENSE"}
+requires-python = ">=3.13"
+
+classifiers = [
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    'Programming Language :: Python :: 3 :: Only',
+    'Programming Language :: Python :: 3.13',
+    'Programming Language :: Python :: 3.14',
+    "Operating System :: OS Independent"
+]
+
+dynamic = ["version"]
+
+dependencies = [
+    "anyschema",
+    "duckdb",
+    "fsspec",
+    "narwhals",
+    "orjson",
+    "pyarrow",
+    "pydantic>2.0",
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest",
+    "pytest-cov"
+]
+
+[project.urls]
+homepage = "https://github.com/nickyoung-github/persistable"
+
+[build-system]
+requires = ["setuptools>=64", "setuptools-scm>=8"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools_scm]
+version_file = "persistable/_version.py"

pypersistable-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

pypersistable-1.0.0/src/pypersistable/__init__.py

@@ -0,0 +1,20 @@
+from .discriminated_union import Discriminated, DiscriminatedUnion, Discriminator
+from .fs_store import FSStore, FSStoreFormat
+from .object_store import ObjectStore
+from .persistable import Identifiable, IdField, Unique
+from .reference import Reference
+from .type_registry import TypeRegistry
+
+__all__ = (
+    "Discriminated",
+    "DiscriminatedUnion",
+    "Discriminator",
+    "FSStore",
+    "FSStoreFormat",
+    "IdField",
+    "Identifiable",
+    "ObjectStore",
+    "Reference",
+    "TypeRegistry",
+    "Unique",
+)

pypersistable-1.0.0/src/pypersistable/anyschema.py

@@ -0,0 +1,75 @@
+from pathlib import Path
+from types import NoneType
+
+import narwhals as nw
+from anyschema._dependencies import is_dataclass
+from anyschema.adapters import dataclass_adapter
+from anyschema.exceptions import UnsupportedDTypeError
+from anyschema.parsers import (
+    AnnotatedStep,
+    ForwardRefStep,
+    PyTypeStep,
+    UnionTypeStep,
+    make_pipeline,
+)
+from anyschema.parsers._base import ParserStep
+from anyschema.parsers.annotated_types import AnnotatedTypesStep
+from anyschema.parsers.pydantic import PydanticTypeStep
+from anyschema.typing import FieldConstraints, FieldMetadata, FieldType
+from narwhals.dtypes import DType
+
+from .persistable import Persistable
+
+
+class DataclassTypeStep(ParserStep):
+    def parse(
+        self,
+        input_type: FieldType,
+        constraints: FieldConstraints,
+        metadata: FieldMetadata,
+    ) -> DType | None:
+        if is_dataclass(input_type):
+            return nw.Struct(
+                [
+                    nw.Field(
+                        name=field_name,
+                        dtype=self.pipeline.parse(field_info, field_constraints, field_metadata, strict=True),
+                    )
+                    for field_name, field_info, field_constraints, field_metadata in dataclass_adapter(input_type)
+                ]
+            )
+
+
+class _UnionTypeStep(UnionTypeStep):
+    # Replaces the anyschema one, so we can handle references
+
+    def _parse_union(self, union: tuple[FieldType, ...]) -> FieldType:
+        if len(union) != 2:
+            msg = "Union with more than two types is not supported."
+            raise UnsupportedDTypeError(msg)
+
+        field0, field1 = union
+
+        if field0 is not NoneType and field1 is not NoneType:
+            msg = "Union with mixed types is not supported."
+
+            for f in (field0, field1):
+                if f is not Path and isinstance(f, type) and not issubclass(f, Persistable):
+                    raise UnsupportedDTypeError(msg)
+
+            return str
+
+        return field1 if field0 is NoneType else field0
+
+
+PARSER_STEPS = (
+    ForwardRefStep(),
+    _UnionTypeStep(),
+    AnnotatedStep(),
+    AnnotatedTypesStep(),
+    DataclassTypeStep(),
+    PydanticTypeStep(),
+    PyTypeStep()
+)
+
+PIPELINE = make_pipeline(PARSER_STEPS)