ic-python-db 0.12.0__tar.gz → 0.14.0__tar.gz

{ic_python_db-0.12.0/ic_python_db.egg-info → ic_python_db-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ic-python-db
-Version: 0.12.0
+Version: 0.14.0
 Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
 Home-page: https://github.com/smart-social-contracts/ic-python-db
 Author: Smart Social Contracts
@@ -60,7 +60,7 @@ A lightweight key-value database with entity relationships and audit logging cap
 ## Features
 
 - **Persistent Storage**: Works with StableBTreeMap stable structure for persistent storage on your canister's stable memory so your data persists automatically across canister upgrades.
-- **Range-based paging**: `load_some`, `find`, `instances` and `rebuild_field_index` read a page of entities with a single ordered `range` call on the stable map (Basilisk `StableBTreeMap.range`), so cost tracks the rows returned rather than the highest id ever issued, and deleted ids are free. Backends without `range` (older CDK builds, plain dict test doubles) transparently fall back to one read per id.
+- **Range-based paging**: `page`, `load_some` and `rebuild_field_index` read a page of entities with a single ordered `range` call on the stable map (Basilisk `StableBTreeMap.range`), so cost tracks the rows returned rather than the highest id ever issued, and deleted ids are free. Backends without `range` (older CDK builds, plain dict test doubles) transparently fall back to one read per id. `instances()` and `find()` load a whole type: they are deprecated and raise `TooLargeError` past 1,000 entities (use `count()`, `find_by()` or `page()`).
 - **Entity-Relational Database**: Create, read and write entities with OneToOne, OneToMany, ManyToOne, and ManyToMany relationships.
 - **Schema Versioning & Upgrade Safety**: Automatic schema introspection, compatibility checking, and auto-migration for safe changes. Breaking changes without a `migrate()` method are rejected, preventing data corruption on canister upgrades.
 - **Entity Hooks**: Intercept and control entity lifecycle events (create, modify, delete) with `on_event` hooks.
@@ -163,11 +163,12 @@ Then use the defined entities to store objects:
 
     assert Person.count() == 3
     assert Person.max_id() == 4
-    assert Person.instances() == [john, alice, eva]
 
-    # Cursor-based pagination
-    assert Person.load_some(0, 2) == [john, alice]
-    assert Person.load_some(2, 2) == [eva]
+    # Cursor-based pagination: (page, next_from_id), None on the last page
+    page, cursor = Person.page(1, 2)
+    assert page == [john, alice]
+    page, cursor = Person.page(cursor, 2)
+    assert page == [eva] and cursor is None
 
     # Retrieve database contents in JSON format
     print(Database.get_instance().dump_json(pretty=True))
@@ -229,7 +230,54 @@ while cursor is not None:
 
 The rebuild is idempotent. The `indexed` flag is not part of the schema descriptor, so toggling it never triggers upgrade-compatibility errors.
 
-Note: each distinct value's ID list is stored in a single storage entry, so extremely hot values are bounded by the backing store's `max_value_size` — the same constraint as relationship reverse indexes.
+Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
+
+## Sorted Fields
+
+Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
+
+```python
+class Listing(Entity):
+    name = String()
+    installs = Integer(sorted=True)
+    likes = Integer(sorted=True)
+    updated_at = Float(sorted=True)
+    __sorted_indexes__ = [("installs", "likes")]   # installs, then likes
+
+top, cursor = Listing.top_by("installs", count=20)              # most installed
+top, cursor = Listing.top_by(("installs", "likes"), count=20)   # ties broken by likes
+recent, cursor = Listing.top_by("updated_at", count=20)
+oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
+more, cursor = Listing.top_by("installs", count=20, cursor=cursor)   # next page; None at the end
+```
+
+The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
+
+`where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
+
+Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
+
+## Large Relations
+
+`OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:
+
+```python
+if section.stands: ...                       # one read
+len(section.stands)                          # one read
+stand in section.stands                      # two reads
+stands, cursor = section.stands.page(1, 50)  # one range read + 50 loads
+group.members.add(user)                      # index writes only
+```
+
+Iterating, indexing and comparing still work, but they load every member, so past `constants.MAX_UNPAGED` (1,000) they raise `TooLargeError` instead of running into the IC instruction limit. Walk a large relation with `page()`, across calls if needed:
+
+```python
+cursor = 1
+while cursor is not None:
+    stands, cursor = section.stands.page(cursor, 50)
+```
+
+`list(view)` gives a plain list. Deleting a parent is O(1): its index entries are purged in bounded steps by later writes, or by `Database.get_instance().purge_step()` from a timer.
 
 ## Namespaces
 

{ic_python_db-0.12.0 → ic_python_db-0.14.0}/README.md

@@ -11,7 +11,7 @@ A lightweight key-value database with entity relationships and audit logging cap
 ## Features
 
 - **Persistent Storage**: Works with StableBTreeMap stable structure for persistent storage on your canister's stable memory so your data persists automatically across canister upgrades.
-- **Range-based paging**: `load_some`, `find`, `instances` and `rebuild_field_index` read a page of entities with a single ordered `range` call on the stable map (Basilisk `StableBTreeMap.range`), so cost tracks the rows returned rather than the highest id ever issued, and deleted ids are free. Backends without `range` (older CDK builds, plain dict test doubles) transparently fall back to one read per id.
+- **Range-based paging**: `page`, `load_some` and `rebuild_field_index` read a page of entities with a single ordered `range` call on the stable map (Basilisk `StableBTreeMap.range`), so cost tracks the rows returned rather than the highest id ever issued, and deleted ids are free. Backends without `range` (older CDK builds, plain dict test doubles) transparently fall back to one read per id. `instances()` and `find()` load a whole type: they are deprecated and raise `TooLargeError` past 1,000 entities (use `count()`, `find_by()` or `page()`).
 - **Entity-Relational Database**: Create, read and write entities with OneToOne, OneToMany, ManyToOne, and ManyToMany relationships.
 - **Schema Versioning & Upgrade Safety**: Automatic schema introspection, compatibility checking, and auto-migration for safe changes. Breaking changes without a `migrate()` method are rejected, preventing data corruption on canister upgrades.
 - **Entity Hooks**: Intercept and control entity lifecycle events (create, modify, delete) with `on_event` hooks.
@@ -114,11 +114,12 @@ Then use the defined entities to store objects:
 
     assert Person.count() == 3
     assert Person.max_id() == 4
-    assert Person.instances() == [john, alice, eva]
 
-    # Cursor-based pagination
-    assert Person.load_some(0, 2) == [john, alice]
-    assert Person.load_some(2, 2) == [eva]
+    # Cursor-based pagination: (page, next_from_id), None on the last page
+    page, cursor = Person.page(1, 2)
+    assert page == [john, alice]
+    page, cursor = Person.page(cursor, 2)
+    assert page == [eva] and cursor is None
 
     # Retrieve database contents in JSON format
     print(Database.get_instance().dump_json(pretty=True))
@@ -180,7 +181,54 @@ while cursor is not None:
 
 The rebuild is idempotent. The `indexed` flag is not part of the schema descriptor, so toggling it never triggers upgrade-compatibility errors.
 
-Note: each distinct value's ID list is stored in a single storage entry, so extremely hot values are bounded by the backing store's `max_value_size` — the same constraint as relationship reverse indexes.
+Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
+
+## Sorted Fields
+
+Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
+
+```python
+class Listing(Entity):
+    name = String()
+    installs = Integer(sorted=True)
+    likes = Integer(sorted=True)
+    updated_at = Float(sorted=True)
+    __sorted_indexes__ = [("installs", "likes")]   # installs, then likes
+
+top, cursor = Listing.top_by("installs", count=20)              # most installed
+top, cursor = Listing.top_by(("installs", "likes"), count=20)   # ties broken by likes
+recent, cursor = Listing.top_by("updated_at", count=20)
+oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
+more, cursor = Listing.top_by("installs", count=20, cursor=cursor)   # next page; None at the end
+```
+
+The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
+
+`where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
+
+Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
+
+## Large Relations
+
+`OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:
+
+```python
+if section.stands: ...                       # one read
+len(section.stands)                          # one read
+stand in section.stands                      # two reads
+stands, cursor = section.stands.page(1, 50)  # one range read + 50 loads
+group.members.add(user)                      # index writes only
+```
+
+Iterating, indexing and comparing still work, but they load every member, so past `constants.MAX_UNPAGED` (1,000) they raise `TooLargeError` instead of running into the IC instruction limit. Walk a large relation with `page()`, across calls if needed:
+
+```python
+cursor = 1
+while cursor is not None:
+    stands, cursor = section.stands.page(cursor, 50)
+```
+
+`list(view)` gives a plain list. Deleting a parent is O(1): its index entries are purged in bounded steps by later writes, or by `Database.get_instance().purge_step()` from a timer.
 
 ## Namespaces
 

{ic_python_db-0.12.0 → ic_python_db-0.14.0}/ic_python_db/__init__.py

@@ -5,6 +5,7 @@ IC Python DB - A lightweight key-value database with entity relationships and au
 from .constants import ACTION_CREATE, ACTION_DELETE, ACTION_MODIFY
 from .db_engine import Database
 from .entity import Entity
+from .errors import TooLargeError
 from .mixins import TimestampedMixin
 from .properties import (
     Boolean,
@@ -14,13 +15,14 @@ from .properties import (
     ManyToOne,
     OneToMany,
     OneToOne,
+    RelationView,
     String,
 )
 from .schema import SchemaIncompatibleError, build_schema, diff_schemas, schema_hash
 from .storage import MemoryStorage, Storage
 from .system_time import SystemTime
 
-__version__ = "0.12.0"
+__version__ = "0.14.0"
 __all__ = [
     "Database",
     "Entity",
@@ -40,6 +42,8 @@ __all__ = [
     "ACTION_MODIFY",
     "ACTION_DELETE",
     "SchemaIncompatibleError",
+    "TooLargeError",
+    "RelationView",
     "build_schema",
     "diff_schemas",
     "schema_hash",

ic_python_db-0.14.0/ic_python_db/constants.py

@@ -0,0 +1,11 @@
+LEVEL_MAX_DEFAULT = 3
+
+# Hook action types
+ACTION_CREATE = "create"
+ACTION_MODIFY = "modify"
+ACTION_DELETE = "delete"
+
+# Largest collection an unpaged operation (iterating a relation, instances())
+# loads in one call; past it they raise TooLargeError. At about 1M
+# instructions per load this stays well inside a query's 5B budget.
+MAX_UNPAGED = 1000