ic-python-db 0.13.0__tar.gz → 0.15.0__tar.gz

{ic_python_db-0.13.0/ic_python_db.egg-info → ic_python_db-0.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ic-python-db
-Version: 0.13.0
+Version: 0.15.0
 Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
 Home-page: https://github.com/smart-social-contracts/ic-python-db
 Author: Smart Social Contracts
@@ -232,6 +232,52 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
 
 Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
 
+### Values that hold `|`
+
+An index entry is a key that starts with the value, then `|`, a letter and the id (`_fi:Ticket:status:open|n000000000042`). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a `|` in it is escaped (`|` as `%7C`, and `%` as `%25`, so that two values never share an escaped form) and filed under `_fx` instead of `_fi`: no value can hold a `|`, so the keys of two values never meet. Every other value keeps the key it always had, one with only a `%` in it included. The escaped key is longer (two more characters for each `|` or `%`), which counts against the storage map's `max_key_size`.
+
+**Upgrading from 0.14.** Entries of values with a `|`, filed before this under `_fi` with the value as it was, stay readable: `find_by`, `count_by` and `field_index_*` also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value `V` and `V|n` and 12 digits, or `V|h` and the digest of a custom id in `V`'s list, converting `V` wrote its entry over the other value's old list: that value's ids are lost. `Entity.rebuild_field_indexes` re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when `Database.field_index_add` is called for it. To move them all, run once after upgrading, in bounded calls:
+
+```python
+db = Database.get_instance()
+cursor = ""
+while cursor is not None:
+    cursor = db.repair_field_indexes(cursor, limit=500)   # keys read per call
+```
+
+`repair_field_indexes` checks each entry of a value with a `|` against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against `limit`; one longer than `limit` is converted whole, alone in its call). Nothing needs to run before it. `Entity.rebuild_field_indexes(from_id, batch)` does the move per type for sequential ids only, and adds entries that are missing; it never removes one.
+
+When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with `|h` and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; `find_by` is not affected). A chain of old lists of `V`, `V|c`, `V|c|c` and so on deeper than about 500 raises `RecursionError` (also a value made for it). Only then may a release drop the old-key reads.
+
+Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a `|` that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(`limit` + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).
+
+**Rolling back to a release before this one, and upgrading again.** While the older release runs, it files values with a `|` under `_fi` (this release reads them there), but it does not know `_fx`: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so `count_by` and `find_by` can be wrong for those values. Run `repair_field_indexes` to its end again, **from `after=""`**: a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.
+
+## Sorted Fields
+
+Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
+
+```python
+class Listing(Entity):
+    name = String()
+    installs = Integer(sorted=True)
+    likes = Integer(sorted=True)
+    updated_at = Float(sorted=True)
+    __sorted_indexes__ = [("installs", "likes")]   # installs, then likes
+
+top, cursor = Listing.top_by("installs", count=20)              # most installed
+top, cursor = Listing.top_by(("installs", "likes"), count=20)   # ties broken by likes
+recent, cursor = Listing.top_by("updated_at", count=20)
+oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
+more, cursor = Listing.top_by("installs", count=20, cursor=cursor)   # next page; None at the end
+```
+
+The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
+
+`where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
+
+Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
+
 ## Large Relations
 
 `OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:

{ic_python_db-0.13.0 → ic_python_db-0.15.0}/README.md

@@ -183,6 +183,52 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
 
 Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
 
+### Values that hold `|`
+
+An index entry is a key that starts with the value, then `|`, a letter and the id (`_fi:Ticket:status:open|n000000000042`). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a `|` in it is escaped (`|` as `%7C`, and `%` as `%25`, so that two values never share an escaped form) and filed under `_fx` instead of `_fi`: no value can hold a `|`, so the keys of two values never meet. Every other value keeps the key it always had, one with only a `%` in it included. The escaped key is longer (two more characters for each `|` or `%`), which counts against the storage map's `max_key_size`.
+
+**Upgrading from 0.14.** Entries of values with a `|`, filed before this under `_fi` with the value as it was, stay readable: `find_by`, `count_by` and `field_index_*` also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value `V` and `V|n` and 12 digits, or `V|h` and the digest of a custom id in `V`'s list, converting `V` wrote its entry over the other value's old list: that value's ids are lost. `Entity.rebuild_field_indexes` re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when `Database.field_index_add` is called for it. To move them all, run once after upgrading, in bounded calls:
+
+```python
+db = Database.get_instance()
+cursor = ""
+while cursor is not None:
+    cursor = db.repair_field_indexes(cursor, limit=500)   # keys read per call
+```
+
+`repair_field_indexes` checks each entry of a value with a `|` against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against `limit`; one longer than `limit` is converted whole, alone in its call). Nothing needs to run before it. `Entity.rebuild_field_indexes(from_id, batch)` does the move per type for sequential ids only, and adds entries that are missing; it never removes one.
+
+When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with `|h` and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; `find_by` is not affected). A chain of old lists of `V`, `V|c`, `V|c|c` and so on deeper than about 500 raises `RecursionError` (also a value made for it). Only then may a release drop the old-key reads.
+
+Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a `|` that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(`limit` + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).
+
+**Rolling back to a release before this one, and upgrading again.** While the older release runs, it files values with a `|` under `_fi` (this release reads them there), but it does not know `_fx`: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so `count_by` and `find_by` can be wrong for those values. Run `repair_field_indexes` to its end again, **from `after=""`**: a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.
+
+## Sorted Fields
+
+Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
+
+```python
+class Listing(Entity):
+    name = String()
+    installs = Integer(sorted=True)
+    likes = Integer(sorted=True)
+    updated_at = Float(sorted=True)
+    __sorted_indexes__ = [("installs", "likes")]   # installs, then likes
+
+top, cursor = Listing.top_by("installs", count=20)              # most installed
+top, cursor = Listing.top_by(("installs", "likes"), count=20)   # ties broken by likes
+recent, cursor = Listing.top_by("updated_at", count=20)
+oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
+more, cursor = Listing.top_by("installs", count=20, cursor=cursor)   # next page; None at the end
+```
+
+The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
+
+`where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
+
+Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
+
 ## Large Relations
 
 `OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:

{ic_python_db-0.13.0 → ic_python_db-0.15.0}/ic_python_db/__init__.py

@@ -22,7 +22,7 @@ from .schema import SchemaIncompatibleError, build_schema, diff_schemas, schema_
 from .storage import MemoryStorage, Storage
 from .system_time import SystemTime
 
-__version__ = "0.13.0"
+__version__ = "0.15.0"
 __all__ = [
     "Database",
     "Entity",