ic-python-db 0.14.0__tar.gz → 0.15.0__tar.gz

{ic_python_db-0.14.0/ic_python_db.egg-info → ic_python_db-0.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ic-python-db
-Version: 0.14.0
+Version: 0.15.0
 Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
 Home-page: https://github.com/smart-social-contracts/ic-python-db
 Author: Smart Social Contracts
@@ -232,6 +232,27 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
 
 Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
 
+### Values that hold `|`
+
+An index entry is a key that starts with the value, then `|`, a letter and the id (`_fi:Ticket:status:open|n000000000042`). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a `|` in it is escaped (`|` as `%7C`, and `%` as `%25`, so that two values never share an escaped form) and filed under `_fx` instead of `_fi`: no value can hold a `|`, so the keys of two values never meet. Every other value keeps the key it always had, one with only a `%` in it included. The escaped key is longer (two more characters for each `|` or `%`), which counts against the storage map's `max_key_size`.
+
+**Upgrading from 0.14.** Entries of values with a `|`, filed before this under `_fi` with the value as it was, stay readable: `find_by`, `count_by` and `field_index_*` also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value `V` and `V|n` and 12 digits, or `V|h` and the digest of a custom id in `V`'s list, converting `V` wrote its entry over the other value's old list: that value's ids are lost. `Entity.rebuild_field_indexes` re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when `Database.field_index_add` is called for it. To move them all, run once after upgrading, in bounded calls:
+
+```python
+db = Database.get_instance()
+cursor = ""
+while cursor is not None:
+    cursor = db.repair_field_indexes(cursor, limit=500)   # keys read per call
+```
+
+`repair_field_indexes` checks each entry of a value with a `|` against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against `limit`; one longer than `limit` is converted whole, alone in its call). Nothing needs to run before it. `Entity.rebuild_field_indexes(from_id, batch)` does the move per type for sequential ids only, and adds entries that are missing; it never removes one.
+
+When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with `|h` and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; `find_by` is not affected). A chain of old lists of `V`, `V|c`, `V|c|c` and so on deeper than about 500 raises `RecursionError` (also a value made for it). Only then may a release drop the old-key reads.
+
+Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a `|` that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(`limit` + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).
+
+**Rolling back to a release before this one, and upgrading again.** While the older release runs, it files values with a `|` under `_fi` (this release reads them there), but it does not know `_fx`: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so `count_by` and `find_by` can be wrong for those values. Run `repair_field_indexes` to its end again, **from `after=""`**: a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.
+
 ## Sorted Fields
 
 Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:

{ic_python_db-0.14.0 → ic_python_db-0.15.0}/README.md

@@ -183,6 +183,27 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
 
 Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
 
+### Values that hold `|`
+
+An index entry is a key that starts with the value, then `|`, a letter and the id (`_fi:Ticket:status:open|n000000000042`). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a `|` in it is escaped (`|` as `%7C`, and `%` as `%25`, so that two values never share an escaped form) and filed under `_fx` instead of `_fi`: no value can hold a `|`, so the keys of two values never meet. Every other value keeps the key it always had, one with only a `%` in it included. The escaped key is longer (two more characters for each `|` or `%`), which counts against the storage map's `max_key_size`.
+
+**Upgrading from 0.14.** Entries of values with a `|`, filed before this under `_fi` with the value as it was, stay readable: `find_by`, `count_by` and `field_index_*` also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value `V` and `V|n` and 12 digits, or `V|h` and the digest of a custom id in `V`'s list, converting `V` wrote its entry over the other value's old list: that value's ids are lost. `Entity.rebuild_field_indexes` re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when `Database.field_index_add` is called for it. To move them all, run once after upgrading, in bounded calls:
+
+```python
+db = Database.get_instance()
+cursor = ""
+while cursor is not None:
+    cursor = db.repair_field_indexes(cursor, limit=500)   # keys read per call
+```
+
+`repair_field_indexes` checks each entry of a value with a `|` against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against `limit`; one longer than `limit` is converted whole, alone in its call). Nothing needs to run before it. `Entity.rebuild_field_indexes(from_id, batch)` does the move per type for sequential ids only, and adds entries that are missing; it never removes one.
+
+When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with `|h` and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; `find_by` is not affected). A chain of old lists of `V`, `V|c`, `V|c|c` and so on deeper than about 500 raises `RecursionError` (also a value made for it). Only then may a release drop the old-key reads.
+
+Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a `|` that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(`limit` + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).
+
+**Rolling back to a release before this one, and upgrading again.** While the older release runs, it files values with a `|` under `_fi` (this release reads them there), but it does not know `_fx`: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so `count_by` and `find_by` can be wrong for those values. Run `repair_field_indexes` to its end again, **from `after=""`**: a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.
+
 ## Sorted Fields
 
 Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:

{ic_python_db-0.14.0 → ic_python_db-0.15.0}/ic_python_db/__init__.py

@@ -22,7 +22,7 @@ from .schema import SchemaIncompatibleError, build_schema, diff_schemas, schema_
 from .storage import MemoryStorage, Storage
 from .system_time import SystemTime
 
-__version__ = "0.14.0"
+__version__ = "0.15.0"
 __all__ = [
     "Database",
     "Entity",

{ic_python_db-0.14.0 → ic_python_db-0.15.0}/ic_python_db/db_engine.py

@@ -22,6 +22,14 @@ _HEX = frozenset("0123456789abcdef")
 _ID_WIDTH = 12
 _HASH_WIDTH = 24
 _SCAN_PAGE = 500
+# Field-index key families repair_field_indexes walks (see "Repairing field
+# indexes")
+_FI_FAMILIES = ("_fi:", "_fx:")
+# Most ids of an old-form list the repair converts in one call: about 245
+# thousand instructions an id, about 345 thousand where other old lists are
+# among its entry keys (one more read an id), so 24.5 billion, or 34.5 billion
+# at worst, in a call of 40 billion
+_CONVERT_MAX_IDS = 100_000
 _BUCKETS = (("n", _ID_WIDTH, ":"), ("h", _HASH_WIDTH, "g"))
 # Purge queue of deleted index sets (see _ix_clear): fixed-width positions
 # keep its entries contiguous in key order
@@ -42,6 +50,72 @@ def _is_sequential_id(id_str: str) -> bool:
     return bool(id_str) and id_str[0] != "0" and all(c in _DIGITS for c in id_str)
 
 
+def _id_digest(entity_id: str) -> str:
+    """The digest a posting-list entry of a custom id is keyed by."""
+    return hashlib.sha256(entity_id.encode("utf-8")).hexdigest()[:_HASH_WIDTH]
+
+
+def _is_hash_entry_key(key: str) -> bool:
+    """True for a key that ends like the entry of a custom (hashed) id."""
+    digest = key[-_HASH_WIDTH:]
+    return key.endswith("|h" + digest) and all(c in _HEX for c in digest)
+
+
+def _is_hash_entry(key: str, value: str) -> bool:
+    """True for the posting-list entry of a custom id: it is keyed by the digest
+    of its value. An old list stored under a value that ends like such a key
+    does not match it; only a value made for it does (``V|h`` and the digest of
+    the list's own JSON text), and that list then reads as an entry of ``V``."""
+    return _is_hash_entry_key(key) and _id_digest(value) == key[-_HASH_WIDTH:]
+
+
+def _escape_key_text(text: str) -> str:
+    """``text`` with no ``|`` in it, and still told apart from every other text.
+
+    ``%`` is written ``%25`` and ``|`` is written ``%7C``. Escaping ``%`` too is
+    what keeps two texts from sharing a result (``a|`` and ``a%7C``).
+    """
+    return text.replace("%", "%25").replace("|", "%7C")
+
+
+def _unescape_key_text(text: str) -> str:
+    """The text ``_escape_key_text`` was given."""
+    return text.replace("%7C", "|").replace("%25", "%")
+
+
+def _entry_base(key: str) -> Optional[str]:
+    """The base a posting-list entry key belongs to; None for any other key
+    (a count, a mark, an old list)."""
+    tail = key[-_ID_WIDTH:]
+    if key.endswith("|n" + tail) and all(c in _DIGITS for c in tail):
+        return key[: -_ID_WIDTH - 2]
+    if _is_hash_entry_key(key):
+        return key[: -_HASH_WIDTH - 2]
+    return None
+
+
+def _old_list_size(key: str, value: str) -> int:
+    """Ids in the list a release before 0.13 stored as one JSON array under
+    ``key``, a key of a value with a "|"; 0 for any other key or value. (The
+    entry of a custom id may read as an array: see ``_is_hash_entry``.)"""
+    if (
+        key.startswith("_fi:")
+        and "|" in key
+        and value.startswith("[")
+        and not _is_hash_entry(key, value)
+    ):
+        return value.count(",") + 1
+    return 0
+
+
+def _listing_order(entity_id: str) -> Tuple[int, int]:
+    """Sort key of the order ``_ix_all`` lists ids in: sequential ids
+    ascending, then the others."""
+    if _is_sequential_id(entity_id) and len(entity_id) <= _ID_WIDTH:
+        return (0, int(entity_id))
+    return (1, 0)
+
+
 def _next_sequential_id(id_str: str, digits: int) -> Optional[int]:
     """Smallest ``digits``-digit sequential id that sorts after ``id_str``.
 
@@ -128,6 +202,9 @@ class Database:
             )
 
         self._entity_types = {}
+        # {target class name: reverse counter names}, built by the first
+        # delete after the registry last changed (see reverse_count_names)
+        self._reverse_count_names: Optional[Dict[str, List[str]]] = None
         # Identity map: {(type_name, entity_id): entity}. Weak, so an entity
         # lives only as long as some caller holds it, and its entry goes with it.
         self._entity_registry = weakref.WeakValueDictionary()
@@ -396,6 +473,16 @@ class Database:
             return type_name.split("::")[-1]
         return type_name
 
+    def _set_entity_type(self, name: str, type_obj) -> None:
+        """Point a registry name at a class, dropping what is derived from the registry.
+
+        Every Entity construction registers its class again, so an unchanged
+        entry must leave the derived data alone.
+        """
+        if self._entity_types.get(name) is not type_obj:
+            self._entity_types[name] = type_obj
+            self._reverse_count_names = None
+
     def register_entity_type(self, type_obj, type_name: str = None):
         """Register an entity type with the database.
 
@@ -410,7 +497,7 @@ class Database:
         )
 
         # Always register under the full type name
-        self._entity_types[type_name] = type_obj
+        self._set_entity_type(type_name, type_obj)
 
         # For backward compatibility, also register under class name if:
         # 1. No namespace (type_name == class name) - allows non-namespaced lookups, OR
@@ -421,7 +508,7 @@ class Database:
             type_name == type_obj.__name__
             or type_obj.__name__ not in self._entity_types
         ):
-            self._entity_types[type_obj.__name__] = type_obj
+            self._set_entity_type(type_obj.__name__, type_obj)
         else:
             # Class name already registered - log warning about potential collision
             existing = self._entity_types[type_obj.__name__]
@@ -550,16 +637,35 @@ class Database:
     # fixed-width suffixes keep one base's entries contiguous, and the "n"
     # entries come back in ascending numeric id order.
     #
+    # A base that holds a "|" can make keys of its own that look like another
+    # base's entries (base "a|n0000000000" has the key "a|n0000000000|c", as
+    # long as the entries "a|n000000000100"). A field index never has one (see
+    # _fi_key). A reverse index base, _ri:{Type}:{parent id}:{relation}, is
+    # not escaped, and a custom parent id may hold a "|": but the relation name
+    # ends the base, so a look-alike has to end with it as well, which is
+    # rarely possible. Either way a range read does not depend on it: it skips
+    # any key that is not an entry of its base and goes on (see _ix_bucket).
+    #
     # Before 0.13 a base key held the whole set as one JSON array, which hit
     # the value-size limit at ~10k ids. Such an array is converted to entries
     # the first time its base is touched (see _ix_count).
 
-    def _scan(self, start: str, end: str, limit: int) -> List[Tuple[str, str]]:
+    def _scan(
+        self, start: str, end: Optional[str], limit: int
+    ) -> List[Tuple[str, str]]:
+        """Up to ``limit`` pairs with ``start <= key < end``, in key order;
+        ``end=None`` is no end."""
         if supports_range(self._db_storage):
             return self._db_storage.range(start, end, limit)
-        lo, hi = storage_sort_key(start), storage_sort_key(end)
+        lo = storage_sort_key(start)
+        hi = None if end is None else storage_sort_key(end)
         keys = sorted(
-            (k for k in self._db_storage.keys() if lo <= storage_sort_key(k) < hi),
+            (
+                k
+                for k in self._db_storage.keys()
+                if lo <= storage_sort_key(k)
+                and (hi is None or storage_sort_key(k) < hi)
+            ),
             key=storage_sort_key,
         )
         return [(k, self._db_storage.get(k)) for k in keys[:limit]]
@@ -568,16 +674,37 @@ class Database:
     def _ix_entry_key(base: str, entity_id: str) -> str:
         if _is_sequential_id(entity_id) and len(entity_id) <= _ID_WIDTH:
             return f"{base}|n{entity_id.zfill(_ID_WIDTH)}"
-        digest = hashlib.sha256(entity_id.encode("utf-8")).hexdigest()
-        return f"{base}|h{digest[:_HASH_WIDTH]}"
+        return f"{base}|h{_id_digest(entity_id)}"
+
+    def _ix_own(self, key: str) -> Optional[str]:
+        """What is stored under a key of this list's own (its count, its mark,
+        an entry of it), after an old list of another value that is stored
+        there is converted, which moves it out of the way.
+
+        Before 0.13 a list was one array under the bare key of its value, and
+        the bare key of the value "V|c" is the key of V's count; "V|d" is V's
+        mark and "V|n" and 12 digits an entry of V.
+        """
+        value = self._db_storage.get(key)
+        if value is not None and value.startswith("["):
+            # (An entry of a custom id that reads as an array is left: see
+            # _is_hash_entry)
+            self._ix_count(key)
+            value = self._db_storage.get(key)
+        return value
 
     def _ix_count(self, base: str) -> int:
-        raw = self._db_storage.get(f"{base}|c")
+        raw = self._ix_own(f"{base}|c")
         if raw is not None:
             return int(raw)
         legacy = self._db_storage.get(base)
         if legacy is None:
             return 0
+        if _is_hash_entry(base, legacy):
+            # An entry holds a custom id, which may read as a list: it is not
+            # the array an old list was stored as, and converting it would
+            # remove the entry
+            return 0
         try:
             ids = json.loads(legacy)
         except ValueError:
@@ -586,8 +713,17 @@ class Database:
             return 0
         self._db_storage.remove(base)
         unique = {str(i) for i in ids}
+        # Old lists of other values among the entry keys (rare: two reads to
+        # know) are converted before an entry is written over them
+        crowded = any(
+            self._scan(f"{base}|{tag}{'0' * width}", f"{base}|{tag}{top * width}", 1)
+            for tag, width, top in _BUCKETS
+        )
         for entity_id in unique:
-            self._db_storage.insert(self._ix_entry_key(base, entity_id), entity_id)
+            key = self._ix_entry_key(base, entity_id)
+            if crowded:
+                self._ix_own(key)
+            self._db_storage.insert(key, entity_id)
         if unique:
             self._db_storage.insert(f"{base}|c", str(len(unique)))
         return len(unique)
@@ -598,7 +734,7 @@ class Database:
         if count == 0:
             self._ix_revive(base)
         key = self._ix_entry_key(base, entity_id)
-        if self._db_storage.get(key) is not None:
+        if self._ix_own(key) == entity_id:
             return
         self._db_storage.insert(key, entity_id)
         self._db_storage.insert(f"{base}|c", str(count + 1))
@@ -609,8 +745,8 @@ class Database:
         if count == 0:
             return
         key = self._ix_entry_key(base, entity_id)
-        if self._db_storage.get(key) is None:
-            return
+        if self._db_storage.get(key) != entity_id:
+            return  # not listed (the key may be an old list of another value)
         self._db_storage.remove(key)
         if count > 1:
             self._db_storage.insert(f"{base}|c", str(count - 1))
@@ -620,7 +756,7 @@ class Database:
     def _ix_contains(self, base: str, entity_id: str) -> bool:
         if self._ix_count(base) == 0:
             return False
-        return self._db_storage.get(self._ix_entry_key(base, entity_id)) is not None
+        return self._db_storage.get(self._ix_entry_key(base, entity_id)) == entity_id
 
     def _ix_page(self, base: str, from_id: int, count: int) -> List[str]:
         """Up to ``count`` sequential ids >= ``from_id``, ascending."""
@@ -630,35 +766,55 @@ class Database:
             raise ValueError("count must be at least 1")
         if self._ix_count(base) == 0 or len(str(from_id)) > _ID_WIDTH:
             return []
-        start = f"{base}|n{str(from_id).zfill(_ID_WIDTH)}"
-        end = f"{base}|n{':' * _ID_WIDTH}"
-        cut = len(base) + 2
-        # A key of another base can fall inside the range only if its suffix
-        # is not all digits (e.g. value "x|n0;..." next to value "x").
-        return [
-            value
-            for key, value in self._scan(start, end, count)
-            if all(c in _DIGITS for c in key[cut:])
-        ]
+        ids: List[str] = []
+        first = str(from_id).zfill(_ID_WIDTH)
+        for _, value in self._ix_bucket(base, "n", _ID_WIDTH, ":", count, first):
+            ids.append(value)
+            if len(ids) == count:
+                break
+        return ids
 
     def _ix_bucket(
-        self, base: str, tag: str, width: int, top: str, page_size: int = _SCAN_PAGE
+        self,
+        base: str,
+        tag: str,
+        width: int,
+        top: str,
+        page_size: int = _SCAN_PAGE,
+        first: Optional[str] = None,
     ):
-        start = f"{base}|{tag}{'0' * width}"
+        """The entries of ``base`` under ``tag`` (from the entry suffix
+        ``first``, when given), in key order.
+
+        A key of another base can fall in the range (see "Posting lists"), and
+        so can the old list of another value (see _ix_own): neither is an entry,
+        which holds the id its key spells. A page that came back short is the
+        end of the range; one that is full but held such keys is not, so the
+        scan goes on from its last key.
+        """
+        start = f"{base}|{tag}{first or '0' * width}"
         end = f"{base}|{tag}{top * width}"
         cut = len(base) + 2
         valid = _DIGITS if tag == "n" else _HEX
+        last = None
         while True:
-            page = self._scan(start, end, page_size)
+            # A range starts at its last key, which is read twice: one more
+            # than a page. (No "next key" is computed: the successor of a
+            # char can be longer in utf-8, so the key after it is not the
+            # same length.)
+            wanted = page_size if last is None else page_size + 1
+            page = self._scan(start, end, wanted)
             for key, value in page:
-                if all(c in valid for c in key[cut:]):
-                    yield key, value
-            if len(page) < page_size:
+                if key != last and all(c in valid for c in key[cut:]):
+                    if tag == "n":
+                        own = value == str(int(key[cut:]))
+                    else:
+                        own = not value.startswith("[") or _is_hash_entry(key, value)
+                    if own:
+                        yield key, value
+            if len(page) < wanted:
                 return
-            # Smallest same-length key after the last one: append the lowest
-            # char and drop the last, i.e. bump by one in the key order.
-            last = page[-1][0]
-            start = last[:-1] + chr(ord(last[-1]) + 1)
+            last = start = page[-1][0]
 
     def _ix_all(self, base: str) -> List[str]:
         """Every id in the set: sequential ids ascending, then the others."""
@@ -710,7 +866,7 @@ class Database:
 
     def _ix_revive(self, base: str) -> None:
         """Before reusing a deleted set: purge its leftovers, drop its queue entry."""
-        position = self._db_storage.get(f"{base}|d")
+        position = self._ix_own(f"{base}|d")
         if position is None:
             return
         while self._ix_drain(base, _SCAN_PAGE) == _SCAN_PAGE:
@@ -828,38 +984,247 @@ class Database:
     # Allow equality lookups (Entity.find_by) without scanning all entities.
     #
     # Base key: _fi:{Type}:{field}:{value} (posting list)
+    #
+    # A value is text a caller chooses, so its base must not hold a "|" (see
+    # "Posting lists"). One that holds a "|" goes under _fx instead, with "|"
+    # and "%" written as %7C and %25 (so that no two values share an escaped
+    # form): _fx:{Type}:{field}:{escaped value}. Every other value, "%" alone
+    # included, keeps the key it always had, so nothing moved for them.
+    #
+    # Such a value used to be filed under _fi with no escape. Its entries stay
+    # readable there: each call below also looks at the old key of a value that
+    # has one, and adding or removing an id takes it off the old list.
+    # repair_field_indexes moves them all, those of custom ids too (which
+    # Entity.rebuild_field_indexes does not reach), and converts the lists
+    # stored before 0.13 as one array. Once it has run to its end on every
+    # realm, _fi_old_key and what uses it can go, but for what it cannot judge
+    # and leaves (see its docstring). (_fx, not _fi, so that an old key can
+    # never be taken for the new key of another value.)
+
+    @staticmethod
+    def _fi_needs_escape(value: str) -> bool:
+        return "|" in value
 
     def _fi_key(self, type_name: str, field_name: str, value: str) -> str:
+        if self._fi_needs_escape(value):
+            return f"_fx:{type_name}:{field_name}:{_escape_key_text(value)}"
         return f"_fi:{type_name}:{field_name}:{value}"
 
+    def _fi_old_key(self, type_name: str, field_name: str, value: str) -> Optional[str]:
+        """Where ``value`` was filed before it was escaped, if not where it is now."""
+        if self._fi_needs_escape(value):
+            return f"_fi:{type_name}:{field_name}:{value}"
+        return None
+
     def field_index_get(self, type_name: str, field_name: str, value: str) -> List[str]:
         """All ids of entities whose ``field_name`` equals ``value``
         (O(matches); prefer ``field_index_page``)."""
-        return self._ix_all(self._fi_key(type_name, field_name, value))
+        ids = self._ix_all(self._fi_key(type_name, field_name, value))
+        old = self._fi_old_key(type_name, field_name, value)
+        if old is None:
+            return ids
+        # An id on both lists (a release that did not know the new keys wrote
+        # to the old one) is listed once
+        ids = list(dict.fromkeys(ids + self._ix_all(old)))
+        return sorted(ids, key=_listing_order)
 
     def field_index_page(
         self, type_name: str, field_name: str, value: str, from_id: int, count: int
     ) -> List[str]:
         """Up to ``count`` sequential ids >= ``from_id`` whose ``field_name``
-        equals ``value``, ascending. One range read."""
-        return self._ix_page(self._fi_key(type_name, field_name, value), from_id, count)
+        equals ``value``, ascending. One range read; two for a value with a
+        "|" while its old key may hold entries, and one more for each key of
+        another value in the way."""
+        key = self._fi_key(type_name, field_name, value)
+        ids = self._ix_page(key, from_id, count)
+        old = self._fi_old_key(type_name, field_name, value)
+        if old is None:
+            return ids
+        # The first ``count`` of both lists together are among the first
+        # ``count`` of each
+        ids += self._ix_page(old, from_id, count)
+        return sorted(set(ids), key=int)[:count]
 
     def field_index_count(self, type_name: str, field_name: str, value: str) -> int:
-        """Number of entities whose ``field_name`` equals ``value`` (one read)."""
-        return self._ix_count(self._fi_key(type_name, field_name, value))
+        """Number of entities whose ``field_name`` equals ``value`` (one read;
+        two for a value with a "|" while its old key may hold entries, and an id
+        on both lists is counted twice until repair_field_indexes has run)."""
+        total = self._ix_count(self._fi_key(type_name, field_name, value))
+        old = self._fi_old_key(type_name, field_name, value)
+        if old is not None:
+            total += self._ix_count(old)
+        return total
 
     def field_index_add(
         self, type_name: str, field_name: str, value: str, entity_id: str
     ) -> None:
         """Add an entity ID to the index entry for ``value``."""
+        old = self._fi_old_key(type_name, field_name, value)
+        if old is not None:
+            self._ix_remove(old, entity_id)
         self._ix_add(self._fi_key(type_name, field_name, value), entity_id)
 
     def field_index_remove(
         self, type_name: str, field_name: str, value: str, entity_id: str
     ) -> None:
         """Remove an entity ID from the index entry for ``value``."""
+        old = self._fi_old_key(type_name, field_name, value)
+        if old is not None:
+            self._ix_remove(old, entity_id)
         self._ix_remove(self._fi_key(type_name, field_name, value), entity_id)
 
+    # ── Repairing field indexes ────────────────────────────────────────────────
+    #
+    # A release that does not know the _fx keys files a value with a "|" under
+    # its _fi key, and does not take an id off the _fx list when the entity's
+    # value changes or the entity is deleted. After an upgrade from it (or
+    # after a rollback to it and a new upgrade) entries are under the old key,
+    # or stale on a list, or on both; a list stored before 0.13 may still be
+    # one JSON array. Entity.rebuild_field_indexes walks the entities, so it
+    # can add or move an entry, but it never finds a stale one and does not
+    # reach custom ids. This walks the keys instead, and checks each entry of a
+    # value with a "|" against its entity.
+
+    def _index_ranges(self, after: str):
+        """The (start, end) key ranges that hold field-index keys, in the order
+        repair_field_indexes walks them, from the one that holds ``after``.
+
+        Storage orders keys by length first, so each length of each family is a
+        range of its own. Only lengths at which the map holds a key are
+        visited (the shortest key at least as long as the last length tells
+        the next), so no length is out of reach.
+        """
+        resume = after[:4] if after else ""
+        first = _FI_FAMILIES.index(resume) if resume else 0
+        for family in _FI_FAMILIES[first:]:
+            length = len(family) + 1
+            if family == resume:
+                length = len(after.encode("utf-8"))
+            while True:
+                shortest = self._scan("\x00" * length, None, 1)
+                if not shortest:
+                    break
+                length = len(shortest[0][0].encode("utf-8"))
+                fill = "\x00" * (length - len(family))
+                resumed = family == resume and length == len(after.encode("utf-8"))
+                yield (after if resumed else family + fill), (
+                    family[:-1] + chr(ord(family[-1]) + 1) + fill
+                )
+                length += 1
+
+    def _field_of_base(self, rest: str) -> Optional[Tuple[str, str, str]]:
+        """(type name, field, text) of ``{Type}:{field}:{text}``, when the type
+        is registered and the field indexed; None otherwise."""
+        for entity_class in self._entity_types.values():
+            type_name = entity_class.get_full_type_name()
+            if not rest.startswith(type_name + ":"):
+                continue
+            skip = len(type_name) + 1
+            field, colon, text = rest[skip:].partition(":")
+            if colon and field in entity_class._indexed_properties():
+                return type_name, field, text
+        return None
+
+    def _repair_entry(self, key: str, value: str) -> None:
+        size = _old_list_size(key, value)
+        if size:
+            if size <= _CONVERT_MAX_IDS:
+                # Converted into entries, which are longer keys: the walk
+                # reaches them later
+                self._ix_count(key)
+            else:
+                logger.warning(
+                    f"repair_field_indexes: the old list {key!r} has {size} ids, "
+                    f"more than a call can convert; left as it is"
+                )
+            return
+        base = _entry_base(key)
+        if base is None:
+            return
+        rest, old_style = base[4:], key.startswith("_fi:")
+        if old_style and "|" not in rest:
+            return  # a value without "|": its key never changed
+        found = self._field_of_base(rest)
+        if found is None:
+            return  # not a type and field this code can judge
+        type_name, field, text = found
+        text = text if old_style else _unescape_key_text(text)
+        row = self.load(type_name, value)  # an entry's value is its entity id
+        if row is not None and field not in row:
+            return  # a row from before the field: nothing to compare with
+        if row is None or row[field] is None or str(row[field]) != text:
+            self._ix_remove(base, value)
+        elif old_style:
+            self.field_index_add(type_name, field, text, value)
+
+    def repair_field_indexes(self, after: str = "", limit: int = 500) -> Optional[str]:
+        """Check the field-index entries of values with a ``|`` against the
+        entities, at most ``limit`` keys at a time; resumable.
+
+        An entry whose entity is gone, or no longer has that value, is removed.
+        One under the old key (before escaping) whose entity has the value is
+        moved to its present key, so an id is listed once. A list a release
+        before 0.13 stored as one array under the old key of such a value is
+        converted into entries first, and they are checked later in the walk.
+        Entries of values without a ``|`` are only looked at.
+
+            cursor = ""
+            while cursor is not None:
+                cursor = Database.get_instance().repair_field_indexes(cursor)
+
+        Run it to its end after upgrading from a release before escaping, and
+        again after a rollback to one and a new upgrade, **from ``after=""``**:
+        a cursor, or a note that it was done, saved before the rollback skips
+        what the older release wrote. Nothing else needs to run first. Once it
+        has run to its end, no old key holds an entry and counts are exact,
+        except for what it cannot judge and leaves as it is: entries of a type
+        or field this code does not index; entries whose row has no such field;
+        an old-form list of more than 100 thousand ids (it is logged; a read of
+        its value converts it where it fits the message, about 160 thousand ids
+        in an update and 20 thousand in a query, and traps where it does not,
+        as before); and an old-form list of a value made to end with ``|h`` and
+        the digest of its own text, which reads as an entry of the value it
+        extends (only such a value matches; ``find_by`` is not affected). A
+        chain of old lists of ``V``, ``V|c``, ``V|c|c`` and so on deeper than
+        about 500 raises ``RecursionError``: also a value made for it.
+
+        Cost, measured on the IC: a key read about 0.1 million instructions; an
+        entry of a value with a ``|`` that is right about 0.73 million (its row
+        is loaded); an old entry moved about 2 million, so a call of 500 moves
+        is about 1 billion; and about 0.58 million for each distinct key length
+        in the whole map that a call walks past (only the lengths at which the
+        map holds a key are visited). A call is O(``limit`` + those lengths). A
+        list it converts counts its ids against ``limit``, and one longer than
+        ``limit`` is converted whole, alone in its call, at about 245 thousand
+        instructions an id: a map with limited value size holds at most about
+        10 thousand ids in a list, but a realm's is not limited (a list of
+        127,000 ids is on record: about 31 billion; it is over the cap, so the
+        repair leaves it, and a read of its value in an update converts it).
+
+        Returns:
+            The ``after`` of the call that continues, None when the walk is done.
+        """
+        if limit < 1:
+            raise ValueError("limit must be at least 1")
+        if after and after[:4] not in _FI_FAMILIES:
+            raise ValueError("after is not a cursor of repair_field_indexes")
+        seen, last = 0, after
+        for start, end in self._index_ranges(after):
+            # One more than is left: a range starts at its last key. A range
+            # with more keys than that fills the call, and ends it below
+            wanted = limit - seen + 1
+            for key, value in self._scan(start, end, wanted):
+                if key == last:
+                    continue
+                cost = _old_list_size(key, value) or 1
+                if seen and seen + cost > limit:
+                    return last
+                self._repair_entry(key, value)
+                last, seen = key, seen + cost
+            if seen >= limit:
+                return last
+        return None
+
     # ── Sorted indexes ─────────────────────────────────────────────────────────
     #
     # A sorted index keeps one entry per entity whose key starts with the
@@ -988,6 +1353,46 @@ class Database:
         if self._db_storage.get(key) is not None:
             self._db_storage.remove(key)
 
+    def reverse_count_names(self, type_name: str) -> List[str]:
+        """Names of the reverse counters that entities of a type carry.
+
+        These are the ``reverse_name`` of every unidirectional ManyToMany that
+        a registered class declares towards ``type_name``; a deleted entity
+        must delete its counters. Finding them reads every attribute of every
+        registered class, which costs millions of IC instructions with a
+        realm's 93 types, so the answer is kept until ``register_entity_type``
+        changes the registry.
+        """
+        if self._reverse_count_names is None:
+            self._reverse_count_names = self._find_reverse_count_names()
+        return self._reverse_count_names.get(self._extract_class_name(type_name), [])
+
+    def _find_reverse_count_names(self) -> Dict[str, List[str]]:
+        """{target class name: reverse counter names}, from the registered classes."""
+        from .properties import ManyToMany
+
+        names: Dict[str, List[str]] = {}
+        read = set()  # a base class is shared by many registered classes
+        for entity_class in self._entity_types.values():
+            if not isinstance(entity_class, type):
+                continue
+            for cls in entity_class.__mro__:
+                if cls in read:
+                    continue
+                read.add(cls)
+                for attr_name, attr in cls.__dict__.items():
+                    if attr_name.startswith("_"):
+                        continue
+                    if not isinstance(attr, ManyToMany) or not attr.unidirectional:
+                        continue
+                    for target in attr._get_allowed_types():
+                        reverse_names = names.setdefault(
+                            self._extract_class_name(target), []
+                        )
+                        if attr.reverse_name not in reverse_names:
+                            reverse_names.append(attr.reverse_name)
+        return names
+
     def get_audit(
         self, id_from: Optional[int] = None, id_to: Optional[int] = None
     ) -> Dict[str, str]:

{ic_python_db-0.14.0 → ic_python_db-0.15.0}/ic_python_db/entity.py

@@ -657,7 +657,9 @@ class Entity:
             while cursor is not None:
                 cursor = Proposal.rebuild_field_index("org_scope", from_id=cursor)
 
-        Idempotent — re-indexing an already-indexed entity is a no-op.
+        Idempotent — re-indexing an already-indexed entity is a no-op. An
+        entity whose value holds ``|`` and was indexed under the key such a
+        value had before it was escaped is moved to its present key.
 
         Returns:
             The from_id for the next batch, or None when the rebuild is done.
@@ -666,6 +668,33 @@ class Entity:
             ValueError: If the field is not declared with indexed=True
         """
         cls._require_indexed_field(field)
+        return cls._rebuild_field_indexes([field], from_id, batch)
+
+    @classmethod
+    def rebuild_field_indexes(
+        cls: Type[T],
+        from_id: int = 1,
+        batch: int = 200,
+    ) -> Optional[int]:
+        """``rebuild_field_index`` for every indexed field of the type at once.
+
+        Moves the entries of values holding ``|`` that an earlier version filed
+        under their own text, for entities with sequential ids: until they are
+        moved they are still found, under the old key. It adds an entry that is
+        missing but never removes one, and does not reach custom ids:
+        ``Database.repair_field_indexes`` does both, and is what completes the
+        move after an upgrade.
+        Same cursor and batch as ``rebuild_field_index``; one batch costs the
+        entities it loads once, and a write per indexed field of each.
+        """
+        return cls._rebuild_field_indexes(
+            list(cls._indexed_properties()), from_id, batch
+        )
+
+    @classmethod
+    def _rebuild_field_indexes(
+        cls: Type[T], fields: List[str], from_id: int, batch: int
+    ) -> Optional[int]:
         if from_id < 1:
             raise ValueError("from_id must be at least 1")
         if batch < 1:
@@ -680,9 +709,10 @@ class Entity:
         # One range read per batch of *live* entities (deleted ids cost nothing)
         entities = cls.load_some(from_id, batch)
         for entity in entities:
-            value = getattr(entity, field, None)
-            if value is not None:
-                db.field_index_add(type_name, field, str(value), entity._id)
+            for field in fields:
+                value = getattr(entity, field, None)
+                if value is not None:
+                    db.field_index_add(type_name, field, str(value), entity._id)
 
         if len(entities) < batch:
             return None
@@ -1129,29 +1159,8 @@ class Entity:
         # via unidirectional ManyToMany relations targeting our type. Their
         # forward indexes may retain a dangling ID (skipped on load); the
         # counter itself must not outlive us.
-        seen_classes = set()
-        for entity_class in db._entity_types.values():
-            if entity_class in seen_classes or not isinstance(entity_class, type):
-                continue
-            seen_classes.add(entity_class)
-            for cls in reversed(entity_class.__mro__):
-                for attr_name, attr_value in cls.__dict__.items():
-                    if attr_name.startswith("_"):
-                        continue
-                    if isinstance(attr_value, ManyToMany) and attr_value.unidirectional:
-                        my_short = (
-                            self._type.split("::")[-1]
-                            if "::" in self._type
-                            else self._type
-                        )
-                        targets = [
-                            t.split("::")[-1] if "::" in t else t
-                            for t in attr_value._get_allowed_types()
-                        ]
-                        if my_short in targets:
-                            db.reverse_count_delete(
-                                self._type, self._id, attr_value.reverse_name
-                            )
+        for reverse_name in db.reverse_count_names(self._type):
+            db.reverse_count_delete(self._type, self._id, reverse_name)
 
         # Remove from field indexes (issue #11)
         for field_name in self._indexed_properties():

{ic_python_db-0.14.0 → ic_python_db-0.15.0/ic_python_db.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ic-python-db
-Version: 0.14.0
+Version: 0.15.0
 Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
 Home-page: https://github.com/smart-social-contracts/ic-python-db
 Author: Smart Social Contracts
@@ -232,6 +232,27 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
 
 Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
 
+### Values that hold `|`
+
+An index entry is a key that starts with the value, then `|`, a letter and the id (`_fi:Ticket:status:open|n000000000042`). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a `|` in it is escaped (`|` as `%7C`, and `%` as `%25`, so that two values never share an escaped form) and filed under `_fx` instead of `_fi`: no value can hold a `|`, so the keys of two values never meet. Every other value keeps the key it always had, one with only a `%` in it included. The escaped key is longer (two more characters for each `|` or `%`), which counts against the storage map's `max_key_size`.
+
+**Upgrading from 0.14.** Entries of values with a `|`, filed before this under `_fi` with the value as it was, stay readable: `find_by`, `count_by` and `field_index_*` also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value `V` and `V|n` and 12 digits, or `V|h` and the digest of a custom id in `V`'s list, converting `V` wrote its entry over the other value's old list: that value's ids are lost. `Entity.rebuild_field_indexes` re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when `Database.field_index_add` is called for it. To move them all, run once after upgrading, in bounded calls:
+
+```python
+db = Database.get_instance()
+cursor = ""
+while cursor is not None:
+    cursor = db.repair_field_indexes(cursor, limit=500)   # keys read per call
+```
+
+`repair_field_indexes` checks each entry of a value with a `|` against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against `limit`; one longer than `limit` is converted whole, alone in its call). Nothing needs to run before it. `Entity.rebuild_field_indexes(from_id, batch)` does the move per type for sequential ids only, and adds entries that are missing; it never removes one.
+
+When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with `|h` and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; `find_by` is not affected). A chain of old lists of `V`, `V|c`, `V|c|c` and so on deeper than about 500 raises `RecursionError` (also a value made for it). Only then may a release drop the old-key reads.
+
+Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a `|` that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(`limit` + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).
+
+**Rolling back to a release before this one, and upgrading again.** While the older release runs, it files values with a `|` under `_fi` (this release reads them there), but it does not know `_fx`: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so `count_by` and `find_by` can be wrong for those values. Run `repair_field_indexes` to its end again, **from `after=""`**: a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.
+
 ## Sorted Fields
 
 Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:

{ic_python_db-0.14.0 → ic_python_db-0.15.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ic-python-db"
-version = "0.14.0"
+version = "0.15.0"
 description = "A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)"
 readme = "README.md"
 authors = [

{ic_python_db-0.14.0 → ic_python_db-0.15.0}/setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
 
 setup(
     name="ic-python-db",
-    version="0.14.0",
+    version="0.15.0",
     author="Smart Social Contracts",
     author_email="smartsocialcontracts@gmail.com",
     description="A lightweight key-value database with entity relationships and audit logging",