ic-python-db 0.13.0__tar.gz → 0.14.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (26) hide show
  1. {ic_python_db-0.13.0/ic_python_db.egg-info → ic_python_db-0.14.0}/PKG-INFO +26 -1
  2. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/README.md +25 -0
  3. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/__init__.py +1 -1
  4. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/db_engine.py +80 -0
  5. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/entity.py +219 -2
  6. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/properties.py +20 -2
  7. {ic_python_db-0.13.0 → ic_python_db-0.14.0/ic_python_db.egg-info}/PKG-INFO +26 -1
  8. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/pyproject.toml +1 -1
  9. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/setup.py +1 -1
  10. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/LICENSE +0 -0
  11. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/MANIFEST.in +0 -0
  12. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/_cdk.py +0 -0
  13. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/constants.py +0 -0
  14. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/context.py +0 -0
  15. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/errors.py +0 -0
  16. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/hooks.py +0 -0
  17. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/mixins.py +0 -0
  18. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/py.typed +0 -0
  19. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/schema.py +0 -0
  20. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/storage.py +0 -0
  21. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db/system_time.py +0 -0
  22. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db.egg-info/SOURCES.txt +0 -0
  23. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db.egg-info/dependency_links.txt +0 -0
  24. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/ic_python_db.egg-info/top_level.txt +0 -0
  25. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/requirements-dev.txt +0 -0
  26. {ic_python_db-0.13.0 → ic_python_db-0.14.0}/setup.cfg +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: ic-python-db
3
- Version: 0.13.0
3
+ Version: 0.14.0
4
4
  Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
5
5
  Home-page: https://github.com/smart-social-contracts/ic-python-db
6
6
  Author: Smart Social Contracts
@@ -232,6 +232,31 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
232
232
 
233
233
  Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
234
234
 
235
+ ## Sorted Fields
236
+
237
+ Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
238
+
239
+ ```python
240
+ class Listing(Entity):
241
+ name = String()
242
+ installs = Integer(sorted=True)
243
+ likes = Integer(sorted=True)
244
+ updated_at = Float(sorted=True)
245
+ __sorted_indexes__ = [("installs", "likes")] # installs, then likes
246
+
247
+ top, cursor = Listing.top_by("installs", count=20) # most installed
248
+ top, cursor = Listing.top_by(("installs", "likes"), count=20) # ties broken by likes
249
+ recent, cursor = Listing.top_by("updated_at", count=20)
250
+ oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
251
+ more, cursor = Listing.top_by("installs", count=20, cursor=cursor) # next page; None at the end
252
+ ```
253
+
254
+ The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
255
+
256
+ `where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
257
+
258
+ Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
259
+
235
260
  ## Large Relations
236
261
 
237
262
  `OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:
@@ -183,6 +183,31 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
183
183
 
184
184
  Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
185
185
 
186
+ ## Sorted Fields
187
+
188
+ Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
189
+
190
+ ```python
191
+ class Listing(Entity):
192
+ name = String()
193
+ installs = Integer(sorted=True)
194
+ likes = Integer(sorted=True)
195
+ updated_at = Float(sorted=True)
196
+ __sorted_indexes__ = [("installs", "likes")] # installs, then likes
197
+
198
+ top, cursor = Listing.top_by("installs", count=20) # most installed
199
+ top, cursor = Listing.top_by(("installs", "likes"), count=20) # ties broken by likes
200
+ recent, cursor = Listing.top_by("updated_at", count=20)
201
+ oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
202
+ more, cursor = Listing.top_by("installs", count=20, cursor=cursor) # next page; None at the end
203
+ ```
204
+
205
+ The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
206
+
207
+ `where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
208
+
209
+ Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
210
+
186
211
  ## Large Relations
187
212
 
188
213
  `OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:
@@ -22,7 +22,7 @@ from .schema import SchemaIncompatibleError, build_schema, diff_schemas, schema_
22
22
  from .storage import MemoryStorage, Storage
23
23
  from .system_time import SystemTime
24
24
 
25
- __version__ = "0.13.0"
25
+ __version__ = "0.14.0"
26
26
  __all__ = [
27
27
  "Database",
28
28
  "Entity",
@@ -30,6 +30,11 @@ _PQ_HEAD = "_pq:head"
30
30
  _PQ_TAIL = "_pq:tail"
31
31
  # Entries of deleted sets each index write removes on the side
32
32
  _PURGE_PER_WRITE = 20
33
+ # Sorted indexes (see "Sorted indexes" below): each field's value is encoded
34
+ # in _SORT_WIDTH decimal digits, an entity id in _SORT_ID_WIDTH characters
35
+ _SORT_WIDTH = 20
36
+ _SORT_ID_WIDTH = 13
37
+ _INVERT = str.maketrans("0123456789", "9876543210")
33
38
 
34
39
 
35
40
  def _is_sequential_id(id_str: str) -> bool:
@@ -855,6 +860,81 @@ class Database:
855
860
  """Remove an entity ID from the index entry for ``value``."""
856
861
  self._ix_remove(self._fi_key(type_name, field_name, value), entity_id)
857
862
 
863
+ # ── Sorted indexes ─────────────────────────────────────────────────────────
864
+ #
865
+ # A sorted index keeps one entry per entity whose key starts with the
866
+ # entity's sort values, so the storage's own key order is the sort order
867
+ # and a "top N" is one range read:
868
+ #
869
+ # _si:{Type}:{fields}:a|{values}{id} -> entity id lowest first
870
+ # _si:{Type}:{fields}:d|{values}{id} -> entity id highest first
871
+ # _sb:{Type}:{fields}:{entity id} -> {values} what is indexed now
872
+ #
873
+ # {values} is each field's value as _SORT_WIDTH digits (Entity encodes
874
+ # them), {id} the id in _SORT_ID_WIDTH characters. Every key of one index
875
+ # and direction has the same length, which keeps them contiguous in key
876
+ # order (storage_sort_key). The "d" entry stores every digit inverted, so
877
+ # ascending key order is descending value order; ties go to the higher id.
878
+ #
879
+ # The back pointer (_sb) lets an update or a rebuild remove the stale
880
+ # entry without knowing the old value, so an index survives data
881
+ # migrations and type changes.
882
+
883
+ def _si_prefix(self, type_name: str, index: str, descending: bool) -> str:
884
+ return f"_si:{type_name}:{index}:{'d' if descending else 'a'}|"
885
+
886
+ @staticmethod
887
+ def _si_id(entity_id: str, descending: bool) -> str:
888
+ if _is_sequential_id(entity_id) and len(entity_id) < _SORT_ID_WIDTH:
889
+ digits = entity_id.zfill(_SORT_ID_WIDTH - 1)
890
+ return "n" + (digits.translate(_INVERT) if descending else digits)
891
+ digest = hashlib.sha256(entity_id.encode("utf-8")).hexdigest()
892
+ return "h" + digest[: _SORT_ID_WIDTH - 1]
893
+
894
+ def sorted_index_set(
895
+ self, type_name: str, index: str, entity_id: str, values: Optional[str]
896
+ ) -> None:
897
+ """Index ``entity_id`` under ``values`` (encoded sort values), or drop
898
+ it when ``values`` is None. A no-op when nothing changed."""
899
+ back = f"_sb:{type_name}:{index}:{entity_id}"
900
+ old = self._db_storage.get(back)
901
+ if old == values:
902
+ return
903
+ storage = self._db_storage
904
+ for descending in (False, True):
905
+ prefix = self._si_prefix(type_name, index, descending)
906
+ ident = self._si_id(entity_id, descending)
907
+ if old is not None:
908
+ key = prefix + (old.translate(_INVERT) if descending else old) + ident
909
+ if storage.get(key) is not None:
910
+ storage.remove(key)
911
+ if values is not None:
912
+ key = prefix + (values.translate(_INVERT) if descending else values)
913
+ storage.insert(key + ident, entity_id)
914
+ if values is None:
915
+ storage.remove(back)
916
+ else:
917
+ storage.insert(back, values)
918
+
919
+ def sorted_index_scan(
920
+ self,
921
+ type_name: str,
922
+ index: str,
923
+ fields: int,
924
+ descending: bool,
925
+ start: Optional[str],
926
+ limit: int,
927
+ ) -> List[Tuple[str, str]]:
928
+ """Up to ``limit`` (key, entity id) pairs of an index in sort order,
929
+ from key ``start`` (inclusive; None for the first)."""
930
+ prefix = self._si_prefix(type_name, index, descending)
931
+ width = _SORT_WIDTH * fields + _SORT_ID_WIDTH
932
+ if start is None:
933
+ start = prefix + "0" * width
934
+ elif not start.startswith(prefix) or len(start) != len(prefix) + width:
935
+ raise ValueError("cursor does not belong to this sorted index")
936
+ return self._scan(start, prefix + "~" * width, limit)
937
+
858
938
  # Reverse counters (unidirectional relations)
859
939
  # Key format: "_rc:{parent_type}:{parent_id}:{relation_name}"
860
940
  # Value: integer count of related entities.
@@ -1,6 +1,6 @@
1
1
  """Enhanced entity implementation with support for mixins and entity types."""
2
2
 
3
- from typing import Any, Dict, List, Optional, Set, Tuple, Type, TypeVar
3
+ from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Type, TypeVar
4
4
 
5
5
  from ic_python_logging import get_logger
6
6
 
@@ -12,6 +12,44 @@ logger = get_logger(__name__)
12
12
 
13
13
  T = TypeVar("T", bound="Entity")
14
14
 
15
+ _I64_MIN, _I64_LIMIT = -(1 << 63), 1 << 63
16
+ _INF = float("inf")
17
+
18
+
19
+ def _float_bits(value: float) -> int:
20
+ """The IEEE 754 bit pattern of abs(value), read from float.hex().
21
+
22
+ The canister's ``struct`` module is a stub without pack/unpack.
23
+ """
24
+ mantissa, exponent = float.hex(abs(value))[2:].split("p")
25
+ lead, fraction = mantissa.split(".")
26
+ fraction_bits = int(fraction.ljust(13, "0"), 16)
27
+ if lead == "0":
28
+ return fraction_bits # zero or subnormal: biased exponent 0
29
+ return ((int(exponent) + 1023) << 52) | fraction_bits
30
+
31
+
32
+ def _sort_encode(value: Any) -> str:
33
+ """A value as 20 decimal digits whose string order is the value's order.
34
+
35
+ Integers are offset into 0..2**64-1; finite floats use their IEEE 754
36
+ bits around 2**63, so that negative numbers come first.
37
+ """
38
+ if isinstance(value, bool):
39
+ value = int(value)
40
+ if isinstance(value, int):
41
+ if not _I64_MIN <= value < _I64_LIMIT:
42
+ raise ValueError(f"sorted Integer values must fit in 64 bits: {value}")
43
+ unsigned = value - _I64_MIN
44
+ elif isinstance(value, float):
45
+ if value != value or value in (_INF, -_INF):
46
+ raise ValueError(f"sorted Float values must be finite: {value}")
47
+ bits = _float_bits(value)
48
+ unsigned = (1 << 63) - bits if value < 0 else (1 << 63) + bits
49
+ else:
50
+ raise TypeError(f"sorted fields hold numbers, got {type(value).__name__}")
51
+ return str(unsigned).zfill(20)
52
+
15
53
 
16
54
  class Entity:
17
55
  """Base class for database entities with enhanced features.
@@ -439,6 +477,7 @@ class Entity:
439
477
  f"Version mismatch for {type_name}@{entity_id}: "
440
478
  f"stored={stored_version}, current={current_version}"
441
479
  )
480
+ stored = dict(data) # migrate() may change data in place
442
481
  # Apply custom migration first (developer logic takes priority)
443
482
  data = cls.migrate(data, stored_version, current_version)
444
483
  # Auto-inject defaults for new fields not handled by migrate()
@@ -451,12 +490,25 @@ class Entity:
451
490
  # Create instance first
452
491
  entity = cls(**data, _loaded=True)
453
492
 
454
- # If migration was applied, persist the migrated data
493
+ # If migration was applied, persist the migrated data and move its
494
+ # index entries along with any value migrate() changed
455
495
  if stored_version != current_version:
456
496
  entity._save()
497
+ entity._reindex_after_migration(stored)
457
498
 
458
499
  return entity
459
500
 
501
+ def _reindex_after_migration(self, stored: dict) -> None:
502
+ db = self.db()
503
+ for field in self._indexed_properties():
504
+ old, new = stored.get(field), getattr(self, field, None)
505
+ if old != new:
506
+ if old is not None:
507
+ db.field_index_remove(self._type, field, str(old), self._id)
508
+ if new is not None:
509
+ db.field_index_add(self._type, field, str(new), self._id)
510
+ self._sync_sorted()
511
+
460
512
  @classmethod
461
513
  def _require_unpaged(cls, method: str) -> None:
462
514
  """Raise TooLargeError before ``method`` loads more than MAX_UNPAGED entities."""
@@ -723,6 +775,168 @@ class Entity:
723
775
  next_from_id = int(rows[count][0]) if len(rows) > count else None
724
776
  return entities, next_from_id
725
777
 
778
+ @classmethod
779
+ def _sorted_indexes(cls) -> Dict[str, Tuple[str, ...]]:
780
+ """{index name: fields} for this type's sorted indexes.
781
+
782
+ A property declared ``sorted=True`` gets its own index; a class's
783
+ ``__sorted_indexes__`` adds indexes over several fields, compared in
784
+ order (the second breaks ties in the first, and so on).
785
+ """
786
+ cached = cls.__dict__.get("_sorted_cache")
787
+ if cached is not None:
788
+ return cached
789
+ from .properties import Property
790
+
791
+ props: Dict[str, Any] = {}
792
+ for klass in reversed(cls.__mro__):
793
+ for attr_name, attr_value in klass.__dict__.items():
794
+ if not attr_name.startswith("_") and isinstance(attr_value, Property):
795
+ props[attr_name] = attr_value
796
+ indexes: Dict[str, Tuple[str, ...]] = {
797
+ name: (name,) for name, prop in props.items() if prop.sorted
798
+ }
799
+ for spec in getattr(cls, "__sorted_indexes__", ()) or ():
800
+ fields = (spec,) if isinstance(spec, str) else tuple(spec)
801
+ for field in fields:
802
+ prop = props.get(field)
803
+ if prop is None or prop.type not in (int, float):
804
+ raise TypeError(
805
+ f"{cls.__name__}.__sorted_indexes__: {field} is not an "
806
+ f"Integer or Float property"
807
+ )
808
+ indexes[",".join(fields)] = fields
809
+ cls._sorted_cache = indexes
810
+ return indexes
811
+
812
+ def _sync_sorted(
813
+ self, field: Optional[str] = None, index: Optional[str] = None
814
+ ) -> None:
815
+ """Bring this entity's sorted index entries up to date: the indexes
816
+ that contain ``field``, only ``index``, or all of them."""
817
+ if self._id is None:
818
+ return
819
+ db = self.db()
820
+ for name, fields in self._sorted_indexes().items():
821
+ if (field is not None and field not in fields) or (
822
+ index is not None and name != index
823
+ ):
824
+ continue
825
+ values = [getattr(self, f, None) for f in fields]
826
+ encoded = (
827
+ None
828
+ if any(v is None for v in values)
829
+ else "".join(_sort_encode(v) for v in values)
830
+ )
831
+ db.sorted_index_set(self._type, name, self._id, encoded)
832
+
833
+ @classmethod
834
+ def _sorted_index_name(cls, fields) -> Tuple[str, Tuple[str, ...]]:
835
+ key = (fields,) if isinstance(fields, str) else tuple(fields)
836
+ name = ",".join(key)
837
+ if name not in cls._sorted_indexes():
838
+ raise ValueError(
839
+ f"{cls.__name__} has no sorted index on {name}: declare the "
840
+ f"property sorted=True, or list the fields in __sorted_indexes__"
841
+ )
842
+ return name, key
843
+
844
+ @classmethod
845
+ def top_by(
846
+ cls: Type[T],
847
+ fields,
848
+ count: int = 20,
849
+ cursor: Optional[str] = None,
850
+ descending: bool = True,
851
+ where: Optional[Callable[[Any], bool]] = None,
852
+ scan_limit: int = 200,
853
+ ) -> Tuple[List[T], Optional[str]]:
854
+ """One page of entities in the order of a sorted index.
855
+
856
+ ``fields`` is a field declared ``sorted=True``, or a tuple listed in
857
+ ``__sorted_indexes__``. Highest first unless ``descending=False``;
858
+ ties go to the higher id. Entities with None in any of the fields are
859
+ not indexed.
860
+
861
+ Returns (entities, cursor); pass the cursor back for the next page,
862
+ None once the index is exhausted. ``where`` drops entities that fail
863
+ it. At most ``scan_limit`` entries are read per call, so with a
864
+ selective ``where`` a page can come back short, or empty, with a
865
+ cursor: call again. Cost is bounded by ``scan_limit`` loads, however
866
+ many entities exist::
867
+
868
+ top, cursor = Listing.top_by("installs", count=20)
869
+ top, cursor = Listing.top_by(("installs", "likes"), count=20)
870
+ """
871
+ if count < 1:
872
+ raise ValueError("count must be at least 1")
873
+ if scan_limit < 1:
874
+ raise ValueError("scan_limit must be at least 1")
875
+ name, key = cls._sorted_index_name(fields)
876
+ db = cls.db()
877
+ type_name = cls.get_full_type_name()
878
+ found: List[T] = []
879
+ read = 0
880
+ start = cursor
881
+ chunk = count # with `where`, doubled each round: rows read track rows kept
882
+ while len(found) < count and read < scan_limit:
883
+ want = min(
884
+ count - len(found) if where is None else chunk, scan_limit - read, 100
885
+ )
886
+ chunk *= 2
887
+ rows = db.sorted_index_scan(
888
+ type_name, name, len(key), descending, start, want
889
+ )
890
+ for row_key, entity_id in rows:
891
+ read += 1
892
+ # The next same-length key: where a later page starts
893
+ start = row_key[:-1] + chr(ord(row_key[-1]) + 1)
894
+ entity = cls.load(entity_id)
895
+ if entity is not None and (where is None or where(entity)):
896
+ found.append(entity)
897
+ if len(found) >= count:
898
+ break
899
+ else:
900
+ if len(rows) < want:
901
+ return found, None # index exhausted
902
+ if start is None or not db.sorted_index_scan(
903
+ type_name, name, len(key), descending, start, 1
904
+ ):
905
+ return found, None
906
+ return found, start
907
+
908
+ @classmethod
909
+ def rebuild_sorted_index(
910
+ cls: Type[T], fields=None, from_id: int = 1, batch: int = 200
911
+ ) -> Optional[int]:
912
+ """Index one batch of existing entities in a sorted index (all of
913
+ them when ``fields`` is None); resumable and idempotent.
914
+
915
+ Use it after declaring a field sorted, or after changing a sorted
916
+ field's type: each entity's stale entry is replaced, whatever it was::
917
+
918
+ cursor = 1
919
+ while cursor is not None:
920
+ cursor = Listing.rebuild_sorted_index("installs", from_id=cursor)
921
+
922
+ Returns the from_id for the next batch, or None when done.
923
+ """
924
+ if from_id < 1:
925
+ raise ValueError("from_id must be at least 1")
926
+ if batch < 1:
927
+ raise ValueError("batch must be at least 1")
928
+ name = None if fields is None else cls._sorted_index_name(fields)[0]
929
+ max_id = cls.max_id()
930
+ if max_id == 0 or from_id > max_id:
931
+ return None
932
+ entities = cls.load_some(from_id, batch)
933
+ for entity in entities:
934
+ entity._sync_sorted(index=name)
935
+ if len(entities) < batch:
936
+ return None
937
+ next_from_id = int(entities[-1]._id) + 1
938
+ return next_from_id if next_from_id <= max_id else None
939
+
726
940
  @classmethod
727
941
  def count(cls: Type[T]) -> int:
728
942
  """Get the total count of entities of this type.
@@ -945,6 +1159,9 @@ class Entity:
945
1159
  if value is not None:
946
1160
  db.field_index_remove(self._type, field_name, str(value), self._id)
947
1161
 
1162
+ for name in self._sorted_indexes():
1163
+ db.sorted_index_set(self._type, name, self._id, None)
1164
+
948
1165
  db.delete(self._type, self._id)
949
1166
 
950
1167
  # Remove from entity registry
@@ -42,6 +42,7 @@ class Property(Generic[T]):
42
42
  default: Optional[T]
43
43
  validator: Optional[Callable[[T], bool]]
44
44
  indexed: bool
45
+ sorted: bool
45
46
 
46
47
  def __init__(
47
48
  self,
@@ -50,12 +51,14 @@ class Property(Generic[T]):
50
51
  default: Optional[T] = None,
51
52
  validator: Optional[Callable[[T], bool]] = None,
52
53
  indexed: bool = False,
54
+ sorted: bool = False,
53
55
  ):
54
56
  self.name = name
55
57
  self.type = type
56
58
  self.default = default
57
59
  self.validator = validator
58
60
  self.indexed = indexed
61
+ self.sorted = sorted
59
62
 
60
63
  def __set_name__(self, owner: type, name: str) -> None:
61
64
  self.name = name
@@ -121,6 +124,9 @@ class Property(Generic[T]):
121
124
  if value is not None:
122
125
  db.field_index_add(obj._type, self.name, str(value), obj._id)
123
126
 
127
+ if old_value != value and obj._id is not None and hasattr(obj, "_sync_sorted"):
128
+ obj._sync_sorted(self.name)
129
+
124
130
  obj._save()
125
131
 
126
132
 
@@ -155,6 +161,7 @@ class Integer(Property[int]):
155
161
  max_value: Optional[int] = None,
156
162
  default: Optional[int] = None,
157
163
  indexed: bool = False,
164
+ sorted: bool = False,
158
165
  ):
159
166
  def validator(value: int) -> bool:
160
167
  if min_value is not None and value < min_value:
@@ -164,7 +171,12 @@ class Integer(Property[int]):
164
171
  return True
165
172
 
166
173
  super().__init__(
167
- name="", type=int, default=default, validator=validator, indexed=indexed
174
+ name="",
175
+ type=int,
176
+ default=default,
177
+ validator=validator,
178
+ indexed=indexed,
179
+ sorted=sorted,
168
180
  )
169
181
 
170
182
 
@@ -177,6 +189,7 @@ class Float(Property[float]):
177
189
  max_value: Optional[float] = None,
178
190
  default: Optional[float] = None,
179
191
  indexed: bool = False,
192
+ sorted: bool = False,
180
193
  ):
181
194
  def validator(value: float) -> bool:
182
195
  if min_value is not None and value < min_value:
@@ -186,7 +199,12 @@ class Float(Property[float]):
186
199
  return True
187
200
 
188
201
  super().__init__(
189
- name="", type=float, default=default, validator=validator, indexed=indexed
202
+ name="",
203
+ type=float,
204
+ default=default,
205
+ validator=validator,
206
+ indexed=indexed,
207
+ sorted=sorted,
190
208
  )
191
209
 
192
210
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: ic-python-db
3
- Version: 0.13.0
3
+ Version: 0.14.0
4
4
  Summary: A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)
5
5
  Home-page: https://github.com/smart-social-contracts/ic-python-db
6
6
  Author: Smart Social Contracts
@@ -232,6 +232,31 @@ The rebuild is idempotent. The `indexed` flag is not part of the schema descript
232
232
 
233
233
  Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's `max_value_size`.
234
234
 
235
+ ## Sorted Fields
236
+
237
+ Declare `sorted=True` on an `Integer` or `Float` property to read entities in its order, highest or lowest first, without loading and sorting them all. `__sorted_indexes__` adds indexes over several fields, where each field breaks ties in the one before:
238
+
239
+ ```python
240
+ class Listing(Entity):
241
+ name = String()
242
+ installs = Integer(sorted=True)
243
+ likes = Integer(sorted=True)
244
+ updated_at = Float(sorted=True)
245
+ __sorted_indexes__ = [("installs", "likes")] # installs, then likes
246
+
247
+ top, cursor = Listing.top_by("installs", count=20) # most installed
248
+ top, cursor = Listing.top_by(("installs", "likes"), count=20) # ties broken by likes
249
+ recent, cursor = Listing.top_by("updated_at", count=20)
250
+ oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
251
+ more, cursor = Listing.top_by("installs", count=20, cursor=cursor) # next page; None at the end
252
+ ```
253
+
254
+ The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus `count` loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
255
+
256
+ `where=` drops entities that fail a test. At most `scan_limit` entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
257
+
258
+ Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (`Listing.rebuild_sorted_index("installs", from_id=cursor)`, until it returns None). A `migrate()` that changes a sorted or indexed value moves the entity's entries when it is loaded.
259
+
235
260
  ## Large Relations
236
261
 
237
262
  `OneToMany` and `ManyToMany` getters return a `RelationView`, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "ic-python-db"
7
- version = "0.13.0"
7
+ version = "0.14.0"
8
8
  description = "A lightweight key-value database written in Python, intended for use on the Internet Computer (IC)"
9
9
  readme = "README.md"
10
10
  authors = [
@@ -5,7 +5,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
5
5
 
6
6
  setup(
7
7
  name="ic-python-db",
8
- version="0.13.0",
8
+ version="0.14.0",
9
9
  author="Smart Social Contracts",
10
10
  author_email="smartsocialcontracts@gmail.com",
11
11
  description="A lightweight key-value database with entity relationships and audit logging",
File without changes
File without changes
File without changes