dray 0.1__tar.gz → 0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: dray
3
- Version: 0.1
3
+ Version: 0.2
4
4
  Summary: A Python record layer for Amazon Aurora DSQL
5
5
  Author: Rodos
6
6
  License-Expression: MIT
@@ -14,7 +14,7 @@ Classifier: Topic :: Database
14
14
  Classifier: Topic :: Database :: Front-Ends
15
15
  Classifier: Operating System :: OS Independent
16
16
  Classifier: Typing :: Typed
17
- Requires-Dist: psycopg>=3.3.4
17
+ Requires-Dist: psycopg>=3.3.4,<4
18
18
  Requires-Dist: cachetools>=5.3
19
19
  Requires-Dist: aurora-dsql-python-connector[psycopg]>=0.2.7,<0.3 ; extra == 'dsql'
20
20
  Requires-Dist: psycopg-pool>=3.3.1 ; extra == 'dsql'
@@ -30,12 +30,12 @@ Description-Content-Type: text/markdown
30
30
 
31
31
  **A Python record layer for Amazon Aurora DSQL.**
32
32
 
33
- dray is built out of what DSQL doesn't have. No foreign keys, no savepoints, no
34
- locking, three thousand rows to a transaction. The limits are the design rather
35
- than obstacles to route around.
33
+ dray is built out of how DSQL works. No savepoints, no locks to take, three
34
+ thousand rows to a transaction, and a commit that can be refused. The limits
35
+ are the design rather than obstacles to route around.
36
36
 
37
37
  You can already run SQLAlchemy or Django against DSQL — AWS ships adapters for
38
- both — but you spend the time talking them out of foreign keys, pessimistic
38
+ both — but you spend the time talking them out of savepoints, pessimistic
39
39
  locking and heap storage. dray assumes none of those to begin with.
40
40
 
41
41
  ## Install
@@ -82,14 +82,19 @@ With dray you get:
82
82
 
83
83
  - **DSQL's limits handled for you.** A transaction there holds 3,000 rows, so a
84
84
  big write is split to fit; a commit it refuses under contention is replayed
85
- rather than raised at you; and a delete reaches the records underneath.
85
+ rather than raised at you; a save that meets somebody's migration rides it
86
+ out; and a delete reaches the records underneath.
86
87
  - **Records that are plain dataclasses**, declared with a decorator. No base
87
88
  class, no metaclass, and no connection — you can build one, hand it to a
88
89
  function and test it with the database nowhere in sight.
89
90
  - **A column or a JSON document, decided per field.** Give a field its own
90
91
  column, or keep it in a `jsonb` document beside the others — adding one of
91
- those is a write rather than a migration, and a filter is written the same
92
- way either side.
92
+ those is a write rather than a migration, and a filter and an index are
93
+ written the same way either side.
94
+ - **Indexes said on the class, down to which rows.** `index("family_name",
95
+ where={"status": "volunteer"})` covers the few anybody asks about, and with
96
+ `unique=True` it is a rule about those alone: one open signup per person,
97
+ any number of cancelled ones.
93
98
  - **Somewhere for your queries to live.** A collection is a class you own, with
94
99
  `find` for matching on values and SQL you wrote for everything else. Both
95
100
  hand back records rather than rows.
@@ -103,8 +108,12 @@ With dray you get:
103
108
  out of the cache they filled.
104
109
  - **Sub-records that travel with their parent.** Notes on a person, lines in an
105
110
  order: queued up, written in the same transaction as the change they explain,
106
- and deleted when it is. As DSQL does not have foreign keys, dray does it for
107
- you.
111
+ and deleted when it is. One child table serves every kind of parent, which
112
+ no foreign key could, so dray does the cascade for you.
113
+ - **A field that names another record, held to it.** Say `references=Person`
114
+ on an `organiser_id` and DSQL refuses an id that resolves to nobody, and
115
+ refuses to delete a person an event still names. dray tells you which field,
116
+ and changes nothing behind your back.
108
117
  - **Business rules that live on the field, not at the call site.** What a value
109
118
  will accept, what it tidies on the way in, what to fill in on a write, and
110
119
  what to run when it changes. Said once on the class and applied at every door
@@ -116,6 +125,10 @@ With dray you get:
116
125
  quietly painting over it. Every record carries an `etag` dray mints and
117
126
  checks on the way back in, so you get a `RecordHasChanged` to handle rather
118
127
  than a mystery change to account for weeks later.
128
+ - **Two answers when DSQL is briefly not there, rather than one error.**
129
+ `Unavailable` where nothing was committed and the work can be run again,
130
+ and `CommitUncertain` where the commit went unanswered and the rows may have
131
+ landed — with the keys of the records to go and look for.
119
132
  - **The `create table` your classes imply**, as statements to paste into a
120
133
  migration — and a way to ask a live database where it has drifted from
121
134
  them.
@@ -145,8 +158,7 @@ page read from the top cannot do. Open it in a browser — it is generated by
145
158
  **Building with an agent?** Hand it [the manual][manual]. The page is one page
146
159
  for this reason as much as for reading: the whole promise fits in a single
147
160
  fetch, in order, with the reasoning attached — so an agent is not guessing at
148
- what a call takes, or reaching for a foreign key, a savepoint or a lock that is
149
- not there. Pointing at it from whatever file your agent reads for project
161
+ what a call takes, or reaching for a savepoint or a lock that is not there. Pointing at it from whatever file your agent reads for project
150
162
  context is enough, and [the reference][reference] is worth adding where the
151
163
  work is mostly about which door runs which rule.
152
164
 
@@ -2,12 +2,12 @@
2
2
 
3
3
  **A Python record layer for Amazon Aurora DSQL.**
4
4
 
5
- dray is built out of what DSQL doesn't have. No foreign keys, no savepoints, no
6
- locking, three thousand rows to a transaction. The limits are the design rather
7
- than obstacles to route around.
5
+ dray is built out of how DSQL works. No savepoints, no locks to take, three
6
+ thousand rows to a transaction, and a commit that can be refused. The limits
7
+ are the design rather than obstacles to route around.
8
8
 
9
9
  You can already run SQLAlchemy or Django against DSQL — AWS ships adapters for
10
- both — but you spend the time talking them out of foreign keys, pessimistic
10
+ both — but you spend the time talking them out of savepoints, pessimistic
11
11
  locking and heap storage. dray assumes none of those to begin with.
12
12
 
13
13
  ## Install
@@ -54,14 +54,19 @@ With dray you get:
54
54
 
55
55
  - **DSQL's limits handled for you.** A transaction there holds 3,000 rows, so a
56
56
  big write is split to fit; a commit it refuses under contention is replayed
57
- rather than raised at you; and a delete reaches the records underneath.
57
+ rather than raised at you; a save that meets somebody's migration rides it
58
+ out; and a delete reaches the records underneath.
58
59
  - **Records that are plain dataclasses**, declared with a decorator. No base
59
60
  class, no metaclass, and no connection — you can build one, hand it to a
60
61
  function and test it with the database nowhere in sight.
61
62
  - **A column or a JSON document, decided per field.** Give a field its own
62
63
  column, or keep it in a `jsonb` document beside the others — adding one of
63
- those is a write rather than a migration, and a filter is written the same
64
- way either side.
64
+ those is a write rather than a migration, and a filter and an index are
65
+ written the same way either side.
66
+ - **Indexes said on the class, down to which rows.** `index("family_name",
67
+ where={"status": "volunteer"})` covers the few anybody asks about, and with
68
+ `unique=True` it is a rule about those alone: one open signup per person,
69
+ any number of cancelled ones.
65
70
  - **Somewhere for your queries to live.** A collection is a class you own, with
66
71
  `find` for matching on values and SQL you wrote for everything else. Both
67
72
  hand back records rather than rows.
@@ -75,8 +80,12 @@ With dray you get:
75
80
  out of the cache they filled.
76
81
  - **Sub-records that travel with their parent.** Notes on a person, lines in an
77
82
  order: queued up, written in the same transaction as the change they explain,
78
- and deleted when it is. As DSQL does not have foreign keys, dray does it for
79
- you.
83
+ and deleted when it is. One child table serves every kind of parent, which
84
+ no foreign key could, so dray does the cascade for you.
85
+ - **A field that names another record, held to it.** Say `references=Person`
86
+ on an `organiser_id` and DSQL refuses an id that resolves to nobody, and
87
+ refuses to delete a person an event still names. dray tells you which field,
88
+ and changes nothing behind your back.
80
89
  - **Business rules that live on the field, not at the call site.** What a value
81
90
  will accept, what it tidies on the way in, what to fill in on a write, and
82
91
  what to run when it changes. Said once on the class and applied at every door
@@ -88,6 +97,10 @@ With dray you get:
88
97
  quietly painting over it. Every record carries an `etag` dray mints and
89
98
  checks on the way back in, so you get a `RecordHasChanged` to handle rather
90
99
  than a mystery change to account for weeks later.
100
+ - **Two answers when DSQL is briefly not there, rather than one error.**
101
+ `Unavailable` where nothing was committed and the work can be run again,
102
+ and `CommitUncertain` where the commit went unanswered and the rows may have
103
+ landed — with the keys of the records to go and look for.
91
104
  - **The `create table` your classes imply**, as statements to paste into a
92
105
  migration — and a way to ask a live database where it has drifted from
93
106
  them.
@@ -117,8 +130,7 @@ page read from the top cannot do. Open it in a browser — it is generated by
117
130
  **Building with an agent?** Hand it [the manual][manual]. The page is one page
118
131
  for this reason as much as for reading: the whole promise fits in a single
119
132
  fetch, in order, with the reasoning attached — so an agent is not guessing at
120
- what a call takes, or reaching for a foreign key, a savepoint or a lock that is
121
- not there. Pointing at it from whatever file your agent reads for project
133
+ what a call takes, or reaching for a savepoint or a lock that is not there. Pointing at it from whatever file your agent reads for project
122
134
  context is enough, and [the reference][reference] is worth adding where the
123
135
  work is mostly about which door runs which rule.
124
136
 
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "dray"
3
- version = "0.1"
3
+ version = "0.2"
4
4
  description = "A Python record layer for Amazon Aurora DSQL"
5
5
  readme = "README.md"
6
6
  requires-python = ">=3.12"
@@ -18,7 +18,7 @@ classifiers = [
18
18
  "Typing :: Typed",
19
19
  ]
20
20
  dependencies = [
21
- "psycopg>=3.3.4",
21
+ "psycopg>=3.3.4,<4",
22
22
  "cachetools>=5.3",
23
23
  ]
24
24
 
@@ -39,10 +39,10 @@ Issues = "https://github.com/draylabs/dray/issues"
39
39
 
40
40
  [dependency-groups]
41
41
  test = [
42
- "pytest",
43
- "pytest-xdist",
44
- "pytest-postgresql",
45
- "psycopg[binary]",
42
+ "pytest>=8",
43
+ "pytest-xdist>=3",
44
+ "pytest-postgresql>=6",
45
+ "psycopg[binary]>=3.3.4",
46
46
  ]
47
47
 
48
48
  [tool.pytest.ini_options]
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "dray"
3
- version = "0.1"
3
+ version = "0.2"
4
4
  description = "A Python record layer for Amazon Aurora DSQL"
5
5
  readme = "README.md"
6
6
  requires-python = ">=3.12"
@@ -22,7 +22,11 @@ classifiers = [
22
22
  "Typing :: Typed",
23
23
  ]
24
24
  dependencies = [
25
- "psycopg>=3.3.4",
25
+ # Capped below 4 because a major may change what a call does, and dray
26
+ # should not promise a version nothing has ever run. The scheduled job
27
+ # installs past this cap deliberately, so the day 4.0 lands we hear it from
28
+ # a check rather than from somebody's traceback.
29
+ "psycopg>=3.3.4,<4",
26
30
  # The read cache's expiry and its size limit. Hand-rolling those is a day's
27
31
  # work and a decade of corner cases, and `functools.lru_cache` cannot do
28
32
  # either — no lifetime, and no way to drop a single key, which is the whole
@@ -61,12 +65,15 @@ Documentation = "https://github.com/draylabs/dray/blob/main/docs/manual.md"
61
65
  Source = "https://github.com/draylabs/dray"
62
66
  Issues = "https://github.com/draylabs/dray/issues"
63
67
 
68
+ # Floors rather than bare names, so a run at `--resolution lowest-direct` asks
69
+ # for something that exists rather than the oldest pytest on PyPI. These bound
70
+ # the environment the suite runs in and reach no user.
64
71
  [dependency-groups]
65
72
  test = [
66
- "pytest",
67
- "pytest-xdist",
68
- "pytest-postgresql",
69
- "psycopg[binary]",
73
+ "pytest>=8",
74
+ "pytest-xdist>=3",
75
+ "pytest-postgresql>=6",
76
+ "psycopg[binary]>=3.3.4",
70
77
  ]
71
78
 
72
79
  [tool.pytest.ini_options]
@@ -43,13 +43,17 @@ from dray.model import (
43
43
  from dray.store import (
44
44
  AfterCommitFailed,
45
45
  CommitRefused,
46
+ CommitUncertain,
46
47
  ConcurrencyExhausted,
47
48
  ConnectionLost,
48
49
  DuplicateRecord,
49
50
  RecordHasChanged,
50
51
  Pool,
51
52
  RecordNotFound,
53
+ ReferenceNotFound,
54
+ StillReferenced,
52
55
  Store,
56
+ Unavailable,
53
57
  replaying,
54
58
  retrying,
55
59
  )
@@ -67,6 +71,7 @@ __all__ = [
67
71
  "Change",
68
72
  "Collection",
69
73
  "CommitRefused",
74
+ "CommitUncertain",
70
75
  "ConcurrencyExhausted",
71
76
  "ConnectionLost",
72
77
  "DrayError",
@@ -76,9 +81,12 @@ __all__ = [
76
81
  "Record",
77
82
  "RecordHasChanged",
78
83
  "RecordNotFound",
84
+ "ReferenceNotFound",
79
85
  "Span",
80
86
  "Sql",
87
+ "StillReferenced",
81
88
  "Store",
89
+ "Unavailable",
82
90
  "ValidationError",
83
91
  "Write",
84
92
  "after_commit",
@@ -36,6 +36,7 @@ from dray.model import (
36
36
  Sql,
37
37
  ValidationError,
38
38
  Write,
39
+ indexed_text,
39
40
  _NOT_YOURS,
40
41
  _checked_most,
41
42
  _checked_ttl,
@@ -49,6 +50,7 @@ from dray.model import (
49
50
  normalised,
50
51
  )
51
52
  from dray.store import (
53
+ CommitUncertain,
52
54
  DuplicateRecord,
53
55
  RecordHasChanged,
54
56
  RecordNotFound,
@@ -58,6 +60,19 @@ from dray.store import (
58
60
  )
59
61
 
60
62
 
63
+ def _ours(failed: BaseException, store: Any) -> bool:
64
+ """Whether what a write raised is its own commit going unanswered.
65
+
66
+ Its own, and the outermost: one raised by another store inside this write,
67
+ or by the commit of a block this write only joined, says nothing about
68
+ which of these records landed."""
69
+ return (
70
+ isinstance(failed, CommitUncertain)
71
+ and getattr(failed, "_store", None) is store
72
+ and not failed.unsettled
73
+ )
74
+
75
+
61
76
  def _as_param(value: Any) -> Any:
62
77
  """One value on its way into a statement against a column.
63
78
 
@@ -254,7 +269,8 @@ class Collection:
254
269
  # number on the other side of its own fetch.
255
270
  dropped = 0 if cache is None else cache.evictions()
256
271
  with cursor(
257
- self.conn, rows=dict_row, watch=self._watch, cls=self.cls
272
+ self.conn, rows=dict_row, watch=self._watch, cls=self.cls,
273
+ theirs=True,
258
274
  ) as cur:
259
275
  cur.execute(statement, list(params))
260
276
  _check_select(self.cls, cur.description)
@@ -331,7 +347,8 @@ class Collection:
331
347
  # row and then saving its defaults over the rest, which cannot happen
332
348
  # where nothing is hydrated.
333
349
  with cursor(
334
- self.conn, rows=dict_row, watch=self._watch, cls=self.cls
350
+ self.conn, rows=dict_row, watch=self._watch, cls=self.cls,
351
+ theirs=True,
335
352
  ) as cur:
336
353
  cur.execute(statement, list(params))
337
354
  return cur.fetchall()
@@ -670,7 +687,9 @@ class Collection:
670
687
  if not any(
671
688
  settled.issuperset(str(column) for column in one.columns)
672
689
  for one in self.cls.__dray_indexes__
673
- if one.unique
690
+ # Unique over every row. One with a predicate says at most one
691
+ # among the rows it covers, and this filter may be about the rest.
692
+ if one.unique and one.where is None
674
693
  ):
675
694
  return None
676
695
  key = tuple(sorted(wanted.items(), key=lambda pair: pair[0]))
@@ -1007,8 +1026,26 @@ class Collection:
1007
1026
  # The name is a declared field, so interpolating it is safe; the
1008
1027
  # value never is and always goes as a parameter.
1009
1028
  inside = name in self.cls.__dray_blob__
1029
+ # A blob field some index is over is asked about twice: as text,
1030
+ # which is the expression the index holds, and as jsonb, which is
1031
+ # how every other blob field is asked about. The second is the
1032
+ # answer and the first is only how to get to it — a row the jsonb
1033
+ # comparison takes has that text, so the text half removes nothing
1034
+ # the other would have kept, and what comes back does not depend
1035
+ # on whether the class declared an index. Where a value has no
1036
+ # text to rely on, the first half is left off and the read scans.
1037
+ #
1038
+ # One row is out of reach of that: a whole number held in the
1039
+ # document as `7.0` equals 7 as jsonb and is not the text `7`. dray
1040
+ # writes an `int` as `7`, so it takes a row written while the
1041
+ # field was a `float`, or by something else.
1042
+ reached = inside and name in self.cls.__dray_blob_indexed__
1010
1043
  if isinstance(value, AnyOf):
1011
1044
  if inside:
1045
+ texts = [indexed_text(each) for each in value]
1046
+ if reached and None not in texts:
1047
+ where.append(f"{self.blob}->>'{name}' = any(%s)")
1048
+ params.append(texts)
1012
1049
  where.append(f"{self.blob}->'{name}' = any(%s)")
1013
1050
  params.append([jsonb(each) for each in value])
1014
1051
  else:
@@ -1028,6 +1065,10 @@ class Collection:
1028
1065
  column = f"{self.blob}->>'{name}'" if inside else name
1029
1066
  where.append(f"{column} is null")
1030
1067
  elif inside:
1068
+ text = indexed_text(value) if reached else None
1069
+ if text is not None:
1070
+ where.append(f"{self.blob}->>'{name}' = %s")
1071
+ params.append(text)
1031
1072
  where.append(f"{self.blob}->'{name}' = %s")
1032
1073
  params.append(jsonb(value))
1033
1074
  else:
@@ -1312,8 +1353,13 @@ class Collection:
1312
1353
  def delete(self, record: Any) -> None:
1313
1354
  """Remove a record and everything hanging off it, in one transaction.
1314
1355
 
1315
- DSQL has no foreign keys and so no cascade of its own; this is the only
1316
- thing standing between the caller and orphaned rows.
1356
+ Nothing in the database ties a child to its parent, so there is no
1357
+ cascade there to do it; this is the only thing standing between the
1358
+ caller and orphaned rows.
1359
+
1360
+ Raises `StillReferenced` where another record names this one, or one
1361
+ of the children going with it, in a field declared `references=` —
1362
+ and removes nothing.
1317
1363
 
1318
1364
  A `@before_delete` on the record runs inside that transaction, so what
1319
1365
  it writes lands with the removal and what it refuses leaves the row
@@ -1324,7 +1370,12 @@ class Collection:
1324
1370
  asks by identity, as `by_id` and a save do, so removing the same record
1325
1371
  twice is a broken assumption rather than a second success. The handler
1326
1372
  has already run and whatever it wrote goes back with the raise."""
1327
- self._delete_batch(record)
1373
+ try:
1374
+ self._delete_batch(record)
1375
+ except CommitUncertain as unsure:
1376
+ if _ours(unsure, self.store):
1377
+ unsure.unsettled = (key_of(record),)
1378
+ raise
1328
1379
  self._forget_removed([key_of(record)])
1329
1380
 
1330
1381
  #
@@ -1565,6 +1616,13 @@ class Collection:
1565
1616
  # whichever of them arrived.
1566
1617
  if durable:
1567
1618
  failed.written = tuple(written)
1619
+ # And the chunk that raised, where what it raised was a
1620
+ # commit that went unanswered. Those are the records to
1621
+ # read back; the ones after it were never sent.
1622
+ if _ours(failed, self.store):
1623
+ failed.unsettled = tuple(
1624
+ key_of(record) for record, *_ in prepared
1625
+ )
1568
1626
  # The rows the chunks before this one wrote are durable, so
1569
1627
  # the cache is now wrong about every one of them — and the
1570
1628
  # raise goes straight past the eviction at the foot of this
@@ -3003,9 +3061,8 @@ def _descendants(cls: type) -> list[tuple[type, ...]]:
3003
3061
  Every kind of record below this one, deepest first, each as the chain of
3004
3062
  classes from the record down to it.
3005
3063
 
3006
- The chain rather than the class alone, because there is no foreign key and
3007
- nothing in an attachment's row says which person it belongs to — only which
3008
- note. Reaching it from the person means going through the note, and the
3064
+ The chain rather than the class alone, because nothing in an attachment's
3065
+ row says which person it belongs to — only which note. Reaching it from the person means going through the note, and the
3009
3066
  chain is what says how.
3010
3067
 
3011
3068
  Classes rather than the table names the delete is written out of, because a
@@ -3032,7 +3089,58 @@ def _descendants(cls: type) -> list[tuple[type, ...]]:
3032
3089
  walk(cls, (cls,), frozenset({cls}))
3033
3090
  # Deepest first, so an attachment goes before the note it hangs off and
3034
3091
  # nothing is orphaned by the row that would have found it being gone.
3035
- return sorted(found, key=len, reverse=True)
3092
+ return _naming_first(sorted(found, key=len, reverse=True))
3093
+
3094
+
3095
+ def _naming_first(chains: list[tuple[type, ...]]) -> list[tuple[type, ...]]:
3096
+ """
3097
+ The generations of a cascade, with one that names another ahead of it.
3098
+
3099
+ Deepest first is the order that finds every row. A field declared
3100
+ `references=` adds a second thing the order has to respect: a phone that
3101
+ names an address has to go before the address does, or the database
3102
+ refuses the address while the phone still names it — a record made
3103
+ undeletable by the order its own children happened to be declared in.
3104
+
3105
+ So a generation waits for the ones below it and for the ones that name it.
3106
+ Where those cannot all be satisfied — two kinds of child naming each other
3107
+ — what is left goes in the order it came, and the database says which key
3108
+ stood in the way.
3109
+ """
3110
+ from dray.model import referenced
3111
+
3112
+ def names(kind: type) -> set[type]:
3113
+ found = set()
3114
+ for name in getattr(kind, "__dray_references__", ()):
3115
+ try:
3116
+ found.add(referenced(kind, name))
3117
+ except TypeError:
3118
+ continue
3119
+ return found - {kind}
3120
+
3121
+ waiting = list(chains)
3122
+ ordered: list[tuple[type, ...]] = []
3123
+ while waiting:
3124
+ for chain in waiting:
3125
+ leaf = chain[-1]
3126
+ held = any(
3127
+ other is not chain
3128
+ and (
3129
+ # Still below it, to be found through it.
3130
+ other[: len(chain)] == chain
3131
+ # Still naming it.
3132
+ or leaf in names(other[-1])
3133
+ )
3134
+ for other in waiting
3135
+ )
3136
+ if not held:
3137
+ ordered.append(chain)
3138
+ waiting.remove(chain)
3139
+ break
3140
+ else:
3141
+ ordered.extend(waiting)
3142
+ break
3143
+ return ordered
3036
3144
 
3037
3145
 
3038
3146
  def _chains_through(parent: type, cls: type) -> list[tuple[type, ...]]: