s3rds 0.2.0__tar.gz → 0.2.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,7 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: s3rds
3
- Version: 0.2.0
4
- Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://.
3
+ Version: 0.2.1
4
+ Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite.
5
5
  Author: Frank Zhang
6
6
  License: MIT
7
7
  Project-URL: Homepage, https://github.com/cloudformer/s3rds
@@ -10,13 +10,14 @@ Project-URL: Deploy, https://github.com/cloudformer/s3rds#quick-start
10
10
  Project-URL: Compatibility, https://github.com/cloudformer/s3rds/blob/main/tests/compat/REPORT.md
11
11
  Project-URL: Changelog, https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md
12
12
  Project-URL: Issues, https://github.com/cloudformer/s3rds/issues
13
- Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy
13
+ Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy,django
14
14
  Classifier: Development Status :: 3 - Alpha
15
15
  Classifier: Intended Audience :: Developers
16
16
  Classifier: License :: OSI Approved :: MIT License
17
17
  Classifier: Programming Language :: Python :: 3
18
18
  Classifier: Programming Language :: Python :: 3.9
19
19
  Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Framework :: Django
20
21
  Classifier: Topic :: Database
21
22
  Classifier: Topic :: Database :: Front-Ends
22
23
  Requires-Python: >=3.9
@@ -25,19 +26,42 @@ Requires-Dist: sqlglot<31,>=30.24
25
26
  Provides-Extra: mysql
26
27
  Provides-Extra: sqlalchemy
27
28
  Requires-Dist: SQLAlchemy<2.1,>=2.0; extra == "sqlalchemy"
29
+ Provides-Extra: django
30
+ Requires-Dist: Django<6.2,>=6.1; extra == "django"
28
31
 
29
32
  # s3rds
30
33
 
34
+ ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
35
+
36
+ ### Switch to s3rds — the only change
37
+
38
+ | Your code uses | Before | With s3rds — **the only change** | What really differs |
39
+ |---|---|---|---|
40
+ | **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
41
+ | **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
42
+ | **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
43
+ | **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
44
+ | **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` — *planned, not available yet* | — |
45
+
46
+ Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
47
+ uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
48
+ backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
49
+ `NotSupportedError` — never silently different.
50
+
51
+ ### The server: deploy once, ≈ $0 when idle
52
+
53
+ You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
54
+ while nobody uses it. Deploy it once with Terraform or CloudFormation from the
55
+ [main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
56
+
31
57
  **Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
32
58
  a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
33
59
 
34
- ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
35
-
36
60
  One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
37
61
  client it mirrors**, so moving to a real database is a one-line import change and nothing else:
38
62
 
39
63
  ```bash
40
- pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
64
+ pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
41
65
  ```
42
66
 
43
67
  ```python
@@ -68,6 +92,7 @@ conn.transaction([ # all-or-nothing, single S3 w
68
92
  | `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
69
93
  | `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
70
94
  | `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
95
+ | `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
71
96
 
72
97
  **SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
73
98
  `s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
@@ -76,6 +101,21 @@ conn.transaction([ # all-or-nothing, single S3 w
76
101
  engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
77
102
  ```
78
103
 
104
+ **Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
105
+
106
+ ```python
107
+ DATABASES = {"default": {
108
+ "ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
109
+ "NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
110
+ }}
111
+ ```
112
+
113
+ Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
114
+ `Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
115
+ conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
116
+ `Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
117
+ `django.db.backends.sqlite3`.
118
+
79
119
  ## Semantics
80
120
 
81
121
  - **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
@@ -83,7 +123,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
83
123
  ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
84
124
  pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
85
125
  statements as one commit and one S3 write.
86
- - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) has sqlite3's transactions: reads see
126
+ - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
127
+ `atomic()`) give sqlite3's transactions: reads see
87
128
  the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
88
129
  writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
89
130
  Keep transactions short — each statement replays the writes before it on the server.
@@ -105,9 +146,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
105
146
  | SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
106
147
  | sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
107
148
  | SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
149
+ | Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
108
150
  | MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
109
151
 
110
- ## Server
111
-
112
- You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
113
- from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
@@ -1,15 +1,36 @@
1
1
  # s3rds
2
2
 
3
+ ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
4
+
5
+ ### Switch to s3rds — the only change
6
+
7
+ | Your code uses | Before | With s3rds — **the only change** | What really differs |
8
+ |---|---|---|---|
9
+ | **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
10
+ | **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
11
+ | **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
12
+ | **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
13
+ | **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` — *planned, not available yet* | — |
14
+
15
+ Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
16
+ uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
17
+ backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
18
+ `NotSupportedError` — never silently different.
19
+
20
+ ### The server: deploy once, ≈ $0 when idle
21
+
22
+ You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
23
+ while nobody uses it. Deploy it once with Terraform or CloudFormation from the
24
+ [main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
25
+
3
26
  **Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
4
27
  a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
5
28
 
6
- ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
7
-
8
29
  One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
9
30
  client it mirrors**, so moving to a real database is a one-line import change and nothing else:
10
31
 
11
32
  ```bash
12
- pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
33
+ pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
13
34
  ```
14
35
 
15
36
  ```python
@@ -40,6 +61,7 @@ conn.transaction([ # all-or-nothing, single S3 w
40
61
  | `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
41
62
  | `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
42
63
  | `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
64
+ | `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
43
65
 
44
66
  **SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
45
67
  `s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
@@ -48,6 +70,21 @@ conn.transaction([ # all-or-nothing, single S3 w
48
70
  engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
49
71
  ```
50
72
 
73
+ **Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
74
+
75
+ ```python
76
+ DATABASES = {"default": {
77
+ "ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
78
+ "NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
79
+ }}
80
+ ```
81
+
82
+ Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
83
+ `Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
84
+ conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
85
+ `Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
86
+ `django.db.backends.sqlite3`.
87
+
51
88
  ## Semantics
52
89
 
53
90
  - **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
@@ -55,7 +92,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
55
92
  ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
56
93
  pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
57
94
  statements as one commit and one S3 write.
58
- - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) has sqlite3's transactions: reads see
95
+ - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
96
+ `atomic()`) give sqlite3's transactions: reads see
59
97
  the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
60
98
  writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
61
99
  Keep transactions short — each statement replays the writes before it on the server.
@@ -77,9 +115,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
77
115
  | SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
78
116
  | sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
79
117
  | SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
118
+ | Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
80
119
  | MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
81
120
 
82
- ## Server
83
-
84
- You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
85
- from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "s3rds"
7
- version = "0.2.0"
8
- description = "A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://."
7
+ version = "0.2.1"
8
+ description = "A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.9"
11
11
  license = { text = "MIT" }
12
12
  authors = [{ name = "Frank Zhang" }]
13
- keywords = ["serverless", "rds", "sqlite", "s3", "lambda", "database", "pymysql", "dbapi", "sqlalchemy"]
13
+ keywords = ["serverless", "rds", "sqlite", "s3", "lambda", "database", "pymysql", "dbapi", "sqlalchemy", "django"]
14
14
  classifiers = [
15
15
  "Development Status :: 3 - Alpha",
16
16
  "Intended Audience :: Developers",
@@ -18,6 +18,7 @@ classifiers = [
18
18
  "Programming Language :: Python :: 3",
19
19
  "Programming Language :: Python :: 3.9",
20
20
  "Programming Language :: Python :: 3.13",
21
+ "Framework :: Django",
21
22
  "Topic :: Database",
22
23
  "Topic :: Database :: Front-Ends",
23
24
  ]
@@ -29,6 +30,7 @@ dependencies = ["sqlglot>=30.24,<31"]
29
30
  [project.optional-dependencies]
30
31
  mysql = [] # kept so that pip install "s3rds[mysql]" from 0.1.x keeps working
31
32
  sqlalchemy = ["SQLAlchemy>=2.0,<2.1"] # the series the compatibility report verified; pip install "s3rds[sqlalchemy]"
33
+ django = ["Django>=6.1,<6.2"] # the same for Django (ENGINE "s3rds.sqlite"); Django 6.1 needs Python >= 3.12
32
34
 
33
35
  [project.entry-points."sqlalchemy.dialects"]
34
36
  "sqlite.s3rds" = "s3rds.sqlite:SQLAlchemyDialect"
@@ -42,7 +44,7 @@ Changelog = "https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md"
42
44
  Issues = "https://github.com/cloudformer/s3rds/issues"
43
45
 
44
46
  [tool.setuptools]
45
- packages = ["s3rds"]
47
+ packages = ["s3rds", "s3rds.sqlite"]
46
48
 
47
49
  # The versions tests/compat/REPORT.md was verified with; README.md, README.zh.md and client/README.md state the same.
48
50
  [tool.s3rds.verified]
@@ -52,4 +54,5 @@ sqlite = "3.50.4" # the server's engine, shipped in the Lambda
52
54
  sqlean = "3.50.4.5" # sqlean.py, which carries it (src/db/requirements.txt)
53
55
  sqlglot = "30.24.0"
54
56
  sqlalchemy = "2.0.55"
57
+ django = "6.1.2"
55
58
  mysql_reference = "8.0.46" # tests/compat/docker-compose.yml
@@ -11,6 +11,8 @@ real database you change one import line and nothing else:
11
11
 
12
12
  SQLAlchemy (Core / ORM) needs no import: create_engine("sqlite+s3rds://user:password@<lambda-url>/<db>")
13
13
  uses SQLAlchemy's SQLite dialect with s3rds.sqlite as its driver; on RDS the URL becomes mysql+pymysql://.
14
+ Django needs no import either: DATABASES ENGINE "s3rds.sqlite" (s3rds/sqlite/base.py) is Django's SQLite
15
+ backend on s3rds.sqlite; on RDS the ENGINE becomes django.db.backends.mysql.
14
16
  """
15
17
  from . import sqlite # noqa: F401
16
18
  from . import pymysql # noqa: F401 (sqlglot is only needed when a MySQL statement is actually translated)
@@ -200,6 +200,7 @@ _RENAME_TABLE = re.compile(r"^\s*RENAME\s+TABLE\s+(.+)$", re.I | re.S)
200
200
  _RENAME_PAIR = re.compile(r"`?(\w+)`?\s+TO\s+`?(\w+)`?", re.I)
201
201
  _SESSION_NOOP = re.compile(r"^\s*SET\s+(?!@)", re.I) # SET NAMES / SESSION / time_zone / sql_mode ...
202
202
  _USE = re.compile(r"^\s*USE\s+`?(\w+)`?\s*$", re.I)
203
+ _IMPLICIT_COMMIT = re.compile(r"^\s*(CREATE|ALTER|DROP|RENAME|TRUNCATE)\b", re.I) # MySQL commits the open transaction first
203
204
  # LAST_INSERT_ID() / DATABASE(): per-session values the server does not know; the driver fills them in
204
205
  _LAST_ID_TOKEN = "__S3RDS_LAST_INSERT_ID__"
205
206
  _DATABASE_TOKEN = "__S3RDS_DATABASE__"
@@ -743,6 +744,8 @@ class Connection(_sqlite.Connection):
743
744
 
744
745
  def execute(self, sql, params=()):
745
746
  sql = self._prepare(sql)
747
+ if self._tx is not None and _IMPLICIT_COMMIT.match(sql):
748
+ self.commit() # after START TRANSACTION, DDL ends the transaction as in MySQL
746
749
  m = _META.match(sql)
747
750
  if m:
748
751
  return self._meta(m)
@@ -27,6 +27,11 @@ the next statement or commit() raise OperationalError (roll back and run it agai
27
27
 
28
28
  SQLAlchemy: create_engine("sqlite+s3rds://user:password@xxx.lambda-url.us-east-2.on.aws/blog")
29
29
  SQLAlchemy's SQLite dialect with this driver, autocommit=False; SQLAlchemy itself is the application's.
30
+ Django: DATABASES = {"default": {"ENGINE": "s3rds.sqlite", "NAME": "blog", "HOST": "xxx.lambda-url...",
31
+ "USER": "admin", "PASSWORD": "..."}} -- Django's SQLite backend on this driver (s3rds/sqlite/base.py).
32
+
33
+ Connection settings as sqlite3 keeps them per connection: PRAGMA foreign_keys (on by default, as the server
34
+ enforces them) and PRAGMA legacy_alter_table are remembered here and applied by the server to every request.
30
35
 
31
36
  Retry policy -- fail rather than duplicate a write:
32
37
  the request is re-sent exactly once, and only when it provably never reached the server
@@ -44,17 +49,22 @@ import socket
44
49
  import urllib.parse
45
50
  import warnings
46
51
 
47
- __version__ = "0.2.0"
52
+ __version__ = "0.2.1"
48
53
 
49
54
  apilevel = "2.0"
50
55
  threadsafety = 1
51
56
  paramstyle = "format" # %s like pymysql; ? is accepted too
52
- # the engine behind the endpoint: the Lambda runtime's SQLite (tests/compat/TASKS.md, task 11);
53
- # SQLAlchemy reads it to decide what SQL it may use (RETURNING needs 3.35)
54
- sqlite_version = "3.40.0"
55
- sqlite_version_info = (3, 40, 0)
57
+ # the engine behind the endpoint: the server ships its own SQLite (src/db/requirements.txt, sqlean.py);
58
+ # SQLAlchemy and Django read it to decide what SQL they may use (RETURNING needs 3.35)
59
+ sqlite_version = "3.50.4"
60
+ sqlite_version_info = (3, 50, 4)
56
61
  Binary = memoryview # as sqlite3
57
62
 
63
+ # Connection.getlimit(): sqlite3's category numbers, the engine's compile-time defaults (Django sizes batches by them)
64
+ SQLITE_LIMIT_COLUMN = 2
65
+ SQLITE_LIMIT_VARIABLE_NUMBER = 9
66
+ _LIMITS = {SQLITE_LIMIT_COLUMN: 2000, SQLITE_LIMIT_VARIABLE_NUMBER: 32766}
67
+
58
68
 
59
69
  # ---------------- exceptions: the DB-API classes sqlite3 raises ----------------
60
70
 
@@ -273,7 +283,8 @@ class cursors: # so that sqlite.cursors.DictCursor reads like pymysql.cursors.D
273
283
  class Connection:
274
284
  _typed = False # ask the server for column-type markers (the MySQL driver does; sqlite3 returns text)
275
285
 
276
- def __init__(self, url, database, user="", password="", timeout=30, cursorclass=Cursor, autocommit=True):
286
+ def __init__(self, url, database, user="", password="", timeout=30, cursorclass=Cursor, autocommit=True,
287
+ paramstyle="format"):
277
288
  p = urllib.parse.urlparse(url.rstrip("/"))
278
289
  self._host = p.netloc
279
290
  self._base_path = p.path
@@ -286,8 +297,12 @@ class Connection:
286
297
  self.last = {}
287
298
  self._written = False # a write landed in S3 since the last commit() (drives the rollback() warning)
288
299
  self._autocommit = autocommit
289
- # the open transaction (autocommit=False): {"log": statements that may have written, "base": ETag}
300
+ # the open transaction: {"log": statements that may have written, "base": ETag}
290
301
  self._tx = None
302
+ # "qmark": the SQL is SQLite's own (? placeholders, % is a plain character) -- what ORMs write
303
+ self._qmark = paramstyle == "qmark"
304
+ # connection settings the server applies to every request (sqlite3 keeps them per connection)
305
+ self._pragmas = {}
291
306
 
292
307
  # ---- transport ----
293
308
 
@@ -365,23 +380,23 @@ class Connection:
365
380
  return (cursorclass or self.cursorclass)(self)
366
381
 
367
382
  def commit(self):
368
- """autocommit=False: the open transaction's writes commit together, S3 is written once; a conflict
369
- with another writer raises OperationalError and nothing lands.
370
- autocommit=True: every statement is already committed and durable when execute() returns; this only
371
- clears the 'written since last commit' flag (kept for pymysql compatibility)."""
372
- tx, self._tx = self._tx, None
383
+ """An open transaction: its writes commit together, S3 is written once; a conflict with another
384
+ writer (or a deferred constraint) raises and nothing lands -- the transaction is gone either way.
385
+ Without one (autocommit) every statement is already committed and durable when execute() returns; this
386
+ only clears the 'written since last commit' flag (kept for pymysql compatibility)."""
387
+ tx, self._tx, self._written = self._tx, None, False # a failed commit is not "writes rollback() cannot undo"
373
388
  if tx and tx["log"]:
374
389
  payload = {"statements": tx["log"], "replay": len(tx["log"]), "base": tx["base"], "commit": True}
375
390
  self._note(self._replay(payload))
376
- self._written = False
391
+ self._written = False
377
392
 
378
393
  def rollback(self):
379
- """autocommit=False: the open transaction never reached S3; forgetting it is the rollback.
380
- autocommit=True: cannot undo anything, statements are durable as soon as they run. If something was
381
- written since the last commit() this warns (never silent); with nothing written it stays quiet,
394
+ """An open transaction never reached S3; forgetting it is the rollback.
395
+ Without one (autocommit) nothing can be undone, statements are durable as soon as they run. If something
396
+ was written since the last commit() this warns (never silent); with nothing written it stays quiet,
382
397
  because pools and ORMs call rollback() when returning a connection."""
383
- self._tx = None
384
- if self._written and self._autocommit:
398
+ tx, self._tx = self._tx, None
399
+ if tx is None and self._written and self._autocommit:
385
400
  warnings.warn("s3rds: rollback() cannot undo statements that already ran; "
386
401
  "use transaction() for all-or-nothing writes", RuntimeWarning, stacklevel=2)
387
402
  self._written = False
@@ -402,6 +417,12 @@ class Connection:
402
417
  def in_transaction(self):
403
418
  return self._tx is not None
404
419
 
420
+ def getlimit(self, category):
421
+ """As sqlite3.Connection.getlimit, for the limits ORMs size their batches by."""
422
+ if category not in _LIMITS:
423
+ raise ProgrammingError(0, f"limit category {category} is not available over s3rds")
424
+ return _LIMITS[category]
425
+
405
426
  def create_function(self, *a, **kw):
406
427
  raise NotSupportedError(0, "user-defined functions would run on the server; "
407
428
  "not supported (that is the moment to move to RDS)")
@@ -439,18 +460,20 @@ class Connection:
439
460
  always complete. If the database changes while paging the server refuses to mix versions
440
461
  (409) and the whole query is re-run, up to 3 times.
441
462
 
442
- autocommit=False follows sqlite3's rules: INSERT / UPDATE / DELETE / REPLACE (and SAVEPOINT) open a
443
- transaction; BEGIN / COMMIT / ROLLBACK statements act as begin / commit() / rollback(); anything
444
- else outside a transaction runs on its own.
463
+ Transactions follow sqlite3's rules: BEGIN / COMMIT / ROLLBACK statements act as begin / commit() /
464
+ rollback() (also with autocommit, as sqlite3 with isolation_level None); with autocommit=False
465
+ INSERT / UPDATE / DELETE / REPLACE (and SAVEPOINT) open one; anything else outside a transaction runs
466
+ on its own.
445
467
  """
446
- if not self._autocommit:
468
+ r = self._session_pragma(sql)
469
+ if r is not None:
470
+ return r
471
+ if not self._autocommit or self._tx is not None or _BEGIN.match(sql):
447
472
  r = self._tx_statement(sql, params)
448
473
  if r is not None:
449
474
  return r
450
475
  path = f"/api/db/{self.database}/query"
451
- payload = {"sql": _to_qmark(sql), "params": _encode_params(params)}
452
- if self._typed:
453
- payload["types"] = True
476
+ payload = self._options({"sql": self._sql(sql), "params": _encode_params(params)})
454
477
  for attempt in range(3):
455
478
  try:
456
479
  return self._fetch_pages(path, payload)
@@ -465,7 +488,7 @@ class Connection:
465
488
  raise InterfaceError(0, "the server truncated the result (server older than 0.1.1); upgrade the server")
466
489
  rows, page = first["rows"], first
467
490
  while page.get("next_token"):
468
- page = self._post(path, {"next_token": page["next_token"]})
491
+ page = self._post(path, self._options({"next_token": page["next_token"]}))
469
492
  rows += page["rows"]
470
493
  first["rows"] = rows
471
494
  first.pop("truncated", None)
@@ -488,16 +511,45 @@ class Connection:
488
511
 
489
512
  Each item is "sql" or (sql, params). Returns one result per statement.
490
513
  """
491
- stmts = [_statement(s, []) if isinstance(s, str) else _statement(s[0], s[1] if len(s) > 1 else [])
514
+ stmts = [self._statement(s, []) if isinstance(s, str) else self._statement(s[0], s[1] if len(s) > 1 else [])
492
515
  for s in statements]
493
- payload = {"statements": stmts}
494
- if self._typed:
495
- payload["types"] = True
496
- r = self._post(f"/api/db/{self.database}/tx", payload)
516
+ r = self._post(f"/api/db/{self.database}/tx", self._options({"statements": stmts}))
497
517
  self._note(r)
498
518
  return [_decode_result(x) for x in r["results"]]
499
519
 
500
- # ---- open transactions (autocommit=False) ----
520
+ def _sql(self, sql):
521
+ return sql if self._qmark else _to_qmark(sql)
522
+
523
+ def _statement(self, sql, params):
524
+ return {"sql": self._sql(sql), "params": _encode_params(params)}
525
+
526
+ def _options(self, payload):
527
+ """What every request carries besides its statements: type markers, the connection's PRAGMAs."""
528
+ if self._typed:
529
+ payload["types"] = True
530
+ if self._pragmas:
531
+ payload["pragmas"] = dict(self._pragmas)
532
+ return payload
533
+
534
+ def _session_pragma(self, sql):
535
+ """PRAGMA foreign_keys / legacy_alter_table = v: remembered here, sent with every request (the server
536
+ keeps no session). Inside a transaction sqlite3 ignores foreign_keys; legacy_alter_table would change
537
+ the transaction's replay, so it is refused there. Returns None for any other statement."""
538
+ m = _SESSION_PRAGMA.match(sql)
539
+ if not m:
540
+ return None
541
+ name, raw = m.group(1).lower(), m.group(2) or m.group(3)
542
+ value = raw.lower()
543
+ if value not in _PRAGMA_VALUES:
544
+ raise OperationalError(0, f"PRAGMA {name}: unknown value {raw!r}")
545
+ if self._tx is not None:
546
+ if name == "legacy_alter_table":
547
+ raise NotSupportedError(0, "PRAGMA legacy_alter_table inside a transaction; set it before BEGIN")
548
+ return _empty() # as sqlite3: foreign_keys cannot change inside a transaction
549
+ self._pragmas[name] = _PRAGMA_VALUES[value]
550
+ return _empty()
551
+
552
+ # ---- open transactions ----
501
553
  # The server keeps no session: every statement inside a transaction is sent with the log of the
502
554
  # statements that may have written so far. The server replays the log on the version the transaction
503
555
  # began on (base), runs the new statements, returns their results and rolls back; commit() sends the log
@@ -518,7 +570,7 @@ class Connection:
518
570
  self.rollback()
519
571
  return _empty()
520
572
  if self._tx is None:
521
- if not _IMPLICIT_BEGIN.match(sql):
573
+ if self._autocommit or not _IMPLICIT_BEGIN.match(sql):
522
574
  return None
523
575
  self._tx = {"log": [], "base": None}
524
576
  return self._tx_run([(sql, params)])[0]
@@ -527,7 +579,7 @@ class Connection:
527
579
  statements = [(sql, p) for p in seq_of_params]
528
580
  if not statements:
529
581
  return []
530
- if self._autocommit or (self._tx is None and not _IMPLICIT_BEGIN.match(sql)):
582
+ if self._tx is None and (self._autocommit or not _IMPLICIT_BEGIN.match(sql)):
531
583
  return self.transaction(statements)
532
584
  if self._tx is None:
533
585
  self._tx = {"log": [], "base": None}
@@ -535,7 +587,7 @@ class Connection:
535
587
 
536
588
  def _tx_run(self, statements):
537
589
  tx = self._tx
538
- new = [_statement(sql, params) for sql, params in statements]
590
+ new = [self._statement(sql, params) for sql, params in statements]
539
591
  payload = {"statements": tx["log"] + new, "replay": len(tx["log"]), "commit": False}
540
592
  if tx["base"]:
541
593
  payload["base"] = tx["base"]
@@ -546,10 +598,8 @@ class Connection:
546
598
  return [_decode_result(x) for x in r["results"]]
547
599
 
548
600
  def _replay(self, payload):
549
- if self._typed:
550
- payload["types"] = True
551
601
  try:
552
- return self._post(f"/api/db/{self.database}/replay", payload)
602
+ return self._post(f"/api/db/{self.database}/replay", self._options(payload))
553
603
  except Error as e:
554
604
  if e.status == 404 and "no route" in e.message:
555
605
  raise OperationalError(404, "the server has no open transactions (older than this driver); "
@@ -561,11 +611,13 @@ _BEGIN = re.compile(r"\s*BEGIN\b", re.I)
561
611
  _COMMIT = re.compile(r"\s*(COMMIT|END)\b", re.I)
562
612
  _ROLLBACK = re.compile(r"\s*ROLLBACK\b(?!.*\bTO\b)", re.I | re.S) # ROLLBACK TO <savepoint> is a statement
563
613
  _IMPLICIT_BEGIN = re.compile(r"\s*(INSERT|UPDATE|DELETE|REPLACE|SAVEPOINT)\b", re.I)
564
- _READS = re.compile(r"\s*(SELECT|VALUES|EXPLAIN)\b", re.I) # cannot write: never replayed
565
-
566
-
567
- def _statement(sql, params):
568
- return {"sql": _to_qmark(sql), "params": _encode_params(params)}
614
+ # cannot write, never replayed: queries and the PRAGMAs that read (a bare name, or one naming a table / index)
615
+ _READS = re.compile(r"\s*(SELECT|VALUES|EXPLAIN)\b|\s*PRAGMA\s+(\w+\s*;?\s*$|(table_info|table_xinfo|table_list|"
616
+ r"index_list|index_info|index_xinfo|foreign_key_list|foreign_key_check|integrity_check|"
617
+ r"quick_check)\b)", re.I)
618
+ _SESSION_PRAGMA = re.compile(r"\s*PRAGMA\s+(foreign_keys|legacy_alter_table)\s*(?:=\s*(\w+)|\(\s*(\w+)\s*\))\s*;?\s*$",
619
+ re.I)
620
+ _PRAGMA_VALUES = {"1": 1, "on": 1, "true": 1, "yes": 1, "0": 0, "off": 0, "false": 0, "no": 0}
569
621
 
570
622
 
571
623
  def _empty():
@@ -575,15 +627,17 @@ def _empty():
575
627
  # ---------------- module entry point, same signature as pymysql.connect ----------------
576
628
 
577
629
  def connect(host, user="", password="", database="", timeout=30,
578
- cursorclass=Cursor, autocommit=True, **_ignored):
630
+ cursorclass=Cursor, autocommit=True, paramstyle="format", **_ignored):
579
631
  """Extra pymysql arguments (port, charset, ...) are accepted and ignored.
580
632
 
581
633
  host may be given with or without https:// (Lambda Function URLs are https only).
582
634
  autocommit=True (default): every statement commits on its own. autocommit=False: transactions as
583
635
  sqlite3 has them -- commit() / rollback(), reads see the transaction's own writes.
636
+ paramstyle="format" (default): %s or ? placeholders, %% for a literal %. "qmark": SQLite's own SQL,
637
+ ? placeholders only and % left alone (what SQLAlchemy and Django write).
584
638
  """
585
639
  url = host if host.startswith("http") else "https://" + host
586
- return Connection(url, database, user, password, timeout, cursorclass, autocommit)
640
+ return Connection(url, database, user, password, timeout, cursorclass, autocommit, paramstyle)
587
641
 
588
642
 
589
643
  # ---------------- SQLAlchemy: sqlite+s3rds://user:password@<lambda-url>/<database> ----------------
@@ -626,7 +680,7 @@ def _sqlalchemy_dialect():
626
680
  def create_connect_args(self, url):
627
681
  host = url.host + (f":{url.port}" if url.port else "")
628
682
  return [], {"host": host, "user": url.username or "", "password": url.password or "",
629
- "database": url.database or "", "autocommit": False}
683
+ "database": url.database or "", "autocommit": False, "paramstyle": "qmark"}
630
684
 
631
685
  def on_connect(self):
632
686
  return None # pysqlite registers Python functions (regexp, floor) here; they cannot run on the server
@@ -0,0 +1,248 @@
1
+ """Django on s3rds: ENGINE "s3rds.sqlite" -- Django's own SQLite backend with s3rds.sqlite as its driver.
2
+
3
+ DATABASES = {"default": {
4
+ "ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
5
+ "NAME": "blog", # the s3rds database (create it once in the console)
6
+ "HOST": "xxx.lambda-url.us-east-2.on.aws",
7
+ "USER": "admin",
8
+ "PASSWORD": "...",
9
+ }}
10
+
11
+ Django writes the same SQLite SQL as for a local file; only what cannot run over a server that keeps no session
12
+ and runs no Python is changed here:
13
+ - the connection is s3rds.sqlite: Django's transactions (atomic, savepoints) on the driver's open transactions,
14
+ PRAGMA foreign_keys (migrations turn it off) on the driver's connection settings;
15
+ - Django's SQLite backend registers ~35 Python functions on every connection. The date ones (Extract, Trunc,
16
+ __year, __date, ...) are written here with SQLite's own strftime(); the rest would need Python on the server
17
+ and raise NotSupportedError naming the feature: time-zone conversion in queries (USE_TZ with a TIME_ZONE
18
+ other than the connection's), regex lookups, MD5 / SHA*, LPad / RPad / Repeat / Reverse / Cot / Random
19
+ (order_by("?")), StdDev / Variance, duration arithmetic (F("dt") + timedelta, F("end") - F("start")).
20
+ Imported by Django (ENGINE), never by `import s3rds`; Django is the application's.
21
+ """
22
+ import re
23
+ from collections.abc import Mapping
24
+
25
+ from django.conf import settings
26
+ from django.core.exceptions import ImproperlyConfigured
27
+ from django.db import NotSupportedError, models
28
+ from django.db.backends.sqlite3 import base, client, creation, features, operations
29
+ from django.db.models import functions
30
+ from django.utils.asyncio import async_unsafe
31
+
32
+ from s3rds import sqlite as Database
33
+
34
+ _NO_PYTHON = "Django's SQLite backend runs it as a Python function on the connection; s3rds runs no Python " \
35
+ "on the server (move to RDS for it)"
36
+
37
+
38
+ class DatabaseFeatures(features.DatabaseFeatures):
39
+ # the parent reads the version of the local sqlite3 module; the engine is the server's
40
+ supports_aggregate_order_by_clause = Database.sqlite_version_info >= (3, 44, 0)
41
+ can_clone_databases = False
42
+ # Python functions in Django's SQLite backend: Django refuses these features itself
43
+ supports_any_value = False
44
+ supports_uuid4_function = False
45
+
46
+ @property
47
+ def supports_uuid7_function(self):
48
+ return False
49
+
50
+
51
+ # Django functions / aggregates that are Python functions on a sqlite3 connection (django/db/backends/sqlite3/_functions.py)
52
+ _PYTHON_FUNCTIONS = tuple(c for c in (
53
+ functions.MD5, functions.SHA1, functions.SHA224, functions.SHA256, functions.SHA384, functions.SHA512,
54
+ functions.LPad, functions.RPad, functions.Repeat, functions.Reverse, functions.Cot, functions.Random,
55
+ models.StdDev, models.Variance,
56
+ getattr(models, "BitAnd", None), getattr(models, "BitOr", None), getattr(models, "BitXor", None),
57
+ ) if c is not None)
58
+
59
+ # Extract(lookup) -> strftime format; the value is the integer Django's Python function returns
60
+ _EXTRACT = {"year": "%Y", "month": "%m", "day": "%d", "hour": "%H", "minute": "%M", "second": "%S",
61
+ "week": "%V", "iso_year": "%G", "iso_week_day": "%u"}
62
+ # Trunc(lookup) on a DateField / DateTimeField / TimeField -> strftime format (+ modifiers)
63
+ _TRUNC_DATE = {"year": "%Y-01-01", "month": "%Y-%m-01", "week": "%Y-%m-%d", "day": "%Y-%m-%d"}
64
+ _TRUNC_DATETIME = {"year": "%Y-01-01 00:00:00", "month": "%Y-%m-01 00:00:00", "week": "%Y-%m-%d 00:00:00",
65
+ "day": "%Y-%m-%d 00:00:00", "hour": "%Y-%m-%d %H:00:00", "minute": "%Y-%m-%d %H:%M:00",
66
+ "second": "%Y-%m-%d %H:%M:%S"}
67
+ _TRUNC_TIME = {"hour": "%H:00:00", "minute": "%H:%M:00", "second": "%H:%M:%S"}
68
+
69
+
70
+ class DatabaseOperations(operations.DatabaseOperations):
71
+ """Django's SQLite operations with the date functions in SQLite's own SQL (no django_* Python functions).
72
+ Formats travel as parameters, as Django passes its lookup names, so % never appears in the SQL."""
73
+
74
+ def check_expression_support(self, expression):
75
+ if isinstance(expression, _PYTHON_FUNCTIONS):
76
+ raise NotSupportedError(f"{type(expression).__name__}(): {_NO_PYTHON}")
77
+ super().check_expression_support(expression)
78
+
79
+ def _no_tz_conversion(self, tzname):
80
+ if tzname and settings.USE_TZ and tzname != self.connection.timezone_name:
81
+ raise NotSupportedError(
82
+ f"time-zone conversion in a query ({self.connection.timezone_name} -> {tzname}): {_NO_PYTHON}. "
83
+ f"Use TIME_ZONE = '{self.connection.timezone_name}', or timezone.override() around the query")
84
+
85
+ def _extract(self, lookup_type, sql, params):
86
+ lookup_type = lookup_type.lower()
87
+ if lookup_type == "week_day": # Sunday = 1 ... Saturday = 7
88
+ return f"(CAST(strftime(%s, {sql}) AS INTEGER) + 1)", ("%w", *params)
89
+ if lookup_type == "quarter":
90
+ return f"((CAST(strftime(%s, {sql}) AS INTEGER) + 2) / 3)", ("%m", *params)
91
+ if lookup_type not in _EXTRACT:
92
+ raise ValueError(f"Unsupported lookup type: {lookup_type!r}")
93
+ return f"CAST(strftime(%s, {sql}) AS INTEGER)", (_EXTRACT[lookup_type], *params)
94
+
95
+ def _trunc(self, formats, suffix, lookup_type, sql, params):
96
+ lookup_type = lookup_type.lower()
97
+ if lookup_type == "quarter": # the first month of the quarter
98
+ return (f"(strftime(%s, {sql}) || printf(%s, (CAST(strftime(%s, {sql}) AS INTEGER) - 1) / 3 * 3 + 1)"
99
+ f" || %s)", ("%Y-", *params, "%02d", "%m", *params, "-01" + suffix))
100
+ if lookup_type not in formats:
101
+ raise ValueError(f"Unsupported lookup type: {lookup_type!r}")
102
+ if lookup_type == "week": # back to Monday: forward to Sunday, then six days back
103
+ return f"strftime(%s, {sql}, 'weekday 0', '-6 days')", (formats["week"], *params)
104
+ return f"strftime(%s, {sql})", (formats[lookup_type], *params)
105
+
106
+ def date_extract_sql(self, lookup_type, sql, params):
107
+ return self._extract(lookup_type, sql, params)
108
+
109
+ def datetime_extract_sql(self, lookup_type, sql, params, tzname):
110
+ self._no_tz_conversion(tzname)
111
+ return self._extract(lookup_type, sql, params)
112
+
113
+ def time_extract_sql(self, lookup_type, sql, params):
114
+ return self._extract(lookup_type, sql, params)
115
+
116
+ def date_trunc_sql(self, lookup_type, sql, params, tzname=None):
117
+ self._no_tz_conversion(tzname)
118
+ return self._trunc(_TRUNC_DATE, "", lookup_type, sql, params)
119
+
120
+ def datetime_trunc_sql(self, lookup_type, sql, params, tzname):
121
+ self._no_tz_conversion(tzname)
122
+ return self._trunc(_TRUNC_DATETIME, " 00:00:00", lookup_type, sql, params)
123
+
124
+ def time_trunc_sql(self, lookup_type, sql, params, tzname=None):
125
+ self._no_tz_conversion(tzname)
126
+ return self._trunc(_TRUNC_TIME, "", lookup_type, sql, params)
127
+
128
+ def datetime_cast_date_sql(self, sql, params, tzname):
129
+ self._no_tz_conversion(tzname)
130
+ return f"date({sql})", params
131
+
132
+ def datetime_cast_time_sql(self, sql, params, tzname):
133
+ self._no_tz_conversion(tzname)
134
+ # time() drops the microseconds; Django's value keeps them when there are any (HH:MM:SS[.ffffff])
135
+ return (f"(time({sql}) || CASE WHEN substr({sql}, 20, 1) = '.' THEN substr({sql}, 20, 7) ELSE '' END)",
136
+ (*params, *params, *params))
137
+
138
+ def combine_expression(self, connector, sub_expressions):
139
+ if connector == "#":
140
+ raise NotSupportedError(f"bitwise XOR (BITXOR): {_NO_PYTHON}")
141
+ return super().combine_expression(connector, sub_expressions) # ^ is POWER(), built into the engine
142
+
143
+ def combine_duration_expression(self, connector, sub_expressions):
144
+ raise NotSupportedError(f"date / time arithmetic with a timedelta: {_NO_PYTHON}")
145
+
146
+ def subtract_temporals(self, internal_type, lhs, rhs):
147
+ raise NotSupportedError(f"subtracting dates / times: {_NO_PYTHON}")
148
+
149
+ def regex_lookup(self, lookup_type):
150
+ raise NotSupportedError(f"__{lookup_type} lookups (REGEXP): {_NO_PYTHON}")
151
+
152
+ def _quote_params_for_last_executed_query(self, params):
153
+ """connection.queries with DEBUG: quote here as SQLite's QUOTE() would, instead of one more request per query."""
154
+ return tuple(_quote(p) for p in params)
155
+
156
+
157
+ def _quote(value):
158
+ if value is None:
159
+ return "NULL"
160
+ if isinstance(value, bool):
161
+ return str(int(value))
162
+ if isinstance(value, (int, float)):
163
+ return repr(value)
164
+ if isinstance(value, (bytes, bytearray, memoryview)):
165
+ return f"X'{bytes(value).hex().upper()}'"
166
+ return "'" + str(value).replace("'", "''") + "'"
167
+
168
+
169
+ class DatabaseCreation(creation.DatabaseCreation):
170
+ def _create_test_db(self, verbosity, autoclobber, keepdb=False):
171
+ raise NotSupportedError("test databases on s3rds: run the tests on django.db.backends.sqlite3 "
172
+ "(the same SQL on a local file)")
173
+
174
+
175
+ class DatabaseClient(client.DatabaseClient):
176
+ @classmethod
177
+ def settings_to_cmd_args_env(cls, settings_dict, parameters):
178
+ raise NotSupportedError("dbshell: open the s3rds console (the Function URL in a browser)")
179
+
180
+
181
+ _FORMAT = re.compile(r"(?<!%)%s")
182
+ _PYFORMAT = re.compile(r"%\((\w+)\)s")
183
+
184
+
185
+ def _convert(query, params):
186
+ """Django's format (%s) / pyformat (%(name)s) SQL -> ? placeholders and a list, %% -> %
187
+ (as Django's SQLiteCursorWrapper, which turns pyformat into :name for sqlite3)."""
188
+ if isinstance(params, Mapping):
189
+ names = _PYFORMAT.findall(query)
190
+ return _PYFORMAT.sub("?", query).replace("%%", "%"), [params[n] for n in names]
191
+ return _FORMAT.sub("?", query).replace("%%", "%"), params
192
+
193
+
194
+ class CursorWrapper(Database.Cursor):
195
+ """The cursor Django gets; execute() returns the cursor, as sqlite3's does (Django chains .fetchone())."""
196
+
197
+ def execute(self, query, params=None):
198
+ if params is not None:
199
+ query, params = _convert(query, params)
200
+ super().execute(query, params)
201
+ return self
202
+
203
+ def executemany(self, query, param_list):
204
+ param_list = list(param_list)
205
+ if param_list and isinstance(param_list[0], Mapping):
206
+ names = _PYFORMAT.findall(query)
207
+ query = _PYFORMAT.sub("?", query).replace("%%", "%")
208
+ param_list = [[p[n] for n in names] for p in param_list]
209
+ else:
210
+ query = _FORMAT.sub("?", query).replace("%%", "%")
211
+ super().executemany(query, param_list)
212
+ return self
213
+
214
+
215
+ class DatabaseWrapper(base.DatabaseWrapper):
216
+ display_name = "SQLite on s3rds"
217
+ Database = Database
218
+ client_class = DatabaseClient
219
+ creation_class = DatabaseCreation
220
+ features_class = DatabaseFeatures
221
+ ops_class = DatabaseOperations
222
+ # regex / iregex: REGEXP is a Python function on sqlite3; without them Django asks ops.regex_lookup(), which refuses
223
+ operators = {k: v for k, v in base.DatabaseWrapper.operators.items() if k not in ("regex", "iregex")}
224
+
225
+ def get_connection_params(self):
226
+ kwargs = super().get_connection_params() # NAME required; OPTIONS transaction_mode / init_command read
227
+ for key in ("detect_types", "check_same_thread", "uri"):
228
+ kwargs.pop(key, None)
229
+ s = self.settings_dict
230
+ if not s["HOST"]:
231
+ raise ImproperlyConfigured("settings.DATABASES: HOST is the s3rds Function URL host "
232
+ "(xxx.lambda-url.<region>.on.aws)")
233
+ return {**kwargs, "host": s["HOST"] + (f":{s['PORT']}" if s["PORT"] else ""), "user": s["USER"],
234
+ "password": s["PASSWORD"], "database": s["NAME"], "paramstyle": "qmark"}
235
+
236
+ @async_unsafe
237
+ def get_new_connection(self, conn_params):
238
+ conn = Database.connect(**conn_params)
239
+ # no register_functions(conn): Python functions would have to run on the server (DatabaseOperations above)
240
+ conn.execute("PRAGMA foreign_keys = ON")
241
+ conn.execute("PRAGMA legacy_alter_table = OFF")
242
+ for init_command in self.init_commands:
243
+ if init_command := init_command.strip():
244
+ conn.execute(init_command)
245
+ return conn
246
+
247
+ def create_cursor(self, name=None):
248
+ return self.connection.cursor(CursorWrapper)
@@ -1,7 +1,7 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: s3rds
3
- Version: 0.2.0
4
- Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://.
3
+ Version: 0.2.1
4
+ Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite.
5
5
  Author: Frank Zhang
6
6
  License: MIT
7
7
  Project-URL: Homepage, https://github.com/cloudformer/s3rds
@@ -10,13 +10,14 @@ Project-URL: Deploy, https://github.com/cloudformer/s3rds#quick-start
10
10
  Project-URL: Compatibility, https://github.com/cloudformer/s3rds/blob/main/tests/compat/REPORT.md
11
11
  Project-URL: Changelog, https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md
12
12
  Project-URL: Issues, https://github.com/cloudformer/s3rds/issues
13
- Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy
13
+ Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy,django
14
14
  Classifier: Development Status :: 3 - Alpha
15
15
  Classifier: Intended Audience :: Developers
16
16
  Classifier: License :: OSI Approved :: MIT License
17
17
  Classifier: Programming Language :: Python :: 3
18
18
  Classifier: Programming Language :: Python :: 3.9
19
19
  Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Framework :: Django
20
21
  Classifier: Topic :: Database
21
22
  Classifier: Topic :: Database :: Front-Ends
22
23
  Requires-Python: >=3.9
@@ -25,19 +26,42 @@ Requires-Dist: sqlglot<31,>=30.24
25
26
  Provides-Extra: mysql
26
27
  Provides-Extra: sqlalchemy
27
28
  Requires-Dist: SQLAlchemy<2.1,>=2.0; extra == "sqlalchemy"
29
+ Provides-Extra: django
30
+ Requires-Dist: Django<6.2,>=6.1; extra == "django"
28
31
 
29
32
  # s3rds
30
33
 
34
+ ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
35
+
36
+ ### Switch to s3rds — the only change
37
+
38
+ | Your code uses | Before | With s3rds — **the only change** | What really differs |
39
+ |---|---|---|---|
40
+ | **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
41
+ | **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
42
+ | **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
43
+ | **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
44
+ | **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` — *planned, not available yet* | — |
45
+
46
+ Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
47
+ uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
48
+ backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
49
+ `NotSupportedError` — never silently different.
50
+
51
+ ### The server: deploy once, ≈ $0 when idle
52
+
53
+ You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
54
+ while nobody uses it. Deploy it once with Terraform or CloudFormation from the
55
+ [main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
56
+
31
57
  **Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
32
58
  a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
33
59
 
34
- ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
35
-
36
60
  One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
37
61
  client it mirrors**, so moving to a real database is a one-line import change and nothing else:
38
62
 
39
63
  ```bash
40
- pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
64
+ pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
41
65
  ```
42
66
 
43
67
  ```python
@@ -68,6 +92,7 @@ conn.transaction([ # all-or-nothing, single S3 w
68
92
  | `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
69
93
  | `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
70
94
  | `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
95
+ | `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
71
96
 
72
97
  **SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
73
98
  `s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
@@ -76,6 +101,21 @@ conn.transaction([ # all-or-nothing, single S3 w
76
101
  engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
77
102
  ```
78
103
 
104
+ **Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
105
+
106
+ ```python
107
+ DATABASES = {"default": {
108
+ "ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
109
+ "NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
110
+ }}
111
+ ```
112
+
113
+ Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
114
+ `Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
115
+ conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
116
+ `Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
117
+ `django.db.backends.sqlite3`.
118
+
79
119
  ## Semantics
80
120
 
81
121
  - **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
@@ -83,7 +123,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
83
123
  ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
84
124
  pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
85
125
  statements as one commit and one S3 write.
86
- - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) has sqlite3's transactions: reads see
126
+ - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
127
+ `atomic()`) give sqlite3's transactions: reads see
87
128
  the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
88
129
  writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
89
130
  Keep transactions short — each statement replays the writes before it on the server.
@@ -105,9 +146,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
105
146
  | SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
106
147
  | sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
107
148
  | SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
149
+ | Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
108
150
  | MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
109
151
 
110
- ## Server
111
-
112
- You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
113
- from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
@@ -3,10 +3,11 @@ pyproject.toml
3
3
  s3rds/__init__.py
4
4
  s3rds/psycopg2.py
5
5
  s3rds/pymysql.py
6
- s3rds/sqlite.py
7
6
  s3rds.egg-info/PKG-INFO
8
7
  s3rds.egg-info/SOURCES.txt
9
8
  s3rds.egg-info/dependency_links.txt
10
9
  s3rds.egg-info/entry_points.txt
11
10
  s3rds.egg-info/requires.txt
12
- s3rds.egg-info/top_level.txt
11
+ s3rds.egg-info/top_level.txt
12
+ s3rds/sqlite/__init__.py
13
+ s3rds/sqlite/base.py
@@ -1,5 +1,8 @@
1
1
  sqlglot<31,>=30.24
2
2
 
3
+ [django]
4
+ Django<6.2,>=6.1
5
+
3
6
  [mysql]
4
7
 
5
8
  [sqlalchemy]
File without changes
File without changes