s3rds 0.2.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {s3rds-0.2.0 → s3rds-0.2.1}/PKG-INFO +49 -11
- {s3rds-0.2.0 → s3rds-0.2.1}/README.md +43 -8
- {s3rds-0.2.0 → s3rds-0.2.1}/pyproject.toml +7 -4
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds/__init__.py +2 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds/pymysql.py +3 -0
- s3rds-0.2.0/s3rds/sqlite.py → s3rds-0.2.1/s3rds/sqlite/__init__.py +100 -46
- s3rds-0.2.1/s3rds/sqlite/base.py +248 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds.egg-info/PKG-INFO +49 -11
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds.egg-info/SOURCES.txt +3 -2
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds.egg-info/requires.txt +3 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds/psycopg2.py +0 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds.egg-info/dependency_links.txt +0 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds.egg-info/entry_points.txt +0 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/s3rds.egg-info/top_level.txt +0 -0
- {s3rds-0.2.0 → s3rds-0.2.1}/setup.cfg +0 -0
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: s3rds
|
|
3
|
-
Version: 0.2.
|
|
4
|
-
Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds
|
|
3
|
+
Version: 0.2.1
|
|
4
|
+
Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite.
|
|
5
5
|
Author: Frank Zhang
|
|
6
6
|
License: MIT
|
|
7
7
|
Project-URL: Homepage, https://github.com/cloudformer/s3rds
|
|
@@ -10,13 +10,14 @@ Project-URL: Deploy, https://github.com/cloudformer/s3rds#quick-start
|
|
|
10
10
|
Project-URL: Compatibility, https://github.com/cloudformer/s3rds/blob/main/tests/compat/REPORT.md
|
|
11
11
|
Project-URL: Changelog, https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md
|
|
12
12
|
Project-URL: Issues, https://github.com/cloudformer/s3rds/issues
|
|
13
|
-
Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy
|
|
13
|
+
Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy,django
|
|
14
14
|
Classifier: Development Status :: 3 - Alpha
|
|
15
15
|
Classifier: Intended Audience :: Developers
|
|
16
16
|
Classifier: License :: OSI Approved :: MIT License
|
|
17
17
|
Classifier: Programming Language :: Python :: 3
|
|
18
18
|
Classifier: Programming Language :: Python :: 3.9
|
|
19
19
|
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Framework :: Django
|
|
20
21
|
Classifier: Topic :: Database
|
|
21
22
|
Classifier: Topic :: Database :: Front-Ends
|
|
22
23
|
Requires-Python: >=3.9
|
|
@@ -25,19 +26,42 @@ Requires-Dist: sqlglot<31,>=30.24
|
|
|
25
26
|
Provides-Extra: mysql
|
|
26
27
|
Provides-Extra: sqlalchemy
|
|
27
28
|
Requires-Dist: SQLAlchemy<2.1,>=2.0; extra == "sqlalchemy"
|
|
29
|
+
Provides-Extra: django
|
|
30
|
+
Requires-Dist: Django<6.2,>=6.1; extra == "django"
|
|
28
31
|
|
|
29
32
|
# s3rds
|
|
30
33
|
|
|
34
|
+

|
|
35
|
+
|
|
36
|
+
### Switch to s3rds — the only change
|
|
37
|
+
|
|
38
|
+
| Your code uses | Before | With s3rds — **the only change** | What really differs |
|
|
39
|
+
|---|---|---|---|
|
|
40
|
+
| **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
|
|
41
|
+
| **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
|
|
42
|
+
| **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
|
|
43
|
+
| **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
|
|
44
|
+
| **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` — *planned, not available yet* | — |
|
|
45
|
+
|
|
46
|
+
Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
|
|
47
|
+
uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
|
|
48
|
+
backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
|
|
49
|
+
`NotSupportedError` — never silently different.
|
|
50
|
+
|
|
51
|
+
### The server: deploy once, ≈ $0 when idle
|
|
52
|
+
|
|
53
|
+
You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
|
|
54
|
+
while nobody uses it. Deploy it once with Terraform or CloudFormation from the
|
|
55
|
+
[main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
|
|
56
|
+
|
|
31
57
|
**Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
|
|
32
58
|
a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
|
|
33
59
|
|
|
34
|
-

|
|
35
|
-
|
|
36
60
|
One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
|
|
37
61
|
client it mirrors**, so moving to a real database is a one-line import change and nothing else:
|
|
38
62
|
|
|
39
63
|
```bash
|
|
40
|
-
pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
|
|
64
|
+
pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
|
|
41
65
|
```
|
|
42
66
|
|
|
43
67
|
```python
|
|
@@ -68,6 +92,7 @@ conn.transaction([ # all-or-nothing, single S3 w
|
|
|
68
92
|
| `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
|
|
69
93
|
| `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
|
|
70
94
|
| `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
|
|
95
|
+
| `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
|
|
71
96
|
|
|
72
97
|
**SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
|
|
73
98
|
`s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
|
|
@@ -76,6 +101,21 @@ conn.transaction([ # all-or-nothing, single S3 w
|
|
|
76
101
|
engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
|
|
77
102
|
```
|
|
78
103
|
|
|
104
|
+
**Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
DATABASES = {"default": {
|
|
108
|
+
"ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
|
|
109
|
+
"NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
|
|
110
|
+
}}
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
|
|
114
|
+
`Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
|
|
115
|
+
conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
|
|
116
|
+
`Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
|
|
117
|
+
`django.db.backends.sqlite3`.
|
|
118
|
+
|
|
79
119
|
## Semantics
|
|
80
120
|
|
|
81
121
|
- **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
|
|
@@ -83,7 +123,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
|
|
|
83
123
|
ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
|
|
84
124
|
pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
|
|
85
125
|
statements as one commit and one S3 write.
|
|
86
|
-
- `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses)
|
|
126
|
+
- `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
|
|
127
|
+
`atomic()`) give sqlite3's transactions: reads see
|
|
87
128
|
the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
|
|
88
129
|
writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
|
|
89
130
|
Keep transactions short — each statement replays the writes before it on the server.
|
|
@@ -105,9 +146,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
|
|
|
105
146
|
| SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
|
|
106
147
|
| sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
|
|
107
148
|
| SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
|
|
149
|
+
| Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
|
|
108
150
|
| MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
|
|
109
151
|
|
|
110
|
-
## Server
|
|
111
|
-
|
|
112
|
-
You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
|
|
113
|
-
from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
|
|
@@ -1,15 +1,36 @@
|
|
|
1
1
|
# s3rds
|
|
2
2
|
|
|
3
|
+

|
|
4
|
+
|
|
5
|
+
### Switch to s3rds — the only change
|
|
6
|
+
|
|
7
|
+
| Your code uses | Before | With s3rds — **the only change** | What really differs |
|
|
8
|
+
|---|---|---|---|
|
|
9
|
+
| **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
|
|
10
|
+
| **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
|
|
11
|
+
| **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
|
|
12
|
+
| **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
|
|
13
|
+
| **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` — *planned, not available yet* | — |
|
|
14
|
+
|
|
15
|
+
Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
|
|
16
|
+
uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
|
|
17
|
+
backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
|
|
18
|
+
`NotSupportedError` — never silently different.
|
|
19
|
+
|
|
20
|
+
### The server: deploy once, ≈ $0 when idle
|
|
21
|
+
|
|
22
|
+
You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
|
|
23
|
+
while nobody uses it. Deploy it once with Terraform or CloudFormation from the
|
|
24
|
+
[main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
|
|
25
|
+
|
|
3
26
|
**Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
|
|
4
27
|
a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
|
|
5
28
|
|
|
6
|
-

|
|
7
|
-
|
|
8
29
|
One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
|
|
9
30
|
client it mirrors**, so moving to a real database is a one-line import change and nothing else:
|
|
10
31
|
|
|
11
32
|
```bash
|
|
12
|
-
pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
|
|
33
|
+
pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
|
|
13
34
|
```
|
|
14
35
|
|
|
15
36
|
```python
|
|
@@ -40,6 +61,7 @@ conn.transaction([ # all-or-nothing, single S3 w
|
|
|
40
61
|
| `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
|
|
41
62
|
| `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
|
|
42
63
|
| `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
|
|
64
|
+
| `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
|
|
43
65
|
|
|
44
66
|
**SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
|
|
45
67
|
`s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
|
|
@@ -48,6 +70,21 @@ conn.transaction([ # all-or-nothing, single S3 w
|
|
|
48
70
|
engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
|
|
49
71
|
```
|
|
50
72
|
|
|
73
|
+
**Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
|
|
74
|
+
|
|
75
|
+
```python
|
|
76
|
+
DATABASES = {"default": {
|
|
77
|
+
"ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
|
|
78
|
+
"NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
|
|
79
|
+
}}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
|
|
83
|
+
`Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
|
|
84
|
+
conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
|
|
85
|
+
`Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
|
|
86
|
+
`django.db.backends.sqlite3`.
|
|
87
|
+
|
|
51
88
|
## Semantics
|
|
52
89
|
|
|
53
90
|
- **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
|
|
@@ -55,7 +92,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
|
|
|
55
92
|
ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
|
|
56
93
|
pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
|
|
57
94
|
statements as one commit and one S3 write.
|
|
58
|
-
- `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses)
|
|
95
|
+
- `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
|
|
96
|
+
`atomic()`) give sqlite3's transactions: reads see
|
|
59
97
|
the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
|
|
60
98
|
writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
|
|
61
99
|
Keep transactions short — each statement replays the writes before it on the server.
|
|
@@ -77,9 +115,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
|
|
|
77
115
|
| SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
|
|
78
116
|
| sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
|
|
79
117
|
| SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
|
|
118
|
+
| Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
|
|
80
119
|
| MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
|
|
81
120
|
|
|
82
|
-
## Server
|
|
83
|
-
|
|
84
|
-
You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
|
|
85
|
-
from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
|
|
@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "s3rds"
|
|
7
|
-
version = "0.2.
|
|
8
|
-
description = "A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds
|
|
7
|
+
version = "0.2.1"
|
|
8
|
+
description = "A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite."
|
|
9
9
|
readme = "README.md"
|
|
10
10
|
requires-python = ">=3.9"
|
|
11
11
|
license = { text = "MIT" }
|
|
12
12
|
authors = [{ name = "Frank Zhang" }]
|
|
13
|
-
keywords = ["serverless", "rds", "sqlite", "s3", "lambda", "database", "pymysql", "dbapi", "sqlalchemy"]
|
|
13
|
+
keywords = ["serverless", "rds", "sqlite", "s3", "lambda", "database", "pymysql", "dbapi", "sqlalchemy", "django"]
|
|
14
14
|
classifiers = [
|
|
15
15
|
"Development Status :: 3 - Alpha",
|
|
16
16
|
"Intended Audience :: Developers",
|
|
@@ -18,6 +18,7 @@ classifiers = [
|
|
|
18
18
|
"Programming Language :: Python :: 3",
|
|
19
19
|
"Programming Language :: Python :: 3.9",
|
|
20
20
|
"Programming Language :: Python :: 3.13",
|
|
21
|
+
"Framework :: Django",
|
|
21
22
|
"Topic :: Database",
|
|
22
23
|
"Topic :: Database :: Front-Ends",
|
|
23
24
|
]
|
|
@@ -29,6 +30,7 @@ dependencies = ["sqlglot>=30.24,<31"]
|
|
|
29
30
|
[project.optional-dependencies]
|
|
30
31
|
mysql = [] # kept so that pip install "s3rds[mysql]" from 0.1.x keeps working
|
|
31
32
|
sqlalchemy = ["SQLAlchemy>=2.0,<2.1"] # the series the compatibility report verified; pip install "s3rds[sqlalchemy]"
|
|
33
|
+
django = ["Django>=6.1,<6.2"] # the same for Django (ENGINE "s3rds.sqlite"); Django 6.1 needs Python >= 3.12
|
|
32
34
|
|
|
33
35
|
[project.entry-points."sqlalchemy.dialects"]
|
|
34
36
|
"sqlite.s3rds" = "s3rds.sqlite:SQLAlchemyDialect"
|
|
@@ -42,7 +44,7 @@ Changelog = "https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md"
|
|
|
42
44
|
Issues = "https://github.com/cloudformer/s3rds/issues"
|
|
43
45
|
|
|
44
46
|
[tool.setuptools]
|
|
45
|
-
packages = ["s3rds"]
|
|
47
|
+
packages = ["s3rds", "s3rds.sqlite"]
|
|
46
48
|
|
|
47
49
|
# The versions tests/compat/REPORT.md was verified with; README.md, README.zh.md and client/README.md state the same.
|
|
48
50
|
[tool.s3rds.verified]
|
|
@@ -52,4 +54,5 @@ sqlite = "3.50.4" # the server's engine, shipped in the Lambda
|
|
|
52
54
|
sqlean = "3.50.4.5" # sqlean.py, which carries it (src/db/requirements.txt)
|
|
53
55
|
sqlglot = "30.24.0"
|
|
54
56
|
sqlalchemy = "2.0.55"
|
|
57
|
+
django = "6.1.2"
|
|
55
58
|
mysql_reference = "8.0.46" # tests/compat/docker-compose.yml
|
|
@@ -11,6 +11,8 @@ real database you change one import line and nothing else:
|
|
|
11
11
|
|
|
12
12
|
SQLAlchemy (Core / ORM) needs no import: create_engine("sqlite+s3rds://user:password@<lambda-url>/<db>")
|
|
13
13
|
uses SQLAlchemy's SQLite dialect with s3rds.sqlite as its driver; on RDS the URL becomes mysql+pymysql://.
|
|
14
|
+
Django needs no import either: DATABASES ENGINE "s3rds.sqlite" (s3rds/sqlite/base.py) is Django's SQLite
|
|
15
|
+
backend on s3rds.sqlite; on RDS the ENGINE becomes django.db.backends.mysql.
|
|
14
16
|
"""
|
|
15
17
|
from . import sqlite # noqa: F401
|
|
16
18
|
from . import pymysql # noqa: F401 (sqlglot is only needed when a MySQL statement is actually translated)
|
|
@@ -200,6 +200,7 @@ _RENAME_TABLE = re.compile(r"^\s*RENAME\s+TABLE\s+(.+)$", re.I | re.S)
|
|
|
200
200
|
_RENAME_PAIR = re.compile(r"`?(\w+)`?\s+TO\s+`?(\w+)`?", re.I)
|
|
201
201
|
_SESSION_NOOP = re.compile(r"^\s*SET\s+(?!@)", re.I) # SET NAMES / SESSION / time_zone / sql_mode ...
|
|
202
202
|
_USE = re.compile(r"^\s*USE\s+`?(\w+)`?\s*$", re.I)
|
|
203
|
+
_IMPLICIT_COMMIT = re.compile(r"^\s*(CREATE|ALTER|DROP|RENAME|TRUNCATE)\b", re.I) # MySQL commits the open transaction first
|
|
203
204
|
# LAST_INSERT_ID() / DATABASE(): per-session values the server does not know; the driver fills them in
|
|
204
205
|
_LAST_ID_TOKEN = "__S3RDS_LAST_INSERT_ID__"
|
|
205
206
|
_DATABASE_TOKEN = "__S3RDS_DATABASE__"
|
|
@@ -743,6 +744,8 @@ class Connection(_sqlite.Connection):
|
|
|
743
744
|
|
|
744
745
|
def execute(self, sql, params=()):
|
|
745
746
|
sql = self._prepare(sql)
|
|
747
|
+
if self._tx is not None and _IMPLICIT_COMMIT.match(sql):
|
|
748
|
+
self.commit() # after START TRANSACTION, DDL ends the transaction as in MySQL
|
|
746
749
|
m = _META.match(sql)
|
|
747
750
|
if m:
|
|
748
751
|
return self._meta(m)
|
|
@@ -27,6 +27,11 @@ the next statement or commit() raise OperationalError (roll back and run it agai
|
|
|
27
27
|
|
|
28
28
|
SQLAlchemy: create_engine("sqlite+s3rds://user:password@xxx.lambda-url.us-east-2.on.aws/blog")
|
|
29
29
|
SQLAlchemy's SQLite dialect with this driver, autocommit=False; SQLAlchemy itself is the application's.
|
|
30
|
+
Django: DATABASES = {"default": {"ENGINE": "s3rds.sqlite", "NAME": "blog", "HOST": "xxx.lambda-url...",
|
|
31
|
+
"USER": "admin", "PASSWORD": "..."}} -- Django's SQLite backend on this driver (s3rds/sqlite/base.py).
|
|
32
|
+
|
|
33
|
+
Connection settings as sqlite3 keeps them per connection: PRAGMA foreign_keys (on by default, as the server
|
|
34
|
+
enforces them) and PRAGMA legacy_alter_table are remembered here and applied by the server to every request.
|
|
30
35
|
|
|
31
36
|
Retry policy -- fail rather than duplicate a write:
|
|
32
37
|
the request is re-sent exactly once, and only when it provably never reached the server
|
|
@@ -44,17 +49,22 @@ import socket
|
|
|
44
49
|
import urllib.parse
|
|
45
50
|
import warnings
|
|
46
51
|
|
|
47
|
-
__version__ = "0.2.
|
|
52
|
+
__version__ = "0.2.1"
|
|
48
53
|
|
|
49
54
|
apilevel = "2.0"
|
|
50
55
|
threadsafety = 1
|
|
51
56
|
paramstyle = "format" # %s like pymysql; ? is accepted too
|
|
52
|
-
# the engine behind the endpoint: the
|
|
53
|
-
# SQLAlchemy
|
|
54
|
-
sqlite_version = "3.
|
|
55
|
-
sqlite_version_info = (3,
|
|
57
|
+
# the engine behind the endpoint: the server ships its own SQLite (src/db/requirements.txt, sqlean.py);
|
|
58
|
+
# SQLAlchemy and Django read it to decide what SQL they may use (RETURNING needs 3.35)
|
|
59
|
+
sqlite_version = "3.50.4"
|
|
60
|
+
sqlite_version_info = (3, 50, 4)
|
|
56
61
|
Binary = memoryview # as sqlite3
|
|
57
62
|
|
|
63
|
+
# Connection.getlimit(): sqlite3's category numbers, the engine's compile-time defaults (Django sizes batches by them)
|
|
64
|
+
SQLITE_LIMIT_COLUMN = 2
|
|
65
|
+
SQLITE_LIMIT_VARIABLE_NUMBER = 9
|
|
66
|
+
_LIMITS = {SQLITE_LIMIT_COLUMN: 2000, SQLITE_LIMIT_VARIABLE_NUMBER: 32766}
|
|
67
|
+
|
|
58
68
|
|
|
59
69
|
# ---------------- exceptions: the DB-API classes sqlite3 raises ----------------
|
|
60
70
|
|
|
@@ -273,7 +283,8 @@ class cursors: # so that sqlite.cursors.DictCursor reads like pymysql.cursors.D
|
|
|
273
283
|
class Connection:
|
|
274
284
|
_typed = False # ask the server for column-type markers (the MySQL driver does; sqlite3 returns text)
|
|
275
285
|
|
|
276
|
-
def __init__(self, url, database, user="", password="", timeout=30, cursorclass=Cursor, autocommit=True
|
|
286
|
+
def __init__(self, url, database, user="", password="", timeout=30, cursorclass=Cursor, autocommit=True,
|
|
287
|
+
paramstyle="format"):
|
|
277
288
|
p = urllib.parse.urlparse(url.rstrip("/"))
|
|
278
289
|
self._host = p.netloc
|
|
279
290
|
self._base_path = p.path
|
|
@@ -286,8 +297,12 @@ class Connection:
|
|
|
286
297
|
self.last = {}
|
|
287
298
|
self._written = False # a write landed in S3 since the last commit() (drives the rollback() warning)
|
|
288
299
|
self._autocommit = autocommit
|
|
289
|
-
# the open transaction
|
|
300
|
+
# the open transaction: {"log": statements that may have written, "base": ETag}
|
|
290
301
|
self._tx = None
|
|
302
|
+
# "qmark": the SQL is SQLite's own (? placeholders, % is a plain character) -- what ORMs write
|
|
303
|
+
self._qmark = paramstyle == "qmark"
|
|
304
|
+
# connection settings the server applies to every request (sqlite3 keeps them per connection)
|
|
305
|
+
self._pragmas = {}
|
|
291
306
|
|
|
292
307
|
# ---- transport ----
|
|
293
308
|
|
|
@@ -365,23 +380,23 @@ class Connection:
|
|
|
365
380
|
return (cursorclass or self.cursorclass)(self)
|
|
366
381
|
|
|
367
382
|
def commit(self):
|
|
368
|
-
"""
|
|
369
|
-
|
|
370
|
-
autocommit
|
|
371
|
-
clears the 'written since last commit' flag (kept for pymysql compatibility)."""
|
|
372
|
-
tx, self._tx = self._tx, None
|
|
383
|
+
"""An open transaction: its writes commit together, S3 is written once; a conflict with another
|
|
384
|
+
writer (or a deferred constraint) raises and nothing lands -- the transaction is gone either way.
|
|
385
|
+
Without one (autocommit) every statement is already committed and durable when execute() returns; this
|
|
386
|
+
only clears the 'written since last commit' flag (kept for pymysql compatibility)."""
|
|
387
|
+
tx, self._tx, self._written = self._tx, None, False # a failed commit is not "writes rollback() cannot undo"
|
|
373
388
|
if tx and tx["log"]:
|
|
374
389
|
payload = {"statements": tx["log"], "replay": len(tx["log"]), "base": tx["base"], "commit": True}
|
|
375
390
|
self._note(self._replay(payload))
|
|
376
|
-
|
|
391
|
+
self._written = False
|
|
377
392
|
|
|
378
393
|
def rollback(self):
|
|
379
|
-
"""
|
|
380
|
-
autocommit
|
|
381
|
-
written since the last commit() this warns (never silent); with nothing written it stays quiet,
|
|
394
|
+
"""An open transaction never reached S3; forgetting it is the rollback.
|
|
395
|
+
Without one (autocommit) nothing can be undone, statements are durable as soon as they run. If something
|
|
396
|
+
was written since the last commit() this warns (never silent); with nothing written it stays quiet,
|
|
382
397
|
because pools and ORMs call rollback() when returning a connection."""
|
|
383
|
-
self._tx = None
|
|
384
|
-
if self._written and self._autocommit:
|
|
398
|
+
tx, self._tx = self._tx, None
|
|
399
|
+
if tx is None and self._written and self._autocommit:
|
|
385
400
|
warnings.warn("s3rds: rollback() cannot undo statements that already ran; "
|
|
386
401
|
"use transaction() for all-or-nothing writes", RuntimeWarning, stacklevel=2)
|
|
387
402
|
self._written = False
|
|
@@ -402,6 +417,12 @@ class Connection:
|
|
|
402
417
|
def in_transaction(self):
|
|
403
418
|
return self._tx is not None
|
|
404
419
|
|
|
420
|
+
def getlimit(self, category):
|
|
421
|
+
"""As sqlite3.Connection.getlimit, for the limits ORMs size their batches by."""
|
|
422
|
+
if category not in _LIMITS:
|
|
423
|
+
raise ProgrammingError(0, f"limit category {category} is not available over s3rds")
|
|
424
|
+
return _LIMITS[category]
|
|
425
|
+
|
|
405
426
|
def create_function(self, *a, **kw):
|
|
406
427
|
raise NotSupportedError(0, "user-defined functions would run on the server; "
|
|
407
428
|
"not supported (that is the moment to move to RDS)")
|
|
@@ -439,18 +460,20 @@ class Connection:
|
|
|
439
460
|
always complete. If the database changes while paging the server refuses to mix versions
|
|
440
461
|
(409) and the whole query is re-run, up to 3 times.
|
|
441
462
|
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
else outside a transaction runs
|
|
463
|
+
Transactions follow sqlite3's rules: BEGIN / COMMIT / ROLLBACK statements act as begin / commit() /
|
|
464
|
+
rollback() (also with autocommit, as sqlite3 with isolation_level None); with autocommit=False
|
|
465
|
+
INSERT / UPDATE / DELETE / REPLACE (and SAVEPOINT) open one; anything else outside a transaction runs
|
|
466
|
+
on its own.
|
|
445
467
|
"""
|
|
446
|
-
|
|
468
|
+
r = self._session_pragma(sql)
|
|
469
|
+
if r is not None:
|
|
470
|
+
return r
|
|
471
|
+
if not self._autocommit or self._tx is not None or _BEGIN.match(sql):
|
|
447
472
|
r = self._tx_statement(sql, params)
|
|
448
473
|
if r is not None:
|
|
449
474
|
return r
|
|
450
475
|
path = f"/api/db/{self.database}/query"
|
|
451
|
-
payload = {"sql":
|
|
452
|
-
if self._typed:
|
|
453
|
-
payload["types"] = True
|
|
476
|
+
payload = self._options({"sql": self._sql(sql), "params": _encode_params(params)})
|
|
454
477
|
for attempt in range(3):
|
|
455
478
|
try:
|
|
456
479
|
return self._fetch_pages(path, payload)
|
|
@@ -465,7 +488,7 @@ class Connection:
|
|
|
465
488
|
raise InterfaceError(0, "the server truncated the result (server older than 0.1.1); upgrade the server")
|
|
466
489
|
rows, page = first["rows"], first
|
|
467
490
|
while page.get("next_token"):
|
|
468
|
-
page = self._post(path, {"next_token": page["next_token"]})
|
|
491
|
+
page = self._post(path, self._options({"next_token": page["next_token"]}))
|
|
469
492
|
rows += page["rows"]
|
|
470
493
|
first["rows"] = rows
|
|
471
494
|
first.pop("truncated", None)
|
|
@@ -488,16 +511,45 @@ class Connection:
|
|
|
488
511
|
|
|
489
512
|
Each item is "sql" or (sql, params). Returns one result per statement.
|
|
490
513
|
"""
|
|
491
|
-
stmts = [_statement(s, []) if isinstance(s, str) else _statement(s[0], s[1] if len(s) > 1 else [])
|
|
514
|
+
stmts = [self._statement(s, []) if isinstance(s, str) else self._statement(s[0], s[1] if len(s) > 1 else [])
|
|
492
515
|
for s in statements]
|
|
493
|
-
|
|
494
|
-
if self._typed:
|
|
495
|
-
payload["types"] = True
|
|
496
|
-
r = self._post(f"/api/db/{self.database}/tx", payload)
|
|
516
|
+
r = self._post(f"/api/db/{self.database}/tx", self._options({"statements": stmts}))
|
|
497
517
|
self._note(r)
|
|
498
518
|
return [_decode_result(x) for x in r["results"]]
|
|
499
519
|
|
|
500
|
-
|
|
520
|
+
def _sql(self, sql):
|
|
521
|
+
return sql if self._qmark else _to_qmark(sql)
|
|
522
|
+
|
|
523
|
+
def _statement(self, sql, params):
|
|
524
|
+
return {"sql": self._sql(sql), "params": _encode_params(params)}
|
|
525
|
+
|
|
526
|
+
def _options(self, payload):
|
|
527
|
+
"""What every request carries besides its statements: type markers, the connection's PRAGMAs."""
|
|
528
|
+
if self._typed:
|
|
529
|
+
payload["types"] = True
|
|
530
|
+
if self._pragmas:
|
|
531
|
+
payload["pragmas"] = dict(self._pragmas)
|
|
532
|
+
return payload
|
|
533
|
+
|
|
534
|
+
def _session_pragma(self, sql):
|
|
535
|
+
"""PRAGMA foreign_keys / legacy_alter_table = v: remembered here, sent with every request (the server
|
|
536
|
+
keeps no session). Inside a transaction sqlite3 ignores foreign_keys; legacy_alter_table would change
|
|
537
|
+
the transaction's replay, so it is refused there. Returns None for any other statement."""
|
|
538
|
+
m = _SESSION_PRAGMA.match(sql)
|
|
539
|
+
if not m:
|
|
540
|
+
return None
|
|
541
|
+
name, raw = m.group(1).lower(), m.group(2) or m.group(3)
|
|
542
|
+
value = raw.lower()
|
|
543
|
+
if value not in _PRAGMA_VALUES:
|
|
544
|
+
raise OperationalError(0, f"PRAGMA {name}: unknown value {raw!r}")
|
|
545
|
+
if self._tx is not None:
|
|
546
|
+
if name == "legacy_alter_table":
|
|
547
|
+
raise NotSupportedError(0, "PRAGMA legacy_alter_table inside a transaction; set it before BEGIN")
|
|
548
|
+
return _empty() # as sqlite3: foreign_keys cannot change inside a transaction
|
|
549
|
+
self._pragmas[name] = _PRAGMA_VALUES[value]
|
|
550
|
+
return _empty()
|
|
551
|
+
|
|
552
|
+
# ---- open transactions ----
|
|
501
553
|
# The server keeps no session: every statement inside a transaction is sent with the log of the
|
|
502
554
|
# statements that may have written so far. The server replays the log on the version the transaction
|
|
503
555
|
# began on (base), runs the new statements, returns their results and rolls back; commit() sends the log
|
|
@@ -518,7 +570,7 @@ class Connection:
|
|
|
518
570
|
self.rollback()
|
|
519
571
|
return _empty()
|
|
520
572
|
if self._tx is None:
|
|
521
|
-
if not _IMPLICIT_BEGIN.match(sql):
|
|
573
|
+
if self._autocommit or not _IMPLICIT_BEGIN.match(sql):
|
|
522
574
|
return None
|
|
523
575
|
self._tx = {"log": [], "base": None}
|
|
524
576
|
return self._tx_run([(sql, params)])[0]
|
|
@@ -527,7 +579,7 @@ class Connection:
|
|
|
527
579
|
statements = [(sql, p) for p in seq_of_params]
|
|
528
580
|
if not statements:
|
|
529
581
|
return []
|
|
530
|
-
if self.
|
|
582
|
+
if self._tx is None and (self._autocommit or not _IMPLICIT_BEGIN.match(sql)):
|
|
531
583
|
return self.transaction(statements)
|
|
532
584
|
if self._tx is None:
|
|
533
585
|
self._tx = {"log": [], "base": None}
|
|
@@ -535,7 +587,7 @@ class Connection:
|
|
|
535
587
|
|
|
536
588
|
def _tx_run(self, statements):
|
|
537
589
|
tx = self._tx
|
|
538
|
-
new = [_statement(sql, params) for sql, params in statements]
|
|
590
|
+
new = [self._statement(sql, params) for sql, params in statements]
|
|
539
591
|
payload = {"statements": tx["log"] + new, "replay": len(tx["log"]), "commit": False}
|
|
540
592
|
if tx["base"]:
|
|
541
593
|
payload["base"] = tx["base"]
|
|
@@ -546,10 +598,8 @@ class Connection:
|
|
|
546
598
|
return [_decode_result(x) for x in r["results"]]
|
|
547
599
|
|
|
548
600
|
def _replay(self, payload):
|
|
549
|
-
if self._typed:
|
|
550
|
-
payload["types"] = True
|
|
551
601
|
try:
|
|
552
|
-
return self._post(f"/api/db/{self.database}/replay", payload)
|
|
602
|
+
return self._post(f"/api/db/{self.database}/replay", self._options(payload))
|
|
553
603
|
except Error as e:
|
|
554
604
|
if e.status == 404 and "no route" in e.message:
|
|
555
605
|
raise OperationalError(404, "the server has no open transactions (older than this driver); "
|
|
@@ -561,11 +611,13 @@ _BEGIN = re.compile(r"\s*BEGIN\b", re.I)
|
|
|
561
611
|
_COMMIT = re.compile(r"\s*(COMMIT|END)\b", re.I)
|
|
562
612
|
_ROLLBACK = re.compile(r"\s*ROLLBACK\b(?!.*\bTO\b)", re.I | re.S) # ROLLBACK TO <savepoint> is a statement
|
|
563
613
|
_IMPLICIT_BEGIN = re.compile(r"\s*(INSERT|UPDATE|DELETE|REPLACE|SAVEPOINT)\b", re.I)
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
568
|
-
|
|
614
|
+
# cannot write, never replayed: queries and the PRAGMAs that read (a bare name, or one naming a table / index)
|
|
615
|
+
_READS = re.compile(r"\s*(SELECT|VALUES|EXPLAIN)\b|\s*PRAGMA\s+(\w+\s*;?\s*$|(table_info|table_xinfo|table_list|"
|
|
616
|
+
r"index_list|index_info|index_xinfo|foreign_key_list|foreign_key_check|integrity_check|"
|
|
617
|
+
r"quick_check)\b)", re.I)
|
|
618
|
+
_SESSION_PRAGMA = re.compile(r"\s*PRAGMA\s+(foreign_keys|legacy_alter_table)\s*(?:=\s*(\w+)|\(\s*(\w+)\s*\))\s*;?\s*$",
|
|
619
|
+
re.I)
|
|
620
|
+
_PRAGMA_VALUES = {"1": 1, "on": 1, "true": 1, "yes": 1, "0": 0, "off": 0, "false": 0, "no": 0}
|
|
569
621
|
|
|
570
622
|
|
|
571
623
|
def _empty():
|
|
@@ -575,15 +627,17 @@ def _empty():
|
|
|
575
627
|
# ---------------- module entry point, same signature as pymysql.connect ----------------
|
|
576
628
|
|
|
577
629
|
def connect(host, user="", password="", database="", timeout=30,
|
|
578
|
-
cursorclass=Cursor, autocommit=True, **_ignored):
|
|
630
|
+
cursorclass=Cursor, autocommit=True, paramstyle="format", **_ignored):
|
|
579
631
|
"""Extra pymysql arguments (port, charset, ...) are accepted and ignored.
|
|
580
632
|
|
|
581
633
|
host may be given with or without https:// (Lambda Function URLs are https only).
|
|
582
634
|
autocommit=True (default): every statement commits on its own. autocommit=False: transactions as
|
|
583
635
|
sqlite3 has them -- commit() / rollback(), reads see the transaction's own writes.
|
|
636
|
+
paramstyle="format" (default): %s or ? placeholders, %% for a literal %. "qmark": SQLite's own SQL,
|
|
637
|
+
? placeholders only and % left alone (what SQLAlchemy and Django write).
|
|
584
638
|
"""
|
|
585
639
|
url = host if host.startswith("http") else "https://" + host
|
|
586
|
-
return Connection(url, database, user, password, timeout, cursorclass, autocommit)
|
|
640
|
+
return Connection(url, database, user, password, timeout, cursorclass, autocommit, paramstyle)
|
|
587
641
|
|
|
588
642
|
|
|
589
643
|
# ---------------- SQLAlchemy: sqlite+s3rds://user:password@<lambda-url>/<database> ----------------
|
|
@@ -626,7 +680,7 @@ def _sqlalchemy_dialect():
|
|
|
626
680
|
def create_connect_args(self, url):
|
|
627
681
|
host = url.host + (f":{url.port}" if url.port else "")
|
|
628
682
|
return [], {"host": host, "user": url.username or "", "password": url.password or "",
|
|
629
|
-
"database": url.database or "", "autocommit": False}
|
|
683
|
+
"database": url.database or "", "autocommit": False, "paramstyle": "qmark"}
|
|
630
684
|
|
|
631
685
|
def on_connect(self):
|
|
632
686
|
return None # pysqlite registers Python functions (regexp, floor) here; they cannot run on the server
|
|
@@ -0,0 +1,248 @@
|
|
|
1
|
+
"""Django on s3rds: ENGINE "s3rds.sqlite" -- Django's own SQLite backend with s3rds.sqlite as its driver.
|
|
2
|
+
|
|
3
|
+
DATABASES = {"default": {
|
|
4
|
+
"ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
|
|
5
|
+
"NAME": "blog", # the s3rds database (create it once in the console)
|
|
6
|
+
"HOST": "xxx.lambda-url.us-east-2.on.aws",
|
|
7
|
+
"USER": "admin",
|
|
8
|
+
"PASSWORD": "...",
|
|
9
|
+
}}
|
|
10
|
+
|
|
11
|
+
Django writes the same SQLite SQL as for a local file; only what cannot run over a server that keeps no session
|
|
12
|
+
and runs no Python is changed here:
|
|
13
|
+
- the connection is s3rds.sqlite: Django's transactions (atomic, savepoints) on the driver's open transactions,
|
|
14
|
+
PRAGMA foreign_keys (migrations turn it off) on the driver's connection settings;
|
|
15
|
+
- Django's SQLite backend registers ~35 Python functions on every connection. The date ones (Extract, Trunc,
|
|
16
|
+
__year, __date, ...) are written here with SQLite's own strftime(); the rest would need Python on the server
|
|
17
|
+
and raise NotSupportedError naming the feature: time-zone conversion in queries (USE_TZ with a TIME_ZONE
|
|
18
|
+
other than the connection's), regex lookups, MD5 / SHA*, LPad / RPad / Repeat / Reverse / Cot / Random
|
|
19
|
+
(order_by("?")), StdDev / Variance, duration arithmetic (F("dt") + timedelta, F("end") - F("start")).
|
|
20
|
+
Imported by Django (ENGINE), never by `import s3rds`; Django is the application's.
|
|
21
|
+
"""
|
|
22
|
+
import re
|
|
23
|
+
from collections.abc import Mapping
|
|
24
|
+
|
|
25
|
+
from django.conf import settings
|
|
26
|
+
from django.core.exceptions import ImproperlyConfigured
|
|
27
|
+
from django.db import NotSupportedError, models
|
|
28
|
+
from django.db.backends.sqlite3 import base, client, creation, features, operations
|
|
29
|
+
from django.db.models import functions
|
|
30
|
+
from django.utils.asyncio import async_unsafe
|
|
31
|
+
|
|
32
|
+
from s3rds import sqlite as Database
|
|
33
|
+
|
|
34
|
+
_NO_PYTHON = "Django's SQLite backend runs it as a Python function on the connection; s3rds runs no Python " \
|
|
35
|
+
"on the server (move to RDS for it)"
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
class DatabaseFeatures(features.DatabaseFeatures):
|
|
39
|
+
# the parent reads the version of the local sqlite3 module; the engine is the server's
|
|
40
|
+
supports_aggregate_order_by_clause = Database.sqlite_version_info >= (3, 44, 0)
|
|
41
|
+
can_clone_databases = False
|
|
42
|
+
# Python functions in Django's SQLite backend: Django refuses these features itself
|
|
43
|
+
supports_any_value = False
|
|
44
|
+
supports_uuid4_function = False
|
|
45
|
+
|
|
46
|
+
@property
|
|
47
|
+
def supports_uuid7_function(self):
|
|
48
|
+
return False
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
# Django functions / aggregates that are Python functions on a sqlite3 connection (django/db/backends/sqlite3/_functions.py)
|
|
52
|
+
_PYTHON_FUNCTIONS = tuple(c for c in (
|
|
53
|
+
functions.MD5, functions.SHA1, functions.SHA224, functions.SHA256, functions.SHA384, functions.SHA512,
|
|
54
|
+
functions.LPad, functions.RPad, functions.Repeat, functions.Reverse, functions.Cot, functions.Random,
|
|
55
|
+
models.StdDev, models.Variance,
|
|
56
|
+
getattr(models, "BitAnd", None), getattr(models, "BitOr", None), getattr(models, "BitXor", None),
|
|
57
|
+
) if c is not None)
|
|
58
|
+
|
|
59
|
+
# Extract(lookup) -> strftime format; the value is the integer Django's Python function returns
|
|
60
|
+
_EXTRACT = {"year": "%Y", "month": "%m", "day": "%d", "hour": "%H", "minute": "%M", "second": "%S",
|
|
61
|
+
"week": "%V", "iso_year": "%G", "iso_week_day": "%u"}
|
|
62
|
+
# Trunc(lookup) on a DateField / DateTimeField / TimeField -> strftime format (+ modifiers)
|
|
63
|
+
_TRUNC_DATE = {"year": "%Y-01-01", "month": "%Y-%m-01", "week": "%Y-%m-%d", "day": "%Y-%m-%d"}
|
|
64
|
+
_TRUNC_DATETIME = {"year": "%Y-01-01 00:00:00", "month": "%Y-%m-01 00:00:00", "week": "%Y-%m-%d 00:00:00",
|
|
65
|
+
"day": "%Y-%m-%d 00:00:00", "hour": "%Y-%m-%d %H:00:00", "minute": "%Y-%m-%d %H:%M:00",
|
|
66
|
+
"second": "%Y-%m-%d %H:%M:%S"}
|
|
67
|
+
_TRUNC_TIME = {"hour": "%H:00:00", "minute": "%H:%M:00", "second": "%H:%M:%S"}
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
class DatabaseOperations(operations.DatabaseOperations):
|
|
71
|
+
"""Django's SQLite operations with the date functions in SQLite's own SQL (no django_* Python functions).
|
|
72
|
+
Formats travel as parameters, as Django passes its lookup names, so % never appears in the SQL."""
|
|
73
|
+
|
|
74
|
+
def check_expression_support(self, expression):
|
|
75
|
+
if isinstance(expression, _PYTHON_FUNCTIONS):
|
|
76
|
+
raise NotSupportedError(f"{type(expression).__name__}(): {_NO_PYTHON}")
|
|
77
|
+
super().check_expression_support(expression)
|
|
78
|
+
|
|
79
|
+
def _no_tz_conversion(self, tzname):
|
|
80
|
+
if tzname and settings.USE_TZ and tzname != self.connection.timezone_name:
|
|
81
|
+
raise NotSupportedError(
|
|
82
|
+
f"time-zone conversion in a query ({self.connection.timezone_name} -> {tzname}): {_NO_PYTHON}. "
|
|
83
|
+
f"Use TIME_ZONE = '{self.connection.timezone_name}', or timezone.override() around the query")
|
|
84
|
+
|
|
85
|
+
def _extract(self, lookup_type, sql, params):
|
|
86
|
+
lookup_type = lookup_type.lower()
|
|
87
|
+
if lookup_type == "week_day": # Sunday = 1 ... Saturday = 7
|
|
88
|
+
return f"(CAST(strftime(%s, {sql}) AS INTEGER) + 1)", ("%w", *params)
|
|
89
|
+
if lookup_type == "quarter":
|
|
90
|
+
return f"((CAST(strftime(%s, {sql}) AS INTEGER) + 2) / 3)", ("%m", *params)
|
|
91
|
+
if lookup_type not in _EXTRACT:
|
|
92
|
+
raise ValueError(f"Unsupported lookup type: {lookup_type!r}")
|
|
93
|
+
return f"CAST(strftime(%s, {sql}) AS INTEGER)", (_EXTRACT[lookup_type], *params)
|
|
94
|
+
|
|
95
|
+
def _trunc(self, formats, suffix, lookup_type, sql, params):
|
|
96
|
+
lookup_type = lookup_type.lower()
|
|
97
|
+
if lookup_type == "quarter": # the first month of the quarter
|
|
98
|
+
return (f"(strftime(%s, {sql}) || printf(%s, (CAST(strftime(%s, {sql}) AS INTEGER) - 1) / 3 * 3 + 1)"
|
|
99
|
+
f" || %s)", ("%Y-", *params, "%02d", "%m", *params, "-01" + suffix))
|
|
100
|
+
if lookup_type not in formats:
|
|
101
|
+
raise ValueError(f"Unsupported lookup type: {lookup_type!r}")
|
|
102
|
+
if lookup_type == "week": # back to Monday: forward to Sunday, then six days back
|
|
103
|
+
return f"strftime(%s, {sql}, 'weekday 0', '-6 days')", (formats["week"], *params)
|
|
104
|
+
return f"strftime(%s, {sql})", (formats[lookup_type], *params)
|
|
105
|
+
|
|
106
|
+
def date_extract_sql(self, lookup_type, sql, params):
|
|
107
|
+
return self._extract(lookup_type, sql, params)
|
|
108
|
+
|
|
109
|
+
def datetime_extract_sql(self, lookup_type, sql, params, tzname):
|
|
110
|
+
self._no_tz_conversion(tzname)
|
|
111
|
+
return self._extract(lookup_type, sql, params)
|
|
112
|
+
|
|
113
|
+
def time_extract_sql(self, lookup_type, sql, params):
|
|
114
|
+
return self._extract(lookup_type, sql, params)
|
|
115
|
+
|
|
116
|
+
def date_trunc_sql(self, lookup_type, sql, params, tzname=None):
|
|
117
|
+
self._no_tz_conversion(tzname)
|
|
118
|
+
return self._trunc(_TRUNC_DATE, "", lookup_type, sql, params)
|
|
119
|
+
|
|
120
|
+
def datetime_trunc_sql(self, lookup_type, sql, params, tzname):
|
|
121
|
+
self._no_tz_conversion(tzname)
|
|
122
|
+
return self._trunc(_TRUNC_DATETIME, " 00:00:00", lookup_type, sql, params)
|
|
123
|
+
|
|
124
|
+
def time_trunc_sql(self, lookup_type, sql, params, tzname=None):
|
|
125
|
+
self._no_tz_conversion(tzname)
|
|
126
|
+
return self._trunc(_TRUNC_TIME, "", lookup_type, sql, params)
|
|
127
|
+
|
|
128
|
+
def datetime_cast_date_sql(self, sql, params, tzname):
|
|
129
|
+
self._no_tz_conversion(tzname)
|
|
130
|
+
return f"date({sql})", params
|
|
131
|
+
|
|
132
|
+
def datetime_cast_time_sql(self, sql, params, tzname):
|
|
133
|
+
self._no_tz_conversion(tzname)
|
|
134
|
+
# time() drops the microseconds; Django's value keeps them when there are any (HH:MM:SS[.ffffff])
|
|
135
|
+
return (f"(time({sql}) || CASE WHEN substr({sql}, 20, 1) = '.' THEN substr({sql}, 20, 7) ELSE '' END)",
|
|
136
|
+
(*params, *params, *params))
|
|
137
|
+
|
|
138
|
+
def combine_expression(self, connector, sub_expressions):
|
|
139
|
+
if connector == "#":
|
|
140
|
+
raise NotSupportedError(f"bitwise XOR (BITXOR): {_NO_PYTHON}")
|
|
141
|
+
return super().combine_expression(connector, sub_expressions) # ^ is POWER(), built into the engine
|
|
142
|
+
|
|
143
|
+
def combine_duration_expression(self, connector, sub_expressions):
|
|
144
|
+
raise NotSupportedError(f"date / time arithmetic with a timedelta: {_NO_PYTHON}")
|
|
145
|
+
|
|
146
|
+
def subtract_temporals(self, internal_type, lhs, rhs):
|
|
147
|
+
raise NotSupportedError(f"subtracting dates / times: {_NO_PYTHON}")
|
|
148
|
+
|
|
149
|
+
def regex_lookup(self, lookup_type):
|
|
150
|
+
raise NotSupportedError(f"__{lookup_type} lookups (REGEXP): {_NO_PYTHON}")
|
|
151
|
+
|
|
152
|
+
def _quote_params_for_last_executed_query(self, params):
|
|
153
|
+
"""connection.queries with DEBUG: quote here as SQLite's QUOTE() would, instead of one more request per query."""
|
|
154
|
+
return tuple(_quote(p) for p in params)
|
|
155
|
+
|
|
156
|
+
|
|
157
|
+
def _quote(value):
|
|
158
|
+
if value is None:
|
|
159
|
+
return "NULL"
|
|
160
|
+
if isinstance(value, bool):
|
|
161
|
+
return str(int(value))
|
|
162
|
+
if isinstance(value, (int, float)):
|
|
163
|
+
return repr(value)
|
|
164
|
+
if isinstance(value, (bytes, bytearray, memoryview)):
|
|
165
|
+
return f"X'{bytes(value).hex().upper()}'"
|
|
166
|
+
return "'" + str(value).replace("'", "''") + "'"
|
|
167
|
+
|
|
168
|
+
|
|
169
|
+
class DatabaseCreation(creation.DatabaseCreation):
|
|
170
|
+
def _create_test_db(self, verbosity, autoclobber, keepdb=False):
|
|
171
|
+
raise NotSupportedError("test databases on s3rds: run the tests on django.db.backends.sqlite3 "
|
|
172
|
+
"(the same SQL on a local file)")
|
|
173
|
+
|
|
174
|
+
|
|
175
|
+
class DatabaseClient(client.DatabaseClient):
|
|
176
|
+
@classmethod
|
|
177
|
+
def settings_to_cmd_args_env(cls, settings_dict, parameters):
|
|
178
|
+
raise NotSupportedError("dbshell: open the s3rds console (the Function URL in a browser)")
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
_FORMAT = re.compile(r"(?<!%)%s")
|
|
182
|
+
_PYFORMAT = re.compile(r"%\((\w+)\)s")
|
|
183
|
+
|
|
184
|
+
|
|
185
|
+
def _convert(query, params):
|
|
186
|
+
"""Django's format (%s) / pyformat (%(name)s) SQL -> ? placeholders and a list, %% -> %
|
|
187
|
+
(as Django's SQLiteCursorWrapper, which turns pyformat into :name for sqlite3)."""
|
|
188
|
+
if isinstance(params, Mapping):
|
|
189
|
+
names = _PYFORMAT.findall(query)
|
|
190
|
+
return _PYFORMAT.sub("?", query).replace("%%", "%"), [params[n] for n in names]
|
|
191
|
+
return _FORMAT.sub("?", query).replace("%%", "%"), params
|
|
192
|
+
|
|
193
|
+
|
|
194
|
+
class CursorWrapper(Database.Cursor):
|
|
195
|
+
"""The cursor Django gets; execute() returns the cursor, as sqlite3's does (Django chains .fetchone())."""
|
|
196
|
+
|
|
197
|
+
def execute(self, query, params=None):
|
|
198
|
+
if params is not None:
|
|
199
|
+
query, params = _convert(query, params)
|
|
200
|
+
super().execute(query, params)
|
|
201
|
+
return self
|
|
202
|
+
|
|
203
|
+
def executemany(self, query, param_list):
|
|
204
|
+
param_list = list(param_list)
|
|
205
|
+
if param_list and isinstance(param_list[0], Mapping):
|
|
206
|
+
names = _PYFORMAT.findall(query)
|
|
207
|
+
query = _PYFORMAT.sub("?", query).replace("%%", "%")
|
|
208
|
+
param_list = [[p[n] for n in names] for p in param_list]
|
|
209
|
+
else:
|
|
210
|
+
query = _FORMAT.sub("?", query).replace("%%", "%")
|
|
211
|
+
super().executemany(query, param_list)
|
|
212
|
+
return self
|
|
213
|
+
|
|
214
|
+
|
|
215
|
+
class DatabaseWrapper(base.DatabaseWrapper):
|
|
216
|
+
display_name = "SQLite on s3rds"
|
|
217
|
+
Database = Database
|
|
218
|
+
client_class = DatabaseClient
|
|
219
|
+
creation_class = DatabaseCreation
|
|
220
|
+
features_class = DatabaseFeatures
|
|
221
|
+
ops_class = DatabaseOperations
|
|
222
|
+
# regex / iregex: REGEXP is a Python function on sqlite3; without them Django asks ops.regex_lookup(), which refuses
|
|
223
|
+
operators = {k: v for k, v in base.DatabaseWrapper.operators.items() if k not in ("regex", "iregex")}
|
|
224
|
+
|
|
225
|
+
def get_connection_params(self):
|
|
226
|
+
kwargs = super().get_connection_params() # NAME required; OPTIONS transaction_mode / init_command read
|
|
227
|
+
for key in ("detect_types", "check_same_thread", "uri"):
|
|
228
|
+
kwargs.pop(key, None)
|
|
229
|
+
s = self.settings_dict
|
|
230
|
+
if not s["HOST"]:
|
|
231
|
+
raise ImproperlyConfigured("settings.DATABASES: HOST is the s3rds Function URL host "
|
|
232
|
+
"(xxx.lambda-url.<region>.on.aws)")
|
|
233
|
+
return {**kwargs, "host": s["HOST"] + (f":{s['PORT']}" if s["PORT"] else ""), "user": s["USER"],
|
|
234
|
+
"password": s["PASSWORD"], "database": s["NAME"], "paramstyle": "qmark"}
|
|
235
|
+
|
|
236
|
+
@async_unsafe
|
|
237
|
+
def get_new_connection(self, conn_params):
|
|
238
|
+
conn = Database.connect(**conn_params)
|
|
239
|
+
# no register_functions(conn): Python functions would have to run on the server (DatabaseOperations above)
|
|
240
|
+
conn.execute("PRAGMA foreign_keys = ON")
|
|
241
|
+
conn.execute("PRAGMA legacy_alter_table = OFF")
|
|
242
|
+
for init_command in self.init_commands:
|
|
243
|
+
if init_command := init_command.strip():
|
|
244
|
+
conn.execute(init_command)
|
|
245
|
+
return conn
|
|
246
|
+
|
|
247
|
+
def create_cursor(self, name=None):
|
|
248
|
+
return self.connection.cursor(CursorWrapper)
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: s3rds
|
|
3
|
-
Version: 0.2.
|
|
4
|
-
Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds
|
|
3
|
+
Version: 0.2.1
|
|
4
|
+
Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite.
|
|
5
5
|
Author: Frank Zhang
|
|
6
6
|
License: MIT
|
|
7
7
|
Project-URL: Homepage, https://github.com/cloudformer/s3rds
|
|
@@ -10,13 +10,14 @@ Project-URL: Deploy, https://github.com/cloudformer/s3rds#quick-start
|
|
|
10
10
|
Project-URL: Compatibility, https://github.com/cloudformer/s3rds/blob/main/tests/compat/REPORT.md
|
|
11
11
|
Project-URL: Changelog, https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md
|
|
12
12
|
Project-URL: Issues, https://github.com/cloudformer/s3rds/issues
|
|
13
|
-
Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy
|
|
13
|
+
Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy,django
|
|
14
14
|
Classifier: Development Status :: 3 - Alpha
|
|
15
15
|
Classifier: Intended Audience :: Developers
|
|
16
16
|
Classifier: License :: OSI Approved :: MIT License
|
|
17
17
|
Classifier: Programming Language :: Python :: 3
|
|
18
18
|
Classifier: Programming Language :: Python :: 3.9
|
|
19
19
|
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Framework :: Django
|
|
20
21
|
Classifier: Topic :: Database
|
|
21
22
|
Classifier: Topic :: Database :: Front-Ends
|
|
22
23
|
Requires-Python: >=3.9
|
|
@@ -25,19 +26,42 @@ Requires-Dist: sqlglot<31,>=30.24
|
|
|
25
26
|
Provides-Extra: mysql
|
|
26
27
|
Provides-Extra: sqlalchemy
|
|
27
28
|
Requires-Dist: SQLAlchemy<2.1,>=2.0; extra == "sqlalchemy"
|
|
29
|
+
Provides-Extra: django
|
|
30
|
+
Requires-Dist: Django<6.2,>=6.1; extra == "django"
|
|
28
31
|
|
|
29
32
|
# s3rds
|
|
30
33
|
|
|
34
|
+

|
|
35
|
+
|
|
36
|
+
### Switch to s3rds — the only change
|
|
37
|
+
|
|
38
|
+
| Your code uses | Before | With s3rds — **the only change** | What really differs |
|
|
39
|
+
|---|---|---|---|
|
|
40
|
+
| **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
|
|
41
|
+
| **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
|
|
42
|
+
| **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
|
|
43
|
+
| **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
|
|
44
|
+
| **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` — *planned, not available yet* | — |
|
|
45
|
+
|
|
46
|
+
Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
|
|
47
|
+
uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
|
|
48
|
+
backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
|
|
49
|
+
`NotSupportedError` — never silently different.
|
|
50
|
+
|
|
51
|
+
### The server: deploy once, ≈ $0 when idle
|
|
52
|
+
|
|
53
|
+
You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
|
|
54
|
+
while nobody uses it. Deploy it once with Terraform or CloudFormation from the
|
|
55
|
+
[main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
|
|
56
|
+
|
|
31
57
|
**Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
|
|
32
58
|
a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
|
|
33
59
|
|
|
34
|
-

|
|
35
|
-
|
|
36
60
|
One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
|
|
37
61
|
client it mirrors**, so moving to a real database is a one-line import change and nothing else:
|
|
38
62
|
|
|
39
63
|
```bash
|
|
40
|
-
pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
|
|
64
|
+
pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
|
|
41
65
|
```
|
|
42
66
|
|
|
43
67
|
```python
|
|
@@ -68,6 +92,7 @@ conn.transaction([ # all-or-nothing, single S3 w
|
|
|
68
92
|
| `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
|
|
69
93
|
| `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
|
|
70
94
|
| `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
|
|
95
|
+
| `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
|
|
71
96
|
|
|
72
97
|
**SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
|
|
73
98
|
`s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
|
|
@@ -76,6 +101,21 @@ conn.transaction([ # all-or-nothing, single S3 w
|
|
|
76
101
|
engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
|
|
77
102
|
```
|
|
78
103
|
|
|
104
|
+
**Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
DATABASES = {"default": {
|
|
108
|
+
"ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
|
|
109
|
+
"NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
|
|
110
|
+
}}
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
|
|
114
|
+
`Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
|
|
115
|
+
conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
|
|
116
|
+
`Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
|
|
117
|
+
`django.db.backends.sqlite3`.
|
|
118
|
+
|
|
79
119
|
## Semantics
|
|
80
120
|
|
|
81
121
|
- **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
|
|
@@ -83,7 +123,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
|
|
|
83
123
|
ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
|
|
84
124
|
pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
|
|
85
125
|
statements as one commit and one S3 write.
|
|
86
|
-
- `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses)
|
|
126
|
+
- `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
|
|
127
|
+
`atomic()`) give sqlite3's transactions: reads see
|
|
87
128
|
the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
|
|
88
129
|
writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
|
|
89
130
|
Keep transactions short — each statement replays the writes before it on the server.
|
|
@@ -105,9 +146,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
|
|
|
105
146
|
| SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
|
|
106
147
|
| sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
|
|
107
148
|
| SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
|
|
149
|
+
| Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
|
|
108
150
|
| MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
|
|
109
151
|
|
|
110
|
-
## Server
|
|
111
|
-
|
|
112
|
-
You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
|
|
113
|
-
from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
|
|
@@ -3,10 +3,11 @@ pyproject.toml
|
|
|
3
3
|
s3rds/__init__.py
|
|
4
4
|
s3rds/psycopg2.py
|
|
5
5
|
s3rds/pymysql.py
|
|
6
|
-
s3rds/sqlite.py
|
|
7
6
|
s3rds.egg-info/PKG-INFO
|
|
8
7
|
s3rds.egg-info/SOURCES.txt
|
|
9
8
|
s3rds.egg-info/dependency_links.txt
|
|
10
9
|
s3rds.egg-info/entry_points.txt
|
|
11
10
|
s3rds.egg-info/requires.txt
|
|
12
|
-
s3rds.egg-info/top_level.txt
|
|
11
|
+
s3rds.egg-info/top_level.txt
|
|
12
|
+
s3rds/sqlite/__init__.py
|
|
13
|
+
s3rds/sqlite/base.py
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|