s3rds 0.2.0__tar.gz → 0.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,7 +1,7 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: s3rds
3
- Version: 0.2.0
4
- Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://.
3
+ Version: 0.3.0
4
+ Summary: A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql, s3rds.psycopg2; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite.
5
5
  Author: Frank Zhang
6
6
  License: MIT
7
7
  Project-URL: Homepage, https://github.com/cloudformer/s3rds
@@ -10,13 +10,14 @@ Project-URL: Deploy, https://github.com/cloudformer/s3rds#quick-start
10
10
  Project-URL: Compatibility, https://github.com/cloudformer/s3rds/blob/main/tests/compat/REPORT.md
11
11
  Project-URL: Changelog, https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md
12
12
  Project-URL: Issues, https://github.com/cloudformer/s3rds/issues
13
- Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,dbapi,sqlalchemy
13
+ Keywords: serverless,rds,sqlite,s3,lambda,database,pymysql,psycopg2,postgresql,dbapi,sqlalchemy,django
14
14
  Classifier: Development Status :: 3 - Alpha
15
15
  Classifier: Intended Audience :: Developers
16
16
  Classifier: License :: OSI Approved :: MIT License
17
17
  Classifier: Programming Language :: Python :: 3
18
18
  Classifier: Programming Language :: Python :: 3.9
19
19
  Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Framework :: Django
20
21
  Classifier: Topic :: Database
21
22
  Classifier: Topic :: Database :: Front-Ends
22
23
  Requires-Python: >=3.9
@@ -25,19 +26,42 @@ Requires-Dist: sqlglot<31,>=30.24
25
26
  Provides-Extra: mysql
26
27
  Provides-Extra: sqlalchemy
27
28
  Requires-Dist: SQLAlchemy<2.1,>=2.0; extra == "sqlalchemy"
29
+ Provides-Extra: django
30
+ Requires-Dist: Django<6.2,>=6.1; extra == "django"
28
31
 
29
32
  # s3rds
30
33
 
34
+ ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
35
+
36
+ ### Switch to s3rds — the only change
37
+
38
+ | Your code uses | Before | With s3rds — **the only change** | What really differs |
39
+ |---|---|---|---|
40
+ | **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
41
+ | **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
42
+ | **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
43
+ | **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
44
+ | **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` | PostgreSQL is translated to SQLite on the client; `.extras` / `.errors` as psycopg2's; no locks, functions, sequences, arrays; BOOLEAN comes back as 1 / 0 |
45
+
46
+ Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
47
+ uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
48
+ backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
49
+ `NotSupportedError` — never silently different.
50
+
51
+ ### The server: deploy once, ≈ $0 when idle
52
+
53
+ You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
54
+ while nobody uses it. Deploy it once with Terraform or CloudFormation from the
55
+ [main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
56
+
31
57
  **Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
32
58
  a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
33
59
 
34
- ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
35
-
36
60
  One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
37
61
  client it mirrors**, so moving to a real database is a one-line import change and nothing else:
38
62
 
39
63
  ```bash
40
- pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
64
+ pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
41
65
  ```
42
66
 
43
67
  ```python
@@ -66,8 +90,9 @@ conn.transaction([ # all-or-nothing, single S3 w
66
90
  |---|---|---|---|
67
91
  | `s3rds.sqlite` | SQLite (no translation) | pymysql-shaped API | available |
68
92
  | `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
69
- | `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
93
+ | `s3rds.psycopg2` | PostgreSQL, translated on the client | psycopg2 (`extras`, `errors`, `extensions`) | available |
70
94
  | `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
95
+ | `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
71
96
 
72
97
  **SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
73
98
  `s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
@@ -76,6 +101,21 @@ conn.transaction([ # all-or-nothing, single S3 w
76
101
  engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
77
102
  ```
78
103
 
104
+ **Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
105
+
106
+ ```python
107
+ DATABASES = {"default": {
108
+ "ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
109
+ "NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
110
+ }}
111
+ ```
112
+
113
+ Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
114
+ `Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
115
+ conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
116
+ `Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
117
+ `django.db.backends.sqlite3`.
118
+
79
119
  ## Semantics
80
120
 
81
121
  - **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
@@ -83,7 +123,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
83
123
  ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
84
124
  pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
85
125
  statements as one commit and one S3 write.
86
- - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) has sqlite3's transactions: reads see
126
+ - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
127
+ `atomic()`) give sqlite3's transactions: reads see
87
128
  the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
88
129
  writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
89
130
  Keep transactions short — each statement replays the writes before it on the server.
@@ -105,9 +146,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
105
146
  | SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
106
147
  | sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
107
148
  | SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
149
+ | Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
108
150
  | MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
109
151
 
110
- ## Server
111
-
112
- You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
113
- from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
@@ -1,15 +1,36 @@
1
1
  # s3rds
2
2
 
3
+ ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
4
+
5
+ ### Switch to s3rds — the only change
6
+
7
+ | Your code uses | Before | With s3rds — **the only change** | What really differs |
8
+ |---|---|---|---|
9
+ | **sqlite3** | `import sqlite3`<br>`sqlite3.connect("app.db")` | `from s3rds import sqlite`<br>`sqlite.connect(host=URL, user=…, password=…, database="app")` | the file lives on S3 behind HTTPS; `create_function` is refused |
10
+ | **pymysql** | `import pymysql` | `from s3rds import pymysql` | MySQL is translated to SQLite on the client; no locks, procedures, triggers |
11
+ | **SQLAlchemy** | `create_engine("sqlite:///app.db")` | `create_engine("sqlite+s3rds://user:pw@URL/app")` | Core and ORM unchanged; `regexp_match` is refused |
12
+ | **Django** | `"ENGINE": "django.db.backends.sqlite3"` | `"ENGINE": "s3rds.sqlite"` + `HOST` / `USER` / `PASSWORD` | `__regex`, time-zone conversion in queries, `order_by("?")` are refused; run tests on sqlite3 |
13
+ | **psycopg2** | `import psycopg2` | `from s3rds import psycopg2` | PostgreSQL is translated to SQLite on the client; `.extras` / `.errors` as psycopg2's; no locks, functions, sequences, arrays; BOOLEAN comes back as 1 / 0 |
14
+
15
+ Everything else stays: models, queries, cursors, transactions. For all of them: one writer at a time, every commit
16
+ uploads the whole file (a few hundred ms), so keep a database under ~20 MB. Moving to RDS later is the same change
17
+ backwards (`import pymysql`, `mysql+pymysql://`, `django.db.backends.mysql`). Refused features raise
18
+ `NotSupportedError` — never silently different.
19
+
20
+ ### The server: deploy once, ≈ $0 when idle
21
+
22
+ You need an s3rds server in your own AWS account — a Lambda Function URL and an S3 bucket, nothing running
23
+ while nobody uses it. Deploy it once with Terraform or CloudFormation from the
24
+ [main repository](https://github.com/cloudformer/s3rds#quick-start). Same as any driver needs its database.
25
+
3
26
  **Python drivers for [s3rds](https://github.com/cloudformer/s3rds) — a truly serverless RDS:
4
27
  a Lambda URL as your database endpoint, SQLite on S3, ≈ $0 when idle.**
5
28
 
6
- ![s3rds architecture](https://raw.githubusercontent.com/cloudformer/s3rds/main/docs/architecture.svg)
7
-
8
29
  One install, one dependency (sqlglot, used only by `s3rds.pymysql`). Each driver is a submodule **named after the standard
9
30
  client it mirrors**, so moving to a real database is a one-line import change and nothing else:
10
31
 
11
32
  ```bash
12
- pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy
33
+ pip install s3rds # s3rds.sqlite, s3rds.pymysql, sqlite+s3rds:// for SQLAlchemy, ENGINE s3rds.sqlite for Django
13
34
  ```
14
35
 
15
36
  ```python
@@ -38,8 +59,9 @@ conn.transaction([ # all-or-nothing, single S3 w
38
59
  |---|---|---|---|
39
60
  | `s3rds.sqlite` | SQLite (no translation) | pymysql-shaped API | available |
40
61
  | `s3rds.pymysql` | MySQL, translated on the client | pymysql | available |
41
- | `s3rds.psycopg2` | PostgreSQL | psycopg2 | placeholder; `connect()` raises until implemented |
62
+ | `s3rds.psycopg2` | PostgreSQL, translated on the client | psycopg2 (`extras`, `errors`, `extensions`) | available |
42
63
  | `sqlite+s3rds://` | SQLite, written by SQLAlchemy | SQLAlchemy Core / ORM | available — the URL is the only change |
64
+ | `ENGINE: "s3rds.sqlite"` | SQLite, written by Django | Django's SQLite backend | available — the database settings are the only change |
43
65
 
44
66
  **SQLAlchemy** (Core and ORM): only the URL changes — SQLAlchemy's own SQLite dialect writes the SQL,
45
67
  `s3rds.sqlite` replaces the standard library's `sqlite3` underneath.
@@ -48,6 +70,21 @@ conn.transaction([ # all-or-nothing, single S3 w
48
70
  engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aws/blog")
49
71
  ```
50
72
 
73
+ **Django**: Django's own SQLite backend writes the SQL; `atomic()`, savepoints and `migrate` work.
74
+
75
+ ```python
76
+ DATABASES = {"default": {
77
+ "ENGINE": "s3rds.sqlite", # was "django.db.backends.sqlite3"
78
+ "NAME": "blog", "HOST": "xxxx.lambda-url.us-east-2.on.aws", "USER": "admin", "PASSWORD": "...",
79
+ }}
80
+ ```
81
+
82
+ Django's SQLite backend registers Python functions on the connection; its date functions (`__year`, `__date`,
83
+ `Trunc`, `Extract`) are written in SQLite's `strftime()` instead, the rest raise `NotSupportedError`: time-zone
84
+ conversion in queries (keep `TIME_ZONE` equal to the connection's), `__regex`, `MD5` / `SHA*`, `LPad` / `Repeat` /
85
+ `Reverse`, `order_by("?")`, `StdDev` / `Variance`, date arithmetic in the query. Run `manage.py test` on
86
+ `django.db.backends.sqlite3`.
87
+
51
88
  ## Semantics
52
89
 
53
90
  - **Transactions**: by default each `execute()` is its own transaction and is durable in S3 when it
@@ -55,7 +92,8 @@ engine = create_engine("sqlite+s3rds://admin:...@xxxx.lambda-url.us-east-2.on.aw
55
92
  ran** — it warns if anything was written since the last `commit()` (and stays silent otherwise, so
56
93
  pools may call it freely). For all-or-nothing writes use `transaction([...])`, which runs several
57
94
  statements as one commit and one S3 write.
58
- - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) has sqlite3's transactions: reads see
95
+ - `s3rds.sqlite.connect(..., autocommit=False)` (what SQLAlchemy uses) and a `BEGIN` statement (Django's
96
+ `atomic()`) give sqlite3's transactions: reads see
59
97
  the transaction's own uncommitted writes, `commit()` writes S3 once, `rollback()` discards. If another
60
98
  writer commits first, the transaction raises `OperationalError` instead of overwriting: run it again.
61
99
  Keep transactions short — each statement replays the writes before it on the server.
@@ -77,9 +115,6 @@ What the report was verified with; the full report: [REPORT.md](https://github.c
77
115
  | SQLite (server engine) | 3.50.4, shipped in the Lambda zip (the runtime's own is 3.40.0) | `src/db/requirements.txt`: `sqlean.py==3.50.4.5` |
78
116
  | sqlglot (MySQL translation) | 30.24 | dependency `sqlglot>=30.24,<31` |
79
117
  | SQLAlchemy | 2.0, tested on 2.0.55 | extra `pip install "s3rds[sqlalchemy]"` = `SQLAlchemy>=2.0,<2.1` |
118
+ | Django | 6.1, tested on 6.1.2 (Python ≥ 3.12) | extra `pip install "s3rds[django]"` = `Django>=6.1,<6.2` |
80
119
  | MySQL (reference) | 8.0.46 in Docker | `tests/compat/docker-compose.yml`: `mysql:8.0.46` |
81
120
 
82
- ## Server
83
-
84
- You need an s3rds server in your own AWS account — deploy it once with Terraform or CloudFormation
85
- from the [main repository](https://github.com/cloudformer/s3rds). Same as any driver needs its database.
@@ -4,13 +4,13 @@ build-backend = "setuptools.build_meta"
4
4
 
5
5
  [project]
6
6
  name = "s3rds"
7
- version = "0.2.0"
8
- description = "A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql; SQLAlchemy via sqlite+s3rds://."
7
+ version = "0.3.0"
8
+ description = "A truly serverless RDS on your own AWS account: Lambda URL endpoint, SQLite on S3, ≈ $0 when idle. Drivers named after the standard clients: s3rds.sqlite, s3rds.pymysql, s3rds.psycopg2; SQLAlchemy via sqlite+s3rds://, Django via ENGINE s3rds.sqlite."
9
9
  readme = "README.md"
10
10
  requires-python = ">=3.9"
11
11
  license = { text = "MIT" }
12
12
  authors = [{ name = "Frank Zhang" }]
13
- keywords = ["serverless", "rds", "sqlite", "s3", "lambda", "database", "pymysql", "dbapi", "sqlalchemy"]
13
+ keywords = ["serverless", "rds", "sqlite", "s3", "lambda", "database", "pymysql", "psycopg2", "postgresql", "dbapi", "sqlalchemy", "django"]
14
14
  classifiers = [
15
15
  "Development Status :: 3 - Alpha",
16
16
  "Intended Audience :: Developers",
@@ -18,10 +18,11 @@ classifiers = [
18
18
  "Programming Language :: Python :: 3",
19
19
  "Programming Language :: Python :: 3.9",
20
20
  "Programming Language :: Python :: 3.13",
21
+ "Framework :: Django",
21
22
  "Topic :: Database",
22
23
  "Topic :: Database :: Front-Ends",
23
24
  ]
24
- # one install for every driver. sqlglot: s3rds.pymysql's MySQL -> SQLite translation, imported only when a MySQL
25
+ # one install for every driver. sqlglot: s3rds.pymysql's / s3rds.psycopg2's MySQL / PostgreSQL -> SQLite translation, imported only when a MySQL
25
26
  # statement is translated; pinned to the release the compatibility report was verified with (translations
26
27
  # change between releases). SQLAlchemy is not a dependency: the applications that use sqlite+s3rds:// have it.
27
28
  dependencies = ["sqlglot>=30.24,<31"]
@@ -29,6 +30,7 @@ dependencies = ["sqlglot>=30.24,<31"]
29
30
  [project.optional-dependencies]
30
31
  mysql = [] # kept so that pip install "s3rds[mysql]" from 0.1.x keeps working
31
32
  sqlalchemy = ["SQLAlchemy>=2.0,<2.1"] # the series the compatibility report verified; pip install "s3rds[sqlalchemy]"
33
+ django = ["Django>=6.1,<6.2"] # the same for Django (ENGINE "s3rds.sqlite"); Django 6.1 needs Python >= 3.12
32
34
 
33
35
  [project.entry-points."sqlalchemy.dialects"]
34
36
  "sqlite.s3rds" = "s3rds.sqlite:SQLAlchemyDialect"
@@ -42,7 +44,7 @@ Changelog = "https://github.com/cloudformer/s3rds/blob/main/CHANGELOG.md"
42
44
  Issues = "https://github.com/cloudformer/s3rds/issues"
43
45
 
44
46
  [tool.setuptools]
45
- packages = ["s3rds"]
47
+ packages = ["s3rds", "s3rds.sqlite", "s3rds.psycopg2"]
46
48
 
47
49
  # The versions tests/compat/REPORT.md was verified with; README.md, README.zh.md and client/README.md state the same.
48
50
  [tool.s3rds.verified]
@@ -52,4 +54,7 @@ sqlite = "3.50.4" # the server's engine, shipped in the Lambda
52
54
  sqlean = "3.50.4.5" # sqlean.py, which carries it (src/db/requirements.txt)
53
55
  sqlglot = "30.24.0"
54
56
  sqlalchemy = "2.0.55"
57
+ django = "6.1.2"
55
58
  mysql_reference = "8.0.46" # tests/compat/docker-compose.yml
59
+ postgres_reference = "16.4" # the same
60
+ psycopg2 = "2.9.12" # the reference client (psycopg2-binary)
@@ -7,14 +7,16 @@ real database you change one import line and nothing else:
7
7
  import pymysql # later: a real RDS MySQL -- pymysql.connect(...), pymysql.cursors, errors all unchanged
8
8
 
9
9
  from s3rds import sqlite # SQLite dialect, no translation, standard library only
10
- from s3rds import psycopg2 # PostgreSQL dialect: placeholder, connect() raises until implemented
10
+ from s3rds import psycopg2 # PostgreSQL dialect; s3rds.psycopg2.extras / .errors / .extensions as psycopg2's
11
11
 
12
12
  SQLAlchemy (Core / ORM) needs no import: create_engine("sqlite+s3rds://user:password@<lambda-url>/<db>")
13
13
  uses SQLAlchemy's SQLite dialect with s3rds.sqlite as its driver; on RDS the URL becomes mysql+pymysql://.
14
+ Django needs no import either: DATABASES ENGINE "s3rds.sqlite" (s3rds/sqlite/base.py) is Django's SQLite
15
+ backend on s3rds.sqlite; on RDS the ENGINE becomes django.db.backends.mysql.
14
16
  """
15
17
  from . import sqlite # noqa: F401
16
18
  from . import pymysql # noqa: F401 (sqlglot is only needed when a MySQL statement is actually translated)
17
- from . import psycopg2 # noqa: F401 (placeholder: connect() raises NotSupportedError until implemented)
19
+ from . import psycopg2 # noqa: F401
18
20
 
19
21
  __all__ = ["sqlite", "pymysql", "psycopg2"]
20
22
  __version__ = sqlite.__version__
@@ -0,0 +1,276 @@
1
+ """s3rds.psycopg2 — the PostgreSQL-dialect driver for s3rds. Write PostgreSQL; it runs on the s3rds endpoint (SQLite engine).
2
+
3
+ from s3rds import psycopg2 # later: `import psycopg2` on a real RDS, nothing else changes
4
+ from s3rds.psycopg2.extras import RealDictCursor
5
+ conn = psycopg2.connect(host="xxx.lambda-url.us-east-2.on.aws", dbname="blog", user="admin", password="...")
6
+ with conn, conn.cursor(cursor_factory=RealDictCursor) as cur: # `with conn`: commit, or rollback on error
7
+ cur.execute("CREATE TABLE users (id SERIAL PRIMARY KEY, name VARCHAR(50) NOT NULL)")
8
+ cur.execute("INSERT INTO users (name) VALUES (%s) RETURNING id", ["alice"])
9
+ cur.execute("SELECT * FROM users WHERE name = %(n)s", {"n": "alice"})
10
+
11
+ Translated by sqlglot (client side, cached per statement), plus: SERIAL / IDENTITY PRIMARY KEY -> INTEGER PRIMARY KEY
12
+ AUTOINCREMENT, declared column types kept (DATE / TIMESTAMP / NUMERIC come back as date / datetime / Decimal),
13
+ TRUNCATE, START TRANSACTION. Errors are psycopg2's classes with the SQLSTATE in pgcode (s3rds.psycopg2.errors).
14
+ Transactions as psycopg2: autocommit=False by default, commit() / rollback(); a transaction opens at the first write.
15
+
16
+ Not supported, raised as NotSupportedError: functions / procedures / triggers, LISTEN / NOTIFY, COPY, row and table
17
+ locks, roles / GRANT, sequences, arrays, extensions, temporary tables. Known gaps: tests/compat/TASKS.md.
18
+ """
19
+ import functools
20
+ import re
21
+
22
+ from .. import sqlite as _sqlite
23
+ from ..sqlite import Cursor
24
+
25
+ try:
26
+ import sqlglot
27
+ from sqlglot import exp
28
+ from sqlglot.errors import ParseError
29
+ except ImportError: # the error surfaces on first query, with install instructions
30
+ sqlglot = None
31
+
32
+ __version__ = _sqlite.__version__
33
+ apilevel = "2.0"
34
+ threadsafety = 1
35
+ paramstyle = "pyformat"
36
+
37
+
38
+ # ---------------- exceptions: psycopg2's classes, pgcode = SQLSTATE ----------------
39
+
40
+ class Warning(Exception): # noqa: A001 (DB-API name)
41
+ pass
42
+
43
+
44
+ class Error(Exception):
45
+ sqlstate = None
46
+
47
+ def __init__(self, message=""):
48
+ super().__init__(message)
49
+ self.pgcode = self.sqlstate
50
+ self.pgerror = f"ERROR: {message}\n"
51
+
52
+
53
+ class InterfaceError(Error): pass
54
+ class DatabaseError(Error): pass
55
+ class DataError(DatabaseError): pass
56
+ class OperationalError(DatabaseError): pass
57
+ class IntegrityError(DatabaseError): pass
58
+ class InternalError(DatabaseError): pass
59
+ class ProgrammingError(DatabaseError): pass
60
+ class NotSupportedError(DatabaseError): pass
61
+
62
+
63
+ from . import errors, extensions, extras # noqa: E402,F401 (errors needs the classes above)
64
+
65
+ # server message -> the psycopg2.errors class PostgreSQL would have raised (its sqlstate is the pgcode)
66
+ _SQLITE_TO_PG = [
67
+ (r"UNIQUE constraint failed", errors.UniqueViolation),
68
+ (r"NOT NULL constraint failed", errors.NotNullViolation),
69
+ (r"FOREIGN KEY constraint failed", errors.ForeignKeyViolation),
70
+ (r"CHECK constraint failed", errors.CheckViolation),
71
+ (r"no such table", errors.UndefinedTable),
72
+ (r"no such column", errors.UndefinedColumn),
73
+ (r"already exists", errors.DuplicateTable),
74
+ (r"no such function", errors.UndefinedFunction),
75
+ (r"syntax error|incomplete input", errors.SyntaxError),
76
+ (r"readonly|read-only", errors.ReadOnlySqlTransaction),
77
+ ]
78
+
79
+
80
+ def _map_error(e):
81
+ """s3rds.sqlite.Error (HTTP status + server message) -> psycopg2-style exception."""
82
+ msg = e.message or str(e)
83
+ if e.status == 0:
84
+ return OperationalError(f"server closed the connection: {msg}")
85
+ if e.status == 401:
86
+ return OperationalError("password authentication failed")
87
+ if e.status == 404 and "database" in msg:
88
+ return errors.InvalidCatalogName(msg)
89
+ if e.status == 409:
90
+ return errors.SerializationFailure(f"could not serialize access due to concurrent update ({msg})")
91
+ for pattern, cls in _SQLITE_TO_PG:
92
+ if re.search(pattern, msg, re.I):
93
+ return cls(msg)
94
+ return InternalError(msg)
95
+
96
+
97
+ # ---------------- placeholders: %s / %(name)s -> ? (psycopg2 formats the whole string, quotes included) ----------------
98
+
99
+ def _bind(sql, params):
100
+ if not params:
101
+ return sql, []
102
+ names = []
103
+
104
+ def sub(m):
105
+ if m.group(0) == "%%":
106
+ return "%"
107
+ names.append(m.group(1))
108
+ return "?"
109
+ out = re.sub(r"%%|%\((\w+)\)s|%s", sub, sql)
110
+ if isinstance(params, dict):
111
+ return out, [params[k] for k in names]
112
+ params = list(params)
113
+ if len(params) != len(names):
114
+ raise TypeError("not all arguments converted during string formatting" if len(params) > len(names)
115
+ else "not enough arguments for format string")
116
+ return out, params
117
+
118
+
119
+ # ---------------- translation: sqlglot postgres -> sqlite, plus what it does not do ----------------
120
+
121
+ _UNSUPPORTED = [
122
+ (r"^\s*CREATE\s+(OR\s+REPLACE\s+)?(FUNCTION|PROCEDURE|TRIGGER|RULE)\b|^\s*(DO|CALL)\b", "functions / procedures / triggers"),
123
+ (r"^\s*(LISTEN|NOTIFY|UNLISTEN)\b", "LISTEN / NOTIFY"),
124
+ (r"^\s*COPY\b", "COPY"),
125
+ (r"^\s*LOCK\b|\bFOR\s+(NO\s+KEY\s+)?(UPDATE|SHARE)\b", "row and table locks"),
126
+ (r"^\s*(GRANT|REVOKE)\b|^\s*(CREATE|ALTER|DROP)\s+(ROLE|USER|GROUP)\b", "roles / GRANT / REVOKE"),
127
+ (r"^\s*(CREATE|ALTER|DROP)\s+(SEQUENCE|EXTENSION|SCHEMA|TYPE|DOMAIN|DATABASE)\b|\b(nextval|currval|setval)\s*\(",
128
+ "sequences / extensions / schemas / types"),
129
+ (r"^\s*CREATE\s+(TEMP|TEMPORARY|UNLOGGED)\b", "temporary tables"),
130
+ (r"\bARRAY\s*\[|\w\s*\[\]", "arrays"),
131
+ ]
132
+ _STATEMENTS = ("Select", "Union", "Intersect", "Except", "Insert", "Update", "Delete", "Create", "Drop", "Alter")
133
+ _TX = re.compile(r"^\s*(BEGIN|START\s+TRANSACTION|COMMIT|END|ROLLBACK|ABORT|SAVEPOINT|RELEASE)\b", re.I)
134
+ _TRUNCATE = re.compile(r"^\s*TRUNCATE\s+(?:TABLE\s+)?(?:ONLY\s+)?([\w\s,\"]+?)(\s+RESTART\s+IDENTITY)?"
135
+ r"(?:\s+CONTINUE\s+IDENTITY)?\s*$", re.I)
136
+ _SERIAL = {"SERIAL", "BIGSERIAL", "SMALLSERIAL"}
137
+
138
+
139
+ def _fix(node):
140
+ if isinstance(node, exp.ColumnDef) and node.args.get("kind") is not None:
141
+ kind, constraints = node.args["kind"], node.args.get("constraints") or []
142
+ if kind.this.name in _SERIAL or any(isinstance(c.kind, exp.GeneratedAsIdentityColumnConstraint) for c in constraints):
143
+ if not any(isinstance(c.kind, exp.PrimaryKeyColumnConstraint) for c in constraints):
144
+ raise errors.FeatureNotSupported("SERIAL / IDENTITY on a column that is not its PRIMARY KEY")
145
+ node.set("kind", exp.DataType.build("INTEGER"))
146
+ others = [c for c in constraints if not isinstance(
147
+ c.kind, (exp.PrimaryKeyColumnConstraint, exp.GeneratedAsIdentityColumnConstraint))]
148
+ node.set("constraints", others + [exp.ColumnConstraint(kind=exp.PrimaryKeyColumnConstraint()),
149
+ exp.ColumnConstraint(kind=exp.AutoIncrementColumnConstraint())])
150
+ else: # keep the declared type: the server returns DATE / TIMESTAMP / NUMERIC columns typed by it
151
+ node.set("kind", exp.DataType(this=exp.DataType.Type.USERDEFINED, kind=kind.sql(dialect="postgres")))
152
+ elif isinstance(node, exp.OnConflict): # sqlglot writes ON CONFLICT (a NULLS LAST), which SQLite rejects
153
+ node.set("conflict_keys", [k.this if isinstance(k, exp.Ordered) else k for k in node.args.get("conflict_keys") or []])
154
+ elif isinstance(node, exp.Table) and node.db == "public":
155
+ node.set("db", None)
156
+ return node
157
+
158
+
159
+ @functools.lru_cache(maxsize=1024)
160
+ def _translate(sql):
161
+ """PostgreSQL statement -> tuple of SQLite statements."""
162
+ for pattern, what in _UNSUPPORTED:
163
+ if re.search(pattern, sql, re.I | re.S):
164
+ raise errors.FeatureNotSupported(f"{what} need a real PostgreSQL - move to RDS when you do")
165
+ if _TX.match(sql):
166
+ sql = re.sub(r"^\s*ABORT\b", "ROLLBACK", sql, flags=re.I)
167
+ return (re.sub(r"^\s*(START\s+TRANSACTION|BEGIN)\b.*", "BEGIN", sql, flags=re.I | re.S),)
168
+ m = _TRUNCATE.match(sql)
169
+ if m:
170
+ tables = [t.strip().strip('"') for t in m.group(1).split(",")]
171
+ return tuple([f'DELETE FROM "{t}"' for t in tables]
172
+ + ([f"DELETE FROM sqlite_sequence WHERE name = '{t}'" for t in tables] if m.group(2) else []))
173
+ if sqlglot is None:
174
+ raise NotSupportedError("the PostgreSQL dialect needs sqlglot (an s3rds dependency): pip install s3rds")
175
+ try:
176
+ tree = sqlglot.parse_one(sql, read="postgres")
177
+ except ParseError as e:
178
+ raise errors.SyntaxError(f"syntax error ({e})") from None
179
+ if type(tree).__name__ not in _STATEMENTS:
180
+ raise errors.FeatureNotSupported(f"{type(tree).__name__} statements are not supported by s3rds.psycopg2")
181
+ return (tree.transform(_fix).sql(dialect="sqlite"),)
182
+
183
+
184
+ # ---------------- connection ----------------
185
+
186
+ class Connection(_sqlite.Connection):
187
+ _typed = True # the server marks DATE / TIME / TIMESTAMP / NUMERIC columns by their declared type
188
+ _closed = False
189
+
190
+ @property
191
+ def autocommit(self):
192
+ return self._autocommit
193
+
194
+ @autocommit.setter
195
+ def autocommit(self, value):
196
+ if self._tx is not None:
197
+ raise ProgrammingError("set_session cannot be used inside a transaction")
198
+ self._autocommit = bool(value)
199
+
200
+ @property
201
+ def closed(self):
202
+ return int(self._closed)
203
+
204
+ def cursor(self, name=None, cursor_factory=None, **_ignored):
205
+ if self._closed:
206
+ raise InterfaceError("connection already closed")
207
+ return (cursor_factory or self.cursorclass)(self)
208
+
209
+ def close(self):
210
+ self._closed = True
211
+ super().close()
212
+
213
+ def __exit__(self, exc_type, *exc): # psycopg2: `with conn` ends the transaction, it does not close
214
+ if exc_type is None:
215
+ self.commit()
216
+ else:
217
+ self.rollback()
218
+
219
+ def rollback(self):
220
+ self._written = False # no warning: PostgreSQL cannot undo autocommitted statements either
221
+ super().rollback()
222
+
223
+ def execute(self, sql, params=()):
224
+ if self._closed:
225
+ raise InterfaceError("connection already closed")
226
+ sql = sql.decode() if isinstance(sql, bytes) else sql
227
+ stmts, args = self._prepare(sql, params)
228
+ try:
229
+ if len(stmts) == 1:
230
+ r = super().execute(stmts[0], args)
231
+ elif self._tx is not None or not self._autocommit:
232
+ self._tx = self._tx or {"log": [], "base": None}
233
+ r = self._tx_run([(s, []) for s in stmts])[0]
234
+ else:
235
+ r = self.transaction([(s, []) for s in stmts])[0]
236
+ except _sqlite.Error as e:
237
+ raise _map_error(e) from None
238
+ r["lastrowid"] = 0 # psycopg2: the OID, 0 without OIDs; use INSERT ... RETURNING
239
+ if r.get("columns"):
240
+ r["rowcount"] = len(r.get("rows") or [])
241
+ elif len(stmts) > 1 or _TX.match(sql):
242
+ r["rowcount"] = -1 # TRUNCATE and transaction statements report no row count
243
+ return r
244
+
245
+ def _prepare(self, sql, params):
246
+ qmark, args = _bind(sql.strip().rstrip(";"), params)
247
+ return _translate(qmark), args
248
+
249
+ def _executemany(self, sql, seq_of_params):
250
+ prepared = []
251
+ for params in seq_of_params:
252
+ stmts, args = self._prepare(sql, params)
253
+ prepared += [(stmts[0], args)] + [(s, []) for s in stmts[1:]]
254
+ if not prepared:
255
+ return []
256
+ try:
257
+ if self._tx is None and (self._autocommit or not _sqlite._IMPLICIT_BEGIN.match(prepared[0][0])):
258
+ return self.transaction(prepared)
259
+ self._tx = self._tx or {"log": [], "base": None}
260
+ return self._tx_run(prepared)
261
+ except _sqlite.Error as e:
262
+ raise _map_error(e) from None
263
+
264
+
265
+ def connect(dsn=None, connection_factory=None, cursor_factory=None, **kwargs):
266
+ """psycopg2.connect: a DSN ("host=... dbname=..." or postgresql://user:password@host/dbname) and/or keywords.
267
+ host is the Lambda Function URL; port, sslmode and the other libpq options are accepted and ignored."""
268
+ params = extensions.parse_dsn(dsn) if dsn else {}
269
+ params.update({k: v for k, v in kwargs.items() if v is not None})
270
+ host = params.get("host")
271
+ if not host:
272
+ raise OperationalError("s3rds.psycopg2 needs host=<lambda-url>")
273
+ url = host if host.startswith("http") else "https://" + host
274
+ return Connection(url, params.get("dbname") or params.get("database", ""), params.get("user", ""),
275
+ params.get("password", ""), int(params.get("connect_timeout") or 30),
276
+ cursor_factory or Cursor, bool(params.get("autocommit", False)), "qmark")
@@ -0,0 +1,31 @@
1
+ """s3rds.psycopg2.errors — psycopg2.errors: one class per SQLSTATE, under the DB-API classes (only those s3rds raises)."""
2
+ from . import (DataError, Error, IntegrityError, InternalError, NotSupportedError, OperationalError,
3
+ ProgrammingError)
4
+
5
+
6
+ class TransactionRollbackError(OperationalError): pass
7
+
8
+
9
+ class UniqueViolation(IntegrityError): sqlstate = "23505"
10
+ class NotNullViolation(IntegrityError): sqlstate = "23502"
11
+ class ForeignKeyViolation(IntegrityError): sqlstate = "23503"
12
+ class CheckViolation(IntegrityError): sqlstate = "23514"
13
+ class InvalidTextRepresentation(DataError): sqlstate = "22P02"
14
+ class UndefinedTable(ProgrammingError): sqlstate = "42P01"
15
+ class UndefinedColumn(ProgrammingError): sqlstate = "42703"
16
+ class DuplicateTable(ProgrammingError): sqlstate = "42P07"
17
+ class UndefinedFunction(ProgrammingError): sqlstate = "42883"
18
+ class SyntaxError(ProgrammingError): sqlstate = "42601" # noqa: A001 (psycopg2's name)
19
+ class InvalidCatalogName(ProgrammingError): sqlstate = "3D000"
20
+ class FeatureNotSupported(NotSupportedError): sqlstate = "0A000"
21
+ class ReadOnlySqlTransaction(InternalError): sqlstate = "25006"
22
+ class SerializationFailure(TransactionRollbackError): sqlstate = "40001"
23
+
24
+
25
+ _BY_CODE = {c.sqlstate: c for c in list(globals().values())
26
+ if isinstance(c, type) and issubclass(c, Error) and c.sqlstate}
27
+
28
+
29
+ def lookup(code):
30
+ """The class for a SQLSTATE, as psycopg2.errors.lookup."""
31
+ return _BY_CODE[code]
@@ -0,0 +1,19 @@
1
+ """s3rds.psycopg2.extensions — psycopg2.extensions: isolation level constants and parse_dsn."""
2
+ import re
3
+ import urllib.parse
4
+
5
+ ISOLATION_LEVEL_AUTOCOMMIT = 0
6
+ ISOLATION_LEVEL_READ_COMMITTED = 1
7
+ ISOLATION_LEVEL_REPEATABLE_READ = 2
8
+ ISOLATION_LEVEL_SERIALIZABLE = 3 # what s3rds transactions are: a concurrent commit makes them fail
9
+
10
+
11
+ def parse_dsn(dsn):
12
+ """"host=... dbname=..." or postgresql://user:password@host/dbname -> dict."""
13
+ if re.match(r"postgres(ql)?://", dsn):
14
+ u = urllib.parse.urlparse(dsn)
15
+ out = {"host": u.hostname, "user": urllib.parse.unquote(u.username or ""),
16
+ "password": urllib.parse.unquote(u.password or ""), "dbname": u.path.lstrip("/")}
17
+ out.update(urllib.parse.parse_qsl(u.query))
18
+ return {k: v for k, v in out.items() if v}
19
+ return {k: v.strip("'") for k, v in re.findall(r"(\w+)\s*=\s*('[^']*'|\S+)", dsn)}