sqliteproof 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- sqliteproof-0.1.0/LICENSE +21 -0
- sqliteproof-0.1.0/PKG-INFO +127 -0
- sqliteproof-0.1.0/README.md +105 -0
- sqliteproof-0.1.0/pyproject.toml +34 -0
- sqliteproof-0.1.0/setup.cfg +4 -0
- sqliteproof-0.1.0/sqliteproof/__init__.py +2 -0
- sqliteproof-0.1.0/sqliteproof/__main__.py +3 -0
- sqliteproof-0.1.0/sqliteproof/check.py +266 -0
- sqliteproof-0.1.0/sqliteproof/cli.py +75 -0
- sqliteproof-0.1.0/sqliteproof.egg-info/PKG-INFO +127 -0
- sqliteproof-0.1.0/sqliteproof.egg-info/SOURCES.txt +12 -0
- sqliteproof-0.1.0/sqliteproof.egg-info/dependency_links.txt +1 -0
- sqliteproof-0.1.0/sqliteproof.egg-info/entry_points.txt +2 -0
- sqliteproof-0.1.0/sqliteproof.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Carthorne
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sqliteproof
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Structural validation for SQLite databases -- tells you which tables survived, not which pages broke.
|
|
5
|
+
License: MIT
|
|
6
|
+
Project-URL: Homepage, https://github.com/OrbitalKeyAi/sqliteproof
|
|
7
|
+
Project-URL: Source, https://github.com/OrbitalKeyAi/sqliteproof
|
|
8
|
+
Project-URL: Issues, https://github.com/OrbitalKeyAi/sqliteproof/issues
|
|
9
|
+
Keywords: sqlite,corruption,integrity,database,validation,backup,recovery,cli,devops
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: System Administrators
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Topic :: Database
|
|
14
|
+
Classifier: Topic :: System :: Recovery Tools
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Operating System :: OS Independent
|
|
18
|
+
Requires-Python: >=3.9
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Dynamic: license-file
|
|
22
|
+
|
|
23
|
+
# sqliteproof
|
|
24
|
+
|
|
25
|
+
**Find out which tables survived, not which pages broke.**
|
|
26
|
+
|
|
27
|
+
SQLite's own `PRAGMA integrity_check` is good at *detecting* corruption. What it gives you is this:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
*** in database main ***
|
|
31
|
+
Tree 25 page 35 cell 18: Offset 57005 out of range 239..4092
|
|
32
|
+
database disk image is malformed
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
That's a page number. Nobody stores data by page number. What you actually need to know at
|
|
36
|
+
3am is **which tables can I still trust**, **how many rows did I lose**, and **is this worth
|
|
37
|
+
restoring from backup**.
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
$ sqliteproof app.db
|
|
41
|
+
|
|
42
|
+
app.db — DAMAGED
|
|
43
|
+
|
|
44
|
+
table verdict rows lost
|
|
45
|
+
------------------------------------------------------------
|
|
46
|
+
orders damaged 392/400 8
|
|
47
|
+
breaks after row 182: database disk image is malformed
|
|
48
|
+
audit_log intact 400/400 0
|
|
49
|
+
customers intact 400/400 0
|
|
50
|
+
|
|
51
|
+
1192 rows read, 8 unreadable.
|
|
52
|
+
Tables marked intact above are safe to export. Restore the damaged
|
|
53
|
+
ones from backup rather than trusting a partial read.
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Install
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
pip install sqliteproof
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Python 3.9+. **No dependencies** — standard library only.
|
|
63
|
+
|
|
64
|
+
## Usage
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
sqliteproof app.db # full report
|
|
68
|
+
sqliteproof app.db --json # machine-readable
|
|
69
|
+
sqliteproof app.db --quiet # verdict line only
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Exit codes: **0** intact · **1** damaged · **2** unreadable or undetermined. Drops straight
|
|
73
|
+
into a backup script:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
sqliteproof app.db --quiet || echo "corruption detected" | mail -s alert me@example.com
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## It opens the database read-only
|
|
80
|
+
|
|
81
|
+
A tool asked to inspect a damaged file must never be able to damage it further. The
|
|
82
|
+
connection is opened with `mode=ro` and nothing is written to the database, ever.
|
|
83
|
+
|
|
84
|
+
Runs entirely on your machine. No network, nothing uploaded.
|
|
85
|
+
|
|
86
|
+
## It will not bluff
|
|
87
|
+
|
|
88
|
+
| verdict | meaning |
|
|
89
|
+
|---|---|
|
|
90
|
+
| `INTACT` | every table read completely and row counts match |
|
|
91
|
+
| `DAMAGED` | some rows are unreadable — **with the table named and the break point located** |
|
|
92
|
+
| `UNKNOWN` | the damage prevents a determination |
|
|
93
|
+
|
|
94
|
+
`UNKNOWN` is never dressed up as clean. A tool that reports a corrupt database as healthy
|
|
95
|
+
is worse than no tool.
|
|
96
|
+
|
|
97
|
+
## Limitations — read these first
|
|
98
|
+
|
|
99
|
+
**Structural, not semantic.** It verifies rows can be *read*. It cannot detect corruption
|
|
100
|
+
that produces valid-looking values — a flipped bit inside an integer that still parses is
|
|
101
|
+
invisible to it.
|
|
102
|
+
|
|
103
|
+
**Row counts come from the same damaged btree.** When `COUNT(*)` itself fails, the expected
|
|
104
|
+
count is unknown and the verdict degrades to `UNKNOWN` rather than guessing.
|
|
105
|
+
|
|
106
|
+
**It does not repair anything.** It tells you what survived so you can export the good
|
|
107
|
+
tables and restore the rest. Recovery is a different tool.
|
|
108
|
+
|
|
109
|
+
**v0.1.0.** Tested against databases built by SQLite and damaged at known page offsets:
|
|
110
|
+
3/3 corrupted databases localised to the correct table, 0 false alarms, 0 false-clean.
|
|
111
|
+
That corpus is deliberate byte corruption, which is one failure mode among several — real
|
|
112
|
+
corruption also arrives via truncated files, interrupted writes, and failing disks.
|
|
113
|
+
|
|
114
|
+
## Tests
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
python sqliteproof/tests/corrupt.py # build the corpus
|
|
118
|
+
python sqliteproof/tests/score.py # score localisation vs ground truth
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Ground truth comes from SQLite's own page allocation: tables are built one at a time and
|
|
122
|
+
the pages added between "before" and "after" belong to that table. The file format isn't
|
|
123
|
+
my invention and the damage lands where SQLite chose to put the data.
|
|
124
|
+
|
|
125
|
+
## License
|
|
126
|
+
|
|
127
|
+
MIT.
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# sqliteproof
|
|
2
|
+
|
|
3
|
+
**Find out which tables survived, not which pages broke.**
|
|
4
|
+
|
|
5
|
+
SQLite's own `PRAGMA integrity_check` is good at *detecting* corruption. What it gives you is this:
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
*** in database main ***
|
|
9
|
+
Tree 25 page 35 cell 18: Offset 57005 out of range 239..4092
|
|
10
|
+
database disk image is malformed
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
That's a page number. Nobody stores data by page number. What you actually need to know at
|
|
14
|
+
3am is **which tables can I still trust**, **how many rows did I lose**, and **is this worth
|
|
15
|
+
restoring from backup**.
|
|
16
|
+
|
|
17
|
+
```
|
|
18
|
+
$ sqliteproof app.db
|
|
19
|
+
|
|
20
|
+
app.db — DAMAGED
|
|
21
|
+
|
|
22
|
+
table verdict rows lost
|
|
23
|
+
------------------------------------------------------------
|
|
24
|
+
orders damaged 392/400 8
|
|
25
|
+
breaks after row 182: database disk image is malformed
|
|
26
|
+
audit_log intact 400/400 0
|
|
27
|
+
customers intact 400/400 0
|
|
28
|
+
|
|
29
|
+
1192 rows read, 8 unreadable.
|
|
30
|
+
Tables marked intact above are safe to export. Restore the damaged
|
|
31
|
+
ones from backup rather than trusting a partial read.
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Install
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
pip install sqliteproof
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Python 3.9+. **No dependencies** — standard library only.
|
|
41
|
+
|
|
42
|
+
## Usage
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
sqliteproof app.db # full report
|
|
46
|
+
sqliteproof app.db --json # machine-readable
|
|
47
|
+
sqliteproof app.db --quiet # verdict line only
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Exit codes: **0** intact · **1** damaged · **2** unreadable or undetermined. Drops straight
|
|
51
|
+
into a backup script:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
sqliteproof app.db --quiet || echo "corruption detected" | mail -s alert me@example.com
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## It opens the database read-only
|
|
58
|
+
|
|
59
|
+
A tool asked to inspect a damaged file must never be able to damage it further. The
|
|
60
|
+
connection is opened with `mode=ro` and nothing is written to the database, ever.
|
|
61
|
+
|
|
62
|
+
Runs entirely on your machine. No network, nothing uploaded.
|
|
63
|
+
|
|
64
|
+
## It will not bluff
|
|
65
|
+
|
|
66
|
+
| verdict | meaning |
|
|
67
|
+
|---|---|
|
|
68
|
+
| `INTACT` | every table read completely and row counts match |
|
|
69
|
+
| `DAMAGED` | some rows are unreadable — **with the table named and the break point located** |
|
|
70
|
+
| `UNKNOWN` | the damage prevents a determination |
|
|
71
|
+
|
|
72
|
+
`UNKNOWN` is never dressed up as clean. A tool that reports a corrupt database as healthy
|
|
73
|
+
is worse than no tool.
|
|
74
|
+
|
|
75
|
+
## Limitations — read these first
|
|
76
|
+
|
|
77
|
+
**Structural, not semantic.** It verifies rows can be *read*. It cannot detect corruption
|
|
78
|
+
that produces valid-looking values — a flipped bit inside an integer that still parses is
|
|
79
|
+
invisible to it.
|
|
80
|
+
|
|
81
|
+
**Row counts come from the same damaged btree.** When `COUNT(*)` itself fails, the expected
|
|
82
|
+
count is unknown and the verdict degrades to `UNKNOWN` rather than guessing.
|
|
83
|
+
|
|
84
|
+
**It does not repair anything.** It tells you what survived so you can export the good
|
|
85
|
+
tables and restore the rest. Recovery is a different tool.
|
|
86
|
+
|
|
87
|
+
**v0.1.0.** Tested against databases built by SQLite and damaged at known page offsets:
|
|
88
|
+
3/3 corrupted databases localised to the correct table, 0 false alarms, 0 false-clean.
|
|
89
|
+
That corpus is deliberate byte corruption, which is one failure mode among several — real
|
|
90
|
+
corruption also arrives via truncated files, interrupted writes, and failing disks.
|
|
91
|
+
|
|
92
|
+
## Tests
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
python sqliteproof/tests/corrupt.py # build the corpus
|
|
96
|
+
python sqliteproof/tests/score.py # score localisation vs ground truth
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Ground truth comes from SQLite's own page allocation: tables are built one at a time and
|
|
100
|
+
the pages added between "before" and "after" belong to that table. The file format isn't
|
|
101
|
+
my invention and the damage lands where SQLite chose to put the data.
|
|
102
|
+
|
|
103
|
+
## License
|
|
104
|
+
|
|
105
|
+
MIT.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=68", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "sqliteproof"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "Structural validation for SQLite databases -- tells you which tables survived, not which pages broke."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.9"
|
|
11
|
+
license = { text = "MIT" }
|
|
12
|
+
keywords = ["sqlite", "corruption", "integrity", "database", "validation", "backup", "recovery", "cli", "devops"]
|
|
13
|
+
classifiers = [
|
|
14
|
+
"Development Status :: 3 - Alpha",
|
|
15
|
+
"Intended Audience :: System Administrators",
|
|
16
|
+
"Intended Audience :: Developers",
|
|
17
|
+
"Topic :: Database",
|
|
18
|
+
"Topic :: System :: Recovery Tools",
|
|
19
|
+
"License :: OSI Approved :: MIT License",
|
|
20
|
+
"Programming Language :: Python :: 3",
|
|
21
|
+
"Operating System :: OS Independent",
|
|
22
|
+
]
|
|
23
|
+
dependencies = []
|
|
24
|
+
|
|
25
|
+
[project.urls]
|
|
26
|
+
Homepage = "https://github.com/OrbitalKeyAi/sqliteproof"
|
|
27
|
+
Source = "https://github.com/OrbitalKeyAi/sqliteproof"
|
|
28
|
+
Issues = "https://github.com/OrbitalKeyAi/sqliteproof/issues"
|
|
29
|
+
|
|
30
|
+
[project.scripts]
|
|
31
|
+
sqliteproof = "sqliteproof.cli:main"
|
|
32
|
+
|
|
33
|
+
[tool.setuptools]
|
|
34
|
+
packages = ["sqliteproof"]
|
|
@@ -0,0 +1,266 @@
|
|
|
1
|
+
"""Structural validation of a SQLite database, localised to the table and row.
|
|
2
|
+
|
|
3
|
+
SQLite ships `PRAGMA integrity_check`, which is genuinely good at *detecting* damage. What
|
|
4
|
+
it will not tell you is what you actually need to know when a database goes bad:
|
|
5
|
+
|
|
6
|
+
*** in database main ***
|
|
7
|
+
Page 4: btreeInitPage() returns error code 11
|
|
8
|
+
Page 7: free space corruption
|
|
9
|
+
|
|
10
|
+
That is a page number. Nobody stores data by page number. The questions that matter are
|
|
11
|
+
"which tables can I still trust", "how many rows did I lose", and "is this file worth
|
|
12
|
+
restoring from backup or can I salvage it" — and the page list answers none of them.
|
|
13
|
+
|
|
14
|
+
This module answers those. It walks each table, isolates the rows that fail to read,
|
|
15
|
+
and reports damage per table with a per-table verdict.
|
|
16
|
+
|
|
17
|
+
Design principle carried over from statementproof: **it refuses to bluff.** Where the
|
|
18
|
+
damage prevents a determination, the verdict is UNKNOWN, never "clean". A tool that
|
|
19
|
+
reports a corrupt database as healthy is worse than no tool.
|
|
20
|
+
"""
|
|
21
|
+
|
|
22
|
+
from __future__ import annotations
|
|
23
|
+
|
|
24
|
+
import os
|
|
25
|
+
import sqlite3
|
|
26
|
+
from dataclasses import dataclass, field
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
@dataclass
|
|
30
|
+
class TableReport:
|
|
31
|
+
name: str
|
|
32
|
+
verdict: str # "intact" | "damaged" | "unreadable" | "unknown"
|
|
33
|
+
rows_read: int = 0
|
|
34
|
+
rows_expected: int | None = None # None when the count itself could not be read
|
|
35
|
+
bad_rowids: list = field(default_factory=list)
|
|
36
|
+
error: str | None = None
|
|
37
|
+
|
|
38
|
+
@property
|
|
39
|
+
def rows_lost(self) -> int | None:
|
|
40
|
+
if self.rows_expected is None:
|
|
41
|
+
return None
|
|
42
|
+
return max(0, self.rows_expected - self.rows_read)
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
@dataclass
|
|
46
|
+
class Report:
|
|
47
|
+
path: str
|
|
48
|
+
readable: bool
|
|
49
|
+
integrity_check: list = field(default_factory=list)
|
|
50
|
+
tables: list = field(default_factory=list)
|
|
51
|
+
error: str | None = None
|
|
52
|
+
page_size: int | None = None
|
|
53
|
+
encoding: str | None = None
|
|
54
|
+
|
|
55
|
+
@property
|
|
56
|
+
def verdict(self) -> str:
|
|
57
|
+
if not self.readable:
|
|
58
|
+
return "UNREADABLE"
|
|
59
|
+
if self.error:
|
|
60
|
+
return "UNKNOWN"
|
|
61
|
+
if not self.tables:
|
|
62
|
+
return "UNKNOWN"
|
|
63
|
+
v = {t.verdict for t in self.tables}
|
|
64
|
+
if v == {"intact"}:
|
|
65
|
+
return "INTACT"
|
|
66
|
+
if "unknown" in v:
|
|
67
|
+
return "UNKNOWN"
|
|
68
|
+
return "DAMAGED"
|
|
69
|
+
|
|
70
|
+
@property
|
|
71
|
+
def rows_read(self) -> int:
|
|
72
|
+
return sum(t.rows_read for t in self.tables)
|
|
73
|
+
|
|
74
|
+
@property
|
|
75
|
+
def rows_lost(self) -> int:
|
|
76
|
+
return sum(t.rows_lost or 0 for t in self.tables)
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
def _connect(path: str) -> sqlite3.Connection:
|
|
80
|
+
# Read-only, and never create. A validation tool that writes to the file it is
|
|
81
|
+
# inspecting can destroy the evidence it was asked to examine.
|
|
82
|
+
uri = "file:%s?mode=ro" % os.path.abspath(path).replace("?", "%3f").replace("#", "%23")
|
|
83
|
+
con = sqlite3.connect(uri, uri=True, timeout=5)
|
|
84
|
+
con.text_factory = bytes # tolerate invalid UTF-8 in damaged pages
|
|
85
|
+
return con
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
def _table_names(con: sqlite3.Connection) -> list:
|
|
89
|
+
cur = con.execute(
|
|
90
|
+
"SELECT name FROM sqlite_master WHERE type='table' "
|
|
91
|
+
"AND name NOT LIKE 'sqlite_%' ORDER BY name")
|
|
92
|
+
out = []
|
|
93
|
+
for (n,) in cur.fetchall():
|
|
94
|
+
out.append(n.decode("utf-8", "replace") if isinstance(n, bytes) else n)
|
|
95
|
+
return out
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def _quote(ident: str) -> str:
|
|
99
|
+
return '"%s"' % ident.replace('"', '""')
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
def _scan_table(con: sqlite3.Connection, table: str, max_bad: int = 200) -> TableReport:
|
|
103
|
+
"""Read every row, isolating the ones that fail.
|
|
104
|
+
|
|
105
|
+
Rows are pulled one at a time rather than fetchall() so a single bad page damages
|
|
106
|
+
one row's report instead of aborting the whole table — which is the entire point.
|
|
107
|
+
"""
|
|
108
|
+
rep = TableReport(name=table, verdict="unknown")
|
|
109
|
+
|
|
110
|
+
try:
|
|
111
|
+
(expected,) = con.execute("SELECT COUNT(*) FROM %s" % _quote(table)).fetchone()
|
|
112
|
+
rep.rows_expected = int(expected)
|
|
113
|
+
except sqlite3.DatabaseError as e:
|
|
114
|
+
# COUNT walks the btree; if that fails the table is structurally broken.
|
|
115
|
+
rep.rows_expected = None
|
|
116
|
+
rep.error = "%s: %s" % (type(e).__name__, e)
|
|
117
|
+
|
|
118
|
+
# Prefer rowid so failures can be named. Tables declared WITHOUT ROWID have none.
|
|
119
|
+
has_rowid = True
|
|
120
|
+
try:
|
|
121
|
+
con.execute("SELECT rowid FROM %s LIMIT 1" % _quote(table)).fetchone()
|
|
122
|
+
except sqlite3.DatabaseError:
|
|
123
|
+
has_rowid = False
|
|
124
|
+
|
|
125
|
+
sel = ("SELECT rowid, * FROM %s" if has_rowid else "SELECT * FROM %s") % _quote(table)
|
|
126
|
+
try:
|
|
127
|
+
cur = con.execute(sel)
|
|
128
|
+
except sqlite3.DatabaseError as e:
|
|
129
|
+
rep.verdict = "unreadable"
|
|
130
|
+
rep.error = rep.error or "%s: %s" % (type(e).__name__, e)
|
|
131
|
+
return rep
|
|
132
|
+
|
|
133
|
+
n = 0
|
|
134
|
+
while True:
|
|
135
|
+
try:
|
|
136
|
+
row = cur.fetchone()
|
|
137
|
+
except sqlite3.DatabaseError as e:
|
|
138
|
+
# One unreadable row. Record it and keep going -- localisation is the product.
|
|
139
|
+
if len(rep.bad_rowids) < max_bad:
|
|
140
|
+
rep.bad_rowids.append({"after_row": n, "error": str(e)[:90]})
|
|
141
|
+
else:
|
|
142
|
+
break
|
|
143
|
+
# The cursor is not reliably resumable after a btree error; restart past
|
|
144
|
+
# the damage using the last good rowid when we have one.
|
|
145
|
+
if not has_rowid:
|
|
146
|
+
break
|
|
147
|
+
try:
|
|
148
|
+
cur = con.execute(
|
|
149
|
+
"SELECT rowid, * FROM %s WHERE rowid > ?" % _quote(table), (n,))
|
|
150
|
+
continue
|
|
151
|
+
except sqlite3.DatabaseError:
|
|
152
|
+
break
|
|
153
|
+
if row is None:
|
|
154
|
+
break
|
|
155
|
+
n += 1
|
|
156
|
+
if has_rowid and isinstance(row[0], int):
|
|
157
|
+
n = max(n, 0)
|
|
158
|
+
|
|
159
|
+
rep.rows_read = n
|
|
160
|
+
if rep.bad_rowids or rep.error:
|
|
161
|
+
rep.verdict = "damaged"
|
|
162
|
+
elif rep.rows_expected is None:
|
|
163
|
+
rep.verdict = "unknown"
|
|
164
|
+
elif rep.rows_read >= rep.rows_expected:
|
|
165
|
+
rep.verdict = "intact"
|
|
166
|
+
else:
|
|
167
|
+
rep.verdict = "damaged"
|
|
168
|
+
return rep
|
|
169
|
+
|
|
170
|
+
|
|
171
|
+
def check(path: str) -> Report:
|
|
172
|
+
rep = Report(path=path, readable=False)
|
|
173
|
+
if not os.path.isfile(path):
|
|
174
|
+
rep.error = "no such file"
|
|
175
|
+
return rep
|
|
176
|
+
|
|
177
|
+
try:
|
|
178
|
+
with open(path, "rb") as f:
|
|
179
|
+
head = f.read(100)
|
|
180
|
+
if not head.startswith(b"SQLite format 3\x00"):
|
|
181
|
+
rep.error = "not a SQLite database (magic header missing)"
|
|
182
|
+
return rep
|
|
183
|
+
rep.page_size = int.from_bytes(head[16:18], "big") or 65536
|
|
184
|
+
rep.encoding = {1: "UTF-8", 2: "UTF-16le", 3: "UTF-16be"}.get(
|
|
185
|
+
int.from_bytes(head[56:60], "big"), "unknown")
|
|
186
|
+
except OSError as e:
|
|
187
|
+
rep.error = "cannot read file: %s" % e
|
|
188
|
+
return rep
|
|
189
|
+
|
|
190
|
+
try:
|
|
191
|
+
con = _connect(path)
|
|
192
|
+
except sqlite3.Error as e:
|
|
193
|
+
rep.error = "cannot open: %s" % e
|
|
194
|
+
return rep
|
|
195
|
+
|
|
196
|
+
rep.readable = True
|
|
197
|
+
try:
|
|
198
|
+
for (msg,) in con.execute("PRAGMA integrity_check").fetchall():
|
|
199
|
+
m = msg.decode("utf-8", "replace") if isinstance(msg, bytes) else msg
|
|
200
|
+
if m != "ok":
|
|
201
|
+
rep.integrity_check.append(m)
|
|
202
|
+
except sqlite3.DatabaseError as e:
|
|
203
|
+
rep.integrity_check.append("integrity_check itself failed: %s" % e)
|
|
204
|
+
|
|
205
|
+
try:
|
|
206
|
+
names = _table_names(con)
|
|
207
|
+
except sqlite3.DatabaseError as e:
|
|
208
|
+
rep.error = "schema unreadable: %s" % e
|
|
209
|
+
con.close()
|
|
210
|
+
return rep
|
|
211
|
+
|
|
212
|
+
for t in names:
|
|
213
|
+
rep.tables.append(_scan_table(con, t))
|
|
214
|
+
con.close()
|
|
215
|
+
return rep
|
|
216
|
+
|
|
217
|
+
|
|
218
|
+
def format_report(rep: Report) -> str:
|
|
219
|
+
L = []
|
|
220
|
+
L.append("%s — %s" % (os.path.basename(rep.path), rep.verdict))
|
|
221
|
+
if rep.page_size:
|
|
222
|
+
L.append(" page size %d, encoding %s" % (rep.page_size, rep.encoding))
|
|
223
|
+
if rep.error:
|
|
224
|
+
L.append(" %s" % rep.error)
|
|
225
|
+
return "\n".join(L)
|
|
226
|
+
|
|
227
|
+
if rep.integrity_check:
|
|
228
|
+
L.append("")
|
|
229
|
+
L.append(" PRAGMA integrity_check reported %d problem(s):" % len(rep.integrity_check))
|
|
230
|
+
for m in rep.integrity_check[:5]:
|
|
231
|
+
for part in str(m).splitlines():
|
|
232
|
+
if part.strip():
|
|
233
|
+
L.append(" %s" % part.strip()[:78])
|
|
234
|
+
if len(rep.integrity_check) > 5:
|
|
235
|
+
L.append(" ... and %d more" % (len(rep.integrity_check) - 5))
|
|
236
|
+
L.append(" (page numbers. Below is what that means for your data.)")
|
|
237
|
+
|
|
238
|
+
L.append("")
|
|
239
|
+
L.append(" %-28s %-11s %8s %8s" % ("table", "verdict", "rows", "lost"))
|
|
240
|
+
L.append(" " + "-" * 60)
|
|
241
|
+
for t in sorted(rep.tables, key=lambda x: (x.verdict == "intact", x.name)):
|
|
242
|
+
lost = "-" if t.rows_lost is None else str(t.rows_lost)
|
|
243
|
+
exp = "?" if t.rows_expected is None else str(t.rows_expected)
|
|
244
|
+
L.append(" %-28s %-11s %8s %8s" % (
|
|
245
|
+
t.name[:28], t.verdict, "%d/%s" % (t.rows_read, exp), lost))
|
|
246
|
+
if t.error:
|
|
247
|
+
L.append(" %s" % t.error[:88])
|
|
248
|
+
for b in t.bad_rowids[:3]:
|
|
249
|
+
L.append(" breaks after row %s: %s" % (b["after_row"], b["error"][:60]))
|
|
250
|
+
if len(t.bad_rowids) > 3:
|
|
251
|
+
L.append(" ... and %d more break points" % (len(t.bad_rowids) - 3))
|
|
252
|
+
|
|
253
|
+
L.append("")
|
|
254
|
+
if rep.verdict == "INTACT":
|
|
255
|
+
L.append(" Every table read completely and row counts match.")
|
|
256
|
+
L.append(" That means the data is structurally readable. It does NOT mean the")
|
|
257
|
+
L.append(" contents are correct — corruption that produces valid-looking values")
|
|
258
|
+
L.append(" cannot be detected this way.")
|
|
259
|
+
elif rep.verdict == "DAMAGED":
|
|
260
|
+
L.append(" %d rows read, %d unreadable." % (rep.rows_read, rep.rows_lost))
|
|
261
|
+
L.append(" Tables marked intact above are safe to export. Restore the damaged")
|
|
262
|
+
L.append(" ones from backup rather than trusting a partial read.")
|
|
263
|
+
else:
|
|
264
|
+
L.append(" The damage prevents a determination. This is not a clean bill of")
|
|
265
|
+
L.append(" health — it means the question could not be answered.")
|
|
266
|
+
return "\n".join(L)
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
"""sqliteproof CLI — find out which tables survived, not which pages broke.
|
|
2
|
+
|
|
3
|
+
Runs entirely locally and opens the database **read-only**. A tool asked to inspect a
|
|
4
|
+
damaged file must never be able to damage it further, so the connection is opened with
|
|
5
|
+
`mode=ro` and nothing is ever written to the database.
|
|
6
|
+
|
|
7
|
+
sqliteproof app.db
|
|
8
|
+
sqliteproof app.db --json
|
|
9
|
+
sqliteproof app.db --quiet # verdict line only; exit code carries the result
|
|
10
|
+
"""
|
|
11
|
+
|
|
12
|
+
from __future__ import annotations
|
|
13
|
+
|
|
14
|
+
import argparse
|
|
15
|
+
import json
|
|
16
|
+
import os
|
|
17
|
+
import sys
|
|
18
|
+
|
|
19
|
+
from .check import check, format_report
|
|
20
|
+
|
|
21
|
+
BANNER = """
|
|
22
|
+
sqliteproof — which tables survived, not which pages broke
|
|
23
|
+
runs locally | opens the database read-only | nothing uploaded
|
|
24
|
+
"""
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
def to_dict(rep) -> dict:
|
|
28
|
+
return {
|
|
29
|
+
"file": os.path.basename(rep.path),
|
|
30
|
+
"verdict": rep.verdict,
|
|
31
|
+
"readable": rep.readable,
|
|
32
|
+
"page_size": rep.page_size,
|
|
33
|
+
"encoding": rep.encoding,
|
|
34
|
+
"error": rep.error,
|
|
35
|
+
"integrity_check_messages": len(rep.integrity_check),
|
|
36
|
+
"rows_read": rep.rows_read,
|
|
37
|
+
"rows_lost": rep.rows_lost,
|
|
38
|
+
"tables": [
|
|
39
|
+
{"name": t.name, "verdict": t.verdict, "rows_read": t.rows_read,
|
|
40
|
+
"rows_expected": t.rows_expected, "rows_lost": t.rows_lost,
|
|
41
|
+
"break_points": len(t.bad_rowids), "error": t.error}
|
|
42
|
+
for t in rep.tables
|
|
43
|
+
],
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
def main(argv=None):
|
|
48
|
+
ap = argparse.ArgumentParser(
|
|
49
|
+
prog="sqliteproof",
|
|
50
|
+
description="Structural validation for SQLite databases, localised to the table.")
|
|
51
|
+
ap.add_argument("database", help="path to a SQLite database file")
|
|
52
|
+
ap.add_argument("--json", action="store_true", help="machine-readable output")
|
|
53
|
+
ap.add_argument("--quiet", action="store_true", help="verdict line only")
|
|
54
|
+
args = ap.parse_args(argv)
|
|
55
|
+
|
|
56
|
+
if not os.path.isfile(args.database):
|
|
57
|
+
print("error: no such file: %s" % args.database, file=sys.stderr)
|
|
58
|
+
return 2
|
|
59
|
+
|
|
60
|
+
rep = check(args.database)
|
|
61
|
+
|
|
62
|
+
if args.json:
|
|
63
|
+
print(json.dumps(to_dict(rep), indent=2))
|
|
64
|
+
elif args.quiet:
|
|
65
|
+
print(rep.verdict)
|
|
66
|
+
else:
|
|
67
|
+
print(BANNER)
|
|
68
|
+
print(format_report(rep))
|
|
69
|
+
|
|
70
|
+
# 0 intact · 1 damaged · 2 unreadable or undetermined, so it drops into a backup script
|
|
71
|
+
return {"INTACT": 0, "DAMAGED": 1}.get(rep.verdict, 2)
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
if __name__ == "__main__":
|
|
75
|
+
sys.exit(main())
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sqliteproof
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Structural validation for SQLite databases -- tells you which tables survived, not which pages broke.
|
|
5
|
+
License: MIT
|
|
6
|
+
Project-URL: Homepage, https://github.com/OrbitalKeyAi/sqliteproof
|
|
7
|
+
Project-URL: Source, https://github.com/OrbitalKeyAi/sqliteproof
|
|
8
|
+
Project-URL: Issues, https://github.com/OrbitalKeyAi/sqliteproof/issues
|
|
9
|
+
Keywords: sqlite,corruption,integrity,database,validation,backup,recovery,cli,devops
|
|
10
|
+
Classifier: Development Status :: 3 - Alpha
|
|
11
|
+
Classifier: Intended Audience :: System Administrators
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Topic :: Database
|
|
14
|
+
Classifier: Topic :: System :: Recovery Tools
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Operating System :: OS Independent
|
|
18
|
+
Requires-Python: >=3.9
|
|
19
|
+
Description-Content-Type: text/markdown
|
|
20
|
+
License-File: LICENSE
|
|
21
|
+
Dynamic: license-file
|
|
22
|
+
|
|
23
|
+
# sqliteproof
|
|
24
|
+
|
|
25
|
+
**Find out which tables survived, not which pages broke.**
|
|
26
|
+
|
|
27
|
+
SQLite's own `PRAGMA integrity_check` is good at *detecting* corruption. What it gives you is this:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
*** in database main ***
|
|
31
|
+
Tree 25 page 35 cell 18: Offset 57005 out of range 239..4092
|
|
32
|
+
database disk image is malformed
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
That's a page number. Nobody stores data by page number. What you actually need to know at
|
|
36
|
+
3am is **which tables can I still trust**, **how many rows did I lose**, and **is this worth
|
|
37
|
+
restoring from backup**.
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
$ sqliteproof app.db
|
|
41
|
+
|
|
42
|
+
app.db — DAMAGED
|
|
43
|
+
|
|
44
|
+
table verdict rows lost
|
|
45
|
+
------------------------------------------------------------
|
|
46
|
+
orders damaged 392/400 8
|
|
47
|
+
breaks after row 182: database disk image is malformed
|
|
48
|
+
audit_log intact 400/400 0
|
|
49
|
+
customers intact 400/400 0
|
|
50
|
+
|
|
51
|
+
1192 rows read, 8 unreadable.
|
|
52
|
+
Tables marked intact above are safe to export. Restore the damaged
|
|
53
|
+
ones from backup rather than trusting a partial read.
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Install
|
|
57
|
+
|
|
58
|
+
```bash
|
|
59
|
+
pip install sqliteproof
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Python 3.9+. **No dependencies** — standard library only.
|
|
63
|
+
|
|
64
|
+
## Usage
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
sqliteproof app.db # full report
|
|
68
|
+
sqliteproof app.db --json # machine-readable
|
|
69
|
+
sqliteproof app.db --quiet # verdict line only
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Exit codes: **0** intact · **1** damaged · **2** unreadable or undetermined. Drops straight
|
|
73
|
+
into a backup script:
|
|
74
|
+
|
|
75
|
+
```bash
|
|
76
|
+
sqliteproof app.db --quiet || echo "corruption detected" | mail -s alert me@example.com
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
## It opens the database read-only
|
|
80
|
+
|
|
81
|
+
A tool asked to inspect a damaged file must never be able to damage it further. The
|
|
82
|
+
connection is opened with `mode=ro` and nothing is written to the database, ever.
|
|
83
|
+
|
|
84
|
+
Runs entirely on your machine. No network, nothing uploaded.
|
|
85
|
+
|
|
86
|
+
## It will not bluff
|
|
87
|
+
|
|
88
|
+
| verdict | meaning |
|
|
89
|
+
|---|---|
|
|
90
|
+
| `INTACT` | every table read completely and row counts match |
|
|
91
|
+
| `DAMAGED` | some rows are unreadable — **with the table named and the break point located** |
|
|
92
|
+
| `UNKNOWN` | the damage prevents a determination |
|
|
93
|
+
|
|
94
|
+
`UNKNOWN` is never dressed up as clean. A tool that reports a corrupt database as healthy
|
|
95
|
+
is worse than no tool.
|
|
96
|
+
|
|
97
|
+
## Limitations — read these first
|
|
98
|
+
|
|
99
|
+
**Structural, not semantic.** It verifies rows can be *read*. It cannot detect corruption
|
|
100
|
+
that produces valid-looking values — a flipped bit inside an integer that still parses is
|
|
101
|
+
invisible to it.
|
|
102
|
+
|
|
103
|
+
**Row counts come from the same damaged btree.** When `COUNT(*)` itself fails, the expected
|
|
104
|
+
count is unknown and the verdict degrades to `UNKNOWN` rather than guessing.
|
|
105
|
+
|
|
106
|
+
**It does not repair anything.** It tells you what survived so you can export the good
|
|
107
|
+
tables and restore the rest. Recovery is a different tool.
|
|
108
|
+
|
|
109
|
+
**v0.1.0.** Tested against databases built by SQLite and damaged at known page offsets:
|
|
110
|
+
3/3 corrupted databases localised to the correct table, 0 false alarms, 0 false-clean.
|
|
111
|
+
That corpus is deliberate byte corruption, which is one failure mode among several — real
|
|
112
|
+
corruption also arrives via truncated files, interrupted writes, and failing disks.
|
|
113
|
+
|
|
114
|
+
## Tests
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
python sqliteproof/tests/corrupt.py # build the corpus
|
|
118
|
+
python sqliteproof/tests/score.py # score localisation vs ground truth
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Ground truth comes from SQLite's own page allocation: tables are built one at a time and
|
|
122
|
+
the pages added between "before" and "after" belong to that table. The file format isn't
|
|
123
|
+
my invention and the damage lands where SQLite chose to put the data.
|
|
124
|
+
|
|
125
|
+
## License
|
|
126
|
+
|
|
127
|
+
MIT.
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
LICENSE
|
|
2
|
+
README.md
|
|
3
|
+
pyproject.toml
|
|
4
|
+
sqliteproof/__init__.py
|
|
5
|
+
sqliteproof/__main__.py
|
|
6
|
+
sqliteproof/check.py
|
|
7
|
+
sqliteproof/cli.py
|
|
8
|
+
sqliteproof.egg-info/PKG-INFO
|
|
9
|
+
sqliteproof.egg-info/SOURCES.txt
|
|
10
|
+
sqliteproof.egg-info/dependency_links.txt
|
|
11
|
+
sqliteproof.egg-info/entry_points.txt
|
|
12
|
+
sqliteproof.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
sqliteproof
|