paradoxdb 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- paradoxdb-0.1.0/.gitignore +31 -0
- paradoxdb-0.1.0/CHANGELOG.md +43 -0
- paradoxdb-0.1.0/LICENSE +21 -0
- paradoxdb-0.1.0/PKG-INFO +229 -0
- paradoxdb-0.1.0/README.es.md +200 -0
- paradoxdb-0.1.0/README.md +198 -0
- paradoxdb-0.1.0/docs/DECISIONES.md +30 -0
- paradoxdb-0.1.0/docs/INFORME_FASE_2.md +127 -0
- paradoxdb-0.1.0/pyproject.toml +79 -0
- paradoxdb-0.1.0/src/paradoxdb/__init__.py +94 -0
- paradoxdb-0.1.0/src/paradoxdb/__main__.py +10 -0
- paradoxdb-0.1.0/src/paradoxdb/blocks.py +138 -0
- paradoxdb-0.1.0/src/paradoxdb/cli.py +140 -0
- paradoxdb-0.1.0/src/paradoxdb/errors.py +27 -0
- paradoxdb-0.1.0/src/paradoxdb/fields.py +132 -0
- paradoxdb-0.1.0/src/paradoxdb/header.py +329 -0
- paradoxdb-0.1.0/src/paradoxdb/index.py +251 -0
- paradoxdb-0.1.0/src/paradoxdb/memo.py +199 -0
- paradoxdb-0.1.0/src/paradoxdb/numeric.py +128 -0
- paradoxdb-0.1.0/src/paradoxdb/profile.py +284 -0
- paradoxdb-0.1.0/src/paradoxdb/py.typed +0 -0
- paradoxdb-0.1.0/src/paradoxdb/synthetic.py +507 -0
- paradoxdb-0.1.0/src/paradoxdb/table.py +247 -0
- paradoxdb-0.1.0/src/paradoxdb/values.py +337 -0
- paradoxdb-0.1.0/src/paradoxdb/writer.py +233 -0
- paradoxdb-0.1.0/tests/__init__.py +0 -0
- paradoxdb-0.1.0/tests/conftest.py +49 -0
- paradoxdb-0.1.0/tests/test_cli_module.py +27 -0
- paradoxdb-0.1.0/tests/test_encoding.py +206 -0
- paradoxdb-0.1.0/tests/test_header.py +119 -0
- paradoxdb-0.1.0/tests/test_index.py +127 -0
- paradoxdb-0.1.0/tests/test_memo.py +124 -0
- paradoxdb-0.1.0/tests/test_no_leaks.py +174 -0
- paradoxdb-0.1.0/tests/test_profile_cli.py +125 -0
- paradoxdb-0.1.0/tests/test_reader.py +217 -0
- paradoxdb-0.1.0/tests/test_synthetic.py +202 -0
- paradoxdb-0.1.0/tests/test_writer.py +287 -0
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
.eggs/
|
|
6
|
+
build/
|
|
7
|
+
dist/
|
|
8
|
+
.venv/
|
|
9
|
+
venv/
|
|
10
|
+
.pytest_cache/
|
|
11
|
+
.hypothesis/
|
|
12
|
+
.ruff_cache/
|
|
13
|
+
.mypy_cache/
|
|
14
|
+
.coverage
|
|
15
|
+
htmlcov/
|
|
16
|
+
|
|
17
|
+
# Editors / OS
|
|
18
|
+
.idea/
|
|
19
|
+
.vscode/
|
|
20
|
+
.DS_Store
|
|
21
|
+
Thumbs.db
|
|
22
|
+
|
|
23
|
+
# Generated Paradox files (always produced by the synthetic factory, never committed)
|
|
24
|
+
*.DB
|
|
25
|
+
*.db
|
|
26
|
+
*.MB
|
|
27
|
+
*.mb
|
|
28
|
+
*.PX
|
|
29
|
+
*.px
|
|
30
|
+
*.X??
|
|
31
|
+
*.Y??
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented in this file.
|
|
4
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
|
|
5
|
+
and the project adheres to [Semantic Versioning](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- GitHub Actions workflow that builds, checks and publishes the package to PyPI
|
|
12
|
+
through trusted publishing on every published release.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
|
|
16
|
+
- README links are absolute so that they work on the PyPI project page.
|
|
17
|
+
|
|
18
|
+
## [0.1.0] - 2026-11-15
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- Reader for Paradox `.DB` tables, versions 3.0 to 7 (headers with and
|
|
23
|
+
without the 4.0+ extension), with per-table code page and per-cell state
|
|
24
|
+
(`OK` / `NULL` / `INVALID`) plus raw bytes.
|
|
25
|
+
- Decoders and encoders for every field type: alpha, date, short, long,
|
|
26
|
+
currency, number, logical, memo, formatted memo, BLOB, OLE, graphic,
|
|
27
|
+
time, timestamp, autoincrement, BCD and bytes.
|
|
28
|
+
- Reader for `.MB` memo files: embedded, single-block and sub-allocated
|
|
29
|
+
memos.
|
|
30
|
+
- Block-chain walker that returns every record present in the data blocks
|
|
31
|
+
and reports the mismatch with the declared record count.
|
|
32
|
+
- Writer on top of a template header, producing new `.DB`, `.MB` and `.PX`
|
|
33
|
+
files, with optional legacy sort order and primary-key uniqueness check.
|
|
34
|
+
- Primary index (`.PX`) builder and verifier.
|
|
35
|
+
- Table profiler (aggregates per table and field, SHA-256 fingerprints) and
|
|
36
|
+
CSV / JSON export.
|
|
37
|
+
- `paradoxdb` command line: `profile`, `export`, `verify-index`.
|
|
38
|
+
- Synthetic table factory used by the test-suite and the examples.
|
|
39
|
+
- Optional `Decryptor` protocol for encrypted tables (the library never
|
|
40
|
+
implements the cipher itself).
|
|
41
|
+
|
|
42
|
+
[Unreleased]: https://github.com/arielgiamportone/paradox-db-py/compare/v0.1.0...HEAD
|
|
43
|
+
[0.1.0]: https://github.com/arielgiamportone/paradox-db-py/releases/tag/v0.1.0
|
paradoxdb-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 arielgiamportone
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
paradoxdb-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: paradoxdb
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Pure-Python reader and writer for Paradox tables (.DB / .MB / .PX) with explicit per-cell state
|
|
5
|
+
Project-URL: Homepage, https://github.com/arielgiamportone/paradox-db-py
|
|
6
|
+
Project-URL: Repository, https://github.com/arielgiamportone/paradox-db-py
|
|
7
|
+
Project-URL: Issues, https://github.com/arielgiamportone/paradox-db-py/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/arielgiamportone/paradox-db-py/blob/main/CHANGELOG.md
|
|
9
|
+
Author: arielgiamportone
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: b-tree,bcd,bde,borland,database,delphi,file-format,legacy,memo,migration,paradox
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
22
|
+
Classifier: Topic :: Database
|
|
23
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
24
|
+
Classifier: Typing :: Typed
|
|
25
|
+
Requires-Python: >=3.11
|
|
26
|
+
Provides-Extra: dev
|
|
27
|
+
Requires-Dist: hypothesis>=6; extra == 'dev'
|
|
28
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
29
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
30
|
+
Description-Content-Type: text/markdown
|
|
31
|
+
|
|
32
|
+
# paradox-db-py
|
|
33
|
+
|
|
34
|
+
[](https://github.com/arielgiamportone/paradox-db-py/actions/workflows/ci.yml)
|
|
35
|
+

|
|
36
|
+

|
|
37
|
+
|
|
38
|
+
**Pure-Python reader and writer for Paradox tables** (`.DB` data, `.MB` memos
|
|
39
|
+
and BLOBs, `.PX` primary index), versions 3.0 to 7. No native code, no copyleft
|
|
40
|
+
dependencies, no silent conversions: every cell comes back as `OK`, `NULL` or
|
|
41
|
+
`INVALID` together with its raw bytes.
|
|
42
|
+
|
|
43
|
+
*Versión en español: [README.es.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/README.es.md).*
|
|
44
|
+
|
|
45
|
+
> Patterns developed during my work at Intervalor Data and published with permission. Independent clean-room implementation with synthetic data; it contains no code or data from the company or its clients.
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## 1. The problem
|
|
50
|
+
|
|
51
|
+
A fixed-asset management system written in Delphi on top of the Borland
|
|
52
|
+
Database Engine kept decades of history in Paradox tables. Modernising it meant
|
|
53
|
+
two things at once:
|
|
54
|
+
|
|
55
|
+
1. **Extract** every record without loss. Not "most records": the legacy
|
|
56
|
+
engine left records in the data blocks that the header never counted, and
|
|
57
|
+
whole columns of monetary values stored as BCD that no longer decoded.
|
|
58
|
+
2. **Give tables back** to the legacy application during a long transition,
|
|
59
|
+
so the old and the new system could coexist for months.
|
|
60
|
+
|
|
61
|
+
The Python ecosystem offered only bindings over a C library (GPL) that is
|
|
62
|
+
read-only in practice, forces one global text encoding, skips the records the
|
|
63
|
+
header does not declare and corrupts its memory heap after a few dozen write
|
|
64
|
+
operations. For an on-premise product none of that was acceptable.
|
|
65
|
+
|
|
66
|
+
## 2. Constraints
|
|
67
|
+
|
|
68
|
+
| Constraint | Consequence in the design |
|
|
69
|
+
|---|---|
|
|
70
|
+
| Distributed on-premise inside a commercial product | Pure Python, MIT licence, zero runtime dependencies |
|
|
71
|
+
| Real tables hold personal and financial data of third parties | Every fixture is generated by `paradoxdb.synthetic`; the repo contains no `.DB` file |
|
|
72
|
+
| The cipher of encrypted tables is only available in GPL code | Decryption is delegated to an optional external `Decryptor`; this library only decodes plain bytes |
|
|
73
|
+
| The legacy engine must still open what we write | The writer copies an existing header (template) and only rewrites counters; new files are produced, nothing is modified in place |
|
|
74
|
+
| Mixed code pages in one dataset (CP437 core, CP1252 later tables) | Code page per table, read from the header, never global |
|
|
75
|
+
| Monetary amounts and 32-digit BCD | `Decimal` everywhere; `float` only as the on-disk representation |
|
|
76
|
+
|
|
77
|
+
## 3. Architecture
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
paradoxdb/
|
|
81
|
+
├── fields.py FieldType, FieldDef, code pages, physical sizes
|
|
82
|
+
├── numeric.py order-preserving codecs: Paradox int, Paradox double, BCD
|
|
83
|
+
├── values.py CellValue(state, value, raw_bytes, reason); decode_value; encode_*
|
|
84
|
+
├── header.py TableHeader, parse_header, patch_counters (0x06, 0x0A-0x10, 0x49)
|
|
85
|
+
├── blocks.py block-chain walker (every record, anomalies) and block packer
|
|
86
|
+
├── memo.py MemoPointer, MemoFile (types 2 and 3), MemoWriter (type 2)
|
|
87
|
+
├── table.py open_table, ParadoxTable, Record, Decryptor protocol
|
|
88
|
+
├── writer.py write_table on a template, WriteRecord, legacy_sort_key
|
|
89
|
+
├── index.py DataBlock, read_data_blocks, build_index, verify_index
|
|
90
|
+
├── profile.py profile_table, export_csv, export_json
|
|
91
|
+
├── cli.py paradoxdb profile | export | verify-index
|
|
92
|
+
└── synthetic.py table factory used by tests and examples
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Reading pipeline:
|
|
96
|
+
|
|
97
|
+
```
|
|
98
|
+
.DB bytes ──► parse_header ──► iter_blocks ──► decode_value per field ──► Record
|
|
99
|
+
│ ▲
|
|
100
|
+
└── code page, descriptors └── MemoFile (.MB sibling, any case)
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Writing pipeline:
|
|
104
|
+
|
|
105
|
+
```
|
|
106
|
+
template .DB header ──copy──► patch counters ──► pack_blocks(records) ──► new .DB
|
|
107
|
+
template .MB first block ──► MemoWriter (one single-block entry per memo) ──► new .MB
|
|
108
|
+
template .PX header ──► build_index(read_data_blocks(new .DB)) ──► new .PX
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
### Quick start
|
|
112
|
+
|
|
113
|
+
```python
|
|
114
|
+
from paradoxdb import open_table
|
|
115
|
+
|
|
116
|
+
table = open_table("ASSETS.DB") # finds assets.mb / ASSETS.MB too
|
|
117
|
+
print(table.header.version_label, table.header.codec)
|
|
118
|
+
|
|
119
|
+
for record in table.records(): # every record in every block
|
|
120
|
+
amount = record["Amount"] # case-insensitive field lookup
|
|
121
|
+
if amount.is_ok:
|
|
122
|
+
total = amount.value # Decimal, date, int, bool, str, bytes
|
|
123
|
+
else:
|
|
124
|
+
print(amount.state, amount.reason, amount.raw_bytes.hex())
|
|
125
|
+
|
|
126
|
+
print(table.block_record_count(), table.declared_record_count, table.header_mismatch)
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
```python
|
|
130
|
+
from paradoxdb import WriteRecord, write_table, verify_index, read_data_blocks
|
|
131
|
+
|
|
132
|
+
records = [
|
|
133
|
+
WriteRecord(r.raw_bytes, {3: r["Note"].value.encode(table.codec)}) for r in table.records()
|
|
134
|
+
]
|
|
135
|
+
result = write_table("ASSETS.DB", "out/ASSETS.DB", records, sort=False)
|
|
136
|
+
assert verify_index(result.px_path, read_data_blocks(result.db_path)) == []
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
```bash
|
|
140
|
+
paradoxdb profile ./tables --output profile.json # aggregates, SHA-256, mismatches
|
|
141
|
+
paradoxdb export ./tables/ASSETS.DB --format csv # value, state and reason per cell
|
|
142
|
+
paradoxdb verify-index ./tables/ASSETS.DB # exit 1 when the .PX is inconsistent
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
## 4. Decisions and the alternatives they replaced
|
|
146
|
+
|
|
147
|
+
| Decision | Rejected alternative | Why |
|
|
148
|
+
|---|---|---|
|
|
149
|
+
| Own reader in pure Python | Bindings over the C library | GPL licence, one global encoding, skips undeclared records, no 64-bit guarantee on Linux |
|
|
150
|
+
| Decryption delegated to an optional component | Re-implement the cipher | The only public implementation is GPL; encrypted tables are a small minority |
|
|
151
|
+
| Explicit state per cell plus raw bytes | `None` / `0` when in doubt | The earlier approach turned garbage currency values and sentinel dates into `None`: silent loss at scale |
|
|
152
|
+
| `Decimal` for doubles and BCD | `float` | Accounting amounts and 32-digit BCD; arithmetic parity with the legacy engine is the consumer's job |
|
|
153
|
+
| Code page per table | Global `latin1` or `cp1252` | `latin1` accepts every byte but misreads `ñ` / `º` from CP437 tables; datasets mix code pages |
|
|
154
|
+
| Header copied from a template | Build headers from scratch | Keeps version, descriptors, key and code page, including in empty tables, without modelling obscure header fields |
|
|
155
|
+
| Always new files | Modify in place | The C library corrupted memory after ~50 in-place writes |
|
|
156
|
+
| One single-block entry per memo | Sub-allocated blocks | Correctness first; the size cost is measured below and planned for v2 |
|
|
157
|
+
| Source order by default, legacy order on request | Always re-sort | The legacy engine accepts the original order; the reproduced collation matched the real one in most but not all tables |
|
|
158
|
+
| Whole file in memory | Block-by-block streaming | Tables of tens of MB; revisit if hundreds appear |
|
|
159
|
+
|
|
160
|
+
Decisions that the specification left open (error-message language, how the
|
|
161
|
+
`.PX` entry count is defined, how the leak guard matches words, and so on) are
|
|
162
|
+
listed with their rationale in [docs/DECISIONES.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/docs/DECISIONES.md).
|
|
163
|
+
|
|
164
|
+
## 5. What was measured and how to reproduce it
|
|
165
|
+
|
|
166
|
+
Everything below runs on synthetic data produced by the repository itself.
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
pip install -e ".[dev]"
|
|
170
|
+
pytest # 100+ tests, including Hypothesis round-trip properties
|
|
171
|
+
python examples/benchmark.py # timings and .MB growth on a 50 000-row table
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Results on a laptop-class container (Python 3.12):
|
|
175
|
+
|
|
176
|
+
| Measurement | Value |
|
|
177
|
+
|---|---|
|
|
178
|
+
| Open and decode 50 000 mixed-type records | 1.3 s |
|
|
179
|
+
| Write `.DB` + `.PX` for those records on a template | 0.09 s |
|
|
180
|
+
| Verify the resulting two-level `.PX` | < 0.01 s |
|
|
181
|
+
| `.MB` written for 5 000 memos of ~150 bytes each | 20 MiB for 0.7 MiB of payload (×27) |
|
|
182
|
+
|
|
183
|
+
The test-suite covers every case of the original specification: headers with
|
|
184
|
+
and without the 4.0+ extension, CP437 and CP1252 text, every field type with
|
|
185
|
+
sign, null versus zero, unreadable BCD, records beyond the declared count,
|
|
186
|
+
memos in the three storage forms, missing `.MB`, encrypted tables with and
|
|
187
|
+
without a decryptor, incoherent headers, round-trips with memos and index,
|
|
188
|
+
multi-level indexes, legacy collation (`NA, nz, Ñu, Oa, ob`), duplicate keys,
|
|
189
|
+
empty tables, autoincrement, secondary indexes left untouched, the index
|
|
190
|
+
verifier and the profile / CLI output.
|
|
191
|
+
|
|
192
|
+
In the original project the same design was validated against about 120 real
|
|
193
|
+
tables (roughly 225 000 rows) with zero byte or value differences versus the
|
|
194
|
+
C library, and the verifier accepted 106 of 108 original indexes (the other two
|
|
195
|
+
had counts corrupted by interrupted writes). Those tables cannot be published;
|
|
196
|
+
the figures are given only as context.
|
|
197
|
+
|
|
198
|
+
## 6. Known limits
|
|
199
|
+
|
|
200
|
+
- **`.MB` growth.** Every memo gets at least one 4 KiB block. Sub-allocated
|
|
201
|
+
(type 3) blocks are read but never written. Planned for v2.
|
|
202
|
+
- **Not yet opened in a real BDE.** Round-trips are validated with this
|
|
203
|
+
library's reader; opening the written files in the legacy engine is pending.
|
|
204
|
+
- **Secondary indexes** (`.Xnn`, `.Ynn`, `.XGn`, `.YGn`) are detected and
|
|
205
|
+
reported, never regenerated.
|
|
206
|
+
- **Encrypted tables are never written**, and decryption requires an external
|
|
207
|
+
component.
|
|
208
|
+
- Index entry counts are limited to 32 767 per entry, BCD to 32 digits; text
|
|
209
|
+
that does not fit a field is an error, not a truncation.
|
|
210
|
+
- Whole files are loaded in memory.
|
|
211
|
+
|
|
212
|
+
## 7. How it was built
|
|
213
|
+
|
|
214
|
+
Three-phase clean-room process. Phase 1 produced a specification of the format,
|
|
215
|
+
behaviour and invariants from the original (private) code, without copying
|
|
216
|
+
code, identifiers or data. Phase 2, this repository, was implemented in a
|
|
217
|
+
separate session with access to that specification and to public documentation
|
|
218
|
+
only, using an AI coding assistant as pair programmer; every decision not fixed
|
|
219
|
+
by the specification is documented. Phase 3 is an independent review comparing
|
|
220
|
+
this repository against the original before publication. The CI runs a leak
|
|
221
|
+
guard (`tests/test_no_leaks.py`) over the tree and the git history, plus
|
|
222
|
+
`gitleaks`.
|
|
223
|
+
|
|
224
|
+
## 8. Attribution and licence
|
|
225
|
+
|
|
226
|
+
Patterns developed during my work at Intervalor Data and published with permission. Independent clean-room implementation with synthetic data; it contains no code or data from the company or its clients.
|
|
227
|
+
|
|
228
|
+
MIT licence. See [LICENSE](https://github.com/arielgiamportone/paradox-db-py/blob/main/LICENSE), [CHANGELOG.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/CHANGELOG.md) and
|
|
229
|
+
[CONTRIBUTING.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/CONTRIBUTING.md).
|
|
@@ -0,0 +1,200 @@
|
|
|
1
|
+
# paradox-db-py
|
|
2
|
+
|
|
3
|
+
[](https://github.com/arielgiamportone/paradox-db-py/actions/workflows/ci.yml)
|
|
4
|
+

|
|
5
|
+

|
|
6
|
+
|
|
7
|
+
**Lector y escritor en Python puro de tablas Paradox** (`.DB` datos, `.MB`
|
|
8
|
+
memos y BLOB, `.PX` índice primario), versiones 3.0 a 7. Sin código nativo, sin
|
|
9
|
+
dependencias copyleft, sin conversiones silenciosas: cada celda vuelve como
|
|
10
|
+
`OK`, `NULL` o `INVALID` junto con sus bytes crudos.
|
|
11
|
+
|
|
12
|
+
*English version: [README.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/README.md).*
|
|
13
|
+
|
|
14
|
+
> Patrones desarrollados durante mi trabajo en Intervalor Data y publicados con autorización. Implementación independiente (clean-room) con datos sintéticos; no contiene código ni datos de la empresa ni de sus clientes.
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## 1. El problema
|
|
19
|
+
|
|
20
|
+
Un sistema de gestión de activos fijos escrito en Delphi sobre el Borland
|
|
21
|
+
Database Engine guardaba décadas de histórico en tablas Paradox. Modernizarlo
|
|
22
|
+
exigía dos cosas a la vez:
|
|
23
|
+
|
|
24
|
+
1. **Extraer** todos los registros sin pérdida. No «casi todos»: el motor
|
|
25
|
+
antiguo dejaba registros en los bloques de datos que la cabecera nunca
|
|
26
|
+
contó, y columnas enteras de importes en BCD que ya no se decodificaban.
|
|
27
|
+
2. **Devolver tablas** a la aplicación antigua durante una transición larga,
|
|
28
|
+
para que el sistema viejo y el nuevo convivieran meses.
|
|
29
|
+
|
|
30
|
+
El ecosistema Python solo ofrecía *bindings* sobre una librería en C (GPL) que
|
|
31
|
+
en la práctica es de solo lectura, impone una codificación global de texto,
|
|
32
|
+
se salta los registros que la cabecera no declara y corrompe su *heap* tras
|
|
33
|
+
unas decenas de escrituras. Para un producto *on-premise* nada de eso servía.
|
|
34
|
+
|
|
35
|
+
## 2. Restricciones
|
|
36
|
+
|
|
37
|
+
| Restricción | Consecuencia en el diseño |
|
|
38
|
+
|---|---|
|
|
39
|
+
| Se distribuye *on-premise* dentro de un producto comercial | Python puro, licencia MIT, cero dependencias en tiempo de ejecución |
|
|
40
|
+
| Las tablas reales contienen datos personales y patrimoniales de terceros | Todas las *fixtures* las genera `paradoxdb.synthetic`; el repo no contiene ningún `.DB` |
|
|
41
|
+
| El cifrado solo está implementado en código GPL | El descifrado se delega a un `Decryptor` externo opcional; la librería solo decodifica bytes en claro |
|
|
42
|
+
| El motor antiguo debe seguir abriendo lo que escribimos | El escritor copia una cabecera existente (plantilla) y solo reescribe contadores; produce ficheros nuevos, nunca modifica en sitio |
|
|
43
|
+
| Páginas de códigos mezcladas en un mismo conjunto (núcleo CP437, tablas posteriores CP1252) | Codepage por tabla, leída de la cabecera, nunca global |
|
|
44
|
+
| Importes contables y BCD de 32 dígitos | `Decimal` en todas partes; `float` solo como representación en disco |
|
|
45
|
+
|
|
46
|
+
## 3. Arquitectura
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
paradoxdb/
|
|
50
|
+
├── fields.py FieldType, FieldDef, páginas de códigos, tamaños físicos
|
|
51
|
+
├── numeric.py codificaciones que preservan el orden: entero y doble Paradox, BCD
|
|
52
|
+
├── values.py CellValue(state, value, raw_bytes, reason); decode_value; encode_*
|
|
53
|
+
├── header.py TableHeader, parse_header, patch_counters (0x06, 0x0A-0x10, 0x49)
|
|
54
|
+
├── blocks.py recorrido de la cadena de bloques (todos los registros, anomalías) y empaquetado
|
|
55
|
+
├── memo.py MemoPointer, MemoFile (bloques de clase 2 y 3), MemoWriter (clase 2)
|
|
56
|
+
├── table.py open_table, ParadoxTable, Record, protocolo Decryptor
|
|
57
|
+
├── writer.py write_table sobre plantilla, WriteRecord, legacy_sort_key
|
|
58
|
+
├── index.py DataBlock, read_data_blocks, build_index, verify_index
|
|
59
|
+
├── profile.py profile_table, export_csv, export_json
|
|
60
|
+
├── cli.py paradoxdb profile | export | verify-index
|
|
61
|
+
└── synthetic.py fábrica de tablas usada por tests y ejemplos
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Lectura:
|
|
65
|
+
|
|
66
|
+
```
|
|
67
|
+
bytes .DB ──► parse_header ──► iter_blocks ──► decode_value por campo ──► Record
|
|
68
|
+
│ ▲
|
|
69
|
+
└── codepage, descriptores └── MemoFile (.MB hermano, cualquier mayúscula)
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Escritura:
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
cabecera .DB plantilla ──copia──► parchear contadores ──► pack_blocks(registros) ──► .DB nuevo
|
|
76
|
+
primer bloque del .MB plantilla ──► MemoWriter (una entrada de bloque único por memo) ──► .MB nuevo
|
|
77
|
+
cabecera .PX plantilla ──► build_index(read_data_blocks(.DB nuevo)) ──► .PX nuevo
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### Inicio rápido
|
|
81
|
+
|
|
82
|
+
```python
|
|
83
|
+
from paradoxdb import open_table
|
|
84
|
+
|
|
85
|
+
table = open_table("ACTIVOS.DB") # encuentra también activos.mb / ACTIVOS.MB
|
|
86
|
+
print(table.header.version_label, table.header.codec)
|
|
87
|
+
|
|
88
|
+
for record in table.records(): # todos los registros de todos los bloques
|
|
89
|
+
amount = record["Importe"] # búsqueda de campo sin distinguir mayúsculas
|
|
90
|
+
if amount.is_ok:
|
|
91
|
+
total = amount.value # Decimal, date, int, bool, str, bytes
|
|
92
|
+
else:
|
|
93
|
+
print(amount.state, amount.reason, amount.raw_bytes.hex())
|
|
94
|
+
|
|
95
|
+
print(table.block_record_count(), table.declared_record_count, table.header_mismatch)
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
from paradoxdb import WriteRecord, write_table, verify_index, read_data_blocks
|
|
100
|
+
|
|
101
|
+
records = [
|
|
102
|
+
WriteRecord(r.raw_bytes, {3: r["Nota"].value.encode(table.codec)}) for r in table.records()
|
|
103
|
+
]
|
|
104
|
+
result = write_table("ACTIVOS.DB", "salida/ACTIVOS.DB", records, sort=False)
|
|
105
|
+
assert verify_index(result.px_path, read_data_blocks(result.db_path)) == []
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
paradoxdb profile ./tablas --output perfil.json # agregados, SHA-256, descuadres
|
|
110
|
+
paradoxdb export ./tablas/ACTIVOS.DB --format csv # valor, estado y motivo por celda
|
|
111
|
+
paradoxdb verify-index ./tablas/ACTIVOS.DB # salida 1 si el .PX es incoherente
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
## 4. Decisiones y alternativas descartadas
|
|
115
|
+
|
|
116
|
+
| Decisión | Alternativa descartada | Motivo |
|
|
117
|
+
|---|---|---|
|
|
118
|
+
| Lector propio en Python puro | *Bindings* de la librería C | Licencia GPL, codificación global, salta registros no declarados, sin garantía de 64 bits en Linux |
|
|
119
|
+
| Descifrado delegado a un componente opcional | Reimplementar el cifrado | La única implementación pública es GPL; las tablas cifradas son una minoría |
|
|
120
|
+
| Estado explícito por celda más bytes crudos | `None` / `0` ante la duda | La versión anterior convertía importes basura y fechas centinela en `None`: pérdida silenciosa a gran escala |
|
|
121
|
+
| `Decimal` para dobles y BCD | `float` | Importes contables y BCD de 32 dígitos; la paridad aritmética con el legado la hace el consumidor |
|
|
122
|
+
| Codepage por tabla | `latin1` o `cp1252` global | `latin1` acepta cualquier byte pero lee mal `ñ` / `º` de tablas CP437; los conjuntos mezclan páginas |
|
|
123
|
+
| Cabecera copiada de una plantilla | Construir cabeceras desde cero | Conserva versión, descriptores, clave y codepage, también en tablas vacías, sin modelar campos oscuros |
|
|
124
|
+
| Siempre ficheros nuevos | Modificar en sitio | La librería C corrompía memoria tras ~50 escrituras en sitio |
|
|
125
|
+
| Una entrada de bloque único por memo | Bloques subasignados | Correctitud primero; el coste en tamaño se mide abajo y se planifica para v2 |
|
|
126
|
+
| Orden de origen por defecto, orden legado bajo demanda | Reordenar siempre | El motor antiguo acepta el orden original; la colación reproducida coincidió con la real en la mayoría de tablas, no en todas |
|
|
127
|
+
| Fichero completo en memoria | Lectura por bloques | Tablas de decenas de MB; revisar si aparecen cientos |
|
|
128
|
+
|
|
129
|
+
Las decisiones que la especificación dejaba abiertas (idioma de los mensajes,
|
|
130
|
+
definición del recuento de entradas del `.PX`, cómo compara palabras el test de
|
|
131
|
+
fugas, etc.) están en [docs/DECISIONES.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/docs/DECISIONES.md) con su motivo.
|
|
132
|
+
|
|
133
|
+
## 5. Qué se midió y cómo reproducirlo
|
|
134
|
+
|
|
135
|
+
Todo lo siguiente se ejecuta sobre datos sintéticos generados por el propio
|
|
136
|
+
repositorio.
|
|
137
|
+
|
|
138
|
+
```bash
|
|
139
|
+
pip install -e ".[dev]"
|
|
140
|
+
pytest # más de 100 tests, con propiedades de ida y vuelta (Hypothesis)
|
|
141
|
+
python examples/benchmark.py # tiempos y crecimiento del .MB con una tabla de 50 000 filas
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
Resultados en un contenedor de clase portátil (Python 3.12):
|
|
145
|
+
|
|
146
|
+
| Medición | Valor |
|
|
147
|
+
|---|---|
|
|
148
|
+
| Abrir y decodificar 50 000 registros de varios campos | 1,3 s |
|
|
149
|
+
| Escribir `.DB` + `.PX` de esos registros sobre plantilla | 0,09 s |
|
|
150
|
+
| Verificar el `.PX` resultante de dos niveles | < 0,01 s |
|
|
151
|
+
| `.MB` escrito para 5 000 memos de ~150 bytes | 20 MiB para 0,7 MiB de contenido (×27) |
|
|
152
|
+
|
|
153
|
+
La batería de tests cubre todos los casos de la especificación original:
|
|
154
|
+
cabeceras con y sin extensión 4.0+, texto en CP437 y CP1252, cada clase de
|
|
155
|
+
campo con signo, nulo frente a cero, BCD ilegible, registros más allá del
|
|
156
|
+
recuento declarado, memos en sus tres formas, `.MB` ausente, tablas cifradas
|
|
157
|
+
con y sin descifrador, cabeceras incoherentes, ida y vuelta con memos e
|
|
158
|
+
índice, índices de varios niveles, colación legada (`NA, nz, Ñu, Oa, ob`),
|
|
159
|
+
claves repetidas, tablas vacías, autoincremento, índices secundarios sin
|
|
160
|
+
tocar, el verificador de índices y la salida del perfil y la CLI.
|
|
161
|
+
|
|
162
|
+
En el proyecto original el mismo diseño se validó contra unas 120 tablas
|
|
163
|
+
reales (unas 225 000 filas) con cero diferencias de bytes o de valor frente a
|
|
164
|
+
la librería C, y el verificador aceptó 106 de 108 índices originales (los otros
|
|
165
|
+
dos tenían recuentos corrompidos por escrituras interrumpidas). Esas tablas no
|
|
166
|
+
pueden publicarse; las cifras se dan solo como contexto.
|
|
167
|
+
|
|
168
|
+
## 6. Límites conocidos
|
|
169
|
+
|
|
170
|
+
- **Crecimiento del `.MB`.** Cada memo ocupa al menos un bloque de 4 KiB. Los
|
|
171
|
+
bloques subasignados (clase 3) se leen pero no se escriben. Previsto para v2.
|
|
172
|
+
- **Aún no abierto en un BDE real.** Las idas y vueltas se validan con el
|
|
173
|
+
propio lector; abrir los ficheros escritos en el motor antiguo está pendiente.
|
|
174
|
+
- **Índices secundarios** (`.Xnn`, `.Ynn`, `.XGn`, `.YGn`): se detectan e
|
|
175
|
+
informan, nunca se regeneran.
|
|
176
|
+
- **Las tablas cifradas no se escriben**, y descifrarlas exige un componente
|
|
177
|
+
externo.
|
|
178
|
+
- Recuento de índice ≤ 32 767 por entrada, BCD ≤ 32 dígitos; el texto que no
|
|
179
|
+
cabe en un campo es un error, no un truncado.
|
|
180
|
+
- Los ficheros se cargan íntegros en memoria.
|
|
181
|
+
|
|
182
|
+
## 7. Cómo se construyó
|
|
183
|
+
|
|
184
|
+
Proceso *clean-room* en tres fases. La fase 1 produjo una especificación del
|
|
185
|
+
formato, el comportamiento y los invariantes a partir del código original
|
|
186
|
+
(privado), sin copiar código, identificadores ni datos. La fase 2, este
|
|
187
|
+
repositorio, se implementó en una sesión separada con acceso solo a esa
|
|
188
|
+
especificación y a documentación pública, usando un asistente de programación
|
|
189
|
+
con IA como pareja de programación; cada decisión no fijada por la
|
|
190
|
+
especificación está documentada. La fase 3 es una revisión independiente que
|
|
191
|
+
compara este repositorio con el original antes de publicarlo. La CI ejecuta
|
|
192
|
+
un test de fugas (`tests/test_no_leaks.py`) sobre el árbol y el historial de
|
|
193
|
+
git, además de `gitleaks`.
|
|
194
|
+
|
|
195
|
+
## 8. Atribución y licencia
|
|
196
|
+
|
|
197
|
+
Patrones desarrollados durante mi trabajo en Intervalor Data y publicados con autorización. Implementación independiente (clean-room) con datos sintéticos; no contiene código ni datos de la empresa ni de sus clientes.
|
|
198
|
+
|
|
199
|
+
Licencia MIT. Véanse [LICENSE](https://github.com/arielgiamportone/paradox-db-py/blob/main/LICENSE), [CHANGELOG.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/CHANGELOG.md) y
|
|
200
|
+
[CONTRIBUTING.md](https://github.com/arielgiamportone/paradox-db-py/blob/main/CONTRIBUTING.md).
|