durablewrite 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- durablewrite-0.2.0/.gitattributes +1 -0
- durablewrite-0.2.0/.github/FUNDING.yml +1 -0
- durablewrite-0.2.0/.gitignore +12 -0
- durablewrite-0.2.0/CHANGELOG.md +66 -0
- durablewrite-0.2.0/CONTRIBUTING.md +24 -0
- durablewrite-0.2.0/GUIDE.md +207 -0
- durablewrite-0.2.0/IMPROVEMENTS-round.md +93 -0
- durablewrite-0.2.0/LICENSE +21 -0
- durablewrite-0.2.0/PKG-INFO +171 -0
- durablewrite-0.2.0/README.md +144 -0
- durablewrite-0.2.0/pyproject.toml +56 -0
- durablewrite-0.2.0/requirements-dev.txt +3 -0
- durablewrite-0.2.0/scripts/check.ps1 +11 -0
- durablewrite-0.2.0/scripts/release.ps1 +28 -0
- durablewrite-0.2.0/src/durablewrite/__init__.py +26 -0
- durablewrite-0.2.0/src/durablewrite/__main__.py +9 -0
- durablewrite-0.2.0/src/durablewrite/_atomic.py +141 -0
- durablewrite-0.2.0/src/durablewrite/_cleanup.py +63 -0
- durablewrite-0.2.0/src/durablewrite/_cli.py +178 -0
- durablewrite-0.2.0/src/durablewrite/_errors.py +40 -0
- durablewrite-0.2.0/src/durablewrite/_fs.py +134 -0
- durablewrite-0.2.0/src/durablewrite/_helpers.py +132 -0
- durablewrite-0.2.0/src/durablewrite/_options.py +79 -0
- durablewrite-0.2.0/src/durablewrite/_replace.py +94 -0
- durablewrite-0.2.0/src/durablewrite/py.typed +0 -0
- durablewrite-0.2.0/tests/_payload.py +27 -0
- durablewrite-0.2.0/tests/_worker.py +55 -0
- durablewrite-0.2.0/tests/conftest.py +54 -0
- durablewrite-0.2.0/tests/test_atomic.py +118 -0
- durablewrite-0.2.0/tests/test_cli.py +183 -0
- durablewrite-0.2.0/tests/test_concurrency.py +48 -0
- durablewrite-0.2.0/tests/test_contract.py +116 -0
- durablewrite-0.2.0/tests/test_crash.py +50 -0
- durablewrite-0.2.0/tests/test_e2e_wheel.py +107 -0
- durablewrite-0.2.0/tests/test_hardening.py +168 -0
- durablewrite-0.2.0/tests/test_json_cleanup.py +90 -0
- durablewrite-0.2.0/tests/test_metadata.py +163 -0
- durablewrite-0.2.0/tests/test_options.py +61 -0
- durablewrite-0.2.0/tests/test_overwrite.py +74 -0
- durablewrite-0.2.0/tests/test_sharing.py +103 -0
|
@@ -0,0 +1 @@
|
|
|
1
|
+
* text=auto eol=lf
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
buy_me_a_coffee: jlianesglrs
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
|
|
5
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.2.0]
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `write_json_atomic(path, obj, ...)`: serialises before touching the disk and
|
|
14
|
+
writes UTF-8 JSON with a trailing newline and LF line endings. `NaN` and
|
|
15
|
+
infinities are refused unless `allow_nan=True`.
|
|
16
|
+
- `fsync="none" | "file" | "full"` on every writer, with the `FsyncLevel` type.
|
|
17
|
+
The `durable` and `fsync_dir` flags keep working.
|
|
18
|
+
- `overwrite=False`: atomic create-if-absent (`link` on POSIX, `MoveFileExW`
|
|
19
|
+
without the replace flag on Windows).
|
|
20
|
+
- `preserve_mode` (default on), `permissions` and `preserve_owner` options.
|
|
21
|
+
- `follow_symlinks` (default on): writing through a symlink updates the file it
|
|
22
|
+
points to and keeps the link.
|
|
23
|
+
- `RetryPolicy`, `DEFAULT_RETRY` and `NO_RETRY`: bounded backoff when a Windows
|
|
24
|
+
sharing or lock violation blocks the rename.
|
|
25
|
+
- `DurableWriteError` (an `OSError`), `SharingViolationError` (a
|
|
26
|
+
`PermissionError`) and `DirectorySyncError` (the file was replaced but its
|
|
27
|
+
directory could not be fsynced).
|
|
28
|
+
- `cleanup_temp_files(directory, older_than=..., target=...)` for temp files
|
|
29
|
+
left by killed writers.
|
|
30
|
+
- `durablewrite` command line tool (also `python -m durablewrite`): writes
|
|
31
|
+
standard input to a file atomically, with `--append`, `--json`, `--fsync`,
|
|
32
|
+
`--no-overwrite`, `--permissions`, `--no-follow-symlinks` and `--attempts`.
|
|
33
|
+
Exit status 1 on errors, 3 when the file was replaced but not fully synced.
|
|
34
|
+
- `GUIDE.md` with the write protocol, per-platform guarantees and sources.
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
- Temp files are named `.<name>.<8 hex digits>.dwtmp` so they can be cleaned up
|
|
39
|
+
safely.
|
|
40
|
+
- Writing through a symlink no longer replaces the link with a regular file
|
|
41
|
+
(pass `follow_symlinks=False` for the old behaviour).
|
|
42
|
+
|
|
43
|
+
### Fixed
|
|
44
|
+
|
|
45
|
+
- New files were created with mode `0600` and replaced files lost their mode;
|
|
46
|
+
new files now honour the umask like `open()` and replaced files keep their
|
|
47
|
+
permission bits.
|
|
48
|
+
- Very long file names no longer overflow the name limit when the temp name is
|
|
49
|
+
built.
|
|
50
|
+
- Writing to an existing directory raises `IsADirectoryError` before any temp
|
|
51
|
+
file is created, and a path ending with a separator raises `ValueError`.
|
|
52
|
+
- When the final mode is restricted (explicit `permissions` or a replaced
|
|
53
|
+
private file) the temp file is `0600` while it is written.
|
|
54
|
+
- On Windows, removal of a temp file that is briefly locked is retried.
|
|
55
|
+
|
|
56
|
+
## [0.1.0]
|
|
57
|
+
|
|
58
|
+
### Added
|
|
59
|
+
|
|
60
|
+
- `atomic_write(path, mode, *, encoding, newline, durable, fsync_dir)` context
|
|
61
|
+
manager: writes to a temp file in the target directory, flushes and fsyncs it,
|
|
62
|
+
then moves it onto the target with `os.replace` and fsyncs the directory.
|
|
63
|
+
- `write_text_atomic(path, text, encoding, *, newline, durable, fsync_dir)` and
|
|
64
|
+
`write_bytes_atomic(path, data, *, durable, fsync_dir)` one-shot helpers.
|
|
65
|
+
- Durable by default (file and directory fsync), both switchable off.
|
|
66
|
+
- Correct on Windows and POSIX; typed, zero runtime dependencies.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Thanks for taking the time. This project keeps a small, predictable shape.
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
pip install -r requirements-dev.txt
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Before you open a pull request
|
|
12
|
+
|
|
13
|
+
- Keep public API in `src/durablewrite/__init__.py`; internals live in `_`-prefixed modules.
|
|
14
|
+
- Add a test for the behaviour you change. Tests run against the real filesystem and
|
|
15
|
+
real child processes; nothing on the write path is mocked. Only the retry `sleep` is
|
|
16
|
+
injected.
|
|
17
|
+
- Run `scripts/check.ps1` (format, lint, types, tests) and make sure it passes. The
|
|
18
|
+
multi-process and packaging tests are marked `slow`; `pytest -m "not slow"` skips
|
|
19
|
+
them while iterating, but the full suite must pass before a commit.
|
|
20
|
+
- The crash test is seeded: set `DURABLEWRITE_CRASH_SEED` and
|
|
21
|
+
`DURABLEWRITE_CRASH_ITERATIONS` to explore other interleavings.
|
|
22
|
+
- One logical change per commit, imperative subject line.
|
|
23
|
+
|
|
24
|
+
Open an issue first if the change is large or shifts the public API.
|
|
@@ -0,0 +1,207 @@
|
|
|
1
|
+
# durablewrite guide
|
|
2
|
+
|
|
3
|
+
This guide explains what `durablewrite` does on every write, what it guarantees
|
|
4
|
+
on each platform, and where those guarantees stop. Each claim cites its source
|
|
5
|
+
or the test in this repository that exercises it.
|
|
6
|
+
|
|
7
|
+
## The write protocol
|
|
8
|
+
|
|
9
|
+
For `atomic_write(path)` and every helper built on it:
|
|
10
|
+
|
|
11
|
+
1. **Resolve the target.** If `path` is a symlink and `follow_symlinks=True`
|
|
12
|
+
(the default), the file it points to becomes the target, so the link keeps
|
|
13
|
+
working. With `follow_symlinks=False` the link itself is replaced by a
|
|
14
|
+
regular file. An existing directory is rejected with `IsADirectoryError`.
|
|
15
|
+
2. **Create a temp file in the target's own directory**, named
|
|
16
|
+
`.<name>.<8 hex digits>.dwtmp`, with `O_CREAT | O_EXCL`. A new file is created
|
|
17
|
+
with mode `0o666`, so the process umask applies exactly as it does for
|
|
18
|
+
`open()`. When the final mode comes from `permissions` or from the replaced
|
|
19
|
+
file, the temp file starts as `0o600` and is widened only after the data is
|
|
20
|
+
written, so restricted content is never readable by others in between.
|
|
21
|
+
Names longer than 48 characters keep a readable head plus a hash of the full
|
|
22
|
+
name, so the temp name stays within the 255-byte limit. Same directory
|
|
23
|
+
means same filesystem, which `rename` needs to be atomic: Python documents
|
|
24
|
+
that `os.replace` "may fail if src and dst are on different filesystems" [1].
|
|
25
|
+
3. **Write** your data into it.
|
|
26
|
+
4. **Set metadata.** With `preserve_owner=True` (POSIX) the uid and gid of the
|
|
27
|
+
old file are copied first, then the permission bits: either `permissions`
|
|
28
|
+
or, with `preserve_mode=True` (the default), the bits of the old file.
|
|
29
|
+
5. **Flush and fsync the temp file** (`fsync="file"` or `"full"`). This must
|
|
30
|
+
happen before the rename: otherwise the new name can reach disk before the
|
|
31
|
+
data it points to, which is how ext4 and XFS produced empty or zero-filled
|
|
32
|
+
files after a crash [3][4][5].
|
|
33
|
+
6. **Rename** the temp file onto the target with `os.replace`. With
|
|
34
|
+
`overwrite=False`, a create-only rename is used instead (see below).
|
|
35
|
+
7. **Fsync the parent directory** (`fsync="full"`, POSIX only) so the rename
|
|
36
|
+
itself is on disk. Linux `fsync(2)`: "Calling fsync() does not necessarily
|
|
37
|
+
ensure that the entry in the directory containing the file has also reached
|
|
38
|
+
disk. For that an explicit fsync() on a file descriptor for the directory is
|
|
39
|
+
also needed." [2][3]
|
|
40
|
+
|
|
41
|
+
If anything raises before step 6 completes, the temp file is removed and the
|
|
42
|
+
target is untouched. That includes exceptions inside your `with` block and
|
|
43
|
+
`KeyboardInterrupt`.
|
|
44
|
+
|
|
45
|
+
## What is guaranteed, per platform
|
|
46
|
+
|
|
47
|
+
| Promise | Linux / macOS (local fs) | Windows (local NTFS) |
|
|
48
|
+
| --- | --- | --- |
|
|
49
|
+
| Readers see the old or the complete new file, never a mix | Yes, `rename(2)` is atomic by POSIX [6] | Yes on NTFS in our tests; not documented as atomic by Microsoft [7] |
|
|
50
|
+
| A killed writer leaves the old or the new file, never a partial one | Yes (tested) | Yes (tested) |
|
|
51
|
+
| New content survives power loss once the call returns (`"full"`) | Linux: yes, file and directory are fsynced. macOS: see below | File data is flushed with `FlushFileBuffers` [8]; the rename is not explicitly flushed |
|
|
52
|
+
| Permission bits kept on replace | Yes (`preserve_mode`) | Not applicable; see below |
|
|
53
|
+
| Owner kept on replace | Opt-in (`preserve_owner`), needs privilege to change uid | No |
|
|
54
|
+
| Replace while another process has the target open | Yes; the reader keeps the old inode | Retried, then `SharingViolationError` |
|
|
55
|
+
| A reader can always open the target | Yes | No: an `open()` racing a rename can fail with `PermissionError` (never `FileNotFoundError`, never torn) |
|
|
56
|
+
| Replace a read-only target | Yes, if the directory is writable | No, `PermissionError` (not retried) |
|
|
57
|
+
|
|
58
|
+
The crash and concurrency rows are exercised by `tests/test_crash.py` and
|
|
59
|
+
`tests/test_concurrency.py`, which run on both Windows (NTFS) and Linux (ext4)
|
|
60
|
+
in this project's local gate.
|
|
61
|
+
|
|
62
|
+
### Process crash versus power loss
|
|
63
|
+
|
|
64
|
+
The kill-mid-write test terminates a writer process at random moments (seeded,
|
|
65
|
+
`DURABLEWRITE_CRASH_SEED`) and checks that the target is always a complete,
|
|
66
|
+
checksummed payload. That proves atomicity against a process crash. No portable
|
|
67
|
+
test can cut the power, so durability against power loss rests on the fsync
|
|
68
|
+
calls above and on the filesystem honouring them [3].
|
|
69
|
+
|
|
70
|
+
### Linux
|
|
71
|
+
|
|
72
|
+
`fsync="full"` follows the sequence recommended in "Ensuring data reaches disk"
|
|
73
|
+
[3]: fsync the file, rename, fsync the directory. Do not rely on ext4's
|
|
74
|
+
`auto_da_alloc` heuristic for "replace via rename" [4]; it is a mount option and
|
|
75
|
+
other filesystems such as XFS have no equivalent [5]. If the directory fsync
|
|
76
|
+
raises (some network and FUSE filesystems refuse it, and it fails when the
|
|
77
|
+
directory is writable but not readable), the new content is already in place
|
|
78
|
+
and `DirectorySyncError` is raised so you know the rename may not be durable
|
|
79
|
+
(`tests/test_hardening.py`).
|
|
80
|
+
|
|
81
|
+
### macOS
|
|
82
|
+
|
|
83
|
+
On macOS `fsync` hands the data to the drive but does not ask the drive to
|
|
84
|
+
flush its own write cache; Apple documents `F_FULLFSYNC` for that [9].
|
|
85
|
+
`durablewrite` calls `os.fsync` and does not issue `F_FULLFSYNC`, so on macOS
|
|
86
|
+
`"full"` protects against an OS crash but not necessarily against a power loss
|
|
87
|
+
with an unflushed drive cache.
|
|
88
|
+
|
|
89
|
+
### Windows
|
|
90
|
+
|
|
91
|
+
- `os.replace` is `MoveFileExW(src, dst, MOVEFILE_REPLACE_EXISTING)` [1][7].
|
|
92
|
+
Microsoft does not document it as atomic. On a local NTFS volume the rename is
|
|
93
|
+
a single metadata operation and our tests never observed a torn or missing
|
|
94
|
+
target. On FAT volumes and SMB shares the behaviour depends on the
|
|
95
|
+
filesystem or server and is not guaranteed.
|
|
96
|
+
- `os.fsync` calls the CRT `_commit`, which calls `FlushFileBuffers` [8][10].
|
|
97
|
+
- There is no directory fsync on Windows through the APIs Python exposes, so
|
|
98
|
+
`"full"` behaves like `"file"`. The data of the new file is flushed before
|
|
99
|
+
the rename; the rename itself becomes durable when NTFS commits its journal.
|
|
100
|
+
- `MoveFileExW` fails with `ERROR_ACCESS_DENIED`, `ERROR_SHARING_VIOLATION` or
|
|
101
|
+
`ERROR_LOCK_VIOLATION` while any process holds the target open without
|
|
102
|
+
`FILE_SHARE_DELETE` [11]. Python's `open()` does not pass that flag, and
|
|
103
|
+
antivirus scanners and the search indexer briefly open new files. These
|
|
104
|
+
errors are retried with bounded backoff (`RetryPolicy`, about 1.1 seconds by
|
|
105
|
+
default). A read-only target is never retried. When the budget runs out,
|
|
106
|
+
`SharingViolationError` is raised and the target is untouched
|
|
107
|
+
(`tests/test_sharing.py`). A directory ACL that denies the rename produces
|
|
108
|
+
the same error code and ends the same way.
|
|
109
|
+
- The temp file can itself be held open (an antivirus scan of the fresh file).
|
|
110
|
+
On every error path its removal is retried under the same `RetryPolicy`; if
|
|
111
|
+
it is still locked after that, it is left for `cleanup_temp_files`.
|
|
112
|
+
- Permission bits do not exist on Windows. `permissions` only toggles the
|
|
113
|
+
read-only attribute and `preserve_mode` does nothing. The new file inherits
|
|
114
|
+
the ACL of its directory; an explicit ACL on the old file is not copied.
|
|
115
|
+
|
|
116
|
+
### Why not `ReplaceFileW`
|
|
117
|
+
|
|
118
|
+
`ReplaceFileW` [12] keeps the replaced file's ACL, attributes and creation
|
|
119
|
+
time, but it requires the target to exist and it can fail half way: its
|
|
120
|
+
documented `ERROR_UNABLE_TO_MOVE_REPLACEMENT_2` result leaves the target under
|
|
121
|
+
the backup name. A primitive whose failure mode removes the target conflicts
|
|
122
|
+
with the core promise, so `durablewrite` uses `MoveFileExW`.
|
|
123
|
+
|
|
124
|
+
## Create only if absent
|
|
125
|
+
|
|
126
|
+
`overwrite=False` publishes the complete file only if nothing exists at the
|
|
127
|
+
target, and raises `FileExistsError` otherwise, without a check-then-act race:
|
|
128
|
+
|
|
129
|
+
- POSIX: `link(temp, target)` fails with `EEXIST` if the name is taken, then the
|
|
130
|
+
temp name is unlinked. The filesystem must support hard links.
|
|
131
|
+
- Windows: `MoveFileExW` without `MOVEFILE_REPLACE_EXISTING` fails with
|
|
132
|
+
`ERROR_ALREADY_EXISTS`.
|
|
133
|
+
|
|
134
|
+
`tests/test_overwrite.py` races six processes for one name and checks that
|
|
135
|
+
exactly one wins.
|
|
136
|
+
|
|
137
|
+
## Symlinks
|
|
138
|
+
|
|
139
|
+
The default (`follow_symlinks=True`) writes to the file the link points to and
|
|
140
|
+
creates the temp file beside that file, so the rename never crosses a
|
|
141
|
+
filesystem even when the link lives elsewhere. A dangling link creates its
|
|
142
|
+
target. `follow_symlinks=False` replaces the link with a regular file, which is
|
|
143
|
+
what plain `os.replace` does; on POSIX this also works when the link points to
|
|
144
|
+
a directory. The replaced link counts as a new file, so it does not inherit the
|
|
145
|
+
mode of whatever it pointed to.
|
|
146
|
+
|
|
147
|
+
## Side effects of replacing a file
|
|
148
|
+
|
|
149
|
+
Replacing a file creates a new file, by design. As a consequence:
|
|
150
|
+
|
|
151
|
+
- Other hard links to the old file keep the old content.
|
|
152
|
+
- The inode (POSIX) or file ID (Windows) changes. Programs that follow a file
|
|
153
|
+
by handle, such as `tail -f` without `-F`, keep reading the old file.
|
|
154
|
+
- Extended attributes, POSIX ACLs and Windows alternate data streams of the
|
|
155
|
+
old file are not copied.
|
|
156
|
+
|
|
157
|
+
## Leftover temp files
|
|
158
|
+
|
|
159
|
+
A writer that is killed between creating its temp file and renaming it cannot
|
|
160
|
+
clean up. `cleanup_temp_files(directory, older_than=3600)` removes only regular
|
|
161
|
+
files matching `.<name>.<8 hex digits>.dwtmp` older than the given age, by
|
|
162
|
+
modification time. A writer that keeps its temp file open for longer than that
|
|
163
|
+
age could lose it, so choose an age longer than your slowest write. With
|
|
164
|
+
`target=`, only temp files of that exact target name are considered. The crash test uses it to reclaim every temp file its
|
|
165
|
+
kills leave behind.
|
|
166
|
+
|
|
167
|
+
## Command line
|
|
168
|
+
|
|
169
|
+
`durablewrite TARGET` copies standard input into TARGET with the protocol
|
|
170
|
+
above, like `sponge` from moreutils. TARGET is replaced only after standard
|
|
171
|
+
input ends, so `sort f | durablewrite f` is safe on every platform, and
|
|
172
|
+
`durablewrite --append f < f` works on POSIX. On Windows the shell (and the
|
|
173
|
+
console-script launcher) keep a redirected file open without
|
|
174
|
+
`FILE_SHARE_DELETE`, which blocks the rename, so the command detects that
|
|
175
|
+
standard input is TARGET and refuses at once; pipe the file instead.
|
|
176
|
+
|
|
177
|
+
Exit status: 0 on success; 1 on any error, including a usage error, with TARGET
|
|
178
|
+
left as it was; 3 when TARGET was replaced but its directory could not be
|
|
179
|
+
fsynced. `--json` accepts only strict JSON (no `NaN` or `Infinity`). Each
|
|
180
|
+
`--append` run is atomic, but two concurrent runs on the same file can lose
|
|
181
|
+
one run's data, because each starts from the content it read.
|
|
182
|
+
|
|
183
|
+
## Sources
|
|
184
|
+
|
|
185
|
+
1. Python documentation, `os.replace` and `os.fsync`:
|
|
186
|
+
https://docs.python.org/3/library/os.html#os.replace
|
|
187
|
+
2. Linux `fsync(2)` manual page: https://man7.org/linux/man-pages/man2/fsync.2.html
|
|
188
|
+
3. Jeff Moyer, "Ensuring data reaches disk", LWN.net, 2011:
|
|
189
|
+
https://lwn.net/Articles/457667/
|
|
190
|
+
4. Linux kernel documentation, ext4 mount options (`auto_da_alloc`):
|
|
191
|
+
https://docs.kernel.org/admin-guide/ext4.html
|
|
192
|
+
5. XFS FAQ, "Why do I see binary NULLS in some files after recovery when I
|
|
193
|
+
unplugged the power?": https://xfs.org/index.php/XFS_FAQ
|
|
194
|
+
6. POSIX.1-2024, `rename`:
|
|
195
|
+
https://pubs.opengroup.org/onlinepubs/9799919799/functions/rename.html
|
|
196
|
+
7. Microsoft, `MoveFileExW`:
|
|
197
|
+
https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexw
|
|
198
|
+
8. Microsoft, `FlushFileBuffers`:
|
|
199
|
+
https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers
|
|
200
|
+
9. Apple, `fsync(2)` manual page (`F_FULLFSYNC`):
|
|
201
|
+
https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html
|
|
202
|
+
10. Microsoft, `_commit`:
|
|
203
|
+
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/commit
|
|
204
|
+
11. Microsoft, `CreateFileW` (`FILE_SHARE_DELETE`):
|
|
205
|
+
https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew
|
|
206
|
+
12. Microsoft, `ReplaceFileW`:
|
|
207
|
+
https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-replacefilew
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
# Improvements round: durablewrite
|
|
2
|
+
|
|
3
|
+
This round takes `durablewrite` from a correct but minimal context manager to a
|
|
4
|
+
complete atomic-write toolkit that is honest about what each platform does and
|
|
5
|
+
does not guarantee. Every promise below is backed by a test that runs on the real
|
|
6
|
+
filesystem: no mocked `os.replace`, no mocked `fsync`.
|
|
7
|
+
|
|
8
|
+
The public API grows additively. `atomic_write`, `write_text_atomic` and
|
|
9
|
+
`write_bytes_atomic` keep their signatures; new behaviour arrives through new
|
|
10
|
+
keyword arguments, new functions and a command-line tool.
|
|
11
|
+
|
|
12
|
+
## Ranked improvements
|
|
13
|
+
|
|
14
|
+
### 1. Correct permissions for new and replaced files (correctness, high)
|
|
15
|
+
`tempfile.mkstemp` creates files with mode `0600`, so 0.1.0 silently turned a
|
|
16
|
+
`0644` config or a `0755` script into a private, non-executable file on every
|
|
17
|
+
write. The temp file is now created with `O_CREAT | O_EXCL` and mode `0666`, so
|
|
18
|
+
the process umask applies exactly as it does for `open()`. When the target
|
|
19
|
+
already exists its permission bits are copied onto the temp file before the
|
|
20
|
+
rename (`preserve_mode=True`, the default). `permissions=0o600` sets explicit
|
|
21
|
+
bits, and `preserve_owner=True` copies uid and gid on POSIX.
|
|
22
|
+
|
|
23
|
+
### 2. Bounded retry on Windows sharing violations (robustness, high)
|
|
24
|
+
On Windows, `os.replace` is `MoveFileExW(MOVEFILE_REPLACE_EXISTING)`, which fails
|
|
25
|
+
with `ERROR_ACCESS_DENIED` or `ERROR_SHARING_VIOLATION` while another process
|
|
26
|
+
holds the target open without `FILE_SHARE_DELETE`. Python's own `open()` does not
|
|
27
|
+
pass that flag, and antivirus scanners and the search indexer open freshly
|
|
28
|
+
written files for a moment. A frozen `RetryPolicy` (attempts, initial delay,
|
|
29
|
+
multiplier, cap, injectable `sleep`) retries only those Windows error codes, never
|
|
30
|
+
a read-only target or a POSIX `EACCES`, and raises a typed
|
|
31
|
+
`SharingViolationError` (a `PermissionError`) when the budget runs out. Removal
|
|
32
|
+
of the temp file is retried under the same policy.
|
|
33
|
+
|
|
34
|
+
### 3. Explicit fsync levels (clarity, high)
|
|
35
|
+
`fsync="none" | "file" | "full"` replaces the two loosely related booleans with
|
|
36
|
+
one named choice. `full` (the default) fsyncs the temp file before the rename and
|
|
37
|
+
the parent directory after it, which is what ext4 and XFS need for the rename to
|
|
38
|
+
survive power loss. The old `durable` and `fsync_dir` flags keep working. The
|
|
39
|
+
docs state plainly that Windows has no directory fsync and that macOS `fsync`
|
|
40
|
+
does not flush the drive cache.
|
|
41
|
+
|
|
42
|
+
### 4. JSON helper (convenience, high)
|
|
43
|
+
`write_json_atomic(path, obj, ...)` serialises first, so a value that cannot be
|
|
44
|
+
encoded raises before any file is created, then writes atomically with a trailing
|
|
45
|
+
newline. Defaults suit config files: `indent=2`, UTF-8, `ensure_ascii=False`.
|
|
46
|
+
|
|
47
|
+
### 5. Atomic create-if-absent (`overwrite=False`) (capability, medium)
|
|
48
|
+
Publishes the complete file only if nothing exists at the target, without a
|
|
49
|
+
check-then-act race: `os.link` on POSIX and a plain `MoveFileExW` without the
|
|
50
|
+
replace flag on Windows both fail atomically when the name is taken. Useful for
|
|
51
|
+
one-time initialisation and lock-free "first writer wins" files.
|
|
52
|
+
|
|
53
|
+
### 6. Documented symlink policy (correctness, medium)
|
|
54
|
+
Writing through a symlink now updates the file the link points to and keeps the
|
|
55
|
+
link (`follow_symlinks=True`), which is what dotfile managers expect. The temp
|
|
56
|
+
file is created next to the real target, so the rename never crosses a
|
|
57
|
+
filesystem. `follow_symlinks=False` replaces the link itself with a regular file.
|
|
58
|
+
|
|
59
|
+
### 7. Stale temp file cleanup (operations, medium)
|
|
60
|
+
A process killed mid-write cannot remove its temp file. Temp files now carry a
|
|
61
|
+
distinctive `.dwtmp` suffix and `cleanup_temp_files(directory, older_than=...)`
|
|
62
|
+
removes only those, only once they are older than a chosen age (one hour by
|
|
63
|
+
default).
|
|
64
|
+
|
|
65
|
+
### 8. Typed errors and early validation (ergonomics, medium)
|
|
66
|
+
`DurableWriteError` is the base of errors the library raises itself and
|
|
67
|
+
subclasses `OSError`, so existing `except OSError` handlers keep working.
|
|
68
|
+
Writing to an existing directory raises `IsADirectoryError` before any temp file
|
|
69
|
+
exists. Long file names no longer overflow the 255-byte name limit when the temp
|
|
70
|
+
name is built.
|
|
71
|
+
|
|
72
|
+
### 9. A `sponge`-style CLI (tooling, medium)
|
|
73
|
+
`durablewrite TARGET` streams stdin into TARGET atomically, so
|
|
74
|
+
`sort data.txt | durablewrite data.txt` is safe. Flags cover `--append`,
|
|
75
|
+
`--json` (refuse to write invalid JSON), `--fsync`, `--no-overwrite`,
|
|
76
|
+
`--permissions` and `--attempts`. Errors go to stderr with exit status 1.
|
|
77
|
+
|
|
78
|
+
### 10. Guide with sources and real crash tests (trust, high)
|
|
79
|
+
A `GUIDE.md` explains the write protocol step by step with citations, and the
|
|
80
|
+
test suite proves the promise: a seeded kill-mid-write test that terminates a
|
|
81
|
+
writer process at random moments and checks the target is always a complete old
|
|
82
|
+
or complete new payload, a multi-process concurrent writer test, a held-open
|
|
83
|
+
reader test on Windows, and an end-to-end test that builds a wheel, installs it
|
|
84
|
+
into a throwaway virtual environment and drives the installed `durablewrite`
|
|
85
|
+
script.
|
|
86
|
+
|
|
87
|
+
## Out of scope this round
|
|
88
|
+
|
|
89
|
+
- `ReplaceFileW` on Windows. It preserves ACLs and attributes of the replaced
|
|
90
|
+
file, but it is documented to fail half way (`ERROR_UNABLE_TO_MOVE_REPLACEMENT_2`)
|
|
91
|
+
and requires the target to exist, so it would weaken the core promise.
|
|
92
|
+
- `F_FULLFSYNC` on macOS. Documented, not issued, because it cannot be exercised
|
|
93
|
+
by the local test suite.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Javier Lianes
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: durablewrite
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Durable, cross-platform atomic file writes: temp-same-dir plus os.replace plus file and dir fsync
|
|
5
|
+
Project-URL: Homepage, https://lianes8studio.com
|
|
6
|
+
Project-URL: Repository, https://github.com/JaviLianes8/durablewrite
|
|
7
|
+
Project-URL: Issues, https://github.com/JaviLianes8/durablewrite/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/JaviLianes8/durablewrite/blob/main/CHANGELOG.md
|
|
9
|
+
Project-URL: LinkedIn, https://www.linkedin.com/in/jlianes/
|
|
10
|
+
Project-URL: Funding, https://buymeacoffee.com/jlianesglrs
|
|
11
|
+
Author-email: Javier Lianes <javier@lianes8studio.com>
|
|
12
|
+
License-Expression: MIT
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Keywords: atomic,crash-safe,durable,file,fsync,os.replace,write
|
|
15
|
+
Classifier: Development Status :: 4 - Beta
|
|
16
|
+
Classifier: Intended Audience :: Developers
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Typing :: Typed
|
|
19
|
+
Requires-Python: >=3.10
|
|
20
|
+
Provides-Extra: dev
|
|
21
|
+
Requires-Dist: build; extra == 'dev'
|
|
22
|
+
Requires-Dist: mypy; extra == 'dev'
|
|
23
|
+
Requires-Dist: pytest; extra == 'dev'
|
|
24
|
+
Requires-Dist: ruff; extra == 'dev'
|
|
25
|
+
Requires-Dist: twine; extra == 'dev'
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
|
|
28
|
+
# durablewrite
|
|
29
|
+
|
|
30
|
+
Durable, cross-platform atomic file writes for Python. Data goes to a temp file
|
|
31
|
+
in the same directory, is flushed and fsynced, then moved onto the target with
|
|
32
|
+
`os.replace`, and the directory is fsynced. A reader never sees a half-written
|
|
33
|
+
file, and a crash leaves either the previous version or the complete new one.
|
|
34
|
+
|
|
35
|
+
Typed, standard library only, Python 3.10+, Windows and POSIX.
|
|
36
|
+
|
|
37
|
+
## Install
|
|
38
|
+
|
|
39
|
+
```
|
|
40
|
+
pip install durablewrite
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Usage
|
|
44
|
+
|
|
45
|
+
```python
|
|
46
|
+
from durablewrite import atomic_write, write_bytes_atomic, write_json_atomic, write_text_atomic
|
|
47
|
+
|
|
48
|
+
# One-shot helpers.
|
|
49
|
+
write_text_atomic("notes.txt", "hello\n")
|
|
50
|
+
write_bytes_atomic("snapshot.bin", b"\x00\x01\x02")
|
|
51
|
+
write_json_atomic("config.json", {"ready": True}) # serialised before any file is touched
|
|
52
|
+
|
|
53
|
+
# Stream into a file: it is replaced only if the block finishes without an error.
|
|
54
|
+
with atomic_write("report.csv") as f:
|
|
55
|
+
for row in rows:
|
|
56
|
+
f.write(row + "\n")
|
|
57
|
+
|
|
58
|
+
with atomic_write("image.png", "wb") as f:
|
|
59
|
+
f.write(data)
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
### Options (on every writer)
|
|
63
|
+
|
|
64
|
+
| Option | Default | Meaning |
|
|
65
|
+
| --- | --- | --- |
|
|
66
|
+
| `fsync` | `"full"` | `"none"`, `"file"` (fsync the data) or `"full"` (data and directory) |
|
|
67
|
+
| `overwrite` | `True` | `False`: create only if absent, atomically, else `FileExistsError` |
|
|
68
|
+
| `preserve_mode` | `True` | keep the permission bits of the file being replaced (POSIX) |
|
|
69
|
+
| `permissions` | `None` | explicit permission bits, e.g. `0o600` |
|
|
70
|
+
| `preserve_owner` | `False` | keep uid and gid of the replaced file (POSIX) |
|
|
71
|
+
| `follow_symlinks` | `True` | write through a symlink and keep it, or replace the link itself |
|
|
72
|
+
| `retry` | `DEFAULT_RETRY` | bounded retry while Windows reports the target locked |
|
|
73
|
+
|
|
74
|
+
New files get `0o666` minus your umask, exactly like `open()`. When the final mode
|
|
75
|
+
comes from `permissions` or from the replaced file, the temp file is created as
|
|
76
|
+
`0o600`, so a secret is never readable by others while it is being written.
|
|
77
|
+
|
|
78
|
+
```python
|
|
79
|
+
from durablewrite import NO_RETRY, RetryPolicy, SharingViolationError, write_text_atomic
|
|
80
|
+
|
|
81
|
+
write_text_atomic("cache.txt", payload, fsync="none") # atomic, not durable
|
|
82
|
+
write_text_atomic("token", secret, permissions=0o600, overwrite=False)
|
|
83
|
+
|
|
84
|
+
try:
|
|
85
|
+
write_text_atomic("held.txt", "x", retry=RetryPolicy(max_attempts=20, max_delay=0.2))
|
|
86
|
+
except SharingViolationError as exc: # a PermissionError; target untouched
|
|
87
|
+
print("still locked after", exc.attempts, "attempts")
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
A writer that is killed mid-write cannot remove its temp file. Clean those up
|
|
91
|
+
at startup:
|
|
92
|
+
|
|
93
|
+
```python
|
|
94
|
+
from durablewrite import cleanup_temp_files
|
|
95
|
+
|
|
96
|
+
cleanup_temp_files("data/", older_than=3600) # only *.dwtmp files written by durablewrite
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
## Command line
|
|
100
|
+
|
|
101
|
+
`durablewrite TARGET` writes standard input to TARGET atomically, like
|
|
102
|
+
`sponge`. TARGET is replaced only after the input ends.
|
|
103
|
+
|
|
104
|
+
```
|
|
105
|
+
sort names.txt | durablewrite names.txt
|
|
106
|
+
jq '.debug = true' config.json | durablewrite --json config.json
|
|
107
|
+
echo "one more line" | durablewrite --append notes.txt
|
|
108
|
+
printf 's3cret' | durablewrite --permissions 600 --no-overwrite token.txt
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Options: `--append`, `--json` (refuse invalid JSON), `--fsync {none,file,full}`,
|
|
112
|
+
`--no-overwrite`, `--permissions OCTAL`, `--no-follow-symlinks`,
|
|
113
|
+
`--attempts N`. Errors go to standard error with exit status 1, and TARGET is
|
|
114
|
+
left as it was. If TARGET was replaced but its directory could not be fsynced,
|
|
115
|
+
the exit status is 3. `python -m durablewrite` works too.
|
|
116
|
+
|
|
117
|
+
On Windows, redirecting TARGET into the command (`durablewrite f < f`) is refused,
|
|
118
|
+
because the shell keeps the redirected file open and that blocks the rename; pipe
|
|
119
|
+
it instead (`type f | durablewrite f`). Concurrent `--append` runs on one file may
|
|
120
|
+
lose each other's lines: each run is atomic, but they do not merge.
|
|
121
|
+
|
|
122
|
+
## Guarantees and limits
|
|
123
|
+
|
|
124
|
+
The short version (details and sources in [GUIDE.md](GUIDE.md)):
|
|
125
|
+
|
|
126
|
+
- **Atomic for readers and against process crashes** on local filesystems,
|
|
127
|
+
Windows and POSIX. Exercised by a seeded test that kills a writer process at
|
|
128
|
+
random moments and by a multi-process concurrent writer test.
|
|
129
|
+
- **Durable against power loss** on Linux with `fsync="full"`. On macOS, `fsync`
|
|
130
|
+
does not flush the drive cache (`F_FULLFSYNC` is not issued). On Windows the
|
|
131
|
+
file data is flushed but a directory cannot be fsynced.
|
|
132
|
+
- **Windows**: `os.replace` is `MoveFileExW`, not documented as atomic by
|
|
133
|
+
Microsoft; tested on NTFS, not guaranteed on FAT or network shares. An open
|
|
134
|
+
handle elsewhere blocks the rename; it is retried for about a second by
|
|
135
|
+
default, then `SharingViolationError` is raised. While a rename is in flight,
|
|
136
|
+
a concurrent `open()` of the target can fail with `PermissionError`; readers
|
|
137
|
+
that poll should retry. A read-only target cannot be replaced on Windows.
|
|
138
|
+
- **After the rename**, a failing directory fsync raises `DirectorySyncError`:
|
|
139
|
+
the new content is in place, but the rename may not survive a power loss.
|
|
140
|
+
- Replacing a file creates a new one: other hard links keep the old content,
|
|
141
|
+
and extended attributes, ACLs and alternate data streams are not copied.
|
|
142
|
+
|
|
143
|
+
## Why
|
|
144
|
+
|
|
145
|
+
Writing a file in place can corrupt it: a crash, a full disk or a concurrent
|
|
146
|
+
reader can catch it half-written. The safe pattern (temp file in the same
|
|
147
|
+
directory, fsync, `os.replace`, fsync the directory) is well known but fiddly
|
|
148
|
+
to get right on every platform.
|
|
149
|
+
|
|
150
|
+
The options that existed each fall short:
|
|
151
|
+
|
|
152
|
+
- `atomicwrites`, the de-facto package, was **archived by its author in 2022**
|
|
153
|
+
and takes no fixes.
|
|
154
|
+
- `boltons.fileutils.atomic_save` has **no Windows support**.
|
|
155
|
+
- The `atomicfile` and `atomicwrite` packages on PyPI are **abandoned** (last
|
|
156
|
+
releases in 2020 and 2015) and neither is typed.
|
|
157
|
+
- `safewrite` is minimal and obscure.
|
|
158
|
+
|
|
159
|
+
`durablewrite` does the full sequence, keeps permissions, handles Windows
|
|
160
|
+
sharing violations, documents exactly what each platform guarantees, and ships
|
|
161
|
+
a command-line tool.
|
|
162
|
+
|
|
163
|
+
## License
|
|
164
|
+
|
|
165
|
+
MIT. See [LICENSE](LICENSE).
|
|
166
|
+
|
|
167
|
+
## Links
|
|
168
|
+
|
|
169
|
+
- Website: https://lianes8studio.com
|
|
170
|
+
- GitHub: https://github.com/JaviLianes8
|
|
171
|
+
- LinkedIn: https://www.linkedin.com/in/jlianes/
|