durablewrite 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. durablewrite-0.2.0/.gitattributes +1 -0
  2. durablewrite-0.2.0/.github/FUNDING.yml +1 -0
  3. durablewrite-0.2.0/.gitignore +12 -0
  4. durablewrite-0.2.0/CHANGELOG.md +66 -0
  5. durablewrite-0.2.0/CONTRIBUTING.md +24 -0
  6. durablewrite-0.2.0/GUIDE.md +207 -0
  7. durablewrite-0.2.0/IMPROVEMENTS-round.md +93 -0
  8. durablewrite-0.2.0/LICENSE +21 -0
  9. durablewrite-0.2.0/PKG-INFO +171 -0
  10. durablewrite-0.2.0/README.md +144 -0
  11. durablewrite-0.2.0/pyproject.toml +56 -0
  12. durablewrite-0.2.0/requirements-dev.txt +3 -0
  13. durablewrite-0.2.0/scripts/check.ps1 +11 -0
  14. durablewrite-0.2.0/scripts/release.ps1 +28 -0
  15. durablewrite-0.2.0/src/durablewrite/__init__.py +26 -0
  16. durablewrite-0.2.0/src/durablewrite/__main__.py +9 -0
  17. durablewrite-0.2.0/src/durablewrite/_atomic.py +141 -0
  18. durablewrite-0.2.0/src/durablewrite/_cleanup.py +63 -0
  19. durablewrite-0.2.0/src/durablewrite/_cli.py +178 -0
  20. durablewrite-0.2.0/src/durablewrite/_errors.py +40 -0
  21. durablewrite-0.2.0/src/durablewrite/_fs.py +134 -0
  22. durablewrite-0.2.0/src/durablewrite/_helpers.py +132 -0
  23. durablewrite-0.2.0/src/durablewrite/_options.py +79 -0
  24. durablewrite-0.2.0/src/durablewrite/_replace.py +94 -0
  25. durablewrite-0.2.0/src/durablewrite/py.typed +0 -0
  26. durablewrite-0.2.0/tests/_payload.py +27 -0
  27. durablewrite-0.2.0/tests/_worker.py +55 -0
  28. durablewrite-0.2.0/tests/conftest.py +54 -0
  29. durablewrite-0.2.0/tests/test_atomic.py +118 -0
  30. durablewrite-0.2.0/tests/test_cli.py +183 -0
  31. durablewrite-0.2.0/tests/test_concurrency.py +48 -0
  32. durablewrite-0.2.0/tests/test_contract.py +116 -0
  33. durablewrite-0.2.0/tests/test_crash.py +50 -0
  34. durablewrite-0.2.0/tests/test_e2e_wheel.py +107 -0
  35. durablewrite-0.2.0/tests/test_hardening.py +168 -0
  36. durablewrite-0.2.0/tests/test_json_cleanup.py +90 -0
  37. durablewrite-0.2.0/tests/test_metadata.py +163 -0
  38. durablewrite-0.2.0/tests/test_options.py +61 -0
  39. durablewrite-0.2.0/tests/test_overwrite.py +74 -0
  40. durablewrite-0.2.0/tests/test_sharing.py +103 -0
@@ -0,0 +1 @@
1
+ * text=auto eol=lf
@@ -0,0 +1 @@
1
+ buy_me_a_coffee: jlianesglrs
@@ -0,0 +1,12 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ .venv/
4
+ venv/
5
+ build/
6
+ dist/
7
+ *.egg-info/
8
+ .pytest_cache/
9
+ .mypy_cache/
10
+ .ruff_cache/
11
+ .coverage
12
+ htmlcov/
@@ -0,0 +1,66 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
5
+ [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
+
7
+ ## [Unreleased]
8
+
9
+ ## [0.2.0]
10
+
11
+ ### Added
12
+
13
+ - `write_json_atomic(path, obj, ...)`: serialises before touching the disk and
14
+ writes UTF-8 JSON with a trailing newline and LF line endings. `NaN` and
15
+ infinities are refused unless `allow_nan=True`.
16
+ - `fsync="none" | "file" | "full"` on every writer, with the `FsyncLevel` type.
17
+ The `durable` and `fsync_dir` flags keep working.
18
+ - `overwrite=False`: atomic create-if-absent (`link` on POSIX, `MoveFileExW`
19
+ without the replace flag on Windows).
20
+ - `preserve_mode` (default on), `permissions` and `preserve_owner` options.
21
+ - `follow_symlinks` (default on): writing through a symlink updates the file it
22
+ points to and keeps the link.
23
+ - `RetryPolicy`, `DEFAULT_RETRY` and `NO_RETRY`: bounded backoff when a Windows
24
+ sharing or lock violation blocks the rename.
25
+ - `DurableWriteError` (an `OSError`), `SharingViolationError` (a
26
+ `PermissionError`) and `DirectorySyncError` (the file was replaced but its
27
+ directory could not be fsynced).
28
+ - `cleanup_temp_files(directory, older_than=..., target=...)` for temp files
29
+ left by killed writers.
30
+ - `durablewrite` command line tool (also `python -m durablewrite`): writes
31
+ standard input to a file atomically, with `--append`, `--json`, `--fsync`,
32
+ `--no-overwrite`, `--permissions`, `--no-follow-symlinks` and `--attempts`.
33
+ Exit status 1 on errors, 3 when the file was replaced but not fully synced.
34
+ - `GUIDE.md` with the write protocol, per-platform guarantees and sources.
35
+
36
+ ### Changed
37
+
38
+ - Temp files are named `.<name>.<8 hex digits>.dwtmp` so they can be cleaned up
39
+ safely.
40
+ - Writing through a symlink no longer replaces the link with a regular file
41
+ (pass `follow_symlinks=False` for the old behaviour).
42
+
43
+ ### Fixed
44
+
45
+ - New files were created with mode `0600` and replaced files lost their mode;
46
+ new files now honour the umask like `open()` and replaced files keep their
47
+ permission bits.
48
+ - Very long file names no longer overflow the name limit when the temp name is
49
+ built.
50
+ - Writing to an existing directory raises `IsADirectoryError` before any temp
51
+ file is created, and a path ending with a separator raises `ValueError`.
52
+ - When the final mode is restricted (explicit `permissions` or a replaced
53
+ private file) the temp file is `0600` while it is written.
54
+ - On Windows, removal of a temp file that is briefly locked is retried.
55
+
56
+ ## [0.1.0]
57
+
58
+ ### Added
59
+
60
+ - `atomic_write(path, mode, *, encoding, newline, durable, fsync_dir)` context
61
+ manager: writes to a temp file in the target directory, flushes and fsyncs it,
62
+ then moves it onto the target with `os.replace` and fsyncs the directory.
63
+ - `write_text_atomic(path, text, encoding, *, newline, durable, fsync_dir)` and
64
+ `write_bytes_atomic(path, data, *, durable, fsync_dir)` one-shot helpers.
65
+ - Durable by default (file and directory fsync), both switchable off.
66
+ - Correct on Windows and POSIX; typed, zero runtime dependencies.
@@ -0,0 +1,24 @@
1
+ # Contributing
2
+
3
+ Thanks for taking the time. This project keeps a small, predictable shape.
4
+
5
+ ## Setup
6
+
7
+ ```
8
+ pip install -r requirements-dev.txt
9
+ ```
10
+
11
+ ## Before you open a pull request
12
+
13
+ - Keep public API in `src/durablewrite/__init__.py`; internals live in `_`-prefixed modules.
14
+ - Add a test for the behaviour you change. Tests run against the real filesystem and
15
+ real child processes; nothing on the write path is mocked. Only the retry `sleep` is
16
+ injected.
17
+ - Run `scripts/check.ps1` (format, lint, types, tests) and make sure it passes. The
18
+ multi-process and packaging tests are marked `slow`; `pytest -m "not slow"` skips
19
+ them while iterating, but the full suite must pass before a commit.
20
+ - The crash test is seeded: set `DURABLEWRITE_CRASH_SEED` and
21
+ `DURABLEWRITE_CRASH_ITERATIONS` to explore other interleavings.
22
+ - One logical change per commit, imperative subject line.
23
+
24
+ Open an issue first if the change is large or shifts the public API.
@@ -0,0 +1,207 @@
1
+ # durablewrite guide
2
+
3
+ This guide explains what `durablewrite` does on every write, what it guarantees
4
+ on each platform, and where those guarantees stop. Each claim cites its source
5
+ or the test in this repository that exercises it.
6
+
7
+ ## The write protocol
8
+
9
+ For `atomic_write(path)` and every helper built on it:
10
+
11
+ 1. **Resolve the target.** If `path` is a symlink and `follow_symlinks=True`
12
+ (the default), the file it points to becomes the target, so the link keeps
13
+ working. With `follow_symlinks=False` the link itself is replaced by a
14
+ regular file. An existing directory is rejected with `IsADirectoryError`.
15
+ 2. **Create a temp file in the target's own directory**, named
16
+ `.<name>.<8 hex digits>.dwtmp`, with `O_CREAT | O_EXCL`. A new file is created
17
+ with mode `0o666`, so the process umask applies exactly as it does for
18
+ `open()`. When the final mode comes from `permissions` or from the replaced
19
+ file, the temp file starts as `0o600` and is widened only after the data is
20
+ written, so restricted content is never readable by others in between.
21
+ Names longer than 48 characters keep a readable head plus a hash of the full
22
+ name, so the temp name stays within the 255-byte limit. Same directory
23
+ means same filesystem, which `rename` needs to be atomic: Python documents
24
+ that `os.replace` "may fail if src and dst are on different filesystems" [1].
25
+ 3. **Write** your data into it.
26
+ 4. **Set metadata.** With `preserve_owner=True` (POSIX) the uid and gid of the
27
+ old file are copied first, then the permission bits: either `permissions`
28
+ or, with `preserve_mode=True` (the default), the bits of the old file.
29
+ 5. **Flush and fsync the temp file** (`fsync="file"` or `"full"`). This must
30
+ happen before the rename: otherwise the new name can reach disk before the
31
+ data it points to, which is how ext4 and XFS produced empty or zero-filled
32
+ files after a crash [3][4][5].
33
+ 6. **Rename** the temp file onto the target with `os.replace`. With
34
+ `overwrite=False`, a create-only rename is used instead (see below).
35
+ 7. **Fsync the parent directory** (`fsync="full"`, POSIX only) so the rename
36
+ itself is on disk. Linux `fsync(2)`: "Calling fsync() does not necessarily
37
+ ensure that the entry in the directory containing the file has also reached
38
+ disk. For that an explicit fsync() on a file descriptor for the directory is
39
+ also needed." [2][3]
40
+
41
+ If anything raises before step 6 completes, the temp file is removed and the
42
+ target is untouched. That includes exceptions inside your `with` block and
43
+ `KeyboardInterrupt`.
44
+
45
+ ## What is guaranteed, per platform
46
+
47
+ | Promise | Linux / macOS (local fs) | Windows (local NTFS) |
48
+ | --- | --- | --- |
49
+ | Readers see the old or the complete new file, never a mix | Yes, `rename(2)` is atomic by POSIX [6] | Yes on NTFS in our tests; not documented as atomic by Microsoft [7] |
50
+ | A killed writer leaves the old or the new file, never a partial one | Yes (tested) | Yes (tested) |
51
+ | New content survives power loss once the call returns (`"full"`) | Linux: yes, file and directory are fsynced. macOS: see below | File data is flushed with `FlushFileBuffers` [8]; the rename is not explicitly flushed |
52
+ | Permission bits kept on replace | Yes (`preserve_mode`) | Not applicable; see below |
53
+ | Owner kept on replace | Opt-in (`preserve_owner`), needs privilege to change uid | No |
54
+ | Replace while another process has the target open | Yes; the reader keeps the old inode | Retried, then `SharingViolationError` |
55
+ | A reader can always open the target | Yes | No: an `open()` racing a rename can fail with `PermissionError` (never `FileNotFoundError`, never torn) |
56
+ | Replace a read-only target | Yes, if the directory is writable | No, `PermissionError` (not retried) |
57
+
58
+ The crash and concurrency rows are exercised by `tests/test_crash.py` and
59
+ `tests/test_concurrency.py`, which run on both Windows (NTFS) and Linux (ext4)
60
+ in this project's local gate.
61
+
62
+ ### Process crash versus power loss
63
+
64
+ The kill-mid-write test terminates a writer process at random moments (seeded,
65
+ `DURABLEWRITE_CRASH_SEED`) and checks that the target is always a complete,
66
+ checksummed payload. That proves atomicity against a process crash. No portable
67
+ test can cut the power, so durability against power loss rests on the fsync
68
+ calls above and on the filesystem honouring them [3].
69
+
70
+ ### Linux
71
+
72
+ `fsync="full"` follows the sequence recommended in "Ensuring data reaches disk"
73
+ [3]: fsync the file, rename, fsync the directory. Do not rely on ext4's
74
+ `auto_da_alloc` heuristic for "replace via rename" [4]; it is a mount option and
75
+ other filesystems such as XFS have no equivalent [5]. If the directory fsync
76
+ raises (some network and FUSE filesystems refuse it, and it fails when the
77
+ directory is writable but not readable), the new content is already in place
78
+ and `DirectorySyncError` is raised so you know the rename may not be durable
79
+ (`tests/test_hardening.py`).
80
+
81
+ ### macOS
82
+
83
+ On macOS `fsync` hands the data to the drive but does not ask the drive to
84
+ flush its own write cache; Apple documents `F_FULLFSYNC` for that [9].
85
+ `durablewrite` calls `os.fsync` and does not issue `F_FULLFSYNC`, so on macOS
86
+ `"full"` protects against an OS crash but not necessarily against a power loss
87
+ with an unflushed drive cache.
88
+
89
+ ### Windows
90
+
91
+ - `os.replace` is `MoveFileExW(src, dst, MOVEFILE_REPLACE_EXISTING)` [1][7].
92
+ Microsoft does not document it as atomic. On a local NTFS volume the rename is
93
+ a single metadata operation and our tests never observed a torn or missing
94
+ target. On FAT volumes and SMB shares the behaviour depends on the
95
+ filesystem or server and is not guaranteed.
96
+ - `os.fsync` calls the CRT `_commit`, which calls `FlushFileBuffers` [8][10].
97
+ - There is no directory fsync on Windows through the APIs Python exposes, so
98
+ `"full"` behaves like `"file"`. The data of the new file is flushed before
99
+ the rename; the rename itself becomes durable when NTFS commits its journal.
100
+ - `MoveFileExW` fails with `ERROR_ACCESS_DENIED`, `ERROR_SHARING_VIOLATION` or
101
+ `ERROR_LOCK_VIOLATION` while any process holds the target open without
102
+ `FILE_SHARE_DELETE` [11]. Python's `open()` does not pass that flag, and
103
+ antivirus scanners and the search indexer briefly open new files. These
104
+ errors are retried with bounded backoff (`RetryPolicy`, about 1.1 seconds by
105
+ default). A read-only target is never retried. When the budget runs out,
106
+ `SharingViolationError` is raised and the target is untouched
107
+ (`tests/test_sharing.py`). A directory ACL that denies the rename produces
108
+ the same error code and ends the same way.
109
+ - The temp file can itself be held open (an antivirus scan of the fresh file).
110
+ On every error path its removal is retried under the same `RetryPolicy`; if
111
+ it is still locked after that, it is left for `cleanup_temp_files`.
112
+ - Permission bits do not exist on Windows. `permissions` only toggles the
113
+ read-only attribute and `preserve_mode` does nothing. The new file inherits
114
+ the ACL of its directory; an explicit ACL on the old file is not copied.
115
+
116
+ ### Why not `ReplaceFileW`
117
+
118
+ `ReplaceFileW` [12] keeps the replaced file's ACL, attributes and creation
119
+ time, but it requires the target to exist and it can fail half way: its
120
+ documented `ERROR_UNABLE_TO_MOVE_REPLACEMENT_2` result leaves the target under
121
+ the backup name. A primitive whose failure mode removes the target conflicts
122
+ with the core promise, so `durablewrite` uses `MoveFileExW`.
123
+
124
+ ## Create only if absent
125
+
126
+ `overwrite=False` publishes the complete file only if nothing exists at the
127
+ target, and raises `FileExistsError` otherwise, without a check-then-act race:
128
+
129
+ - POSIX: `link(temp, target)` fails with `EEXIST` if the name is taken, then the
130
+ temp name is unlinked. The filesystem must support hard links.
131
+ - Windows: `MoveFileExW` without `MOVEFILE_REPLACE_EXISTING` fails with
132
+ `ERROR_ALREADY_EXISTS`.
133
+
134
+ `tests/test_overwrite.py` races six processes for one name and checks that
135
+ exactly one wins.
136
+
137
+ ## Symlinks
138
+
139
+ The default (`follow_symlinks=True`) writes to the file the link points to and
140
+ creates the temp file beside that file, so the rename never crosses a
141
+ filesystem even when the link lives elsewhere. A dangling link creates its
142
+ target. `follow_symlinks=False` replaces the link with a regular file, which is
143
+ what plain `os.replace` does; on POSIX this also works when the link points to
144
+ a directory. The replaced link counts as a new file, so it does not inherit the
145
+ mode of whatever it pointed to.
146
+
147
+ ## Side effects of replacing a file
148
+
149
+ Replacing a file creates a new file, by design. As a consequence:
150
+
151
+ - Other hard links to the old file keep the old content.
152
+ - The inode (POSIX) or file ID (Windows) changes. Programs that follow a file
153
+ by handle, such as `tail -f` without `-F`, keep reading the old file.
154
+ - Extended attributes, POSIX ACLs and Windows alternate data streams of the
155
+ old file are not copied.
156
+
157
+ ## Leftover temp files
158
+
159
+ A writer that is killed between creating its temp file and renaming it cannot
160
+ clean up. `cleanup_temp_files(directory, older_than=3600)` removes only regular
161
+ files matching `.<name>.<8 hex digits>.dwtmp` older than the given age, by
162
+ modification time. A writer that keeps its temp file open for longer than that
163
+ age could lose it, so choose an age longer than your slowest write. With
164
+ `target=`, only temp files of that exact target name are considered. The crash test uses it to reclaim every temp file its
165
+ kills leave behind.
166
+
167
+ ## Command line
168
+
169
+ `durablewrite TARGET` copies standard input into TARGET with the protocol
170
+ above, like `sponge` from moreutils. TARGET is replaced only after standard
171
+ input ends, so `sort f | durablewrite f` is safe on every platform, and
172
+ `durablewrite --append f < f` works on POSIX. On Windows the shell (and the
173
+ console-script launcher) keep a redirected file open without
174
+ `FILE_SHARE_DELETE`, which blocks the rename, so the command detects that
175
+ standard input is TARGET and refuses at once; pipe the file instead.
176
+
177
+ Exit status: 0 on success; 1 on any error, including a usage error, with TARGET
178
+ left as it was; 3 when TARGET was replaced but its directory could not be
179
+ fsynced. `--json` accepts only strict JSON (no `NaN` or `Infinity`). Each
180
+ `--append` run is atomic, but two concurrent runs on the same file can lose
181
+ one run's data, because each starts from the content it read.
182
+
183
+ ## Sources
184
+
185
+ 1. Python documentation, `os.replace` and `os.fsync`:
186
+ https://docs.python.org/3/library/os.html#os.replace
187
+ 2. Linux `fsync(2)` manual page: https://man7.org/linux/man-pages/man2/fsync.2.html
188
+ 3. Jeff Moyer, "Ensuring data reaches disk", LWN.net, 2011:
189
+ https://lwn.net/Articles/457667/
190
+ 4. Linux kernel documentation, ext4 mount options (`auto_da_alloc`):
191
+ https://docs.kernel.org/admin-guide/ext4.html
192
+ 5. XFS FAQ, "Why do I see binary NULLS in some files after recovery when I
193
+ unplugged the power?": https://xfs.org/index.php/XFS_FAQ
194
+ 6. POSIX.1-2024, `rename`:
195
+ https://pubs.opengroup.org/onlinepubs/9799919799/functions/rename.html
196
+ 7. Microsoft, `MoveFileExW`:
197
+ https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexw
198
+ 8. Microsoft, `FlushFileBuffers`:
199
+ https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers
200
+ 9. Apple, `fsync(2)` manual page (`F_FULLFSYNC`):
201
+ https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html
202
+ 10. Microsoft, `_commit`:
203
+ https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/commit
204
+ 11. Microsoft, `CreateFileW` (`FILE_SHARE_DELETE`):
205
+ https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew
206
+ 12. Microsoft, `ReplaceFileW`:
207
+ https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-replacefilew
@@ -0,0 +1,93 @@
1
+ # Improvements round: durablewrite
2
+
3
+ This round takes `durablewrite` from a correct but minimal context manager to a
4
+ complete atomic-write toolkit that is honest about what each platform does and
5
+ does not guarantee. Every promise below is backed by a test that runs on the real
6
+ filesystem: no mocked `os.replace`, no mocked `fsync`.
7
+
8
+ The public API grows additively. `atomic_write`, `write_text_atomic` and
9
+ `write_bytes_atomic` keep their signatures; new behaviour arrives through new
10
+ keyword arguments, new functions and a command-line tool.
11
+
12
+ ## Ranked improvements
13
+
14
+ ### 1. Correct permissions for new and replaced files (correctness, high)
15
+ `tempfile.mkstemp` creates files with mode `0600`, so 0.1.0 silently turned a
16
+ `0644` config or a `0755` script into a private, non-executable file on every
17
+ write. The temp file is now created with `O_CREAT | O_EXCL` and mode `0666`, so
18
+ the process umask applies exactly as it does for `open()`. When the target
19
+ already exists its permission bits are copied onto the temp file before the
20
+ rename (`preserve_mode=True`, the default). `permissions=0o600` sets explicit
21
+ bits, and `preserve_owner=True` copies uid and gid on POSIX.
22
+
23
+ ### 2. Bounded retry on Windows sharing violations (robustness, high)
24
+ On Windows, `os.replace` is `MoveFileExW(MOVEFILE_REPLACE_EXISTING)`, which fails
25
+ with `ERROR_ACCESS_DENIED` or `ERROR_SHARING_VIOLATION` while another process
26
+ holds the target open without `FILE_SHARE_DELETE`. Python's own `open()` does not
27
+ pass that flag, and antivirus scanners and the search indexer open freshly
28
+ written files for a moment. A frozen `RetryPolicy` (attempts, initial delay,
29
+ multiplier, cap, injectable `sleep`) retries only those Windows error codes, never
30
+ a read-only target or a POSIX `EACCES`, and raises a typed
31
+ `SharingViolationError` (a `PermissionError`) when the budget runs out. Removal
32
+ of the temp file is retried under the same policy.
33
+
34
+ ### 3. Explicit fsync levels (clarity, high)
35
+ `fsync="none" | "file" | "full"` replaces the two loosely related booleans with
36
+ one named choice. `full` (the default) fsyncs the temp file before the rename and
37
+ the parent directory after it, which is what ext4 and XFS need for the rename to
38
+ survive power loss. The old `durable` and `fsync_dir` flags keep working. The
39
+ docs state plainly that Windows has no directory fsync and that macOS `fsync`
40
+ does not flush the drive cache.
41
+
42
+ ### 4. JSON helper (convenience, high)
43
+ `write_json_atomic(path, obj, ...)` serialises first, so a value that cannot be
44
+ encoded raises before any file is created, then writes atomically with a trailing
45
+ newline. Defaults suit config files: `indent=2`, UTF-8, `ensure_ascii=False`.
46
+
47
+ ### 5. Atomic create-if-absent (`overwrite=False`) (capability, medium)
48
+ Publishes the complete file only if nothing exists at the target, without a
49
+ check-then-act race: `os.link` on POSIX and a plain `MoveFileExW` without the
50
+ replace flag on Windows both fail atomically when the name is taken. Useful for
51
+ one-time initialisation and lock-free "first writer wins" files.
52
+
53
+ ### 6. Documented symlink policy (correctness, medium)
54
+ Writing through a symlink now updates the file the link points to and keeps the
55
+ link (`follow_symlinks=True`), which is what dotfile managers expect. The temp
56
+ file is created next to the real target, so the rename never crosses a
57
+ filesystem. `follow_symlinks=False` replaces the link itself with a regular file.
58
+
59
+ ### 7. Stale temp file cleanup (operations, medium)
60
+ A process killed mid-write cannot remove its temp file. Temp files now carry a
61
+ distinctive `.dwtmp` suffix and `cleanup_temp_files(directory, older_than=...)`
62
+ removes only those, only once they are older than a chosen age (one hour by
63
+ default).
64
+
65
+ ### 8. Typed errors and early validation (ergonomics, medium)
66
+ `DurableWriteError` is the base of errors the library raises itself and
67
+ subclasses `OSError`, so existing `except OSError` handlers keep working.
68
+ Writing to an existing directory raises `IsADirectoryError` before any temp file
69
+ exists. Long file names no longer overflow the 255-byte name limit when the temp
70
+ name is built.
71
+
72
+ ### 9. A `sponge`-style CLI (tooling, medium)
73
+ `durablewrite TARGET` streams stdin into TARGET atomically, so
74
+ `sort data.txt | durablewrite data.txt` is safe. Flags cover `--append`,
75
+ `--json` (refuse to write invalid JSON), `--fsync`, `--no-overwrite`,
76
+ `--permissions` and `--attempts`. Errors go to stderr with exit status 1.
77
+
78
+ ### 10. Guide with sources and real crash tests (trust, high)
79
+ A `GUIDE.md` explains the write protocol step by step with citations, and the
80
+ test suite proves the promise: a seeded kill-mid-write test that terminates a
81
+ writer process at random moments and checks the target is always a complete old
82
+ or complete new payload, a multi-process concurrent writer test, a held-open
83
+ reader test on Windows, and an end-to-end test that builds a wheel, installs it
84
+ into a throwaway virtual environment and drives the installed `durablewrite`
85
+ script.
86
+
87
+ ## Out of scope this round
88
+
89
+ - `ReplaceFileW` on Windows. It preserves ACLs and attributes of the replaced
90
+ file, but it is documented to fail half way (`ERROR_UNABLE_TO_MOVE_REPLACEMENT_2`)
91
+ and requires the target to exist, so it would weaken the core promise.
92
+ - `F_FULLFSYNC` on macOS. Documented, not issued, because it cannot be exercised
93
+ by the local test suite.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Javier Lianes
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,171 @@
1
+ Metadata-Version: 2.5
2
+ Name: durablewrite
3
+ Version: 0.2.0
4
+ Summary: Durable, cross-platform atomic file writes: temp-same-dir plus os.replace plus file and dir fsync
5
+ Project-URL: Homepage, https://lianes8studio.com
6
+ Project-URL: Repository, https://github.com/JaviLianes8/durablewrite
7
+ Project-URL: Issues, https://github.com/JaviLianes8/durablewrite/issues
8
+ Project-URL: Changelog, https://github.com/JaviLianes8/durablewrite/blob/main/CHANGELOG.md
9
+ Project-URL: LinkedIn, https://www.linkedin.com/in/jlianes/
10
+ Project-URL: Funding, https://buymeacoffee.com/jlianesglrs
11
+ Author-email: Javier Lianes <javier@lianes8studio.com>
12
+ License-Expression: MIT
13
+ License-File: LICENSE
14
+ Keywords: atomic,crash-safe,durable,file,fsync,os.replace,write
15
+ Classifier: Development Status :: 4 - Beta
16
+ Classifier: Intended Audience :: Developers
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Typing :: Typed
19
+ Requires-Python: >=3.10
20
+ Provides-Extra: dev
21
+ Requires-Dist: build; extra == 'dev'
22
+ Requires-Dist: mypy; extra == 'dev'
23
+ Requires-Dist: pytest; extra == 'dev'
24
+ Requires-Dist: ruff; extra == 'dev'
25
+ Requires-Dist: twine; extra == 'dev'
26
+ Description-Content-Type: text/markdown
27
+
28
+ # durablewrite
29
+
30
+ Durable, cross-platform atomic file writes for Python. Data goes to a temp file
31
+ in the same directory, is flushed and fsynced, then moved onto the target with
32
+ `os.replace`, and the directory is fsynced. A reader never sees a half-written
33
+ file, and a crash leaves either the previous version or the complete new one.
34
+
35
+ Typed, standard library only, Python 3.10+, Windows and POSIX.
36
+
37
+ ## Install
38
+
39
+ ```
40
+ pip install durablewrite
41
+ ```
42
+
43
+ ## Usage
44
+
45
+ ```python
46
+ from durablewrite import atomic_write, write_bytes_atomic, write_json_atomic, write_text_atomic
47
+
48
+ # One-shot helpers.
49
+ write_text_atomic("notes.txt", "hello\n")
50
+ write_bytes_atomic("snapshot.bin", b"\x00\x01\x02")
51
+ write_json_atomic("config.json", {"ready": True}) # serialised before any file is touched
52
+
53
+ # Stream into a file: it is replaced only if the block finishes without an error.
54
+ with atomic_write("report.csv") as f:
55
+ for row in rows:
56
+ f.write(row + "\n")
57
+
58
+ with atomic_write("image.png", "wb") as f:
59
+ f.write(data)
60
+ ```
61
+
62
+ ### Options (on every writer)
63
+
64
+ | Option | Default | Meaning |
65
+ | --- | --- | --- |
66
+ | `fsync` | `"full"` | `"none"`, `"file"` (fsync the data) or `"full"` (data and directory) |
67
+ | `overwrite` | `True` | `False`: create only if absent, atomically, else `FileExistsError` |
68
+ | `preserve_mode` | `True` | keep the permission bits of the file being replaced (POSIX) |
69
+ | `permissions` | `None` | explicit permission bits, e.g. `0o600` |
70
+ | `preserve_owner` | `False` | keep uid and gid of the replaced file (POSIX) |
71
+ | `follow_symlinks` | `True` | write through a symlink and keep it, or replace the link itself |
72
+ | `retry` | `DEFAULT_RETRY` | bounded retry while Windows reports the target locked |
73
+
74
+ New files get `0o666` minus your umask, exactly like `open()`. When the final mode
75
+ comes from `permissions` or from the replaced file, the temp file is created as
76
+ `0o600`, so a secret is never readable by others while it is being written.
77
+
78
+ ```python
79
+ from durablewrite import NO_RETRY, RetryPolicy, SharingViolationError, write_text_atomic
80
+
81
+ write_text_atomic("cache.txt", payload, fsync="none") # atomic, not durable
82
+ write_text_atomic("token", secret, permissions=0o600, overwrite=False)
83
+
84
+ try:
85
+ write_text_atomic("held.txt", "x", retry=RetryPolicy(max_attempts=20, max_delay=0.2))
86
+ except SharingViolationError as exc: # a PermissionError; target untouched
87
+ print("still locked after", exc.attempts, "attempts")
88
+ ```
89
+
90
+ A writer that is killed mid-write cannot remove its temp file. Clean those up
91
+ at startup:
92
+
93
+ ```python
94
+ from durablewrite import cleanup_temp_files
95
+
96
+ cleanup_temp_files("data/", older_than=3600) # only *.dwtmp files written by durablewrite
97
+ ```
98
+
99
+ ## Command line
100
+
101
+ `durablewrite TARGET` writes standard input to TARGET atomically, like
102
+ `sponge`. TARGET is replaced only after the input ends.
103
+
104
+ ```
105
+ sort names.txt | durablewrite names.txt
106
+ jq '.debug = true' config.json | durablewrite --json config.json
107
+ echo "one more line" | durablewrite --append notes.txt
108
+ printf 's3cret' | durablewrite --permissions 600 --no-overwrite token.txt
109
+ ```
110
+
111
+ Options: `--append`, `--json` (refuse invalid JSON), `--fsync {none,file,full}`,
112
+ `--no-overwrite`, `--permissions OCTAL`, `--no-follow-symlinks`,
113
+ `--attempts N`. Errors go to standard error with exit status 1, and TARGET is
114
+ left as it was. If TARGET was replaced but its directory could not be fsynced,
115
+ the exit status is 3. `python -m durablewrite` works too.
116
+
117
+ On Windows, redirecting TARGET into the command (`durablewrite f < f`) is refused,
118
+ because the shell keeps the redirected file open and that blocks the rename; pipe
119
+ it instead (`type f | durablewrite f`). Concurrent `--append` runs on one file may
120
+ lose each other's lines: each run is atomic, but they do not merge.
121
+
122
+ ## Guarantees and limits
123
+
124
+ The short version (details and sources in [GUIDE.md](GUIDE.md)):
125
+
126
+ - **Atomic for readers and against process crashes** on local filesystems,
127
+ Windows and POSIX. Exercised by a seeded test that kills a writer process at
128
+ random moments and by a multi-process concurrent writer test.
129
+ - **Durable against power loss** on Linux with `fsync="full"`. On macOS, `fsync`
130
+ does not flush the drive cache (`F_FULLFSYNC` is not issued). On Windows the
131
+ file data is flushed but a directory cannot be fsynced.
132
+ - **Windows**: `os.replace` is `MoveFileExW`, not documented as atomic by
133
+ Microsoft; tested on NTFS, not guaranteed on FAT or network shares. An open
134
+ handle elsewhere blocks the rename; it is retried for about a second by
135
+ default, then `SharingViolationError` is raised. While a rename is in flight,
136
+ a concurrent `open()` of the target can fail with `PermissionError`; readers
137
+ that poll should retry. A read-only target cannot be replaced on Windows.
138
+ - **After the rename**, a failing directory fsync raises `DirectorySyncError`:
139
+ the new content is in place, but the rename may not survive a power loss.
140
+ - Replacing a file creates a new one: other hard links keep the old content,
141
+ and extended attributes, ACLs and alternate data streams are not copied.
142
+
143
+ ## Why
144
+
145
+ Writing a file in place can corrupt it: a crash, a full disk or a concurrent
146
+ reader can catch it half-written. The safe pattern (temp file in the same
147
+ directory, fsync, `os.replace`, fsync the directory) is well known but fiddly
148
+ to get right on every platform.
149
+
150
+ The options that existed each fall short:
151
+
152
+ - `atomicwrites`, the de-facto package, was **archived by its author in 2022**
153
+ and takes no fixes.
154
+ - `boltons.fileutils.atomic_save` has **no Windows support**.
155
+ - The `atomicfile` and `atomicwrite` packages on PyPI are **abandoned** (last
156
+ releases in 2020 and 2015) and neither is typed.
157
+ - `safewrite` is minimal and obscure.
158
+
159
+ `durablewrite` does the full sequence, keeps permissions, handles Windows
160
+ sharing violations, documents exactly what each platform guarantees, and ships
161
+ a command-line tool.
162
+
163
+ ## License
164
+
165
+ MIT. See [LICENSE](LICENSE).
166
+
167
+ ## Links
168
+
169
+ - Website: https://lianes8studio.com
170
+ - GitHub: https://github.com/JaviLianes8
171
+ - LinkedIn: https://www.linkedin.com/in/jlianes/