magical-py 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- magical_py-0.1.0/CHANGELOG.md +336 -0
- magical_py-0.1.0/Cargo.lock +7 -0
- magical_py-0.1.0/Cargo.toml +24 -0
- magical_py-0.1.0/LICENSE +21 -0
- magical_py-0.1.0/PKG-INFO +169 -0
- magical_py-0.1.0/README.md +140 -0
- magical_py-0.1.0/bindings/python/Cargo.lock +137 -0
- magical_py-0.1.0/bindings/python/Cargo.toml +38 -0
- magical_py-0.1.0/bindings/python/README.md +140 -0
- magical_py-0.1.0/bindings/python/gates.sh +34 -0
- magical_py-0.1.0/bindings/python/gen_kinds.ps1 +279 -0
- magical_py-0.1.0/bindings/python/pyrightconfig.json +25 -0
- magical_py-0.1.0/bindings/python/src/lib.rs +229 -0
- magical_py-0.1.0/bindings/python/tests/test_detect.py +147 -0
- magical_py-0.1.0/bindings/python/tests/test_drift.py +108 -0
- magical_py-0.1.0/bindings/python/tests/test_metadata.py +110 -0
- magical_py-0.1.0/bindings/python/tests/test_readme.py +160 -0
- magical_py-0.1.0/bindings/python/tests/test_stub.py +68 -0
- magical_py-0.1.0/bindings/python/uv.lock +325 -0
- magical_py-0.1.0/pyproject.toml +47 -0
- magical_py-0.1.0/python/magical_py/__init__.py +98 -0
- magical_py-0.1.0/python/magical_py/_kinds.py +525 -0
- magical_py-0.1.0/python/magical_py/_magical_rs.pyi +25 -0
- magical_py-0.1.0/python/magical_py/py.typed +1 -0
- magical_py-0.1.0/readme.md +519 -0
- magical_py-0.1.0/src/lib.rs +24 -0
- magical_py-0.1.0/src/magical/async_dyn_magic.rs +121 -0
- magical_py-0.1.0/src/magical/bytes_read.rs +177 -0
- magical_py-0.1.0/src/magical/dyn_magic.rs +145 -0
- magical_py-0.1.0/src/magical/ext_fn/shebang.rs +62 -0
- magical_py-0.1.0/src/magical/ext_fn/webp.rs +24 -0
- magical_py-0.1.0/src/magical/magic.rs +376 -0
- magical_py-0.1.0/src/magical/magic_custom.rs +802 -0
- magical_py-0.1.0/src/magical/match_rules.rs +4 -0
- magical_py-0.1.0/src/magical/signatures.rs +575 -0
- magical_py-0.1.0/src/magical/signatures_ext.rs +175 -0
|
@@ -0,0 +1,336 @@
|
|
|
1
|
+
# CHANGELOG
|
|
2
|
+
- [CHANGELOG](#changelog)
|
|
3
|
+
- [Version: 0.1.3](#version-013)
|
|
4
|
+
- [Version: 0.2.0](#version-020)
|
|
5
|
+
- [Version: 0.2.1:](#version-021)
|
|
6
|
+
- [Version: 0.3.0:](#version-030)
|
|
7
|
+
- [Version: 0.3.1, `Minor edits`](#version-031-minor-edits)
|
|
8
|
+
- [Version: 0.4.0 `Major API Update`](#version-040-major-api-update)
|
|
9
|
+
- [Version: 0.4.5 `Major API Update`](#version-045-major-api-update)
|
|
10
|
+
- [Version: 0.6.0 `Signature Tightening`](#version-060-signature-tightening)
|
|
11
|
+
- [Version: 0.6.1 `Documentation and Test Coverage`](#version-061-documentation-and-test-coverage)
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
## Version: 0.1.3
|
|
15
|
+
**What has been changed:**
|
|
16
|
+
* Added examples of how to use `magical_rs`.
|
|
17
|
+
* Specifically as follows:
|
|
18
|
+
|
|
19
|
+
| Use case | Can be found at |
|
|
20
|
+
| ------------------ | -------------------------------------- |
|
|
21
|
+
| Basic usage | [normal_usage](examples/normal_usage) |
|
|
22
|
+
| Magic Custom usage | [magic_custom](examples/magic_custom/) |
|
|
23
|
+
|
|
24
|
+
* Added category `"no-std"` to [`[Cargo.toml]`](Cargo.toml)
|
|
25
|
+
* Changed repository URL of `magical_rs` in [`Cargo.toml`](Cargo.toml)
|
|
26
|
+
* Added examples to [`Cargo metadata`](Cargo.toml)
|
|
27
|
+
|
|
28
|
+
## Version: 0.2.0
|
|
29
|
+
**What has been changed:**
|
|
30
|
+
* Added methods to normalize and extends file matching in `CustomRulesMatches`
|
|
31
|
+
* Added documentation and test cases.
|
|
32
|
+
* Still retains backward compatibility for `no_std`.
|
|
33
|
+
* Added some macro to standardize the syntax sugar to make the API more friendly.
|
|
34
|
+
* Some examples of using the macros have also been added. Can be found at: [examples](examples).
|
|
35
|
+
* Some examples of using `DynMagic` have also been added. Can be found at: [examples](examples).
|
|
36
|
+
|
|
37
|
+
**Bellow is a list of macros that have been added:**
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
| Macro name | Support `no_std`, backward compatibility? |
|
|
41
|
+
| ------------------ | ----------------------------------------- |
|
|
42
|
+
| `match_custom!` | Yes |
|
|
43
|
+
| `magic_custom!` | Yes |
|
|
44
|
+
| `with_fn_matches!` | Yes |
|
|
45
|
+
| `any_matches!` | Yes |
|
|
46
|
+
| `all_matches!` | Yes |
|
|
47
|
+
|
|
48
|
+
---
|
|
49
|
+
* The list of supported file in `readme.md` will also synchronized.
|
|
50
|
+
|
|
51
|
+
**Bellow is a list of the signature files have been added:**
|
|
52
|
+
|
|
53
|
+
| Name | Signature | Offset |
|
|
54
|
+
| ----------------------- | ------------------------ | ------ |
|
|
55
|
+
| VMDK File | `0x4B, 0x44, 0x4D` | `0` |
|
|
56
|
+
| Google Chrome Extension | `0x43, 0x72, 0x32, 0x34` | `0` |
|
|
57
|
+
|
|
58
|
+
* Bellow is the development roadmap for version `0.2.0`:
|
|
59
|
+
|
|
60
|
+
| Name | Description | Status |
|
|
61
|
+
| ----------------- | ----------------------------------------------------------------------- | ------ |
|
|
62
|
+
| `Macro Supported` | Allows the use of macros to sugar-syntaxize the API | [x] |
|
|
63
|
+
| `MultipeFn` | Support for multiple `OR`, `AND` type pointer function in `CustomMagic` | [x] |
|
|
64
|
+
|
|
65
|
+
## Version: 0.2.1:
|
|
66
|
+
**What has been changed:**
|
|
67
|
+
|
|
68
|
+
* Fixed the documentation and added use for each module in [readme.md](readme.md)
|
|
69
|
+
* Fixed blank signatures & offsets blank in [magic_custom example](examples/magic_custom/src/v_2_0_0/magic_custom_macro.rs)
|
|
70
|
+
|
|
71
|
+
## Version: 0.3.0:
|
|
72
|
+
**What has been changed:**
|
|
73
|
+
|
|
74
|
+
* Added feature only avalable in version `0.3.0` of `magical_rs`: `AsyncDynMagic`
|
|
75
|
+
* Added documentation and usage warnings to [`lib.rs`](src/lib.rs) and [`readme.md`](readme.md)
|
|
76
|
+
* From this version onwards, `AsyncDynMagic` becomes an optional module. Cargo and flags are required to enable it:
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
cargo add magical_rs --features magical_async_dyn
|
|
80
|
+
```
|
|
81
|
+
* Of course, flag `magical_async_dyn` has also been added to [`Cargo.toml`](Cargo.toml)
|
|
82
|
+
* Instructions on how to use have also added at [`AsyncDynMagic Examples`](examples/async_dyn_magic)
|
|
83
|
+
* Current flags in version `0.3.0` can be used:
|
|
84
|
+
|
|
85
|
+
| Name | Description | Cargo flag |
|
|
86
|
+
| ------------------- | -------------------------------------------------------------- | ----------------------- |
|
|
87
|
+
| `magical_dyn` | Unlock lvl 3 with file dection with infinite rules at run time | `magical_dyn` |
|
|
88
|
+
| `magical_async_dyn` | Has all the features of level 3 but supports asynchronous | `magical_async_dyn` |
|
|
89
|
+
| `no_std` | Used in non-std environments like kernel, emebedded | `--no-default-features` |
|
|
90
|
+
|
|
91
|
+
## Version: 0.3.1, `Minor edits`
|
|
92
|
+
**What has been changed:**
|
|
93
|
+
* Minor edit in [`Cargo.toml`](Cargo.toml), added category slug `asynchronous`
|
|
94
|
+
* Edited some keywords related to the framework in [`Cargo.toml`](Cargo.toml)
|
|
95
|
+
* Changed the description of the framework to better identify it's purpose
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
## Version: 0.4.0 `Major API Update`
|
|
99
|
+
**What has been changed:**
|
|
100
|
+
* Added feature flag `unsafe_context` to [`Cargo.toml`](Cargo.toml)
|
|
101
|
+
* Release new features included in the module `magic_custom` is `WithUnsafeFn`
|
|
102
|
+
- Test can be found at: [`here`](tests/unsafe.rs).
|
|
103
|
+
- Documentation and instructions, security warnings have also added for `magic_custom` module.
|
|
104
|
+
- This unsafe feature is only compiled and used when the `unsafe_context` flag is explicitly enabled via `Cargo`:
|
|
105
|
+
```bash
|
|
106
|
+
cargo add magical_rs --features unsafe_context
|
|
107
|
+
```
|
|
108
|
+
- This version also adds more documentation and warnings for features like `Default`, `WithFn`.
|
|
109
|
+
- However, `no_std` support is still absolutely guaranteed.
|
|
110
|
+
- Edited [`Makefile`](Makefile) rules, allowing testing with `unsafe_context` feature
|
|
111
|
+
- Added example for using `unsafe_context` [`here`](examples/unsafe_context) and [`readme.md`](readme.md)
|
|
112
|
+
- We do a plan to add bindings to Python. However, we can't show them yet. So, the `bindings` folder will be ignored by Git for now. [`.gitignore`](.gitignore)
|
|
113
|
+
|
|
114
|
+
## Version: 0.4.5 `Major API Update`
|
|
115
|
+
**What has been changed:**
|
|
116
|
+
* Added support for multiple unsafe function pointers. It will be disabled by default.
|
|
117
|
+
- Only usable if feature flag is explicitly used by Cargo:
|
|
118
|
+
- ```bash
|
|
119
|
+
cargo add magical_rs --features unsafe_context
|
|
120
|
+
```
|
|
121
|
+
- These new features will not affect `no_std`, and will still be supported.
|
|
122
|
+
- Added testing for the above features. Can be found at [`test`](tests/unsafe.rs)
|
|
123
|
+
- Samples for the above features:
|
|
124
|
+
- `AllMatchesUnsafe`:
|
|
125
|
+
- ```rust
|
|
126
|
+
use core::slice;
|
|
127
|
+
use magical_rs::magical::magic_custom::match_types_custom;
|
|
128
|
+
use magical_rs::magical::magic_custom::{CustomMatchRules, MagicCustom};
|
|
129
|
+
|
|
130
|
+
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
|
|
131
|
+
enum MagicKind {
|
|
132
|
+
MoeMoe,
|
|
133
|
+
UnknownFallback,
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
fn is_shoujo_girl(data: *const ()) -> bool {
|
|
137
|
+
unsafe {
|
|
138
|
+
let slice_ptr = data.cast::<u8>();
|
|
139
|
+
let slice = slice::from_raw_parts(slice_ptr, 100);
|
|
140
|
+
|
|
141
|
+
slice.starts_with(b"MagicalGirl")
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
|
|
145
|
+
fn is_not_shoujo_girl(data: *const ()) -> bool {
|
|
146
|
+
unsafe {
|
|
147
|
+
let slice_ptr = data.cast::<u8>();
|
|
148
|
+
let slice = slice::from_raw_parts(slice_ptr, 100);
|
|
149
|
+
|
|
150
|
+
!slice.starts_with(b"MagicalGirl")
|
|
151
|
+
}
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
let rules: &[MagicCustom<MagicKind>] = &[MagicCustom {
|
|
155
|
+
signatures: &[],
|
|
156
|
+
offsets: &[],
|
|
157
|
+
max_bytes_read: 200,
|
|
158
|
+
kind: MagicKind::MoeMoe,
|
|
159
|
+
rules: CustomMatchRules::AllMatchesUnsafe(&[is_shoujo_girl, is_not_shoujo_girl]),
|
|
160
|
+
}];
|
|
161
|
+
|
|
162
|
+
let result = match_types_custom(b"MagicalGirl", rules, MagicKind::UnknownFallback);
|
|
163
|
+
|
|
164
|
+
assert_ne!(result, MagicKind::MoeMoe);
|
|
165
|
+
assert_eq!(result, MagicKind::UnknownFallback);
|
|
166
|
+
```
|
|
167
|
+
- `AnyMatchesUnsafe`:
|
|
168
|
+
- ```rust
|
|
169
|
+
use core::slice;
|
|
170
|
+
use magical_rs::magical::magic_custom::match_types_custom;
|
|
171
|
+
use magical_rs::magical::magic_custom::{CustomMatchRules, MagicCustom};
|
|
172
|
+
|
|
173
|
+
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
|
|
174
|
+
enum MagicKind {
|
|
175
|
+
MoeMoe,
|
|
176
|
+
UnknownFallback,
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
fn is_shoujo_girl(data: *const ()) -> bool {
|
|
180
|
+
unsafe {
|
|
181
|
+
let slice_ptr = data.cast::<u8>();
|
|
182
|
+
let slice = slice::from_raw_parts(slice_ptr, 100);
|
|
183
|
+
|
|
184
|
+
slice.starts_with(b"MagicalGirl")
|
|
185
|
+
}
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
fn is_not_shoujo_girl(data: *const ()) -> bool {
|
|
189
|
+
unsafe {
|
|
190
|
+
let slice_ptr = data.cast::<u8>();
|
|
191
|
+
let slice = slice::from_raw_parts(slice_ptr, 100);
|
|
192
|
+
|
|
193
|
+
!slice.starts_with(b"MagicalGirl")
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
let rules: &[MagicCustom<MagicKind>] = &[MagicCustom {
|
|
198
|
+
signatures: &[],
|
|
199
|
+
offsets: &[],
|
|
200
|
+
max_bytes_read: 200,
|
|
201
|
+
kind: MagicKind::MoeMoe,
|
|
202
|
+
rules: CustomMatchRules::AnyMatchesUnsafe(&[is_shoujo_girl, is_not_shoujo_girl]),
|
|
203
|
+
}];
|
|
204
|
+
|
|
205
|
+
let result = match_types_custom(b"MagicalGirl", rules, MagicKind::UnknownFallback);
|
|
206
|
+
|
|
207
|
+
assert_eq!(result, MagicKind::MoeMoe);
|
|
208
|
+
assert_ne!(result, MagicKind::UnknownFallback);
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
|
|
212
|
+
## Version: 0.6.1 `Documentation and Test Coverage`
|
|
213
|
+
|
|
214
|
+
**No API change, and no format added or removed.** The signature table is
|
|
215
|
+
byte-for-byte the one in `0.6.0`, and the twelve public items are the same
|
|
216
|
+
twelve. This release exists because `0.6.0` shipped a crate that nothing in
|
|
217
|
+
the repository was checking, and a readme whose claims no test agreed with.
|
|
218
|
+
|
|
219
|
+
**What this fixes:**
|
|
220
|
+
|
|
221
|
+
* The readme is now the single source of the crate's documentation. `src/lib.rs`
|
|
222
|
+
carries `#![doc = include_str!("../readme.md")]` instead of 242 lines of
|
|
223
|
+
duplicated doc comment, so the two can no longer disagree. Nothing but
|
|
224
|
+
documentation was removed from the crate.
|
|
225
|
+
* The format table is checked rather than asserted. `tests/table_size.rs`,
|
|
226
|
+
`tests/signature_coverage.rs` and `tests/readme_coverage.rs` fail if the
|
|
227
|
+
built-in table and the readme stop matching, and the nine two-byte-only
|
|
228
|
+
signatures are listed and checked individually.
|
|
229
|
+
* The `no_std` claim is verified instead of asserted. `make test-nostd` runs
|
|
230
|
+
the suite without `std` and cross compiles to `thumbv7em-none-eabi`. It
|
|
231
|
+
excludes doctests deliberately, since the readme's quick start calls a
|
|
232
|
+
`std`-gated function; the cross compile is the real gate.
|
|
233
|
+
* `crate_dev.yml` now runs on pull requests into `master`, not only into
|
|
234
|
+
`dev`. It had not run at all on the code that shipped as `0.6.0`, because
|
|
235
|
+
`dev` had not moved since the commit before it.
|
|
236
|
+
|
|
237
|
+
**Not part of this release:** the Python bindings are a separate package,
|
|
238
|
+
`magical-py`, and are not reachable through this crate.
|
|
239
|
+
|
|
240
|
+
## Version: 0.6.0 `Signature Tightening` and `Format Table Expansion`
|
|
241
|
+
|
|
242
|
+
**Breaking: three signatures changed.**
|
|
243
|
+
Detection results for the same bytes can differ from `0.5.x`. Review before upgrading.
|
|
244
|
+
|
|
245
|
+
| Format | `0.5.x` | `0.6.0` | Why |
|
|
246
|
+
| --- | --- | --- | --- |
|
|
247
|
+
| `FileKind::Bzip` | `BZ` | `BZh` | `BZ` is only a prefix. The bzip2 block header is `BZh`, so the old rule reported any file starting with those two letters as bzip2. |
|
|
248
|
+
| `FileKind::ScriptExecute` | `#!` | `#!` plus `/` on the same line | `#!` claimed every file starting with those bytes. A real shebang must name an interpreter by path. |
|
|
249
|
+
| `FileKind::Ply` | `ply` | `ply` plus a line break | `ply` claimed any text file starting with that word. The PLY spec puts a line ending straight after the keyword. |
|
|
250
|
+
|
|
251
|
+
**What this fixes:**
|
|
252
|
+
* An AMR audio file is now reported as [`FileKind::Amr`] instead of
|
|
253
|
+
`FileKind::ScriptExecute`. The AMR header `#!AMR` has no path separator, so
|
|
254
|
+
it no longer matches the narrowed shebang rule.
|
|
255
|
+
* Bzip2 detection no longer produces false positives on files that merely
|
|
256
|
+
begin with the letters `BZ`.
|
|
257
|
+
* A `#` comment or `#include` line in a source file is no longer reported as a
|
|
258
|
+
script.
|
|
259
|
+
|
|
260
|
+
**New public module:** `magical::ext_fn::shebang`, exposing `is_shebang`.
|
|
261
|
+
Entries that rely on a predicate rather than a byte signature are now
|
|
262
|
+
`ScriptExecute` and `WEBP`.
|
|
263
|
+
|
|
264
|
+
**Known limitation of the new shebang rule:** a relative interpreter name such
|
|
265
|
+
as `#!python` is not detected, because it contains no path separator. Such a
|
|
266
|
+
script is non-portable in practice. Previously it was also undetected for a
|
|
267
|
+
different reason, so no realistic script is lost.
|
|
268
|
+
|
|
269
|
+
**What was changed:**
|
|
270
|
+
* `magical_rs` is now licensed under the MIT License instead of the GNU General
|
|
271
|
+
Public License v3.0.
|
|
272
|
+
* The built-in format table grew from 48 to 114 formats (142 distinct magic
|
|
273
|
+
signatures), defined in the new `src/magical/signatures_ext.rs`.
|
|
274
|
+
* The 65 new rules are **appended** to `SIGNATURE_KIND`, never interleaved.
|
|
275
|
+
Because `match_types` returns the first match, appending guarantees no
|
|
276
|
+
pre-existing rule can be shadowed. Behaviour of the original 48 formats is
|
|
277
|
+
bit-for-bit unchanged.
|
|
278
|
+
* `readme.md` was rewritten and is now the single source of crate
|
|
279
|
+
documentation, pulled in by `#![doc = include_str!("../readme.md")]`.
|
|
280
|
+
Previously the README and the crate docs were two separate copies that had
|
|
281
|
+
drifted apart; the crate docs contained a doctest referencing `async_std`,
|
|
282
|
+
which is not a dependency, so `cargo test --doc` was already failing on
|
|
283
|
+
`master`.
|
|
284
|
+
* The 65 new rules are **appended** to `SIGNATURE_KIND`, never interleaved.
|
|
285
|
+
Because `match_types` returns the first match, appending guarantees no
|
|
286
|
+
pre-existing rule can be shadowed, apart from the three deliberate changes
|
|
287
|
+
listed above.
|
|
288
|
+
|
|
289
|
+
**Bugs found and fixed in the documentation:**
|
|
290
|
+
* The old format table misdescribed 9 signatures, including XML (the docs
|
|
291
|
+
claimed `<!DOCTYPE` was accepted; the code only matches `<?xml ` with a
|
|
292
|
+
trailing space) and the environment module format (docs said
|
|
293
|
+
`MODULE\0\0\0`; the code checks `#%Module`).
|
|
294
|
+
* The level 5 example in the old README did not compile. It referenced
|
|
295
|
+
`CustomMatchRules::WithFnUnsafe`, which does not exist; the available
|
|
296
|
+
variants are `AllMatchesUnsafe` and `AnyMatchesUnsafe`.
|
|
297
|
+
* `with_bytes_read()` returns 36,870 bytes, not 2,048, because ISO 9660 stores
|
|
298
|
+
its magic at offset 36,865. This is now documented and covered by a test.
|
|
299
|
+
|
|
300
|
+
**New tests:**
|
|
301
|
+
* `tests/signature_coverage.rs` walks `SIGNATURE_KIND` and asserts every entry
|
|
302
|
+
detects itself, that no signature matches at an undeclared offset, that
|
|
303
|
+
padding is never misdetected, and that truncated input never panics. Any
|
|
304
|
+
format added in the future is covered automatically.
|
|
305
|
+
* `tests/signature_tightening.rs` pins the `0.6.0` behaviour change. Each of the
|
|
306
|
+
three tightened formats is tested both ways: real files are still detected,
|
|
307
|
+
and the specific over-match the change was made to fix no longer happens.
|
|
308
|
+
* `tests/readme_coverage.rs` checks the README against the code in both
|
|
309
|
+
directions. Every format the table can return must be named in the README,
|
|
310
|
+
every `FileKind` the README advertises must be one the table can actually
|
|
311
|
+
return, and the stated table size must match `SIGNATURE_KIND.len()`. A guard
|
|
312
|
+
assertion fails the test if the README table layout changes and the parser
|
|
313
|
+
silently stops matching anything.
|
|
314
|
+
* `tests/readme_examples.rs` compiles and runs every code example in the
|
|
315
|
+
README.
|
|
316
|
+
* `tests/table_size.rs` reports the live table size.
|
|
317
|
+
|
|
318
|
+
**Formats evaluated and deliberately excluded:**
|
|
319
|
+
* 3D Studio Max, whose magic is byte-for-byte identical to BigTIFF.
|
|
320
|
+
* Text-based formats, which have no magic bytes.
|
|
321
|
+
|
|
322
|
+
**Why:**
|
|
323
|
+
* The previous GPL-3.0 license capped adoption. Many organizations block GPL-licensed
|
|
324
|
+
dependencies in their build and security policy, which ruled out both corporate
|
|
325
|
+
adoption and the use of `magical_rs` inside permissively licensed tooling.
|
|
326
|
+
* MIT removes that ceiling without changing a single line of library code.
|
|
327
|
+
|
|
328
|
+
**What this does not change:**
|
|
329
|
+
* `magical_rs` is a single-author project. No other contributor holds copyright,
|
|
330
|
+
so no third-party consent was required for this relicense.
|
|
331
|
+
* Versions `0.4.5` and earlier were already distributed under the GPL. Those grants
|
|
332
|
+
are permanent and cannot be revoked, for anyone who already received those
|
|
333
|
+
versions. Anyone depending on `0.4.5` may continue to use it under GPL terms.
|
|
334
|
+
Only versions from `0.5.0` onward are MIT-licensed.
|
|
335
|
+
* No API, behavior, feature flag, or `no_std` support was modified by this change.
|
|
336
|
+
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "magical_rs"
|
|
3
|
+
description = """Rust framework for file recognition, aiming for high extensibility and customization."""
|
|
4
|
+
version = "0.6.1"
|
|
5
|
+
edition = "2024"
|
|
6
|
+
license = "MIT"
|
|
7
|
+
keywords = ["detect-file", "no-dependency", "no-std", "magic", "rlib"]
|
|
8
|
+
categories = ["filesystem", "no-std", "asynchronous"]
|
|
9
|
+
include = ["LICENSE", "src/**/*.rs", "CHANGELOG.md"]
|
|
10
|
+
authors = ["Reim-developer", "contact.kaxtr@gmail.com"]
|
|
11
|
+
readme = "readme.md"
|
|
12
|
+
repository = "https://github.com/reim-developer/magical_rs"
|
|
13
|
+
|
|
14
|
+
[lib]
|
|
15
|
+
crate-type = ["lib"]
|
|
16
|
+
|
|
17
|
+
[features]
|
|
18
|
+
default = ["std"]
|
|
19
|
+
std = []
|
|
20
|
+
magical_dyn = []
|
|
21
|
+
magical_async_dyn = []
|
|
22
|
+
unsafe_context = []
|
|
23
|
+
|
|
24
|
+
[dependencies]
|
magical_py-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Reim-developer
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: magical-py
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Classifier: Development Status :: 3 - Alpha
|
|
5
|
+
Classifier: Intended Audience :: Developers
|
|
6
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
7
|
+
Classifier: Programming Language :: Python :: 3
|
|
8
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
10
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
15
|
+
Classifier: Programming Language :: Rust
|
|
16
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
17
|
+
Classifier: Topic :: Utilities
|
|
18
|
+
Classifier: Typing :: Typed
|
|
19
|
+
Summary: Native Python bindings for magical_rs, a zero-dependency file type detection library.
|
|
20
|
+
Keywords: file-type,magic,mime,detection,no-dependency
|
|
21
|
+
Home-Page: https://github.com/Reim-developer/magical_rs/blob/master/bindings/python
|
|
22
|
+
License: MIT
|
|
23
|
+
Requires-Python: >=3.8
|
|
24
|
+
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
|
25
|
+
Project-URL: Changelog, https://github.com/Reim-developer/magical_rs/blob/master/CHANGELOG.md
|
|
26
|
+
Project-URL: Homepage, https://github.com/Reim-developer/magical_rs
|
|
27
|
+
Project-URL: Repository, https://github.com/Reim-developer/magical_rs
|
|
28
|
+
|
|
29
|
+
# `magical_py`
|
|
30
|
+
|
|
31
|
+
Native Python bindings for
|
|
32
|
+
[`magical_rs`](https://github.com/Reim-developer/magical_rs), a zero-dependency
|
|
33
|
+
Rust library that identifies files by their magic bytes.
|
|
34
|
+
|
|
35
|
+
This is **not** a drop-in replacement for `python-magic`, and it is not trying to
|
|
36
|
+
be one. `python-magic` returns a string like `"image/png"` and leaves you to parse
|
|
37
|
+
it. This returns a typed value that carries its media type and conventional
|
|
38
|
+
extension with it.
|
|
39
|
+
|
|
40
|
+
## Install
|
|
41
|
+
|
|
42
|
+
Not released to PyPI yet. From a checkout of this repository:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
pip install ./bindings/python
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Once published this becomes `pip install magical-py`, and there is a binary
|
|
49
|
+
wheel for every platform we publish for. One `abi3` wheel covers CPython 3.8
|
|
50
|
+
and later, so you will not have to build from source.
|
|
51
|
+
|
|
52
|
+
`FileKind` covers all 114 formats the Rust crate detects, and the enum is
|
|
53
|
+
declared in Python rather than generated, so it is a real `enum.Enum` and your
|
|
54
|
+
editor completes `FileKind.` with every member and its docstring.
|
|
55
|
+
|
|
56
|
+
## Use
|
|
57
|
+
|
|
58
|
+
```python
|
|
59
|
+
from magical_py import FileKind, detect
|
|
60
|
+
|
|
61
|
+
kind = detect("photo.jpg")
|
|
62
|
+
|
|
63
|
+
kind is FileKind.Jpg # True, a real enum member your editor can complete
|
|
64
|
+
kind.value # 'jpg', a stable id for serialising
|
|
65
|
+
kind.mime # 'image/jpeg'
|
|
66
|
+
kind.extension # 'jpg'
|
|
67
|
+
kind.description # 'JPEG'
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
For data you already have in memory:
|
|
71
|
+
|
|
72
|
+
```python
|
|
73
|
+
from magical_py import detect_bytes
|
|
74
|
+
|
|
75
|
+
detect_bytes(header) # FileKind.Png | None
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Both return `None` when nothing matches, and raise the usual `OSError`
|
|
79
|
+
subclasses when a file cannot be read:
|
|
80
|
+
|
|
81
|
+
```python
|
|
82
|
+
try:
|
|
83
|
+
kind = detect(path)
|
|
84
|
+
except FileNotFoundError:
|
|
85
|
+
kind = None
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## What it looks like
|
|
89
|
+
|
|
90
|
+
Each member carries its display name as its `__doc__`, which is the attribute
|
|
91
|
+
the REPL and `help()` read. Python 3.9 and later render it for all 114 members;
|
|
92
|
+
Python 3.8 omits it from `help()`, so on 3.8 read `__doc__` directly as below.
|
|
93
|
+
|
|
94
|
+
```python
|
|
95
|
+
>>> FileKind.Png.__doc__
|
|
96
|
+
'PNG'
|
|
97
|
+
>>> FileKind.ISO.__doc__
|
|
98
|
+
'ISO 9660'
|
|
99
|
+
>>> FileKind.PkgZip.description
|
|
100
|
+
'Zip / JAR / APK'
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Things worth knowing
|
|
104
|
+
|
|
105
|
+
**Detection reads magic bytes only.** The file name is never consulted, so a
|
|
106
|
+
`.jpg` that actually contains PNG data is reported as `FileKind.Png`. That is
|
|
107
|
+
the behaviour you want from a detection library, and it is also why the
|
|
108
|
+
extension is advisory rather than something to switch on.
|
|
109
|
+
|
|
110
|
+
**Some media types are `None`.** `FileKind.mime` and `FileKind.extension`
|
|
111
|
+
return `None` when no media type is registered or verified for that format.
|
|
112
|
+
`None` means "there isn't one", never "we guessed". A wrong media type would be
|
|
113
|
+
a documentation bug of the same kind as a wrong magic byte, and this project
|
|
114
|
+
does not ship those.
|
|
115
|
+
|
|
116
|
+
**`detect` reads 36,870 bytes.** ISO 9660 stores its magic at offset 36,865, so
|
|
117
|
+
detecting it means reading that far into the file. `detect` does that for you;
|
|
118
|
+
`magical_py.bytes_read()` reports the figure if you are reading headers
|
|
119
|
+
yourself. Do not hardcode 2048 and assume you are done.
|
|
120
|
+
|
|
121
|
+
**Some signatures are only two bytes.** 9 formats match on nothing but a
|
|
122
|
+
two-byte prefix: `Arj`, `Bitmap`, `Gzip`, `MP3`, `MSDOS`, `Pcx`, `Pickle`,
|
|
123
|
+
`SerializedJavaData` and `Zlib`. Those are the values the formats themselves
|
|
124
|
+
specify, so a file beginning with those bytes is reported as that format even
|
|
125
|
+
if it is not really one. Each one is listed with its bytes in the
|
|
126
|
+
[`magical_rs` README](https://github.com/Reim-developer/magical_rs#two-things-to-know-before-you-rely-on-it).
|
|
127
|
+
|
|
128
|
+
**Two rules look at more than a fixed pattern.** `FileKind.ScriptExecute`
|
|
129
|
+
requires a `/` later on the first line, so a bare `#!` does not count;
|
|
130
|
+
`FileKind.WEBP` requires the full `RIFF....WEBP` layout. The shebang rule
|
|
131
|
+
exists because `#!AMR` is the literal magic of AMR audio: without the `/`
|
|
132
|
+
check, the script rule claimed those files and they became undetectable.
|
|
133
|
+
|
|
134
|
+
## Type checking
|
|
135
|
+
|
|
136
|
+
The package ships a `py.typed` marker and stubs for the compiled module, and it
|
|
137
|
+
is checked with `pyright` in strict mode. `pythonVersion` is pinned to 3.8 so the
|
|
138
|
+
checker enforces the same floor as the wheel.
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from magical_py import FileKind
|
|
142
|
+
|
|
143
|
+
# Narrowed to FileKind after the None check.
|
|
144
|
+
kind: FileKind | None = detect("photo.jpg")
|
|
145
|
+
if kind is not None:
|
|
146
|
+
reveal_type(kind) # FileKind
|
|
147
|
+
reveal_type(kind.mime) # str | None
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
## Development
|
|
151
|
+
|
|
152
|
+
Managed with [uv](https://docs.astral.sh/uv/).
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
uv sync # create the venv and install maturin, pytest, pyright
|
|
156
|
+
uv run maturin develop # build the extension into the venv
|
|
157
|
+
uv run pytest # run the tests
|
|
158
|
+
uv run pyright # type check in strict mode
|
|
159
|
+
uv build # build the wheel and the sdist
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
`python/magical_py/_kinds.py` is generated. Edit the metadata table in
|
|
163
|
+
`gen_kinds.ps1` and re-run it; `tests/test_drift.py` fails if the enum and the
|
|
164
|
+
Rust detection table disagree.
|
|
165
|
+
|
|
166
|
+
## License
|
|
167
|
+
|
|
168
|
+
MIT, the same as `magical_rs`.
|
|
169
|
+
|