import-zig 0.13.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ Copyright (c) 2024 Felix Graßl.
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
4
+
5
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
6
+
7
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,90 @@
1
+ Metadata-Version: 2.1
2
+ Name: import_zig
3
+ Version: 0.13.0
4
+ Summary: Compile and import Zig functions at runtime without building a package
5
+ Author: Felix Graßl
6
+ License: MIT
7
+ Project-URL: Homepage, https://github.com/ffelixg/import_zig
8
+ Project-URL: Issues, https://github.com/ffelixg/import_zig/issues
9
+ Keywords: zig,ziglang,import,compile
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Programming Language :: Python
12
+ Classifier: Programming Language :: Python :: 3
13
+ Requires-Python: >=3.10
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE
16
+ Requires-Dist: ziglang==0.13
17
+
18
+ # import-zig
19
+
20
+ pip install import-zig
21
+
22
+ This module provides a way to import Zig code directly into Python using a single function call to `import_zig`. Here is an example:
23
+
24
+ ```py
25
+ from import_zig import import_zig
26
+
27
+ mod = import_zig(source_code = """
28
+ pub fn Q_rsqrt(number: f32) f32 {
29
+ const threehalfs: f32 = 1.5;
30
+ const x2 = number * 0.5;
31
+ var y = number;
32
+ var i: i32 = @bitCast(y);
33
+ i = 0x5f3759df - (i >> 1);
34
+ y = @bitCast(i);
35
+ y = y * (threehalfs - (x2 * y * y));
36
+
37
+ return y;
38
+ }
39
+ """)
40
+
41
+ print(f"1 / sqrt(1.234) = {mod.Q_rsqrt(1.234)}")
42
+ ```
43
+
44
+ The main goal of this module is to make it easy to prototype Zig extensions for Python without having to engage with the Python build system. When building larger Zig extensions it is likely preferable to write your own build process with setuptools or to use a ziggy-pydust template, which provides a comptime abstraction over many aspects of the Python C API. Zig and Zig extensions are still very new though, so things will likely change. Technically you could also use this module as part of a packaging step by calling the `compile_to` function to build the binary and moving it into the packages build folder.
45
+
46
+ One approach that I expect to stay the same is the comptime wrapping for conversion between Zig and Python types as well as exception handling. This is conveniently packed into the file `py_utils.zig` and could be copy pasted into a new setuptools based project and maybe adjusted.
47
+
48
+ See the docs of the import_zig function for more details or check out the examples directory.
49
+
50
+ # File structure
51
+
52
+ The file structure that will be generated to compile the Zig code looks as follows.
53
+
54
+ ```bash
55
+ project_folder
56
+ ├── build.zig
57
+ ├── generated.zig
58
+ ├── inner
59
+ │   └── import_fns.zig
60
+ ├── py_utils.zig
61
+ └── zig_ext.zig
62
+ ```
63
+
64
+ The `inner` directory is where your code lives. When you pass a source code string or file path, it will be written / linked as the `import_fns.zig` file. When you pass a directory path, the directory will be linked as the `inner` directory above, enabling references to other files in the directory path.
65
+
66
+ The above file structure can be generated with:
67
+
68
+ ```py
69
+ import_zig.prepare("/path/to/project_folder", "module_name")
70
+ ```
71
+
72
+ This enables ZLS support for the Python C API when importing `py_utils` from `import_fns.zig`.
73
+
74
+ # Type mapping
75
+
76
+ The conversion is defined in `py_utils.zig` and applied based on the parameter / return types of the exported function. Errors are also forwarded. The solution to passing variable length data back to Python is a bit of a hack: When an exported function specifies `std.mem.Allocator` as a parameter type, then an arena allocator - which gets deallocated after the function call - will be passed into the function. The allocator can then be used to allocate and return new slices for example.
77
+
78
+ For nested types, the conversion is applied recursively.
79
+
80
+ | Conversion from Python | Zig datatype | Conversion to Python |
81
+ | --------------------------------------- | --------------------------------- | --------------------------------------------- |
82
+ | int | integer (any size / sign) | int |
83
+ | float | float (any size) | float |
84
+ | - | void | None |
85
+ | evaluated like bool() | bool | bool |
86
+ | sequence | array | list |
87
+ | sequence | non u8 const slice | list |
88
+ | str | u8 const slice | str |
89
+ | dict or sequence | struct | tuple if struct is a tuple or named tuple |
90
+ | comparison with None | optional | null -> None |
@@ -0,0 +1,73 @@
1
+ # import-zig
2
+
3
+ pip install import-zig
4
+
5
+ This module provides a way to import Zig code directly into Python using a single function call to `import_zig`. Here is an example:
6
+
7
+ ```py
8
+ from import_zig import import_zig
9
+
10
+ mod = import_zig(source_code = """
11
+ pub fn Q_rsqrt(number: f32) f32 {
12
+ const threehalfs: f32 = 1.5;
13
+ const x2 = number * 0.5;
14
+ var y = number;
15
+ var i: i32 = @bitCast(y);
16
+ i = 0x5f3759df - (i >> 1);
17
+ y = @bitCast(i);
18
+ y = y * (threehalfs - (x2 * y * y));
19
+
20
+ return y;
21
+ }
22
+ """)
23
+
24
+ print(f"1 / sqrt(1.234) = {mod.Q_rsqrt(1.234)}")
25
+ ```
26
+
27
+ The main goal of this module is to make it easy to prototype Zig extensions for Python without having to engage with the Python build system. When building larger Zig extensions it is likely preferable to write your own build process with setuptools or to use a ziggy-pydust template, which provides a comptime abstraction over many aspects of the Python C API. Zig and Zig extensions are still very new though, so things will likely change. Technically you could also use this module as part of a packaging step by calling the `compile_to` function to build the binary and moving it into the packages build folder.
28
+
29
+ One approach that I expect to stay the same is the comptime wrapping for conversion between Zig and Python types as well as exception handling. This is conveniently packed into the file `py_utils.zig` and could be copy pasted into a new setuptools based project and maybe adjusted.
30
+
31
+ See the docs of the import_zig function for more details or check out the examples directory.
32
+
33
+ # File structure
34
+
35
+ The file structure that will be generated to compile the Zig code looks as follows.
36
+
37
+ ```bash
38
+ project_folder
39
+ ├── build.zig
40
+ ├── generated.zig
41
+ ├── inner
42
+ │   └── import_fns.zig
43
+ ├── py_utils.zig
44
+ └── zig_ext.zig
45
+ ```
46
+
47
+ The `inner` directory is where your code lives. When you pass a source code string or file path, it will be written / linked as the `import_fns.zig` file. When you pass a directory path, the directory will be linked as the `inner` directory above, enabling references to other files in the directory path.
48
+
49
+ The above file structure can be generated with:
50
+
51
+ ```py
52
+ import_zig.prepare("/path/to/project_folder", "module_name")
53
+ ```
54
+
55
+ This enables ZLS support for the Python C API when importing `py_utils` from `import_fns.zig`.
56
+
57
+ # Type mapping
58
+
59
+ The conversion is defined in `py_utils.zig` and applied based on the parameter / return types of the exported function. Errors are also forwarded. The solution to passing variable length data back to Python is a bit of a hack: When an exported function specifies `std.mem.Allocator` as a parameter type, then an arena allocator - which gets deallocated after the function call - will be passed into the function. The allocator can then be used to allocate and return new slices for example.
60
+
61
+ For nested types, the conversion is applied recursively.
62
+
63
+ | Conversion from Python | Zig datatype | Conversion to Python |
64
+ | --------------------------------------- | --------------------------------- | --------------------------------------------- |
65
+ | int | integer (any size / sign) | int |
66
+ | float | float (any size) | float |
67
+ | - | void | None |
68
+ | evaluated like bool() | bool | bool |
69
+ | sequence | array | list |
70
+ | sequence | non u8 const slice | list |
71
+ | str | u8 const slice | str |
72
+ | dict or sequence | struct | tuple if struct is a tuple or named tuple |
73
+ | comparison with None | optional | null -> None |
@@ -0,0 +1,168 @@
1
+ from pathlib import Path
2
+ from shutil import copyfile, copytree
3
+ from tempfile import TemporaryDirectory
4
+ from importlib import import_module
5
+ import sysconfig
6
+ import sys
7
+ import subprocess
8
+ import random
9
+ import platform
10
+
11
+ _copy_paths = [
12
+ Path(__file__).parent / "build.zig",
13
+ Path(__file__).parent / "py_utils.zig",
14
+ Path(__file__).parent / "zig_ext.zig",
15
+ ]
16
+
17
+
18
+ _is_windows = platform.system() == "Windows"
19
+
20
+
21
+ def _escape(path: str) -> str:
22
+ return path.replace("\\", "\\\\")
23
+
24
+
25
+ def prepare(path: str | Path, module_name: str, hardlink_only: bool = False):
26
+ """
27
+ Link/Create files at path needed to compile the Zig code
28
+
29
+ In order to get ZLS support for the Python C API, you can execute this and
30
+ develop inside the "inner" directory.
31
+ """
32
+ path = Path(path)
33
+ if not path.exists():
34
+ path.mkdir()
35
+
36
+ for src in _copy_paths:
37
+ tgt = path / src.name
38
+ if hardlink_only:
39
+ # Symlinks are buggy on windows
40
+ (tgt).hardlink_to(src)
41
+ else:
42
+ copyfile(src, tgt)
43
+
44
+ (path / "inner").mkdir()
45
+
46
+ include_dirs = [sysconfig.get_path("include")]
47
+ lib_paths = [
48
+ str(Path(sysconfig.get_config_var("installed_base"), "Libs").absolute())
49
+ ]
50
+
51
+ with (path / "generated.zig").open("w", encoding="utf-8") as f:
52
+ f.write(
53
+ f"pub const include: [{len(include_dirs)}][]const u8 = .{{\n"
54
+ + "".join(f' "{p}",\n' for p in map(_escape, include_dirs))
55
+ + "};\n"
56
+ + f"pub const lib: [{len(lib_paths)}][]const u8 = .{{\n"
57
+ + "".join(f' "{p}",\n' for p in map(_escape, lib_paths))
58
+ + "};\n"
59
+ + f'pub const module_name = "{module_name}";\n'
60
+ )
61
+
62
+
63
+ def compile_to(
64
+ target_dir: str | Path,
65
+ module_name: str = "zig_ext",
66
+ source_code: str | None = None,
67
+ file: Path | str | None = None,
68
+ directory: Path | str | None = None,
69
+ ):
70
+ """
71
+ Same as import_zig, except that the module will not be imported an instead
72
+ copied into the directory specified by `path_target`.
73
+
74
+ Further, `module_name` is not randomized.
75
+ """
76
+ if (source_code is not None) + (file is not None) + (directory is not None) != 1:
77
+ raise Exception(
78
+ "Exactly one method must be used to specify location of Zig file(s)."
79
+ )
80
+
81
+ with TemporaryDirectory(prefix="import_zig_compile_") as tempdir:
82
+ temppath = Path(tempdir)
83
+ prepare(temppath, module_name, hardlink_only=True)
84
+
85
+ temppath_inner = temppath / "inner"
86
+ if directory is not None:
87
+ temppath_inner.rmdir()
88
+ if _is_windows:
89
+ copytree(Path(directory).absolute(), temppath_inner)
90
+ else:
91
+ temppath_inner.symlink_to(Path(directory).absolute())
92
+ elif file is not None:
93
+ (temppath_inner / "import_fns.zig").hardlink_to(Path(file).absolute())
94
+ else:
95
+ assert source_code is not None
96
+ with (temppath_inner / "import_fns.zig").open("w", encoding="utf-8") as f:
97
+ f.write(source_code)
98
+
99
+ args = [
100
+ sys.executable,
101
+ "-m",
102
+ "ziglang",
103
+ "build",
104
+ *(["-Dtarget=x86_64-windows"] if _is_windows else []),
105
+ ]
106
+ subprocess.run(args, cwd=tempdir, check=True)
107
+
108
+ (binary,) = (
109
+ p
110
+ for p in (temppath / "zig-out").glob(f"**/*{'.dll' if _is_windows else ''}")
111
+ if p.is_file()
112
+ )
113
+
114
+ binary.rename(
115
+ Path(target_dir) / (module_name + sysconfig.get_config_var("EXT_SUFFIX"))
116
+ )
117
+
118
+
119
+ def import_zig(
120
+ module_name: str | None = None,
121
+ source_code: str | None = None,
122
+ file: Path | str | None = None,
123
+ directory: Path | str | None = None,
124
+ ):
125
+ """
126
+ This function takes in Zig code, wraps it in the Python C API, compiles the
127
+ code and returns the imported binary.
128
+
129
+ Assumptions on the code:
130
+ The Zig source can be specified as a source code string, a file or a directory.
131
+ If it is specified as a directory, the file containin the functions which get
132
+ exported to Python must be named `import_fns.zig`, however that file may use
133
+ any other files present in the directory.
134
+
135
+ A function gets exposed to Python if it is marked pub.
136
+
137
+ It is possible to use
138
+ ```
139
+ const pyu = @import("../py_utils.zig");
140
+ const py = pyu.py;
141
+ ```
142
+ in order to access the Python C API with `py` and utilities with `pyu`. This
143
+ allows for example raising exceptions or passing Python objects with
144
+ `*py.PyObject`.
145
+
146
+ If module_name is left blank, a random name will be assigned.
147
+ """
148
+ if module_name is None:
149
+ module_name = f"zig_ext_{hex(random.randint(0, 2**128))[2:]}"
150
+
151
+ # For some reason the binary can't be deleted on windows, so it will live on
152
+ # due to ignore_cleanup_errors. Hopefully the OS takes care of it eventually.
153
+ with TemporaryDirectory(
154
+ prefix="import_zig_", ignore_cleanup_errors=True
155
+ ) as tempdir:
156
+ compile_to(
157
+ tempdir,
158
+ source_code=source_code,
159
+ file=file,
160
+ directory=directory,
161
+ module_name=module_name,
162
+ )
163
+ sys.path.append(tempdir)
164
+ try:
165
+ module = import_module(module_name)
166
+ finally:
167
+ sys.path.remove(tempdir)
168
+ return module
@@ -0,0 +1,27 @@
1
+ const std = @import("std");
2
+ const generated = @import("generated.zig");
3
+
4
+ pub fn build(b: *std.Build) void {
5
+ const target = b.standardTargetOptions(.{});
6
+ const optimize = b.standardOptimizeOption(.{ .preferred_optimize_mode = .Debug });
7
+
8
+ const lib = b.addSharedLibrary(.{
9
+ .name = "zig_ext",
10
+ // .root_source_file = b.path("zig_ext_export.zig"),
11
+ .root_source_file = b.path("zig_ext.zig"),
12
+ .target = target,
13
+ .optimize = optimize,
14
+ });
15
+ lib.linkLibC();
16
+ inline for (generated.include) |path| {
17
+ lib.addIncludePath(.{ .cwd_relative = path });
18
+ }
19
+ inline for (generated.lib) |path| {
20
+ lib.addLibraryPath(.{ .cwd_relative = path });
21
+ }
22
+ if (target.query.os_tag == .windows) {
23
+ lib.linkSystemLibrary2("python3", .{ .needed = true, .preferred_link_mode = .static });
24
+ }
25
+ lib.linker_allow_shlib_undefined = true;
26
+ b.installArtifact(lib);
27
+ }
@@ -0,0 +1,275 @@
1
+ const std = @import("std");
2
+ pub const py = @cImport({
3
+ @cDefine("Py_LIMITED_API", "0x030a00f0");
4
+ @cDefine("PY_SSIZE_T_CLEAN", {});
5
+ @cInclude("Python.h");
6
+ });
7
+
8
+ var gpa = std.heap.GeneralPurposeAllocator(.{ .safety = true }){};
9
+ pub const gp_allocator = gpa.allocator();
10
+
11
+ pub const PyErr = error{PyErr};
12
+ const Exceptions = enum { Exception, NotImplemented, TypeError, ValueError };
13
+
14
+ pub fn raise(exc: Exceptions, comptime msg: []const u8, args: anytype) PyErr {
15
+ @setCold(true);
16
+ const pyexc = switch (exc) {
17
+ .Exception => py.PyExc_Exception,
18
+ .NotImplemented => py.PyExc_NotImplementedError,
19
+ .TypeError => py.PyExc_TypeError,
20
+ .ValueError => py.PyExc_ValueError,
21
+ };
22
+ const formatted = std.fmt.allocPrintZ(gp_allocator, msg, args) catch "Error formatting error message";
23
+ defer gp_allocator.free(formatted);
24
+
25
+ // new in Python 3.12, for older versions we just overwrite exceptions.
26
+ if (@hasField(py, "PyErr_GetRaisedException")) {
27
+ const cause = py.PyErr_GetRaisedException();
28
+ py.PyErr_SetString(pyexc, formatted.ptr);
29
+ if (cause) |_| {
30
+ const consequence = py.PyErr_GetRaisedException();
31
+ py.PyException_SetCause(consequence, cause);
32
+ py.PyErr_SetRaisedException(consequence);
33
+ }
34
+ } else {
35
+ py.PyErr_SetString(pyexc, formatted.ptr);
36
+ }
37
+ return PyErr.PyErr;
38
+ }
39
+
40
+ fn toPyList(value: anytype) !*py.PyObject {
41
+ const pylist = py.PyList_New(@intCast(value.len)) orelse return PyErr.PyErr;
42
+ errdefer py.Py_DECREF(pylist);
43
+ for (value, 0..) |entry, i_entry| {
44
+ const py_entry = try zig_to_py(entry);
45
+ if (py.PyList_SetItem(pylist, @intCast(i_entry), py_entry) == -1) {
46
+ py.Py_DECREF(py_entry);
47
+ return PyErr.PyErr;
48
+ }
49
+ }
50
+ return pylist;
51
+ }
52
+
53
+ var struct_tuple_map = std.StringHashMap(?*py.PyTypeObject).init(gp_allocator);
54
+
55
+ /// Steals a reference when passed PyObjects
56
+ pub fn zig_to_py(value: anytype) !*py.PyObject {
57
+ return switch (@typeInfo(@TypeOf(value))) {
58
+ .Int => |info| if (info.signedness == .signed) py.PyLong_FromLongLong(@as(c_longlong, value)) else py.PyLong_FromUnsignedLongLong(@as(c_ulonglong, value)),
59
+ .ComptimeInt => if (value < 0) py.PyLong_FromLongLong(@as(c_longlong, value)) else py.PyLong_FromUnsignedLongLong(@as(c_ulonglong, value)),
60
+ .Void => py.Py_NewRef(py.Py_None()),
61
+ .Float => py.PyFloat_FromDouble(@floatCast(value)),
62
+ .ComptimeFloat => py.PyFloat_FromDouble(@floatCast(value)),
63
+ .Bool => py.PyBool_FromLong(@intFromBool(value)),
64
+ .Optional => if (value) |v| zig_to_py(v) catch null else py.Py_NewRef(py.Py_None()),
65
+ .Array => |info| if (info.sentinel) |_|
66
+ @compileError("Sentinel is not supported")
67
+ else
68
+ toPyList(value) catch null,
69
+ .Pointer => |info| if (info.child == u8 and info.size == .Slice)
70
+ py.PyUnicode_FromStringAndSize(value.ptr, @intCast(value.len))
71
+ else if (info.child == py.PyObject and info.size == .One)
72
+ @as(?*py.PyObject, value)
73
+ else if (info.size == .Slice)
74
+ toPyList(value) catch null
75
+ else
76
+ unreachable,
77
+ .Struct => |info| blk: {
78
+ if (info.is_tuple) {
79
+ const tuple = py.PyTuple_New(info.fields.len) orelse return PyErr.PyErr;
80
+ errdefer py.Py_DECREF(tuple);
81
+ inline for (info.fields, 0..) |field, i_field| {
82
+ const py_value = try zig_to_py(@field(value, field.name));
83
+ if (py.PyTuple_SetItem(tuple, @intCast(i_field), py_value) == -1) {
84
+ py.Py_DECREF(py_value);
85
+ return PyErr.PyErr;
86
+ }
87
+ }
88
+ break :blk tuple;
89
+ } else {
90
+ const type_name = @typeName(@TypeOf(value));
91
+ const tuple_type = struct_tuple_map.get(type_name) orelse blk_tp: {
92
+ var fields: [info.fields.len + 1]py.PyStructSequence_Field = undefined;
93
+ fields[fields.len - 1] = py.PyStructSequence_Field{ .doc = null, .name = null };
94
+ inline for (info.fields, 0..) |field, i_field| {
95
+ fields[i_field] = py.PyStructSequence_Field{
96
+ .doc = "Zig type for this field is " ++ @typeName(field.type),
97
+ .name = field.name,
98
+ };
99
+ }
100
+
101
+ var desc: py.PyStructSequence_Desc = .{
102
+ .doc = "Generated in order to convert Zig struct " ++ type_name ++ " to Python object",
103
+ .n_in_sequence = fields.len - 1,
104
+ // Fully qualified name would be too verbose
105
+ .name = comptime name: {
106
+ var name: []const u8 = undefined;
107
+ var tokenizer = std.mem.tokenize(u8, type_name, ".");
108
+ while (tokenizer.next()) |token| {
109
+ name = token;
110
+ }
111
+ break :name "import_zig." ++ name ++ "";
112
+ },
113
+ .fields = &fields,
114
+ };
115
+ const tp = py.PyStructSequence_NewType(&desc) orelse return PyErr.PyErr;
116
+
117
+ try struct_tuple_map.put(type_name, tp);
118
+
119
+ break :blk_tp tp;
120
+ };
121
+
122
+ const tuple = py.PyStructSequence_New(tuple_type) orelse return PyErr.PyErr;
123
+ errdefer py.Py_DECREF(tuple);
124
+ inline for (info.fields, 0..) |field, i_field| {
125
+ const py_value = try zig_to_py(@field(value, field.name));
126
+ py.PyStructSequence_SetItem(tuple, @intCast(i_field), py_value);
127
+ }
128
+ break :blk tuple;
129
+ }
130
+ },
131
+ else => |info| {
132
+ @compileLog("unsupported py-type conversion", info);
133
+ comptime unreachable;
134
+ },
135
+ } orelse return PyErr.PyErr;
136
+ }
137
+
138
+ /// Parse Python value into Zig type. Memory management for strings is handled by Python.
139
+ /// This also means that once the original Python string is garbage collected the pointer is dangling.
140
+ /// Similary, when a PyObject is requested, the reference is borrowed.
141
+ pub fn py_to_zig(zig_type: type, py_value: *py.PyObject, allocator: ?std.mem.Allocator) !zig_type {
142
+ switch (@typeInfo(zig_type)) {
143
+ .Int => |info| {
144
+ const val = if (info.signedness == .signed) py.PyLong_AsLongLong(py_value) else py.PyLong_AsUnsignedLongLong(py_value);
145
+ if (py.PyErr_Occurred() != null) {
146
+ return PyErr.PyErr;
147
+ }
148
+ return std.math.cast(zig_type, val) orelse return raise(.ValueError, "Expected integer to fit into {any}", .{zig_type});
149
+ },
150
+ .Float => {
151
+ const val: zig_type = @floatCast(py.PyFloat_AsDouble(py_value));
152
+ if (py.PyErr_Occurred() != null) {
153
+ return PyErr.PyErr;
154
+ }
155
+ return val;
156
+ },
157
+ .Bool => {
158
+ switch (py.PyObject_IsTrue(py_value)) {
159
+ -1 => return PyErr.PyErr,
160
+ 0 => return false,
161
+ 1 => return true,
162
+ else => unreachable,
163
+ }
164
+ },
165
+ .Optional => |info| {
166
+ switch (py.Py_IsNone(py_value)) {
167
+ 1 => return null,
168
+ 0 => return try py_to_zig(info.child, py_value, allocator),
169
+ else => unreachable,
170
+ }
171
+ },
172
+ .Array => |info| {
173
+ if (info.sentinel) |_| @compileError("Sentinel is not supported");
174
+ switch (py.PyObject_Length(py_value)) {
175
+ -1 => return PyErr.PyErr,
176
+ info.len => {},
177
+ else => |len| return raise(.TypeError, "Sequence had length {}, expected {}", .{ len, info.len }),
178
+ }
179
+ var zig_value: zig_type = undefined;
180
+ for (0..info.len) |i| {
181
+ const py_value_inner = py.PySequence_GetItem(py_value, @intCast(i)) orelse return PyErr.PyErr;
182
+ defer py.Py_DECREF(py_value_inner);
183
+ zig_value[i] = try py_to_zig(info.child, py_value_inner, allocator);
184
+ }
185
+ return zig_value;
186
+ },
187
+ .Pointer => |info| {
188
+ switch (info.size) {
189
+ .One => {
190
+ if (info.child == py.PyObject) {
191
+ return py_value;
192
+ } else @compileError("Only PyObject is supported for One-Pointer");
193
+ },
194
+ .Many => @compileError("Many Pointer not supported"),
195
+ .Slice => {
196
+ if (info.child == u8) {
197
+ var size: py.Py_ssize_t = -1;
198
+ const char_ptr = py.PyUnicode_AsUTF8AndSize(py_value, &size) orelse return PyErr.PyErr;
199
+ if (size < 0) {
200
+ return PyErr.PyErr;
201
+ }
202
+ return char_ptr[0..@intCast(size)];
203
+ } else {
204
+ const len: usize = blk: {
205
+ const py_len = py.PyObject_Length(py_value);
206
+ if (py_len < 0) {
207
+ return PyErr.PyErr;
208
+ }
209
+ break :blk @intCast(py_len);
210
+ };
211
+ const slice = allocator.?.alloc(info.child, len) catch {
212
+ _ = py.PyErr_NoMemory();
213
+ return PyErr.PyErr;
214
+ };
215
+ for (slice, 0..) |*entry, i_entry| {
216
+ const py_entry = py.PySequence_GetItem(py_value, @intCast(i_entry)) orelse return PyErr.PyErr;
217
+ entry.* = try py_to_zig(info.child, py_entry, allocator);
218
+ }
219
+ return slice;
220
+ }
221
+ },
222
+ .C => @compileError("C Pointer not supported"),
223
+ }
224
+ },
225
+ .Struct => |info| {
226
+ var zig_value: zig_type = undefined;
227
+ if (info.fields.len == 0) {
228
+ return zig_value;
229
+ }
230
+ if (py.PyDict_Check(py_value) != 0) {
231
+ comptime var n_fields = 0;
232
+ inline for (info.fields) |field| {
233
+ const py_value_inner = py.PyDict_GetItemString(
234
+ py_value,
235
+ field.name,
236
+ ) orelse {
237
+ return raise(.TypeError, "Could not get dict value for key={s}", .{field.name});
238
+ };
239
+ @field(zig_value, field.name) = try py_to_zig(
240
+ field.type,
241
+ py_value_inner,
242
+ allocator,
243
+ );
244
+ n_fields += 1;
245
+ }
246
+ switch (py.PyObject_Length(py_value)) {
247
+ -1 => return PyErr.PyErr,
248
+ n_fields => return zig_value,
249
+ else => |len| return raise(.TypeError, "Dict had length {}, expected {}", .{ len, n_fields }),
250
+ }
251
+ } else {
252
+ comptime var n_fields = 0;
253
+ inline for (info.fields) |field| {
254
+ const py_value_inner = py.PySequence_GetItem(py_value, n_fields) orelse return PyErr.PyErr;
255
+ defer py.Py_DECREF(py_value_inner);
256
+ @field(zig_value, field.name) = try py_to_zig(
257
+ field.type,
258
+ py_value_inner,
259
+ allocator,
260
+ );
261
+ n_fields += 1;
262
+ }
263
+ switch (py.PyObject_Length(py_value)) {
264
+ -1 => return PyErr.PyErr,
265
+ n_fields => return zig_value,
266
+ else => |len| return raise(.TypeError, "Sequence had length {}, expected {}", .{ len, n_fields }),
267
+ }
268
+ return zig_value;
269
+ }
270
+ },
271
+ else => {},
272
+ }
273
+ @compileLog("Unsupported conversion from py to zig", @typeInfo(zig_type));
274
+ comptime unreachable;
275
+ }
@@ -0,0 +1,136 @@
1
+ const std = @import("std");
2
+ const builtin = @import("builtin");
3
+ const generated = @import("generated.zig");
4
+ const pyu = @import("py_utils.zig");
5
+ const py = pyu.py;
6
+ const zig_file = @import("inner/import_fns.zig");
7
+
8
+ fn list_to_arr(T: type, list: *std.SinglyLinkedList(T)) [list.len()]T {
9
+ var arr: [list.len()]T = undefined;
10
+ var idx: usize = list.len();
11
+ while (list.popFirst()) |node| {
12
+ idx -= 1;
13
+ arr[idx] = node.data;
14
+ }
15
+ std.debug.assert(idx == 0);
16
+ return arr;
17
+ }
18
+
19
+ var zig_ext_methods = blk: {
20
+ var methods = std.SinglyLinkedList(py.PyMethodDef){ .first = null };
21
+ const methods_node = @TypeOf(methods).Node;
22
+
23
+ for (@typeInfo(zig_file).Struct.decls) |fn_decl| {
24
+ const zig_func = @field(zig_file, fn_decl.name);
25
+ if (@typeInfo(@TypeOf(zig_func)) != .Fn) continue;
26
+ const fn_info = @typeInfo(@TypeOf(zig_func)).Fn;
27
+
28
+ var i_allocator: isize = -1;
29
+ const arg_type = std.meta.Tuple(&T: {
30
+ var types: [fn_info.params.len]type = undefined;
31
+ for (fn_info.params, 0..) |param, i_type| {
32
+ const T = param.type.?;
33
+ types[i_type] = T;
34
+ if (T == std.mem.Allocator) {
35
+ if (i_allocator != -1) {
36
+ @compileError("Can only request allocator once per function");
37
+ }
38
+ i_allocator = i_type;
39
+ }
40
+ }
41
+ break :T types;
42
+ });
43
+
44
+ const n_py_args = if (i_allocator != -1) fn_info.params.len - 1 else fn_info.params.len;
45
+
46
+ const wrapper = struct {
47
+ fn wrapper(_: ?*py.PyObject, py_args: [*]*py.PyObject, n_py_args_runtime: isize) callconv(.C) ?*py.PyObject {
48
+ var arena = std.heap.ArenaAllocator.init(pyu.gp_allocator);
49
+ defer arena.deinit();
50
+ const allocator = arena.allocator();
51
+ var args: arg_type = undefined;
52
+ if (n_py_args != n_py_args_runtime) {
53
+ pyu.raise(.Exception, "Expected {} arguments, received {}", .{ n_py_args, n_py_args_runtime }) catch {};
54
+ return null;
55
+ }
56
+ inline for (@typeInfo(arg_type).Struct.fields, 0..) |field, i_field| {
57
+ if (i_field == i_allocator) {
58
+ @field(args, field.name) = allocator;
59
+ continue;
60
+ }
61
+ const py_arg = py_args[i_field - @intFromBool(i_allocator != -1 and i_field > i_allocator)];
62
+ @field(args, field.name) = pyu.py_to_zig(field.type, py_arg, allocator) catch {
63
+ pyu.raise(.Exception, "Error converting function arguments to zig types", .{}) catch {};
64
+ return null;
65
+ };
66
+ }
67
+
68
+ const zig_ret = @call(.always_inline, zig_func, args);
69
+
70
+ const zig_ret_unwrapped = if (@typeInfo(@TypeOf(zig_ret)) == .ErrorUnion)
71
+ zig_ret catch |err| {
72
+ if (err != pyu.PyErr.PyErr) {
73
+ pyu.raise(.Exception, "Zig function returned an error: {any}", .{err}) catch {};
74
+ }
75
+ return null;
76
+ }
77
+ else
78
+ zig_ret;
79
+
80
+ return pyu.zig_to_py(zig_ret_unwrapped) catch {
81
+ pyu.raise(.Exception, "Error converting zig return values to python types", .{}) catch {};
82
+ return null;
83
+ };
84
+ }
85
+ }.wrapper;
86
+
87
+ var node = methods_node{
88
+ .data = py.PyMethodDef{
89
+ .ml_name = fn_decl.name,
90
+ .ml_meth = @ptrCast(&wrapper),
91
+ .ml_flags = py.METH_FASTCALL,
92
+ .ml_doc = null,
93
+ },
94
+ };
95
+ methods.prepend(&node);
96
+ }
97
+ var node = methods_node{
98
+ .data = py.PyMethodDef{
99
+ .ml_name = null,
100
+ .ml_meth = null,
101
+ .ml_flags = 0,
102
+ .ml_doc = null,
103
+ },
104
+ };
105
+ methods.prepend(&node);
106
+ break :blk list_to_arr(py.PyMethodDef, &methods);
107
+ };
108
+
109
+ var zig_ext_module = py.PyModuleDef{
110
+ .m_base = py.PyModuleDef_Base{
111
+ .ob_base = py.PyObject{
112
+ // .ob_refcnt = 1,
113
+ .ob_type = null,
114
+ },
115
+ .m_init = null,
116
+ .m_index = 0,
117
+ .m_copy = null,
118
+ },
119
+ .m_name = "zig_ext",
120
+ .m_doc = null,
121
+ .m_size = -1,
122
+ .m_methods = &zig_ext_methods,
123
+ .m_slots = null,
124
+ .m_traverse = null,
125
+ .m_clear = null,
126
+ .m_free = null,
127
+ };
128
+
129
+ fn init() callconv(.C) ?*py.PyObject {
130
+ const module = py.PyModule_Create(&zig_ext_module);
131
+ return module;
132
+ }
133
+
134
+ comptime {
135
+ @export(init, .{ .name = "PyInit_" ++ generated.module_name, .linkage = .strong });
136
+ }
@@ -0,0 +1,90 @@
1
+ Metadata-Version: 2.1
2
+ Name: import_zig
3
+ Version: 0.13.0
4
+ Summary: Compile and import Zig functions at runtime without building a package
5
+ Author: Felix Graßl
6
+ License: MIT
7
+ Project-URL: Homepage, https://github.com/ffelixg/import_zig
8
+ Project-URL: Issues, https://github.com/ffelixg/import_zig/issues
9
+ Keywords: zig,ziglang,import,compile
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Programming Language :: Python
12
+ Classifier: Programming Language :: Python :: 3
13
+ Requires-Python: >=3.10
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE
16
+ Requires-Dist: ziglang==0.13
17
+
18
+ # import-zig
19
+
20
+ pip install import-zig
21
+
22
+ This module provides a way to import Zig code directly into Python using a single function call to `import_zig`. Here is an example:
23
+
24
+ ```py
25
+ from import_zig import import_zig
26
+
27
+ mod = import_zig(source_code = """
28
+ pub fn Q_rsqrt(number: f32) f32 {
29
+ const threehalfs: f32 = 1.5;
30
+ const x2 = number * 0.5;
31
+ var y = number;
32
+ var i: i32 = @bitCast(y);
33
+ i = 0x5f3759df - (i >> 1);
34
+ y = @bitCast(i);
35
+ y = y * (threehalfs - (x2 * y * y));
36
+
37
+ return y;
38
+ }
39
+ """)
40
+
41
+ print(f"1 / sqrt(1.234) = {mod.Q_rsqrt(1.234)}")
42
+ ```
43
+
44
+ The main goal of this module is to make it easy to prototype Zig extensions for Python without having to engage with the Python build system. When building larger Zig extensions it is likely preferable to write your own build process with setuptools or to use a ziggy-pydust template, which provides a comptime abstraction over many aspects of the Python C API. Zig and Zig extensions are still very new though, so things will likely change. Technically you could also use this module as part of a packaging step by calling the `compile_to` function to build the binary and moving it into the packages build folder.
45
+
46
+ One approach that I expect to stay the same is the comptime wrapping for conversion between Zig and Python types as well as exception handling. This is conveniently packed into the file `py_utils.zig` and could be copy pasted into a new setuptools based project and maybe adjusted.
47
+
48
+ See the docs of the import_zig function for more details or check out the examples directory.
49
+
50
+ # File structure
51
+
52
+ The file structure that will be generated to compile the Zig code looks as follows.
53
+
54
+ ```bash
55
+ project_folder
56
+ ├── build.zig
57
+ ├── generated.zig
58
+ ├── inner
59
+ │   └── import_fns.zig
60
+ ├── py_utils.zig
61
+ └── zig_ext.zig
62
+ ```
63
+
64
+ The `inner` directory is where your code lives. When you pass a source code string or file path, it will be written / linked as the `import_fns.zig` file. When you pass a directory path, the directory will be linked as the `inner` directory above, enabling references to other files in the directory path.
65
+
66
+ The above file structure can be generated with:
67
+
68
+ ```py
69
+ import_zig.prepare("/path/to/project_folder", "module_name")
70
+ ```
71
+
72
+ This enables ZLS support for the Python C API when importing `py_utils` from `import_fns.zig`.
73
+
74
+ # Type mapping
75
+
76
+ The conversion is defined in `py_utils.zig` and applied based on the parameter / return types of the exported function. Errors are also forwarded. The solution to passing variable length data back to Python is a bit of a hack: When an exported function specifies `std.mem.Allocator` as a parameter type, then an arena allocator - which gets deallocated after the function call - will be passed into the function. The allocator can then be used to allocate and return new slices for example.
77
+
78
+ For nested types, the conversion is applied recursively.
79
+
80
+ | Conversion from Python | Zig datatype | Conversion to Python |
81
+ | --------------------------------------- | --------------------------------- | --------------------------------------------- |
82
+ | int | integer (any size / sign) | int |
83
+ | float | float (any size) | float |
84
+ | - | void | None |
85
+ | evaluated like bool() | bool | bool |
86
+ | sequence | array | list |
87
+ | sequence | non u8 const slice | list |
88
+ | str | u8 const slice | str |
89
+ | dict or sequence | struct | tuple if struct is a tuple or named tuple |
90
+ | comparison with None | optional | null -> None |
@@ -0,0 +1,12 @@
1
+ LICENSE
2
+ README.md
3
+ pyproject.toml
4
+ import_zig/__init__.py
5
+ import_zig/build.zig
6
+ import_zig/py_utils.zig
7
+ import_zig/zig_ext.zig
8
+ import_zig.egg-info/PKG-INFO
9
+ import_zig.egg-info/SOURCES.txt
10
+ import_zig.egg-info/dependency_links.txt
11
+ import_zig.egg-info/requires.txt
12
+ import_zig.egg-info/top_level.txt
@@ -0,0 +1 @@
1
+ ziglang==0.13
@@ -0,0 +1 @@
1
+ import_zig
@@ -0,0 +1,28 @@
1
+ [build-system]
2
+ requires = ["setuptools"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "import_zig"
7
+ authors = [{name = "Felix Graßl"}]
8
+ description = "Compile and import Zig functions at runtime without building a package"
9
+ readme = "README.md"
10
+ requires-python = ">=3.10"
11
+ keywords = ["zig", "ziglang", "import", "compile"]
12
+ license = {text = "MIT"}
13
+ classifiers = [
14
+ "License :: OSI Approved :: MIT License",
15
+ "Programming Language :: Python",
16
+ "Programming Language :: Python :: 3",
17
+ ]
18
+ dependencies = ["ziglang==0.13"]
19
+ version = "0.13.0"
20
+
21
+ [project.urls]
22
+ Homepage = "https://github.com/ffelixg/import_zig"
23
+ Issues = "https://github.com/ffelixg/import_zig/issues"
24
+
25
+ [tool.setuptools]
26
+ py-modules = ["import_zig"]
27
+ packages = ["import_zig"]
28
+ package-data = {import_zig = ["*.zig"]}
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+