pai-turbox 0.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of pai-turbox might be problematic. Click here for more details.

@@ -0,0 +1,6 @@
1
+ include installer_backend.py _pai_turbox_loader.py index-url.txt VERSION README.md pyproject.toml setup.py
2
+ exclude releases.json make_index.py
3
+ exclude publish_oss_index.py release-policy.json requirements-publish.txt
4
+ exclude install.py
5
+ recursive-exclude tests *
6
+ recursive-exclude .wheel-cache *
@@ -0,0 +1,118 @@
1
+ Metadata-Version: 2.4
2
+ Name: pai-turbox
3
+ Version: 0.1.1
4
+ Summary: Environment-aware installation entry for TurboX binary wheels
5
+ Author: TurboX Team
6
+ Classifier: Operating System :: POSIX :: Linux
7
+ Requires-Python: >=3.8
8
+ Description-Content-Type: text/markdown
9
+ Dynamic: author
10
+ Dynamic: classifier
11
+ Dynamic: description
12
+ Dynamic: description-content-type
13
+ Dynamic: requires-python
14
+ Dynamic: summary
15
+ Dynamic: Requires-Dist
16
+
17
+ # pai-turbox pip installation and OSS release index
18
+
19
+ The package name is `pai-turbox`; usage remains `import turbox`, existing console
20
+ commands and `ENABLE_TURBOX=1`. Installation finishes through pip, without a
21
+ second script or first-import binary download. Supported GPU PyTorch must already
22
+ be installed in the active environment.
23
+
24
+ ## Customer installation
25
+
26
+ ```sh
27
+ python -m pip install --no-cache-dir pai-turbox
28
+ ```
29
+
30
+ Install into the active GPU PyTorch environment. The archive contains installation
31
+ code only. Customers do not supply wheel-index or OSS credentials. Its embedded
32
+ public index is:
33
+
34
+ https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
35
+
36
+ The installer version is independent of the selected TurboX binary version.
37
+ Existing same-version installations are not automatically replaced when the
38
+ external index changes; see the update and cache rules below.
39
+
40
+ ## Build and installation
41
+
42
+ The PEP 517 backend inspects the target interpreter's Torch/CUDA/device, reads the
43
+ public index and selects an existing wheel. It verifies the original archive's
44
+ SHA256 and metadata, then builds a pai-turbox wheel owning that binary payload.
45
+ The compiled files stay unchanged under `_pai_turbox_payload/`. A lightweight
46
+ loader exposes them as `turbox` and preserves original startup and license
47
+ checks, without sharing files with an existing native turbox distribution.
48
+
49
+ Public METADATA, dist-info and wheel filename use pai-turbox and VERSION.
50
+ Original dependencies are resolved alongside exact target Torch and index policy
51
+ constraints. There is no separate turbox distribution dependency. Source metadata
52
+ marks Requires-Dist dynamic. RECORD is rebuilt for all owned files. Selection is
53
+ frozen between metadata generation and wheel generation.
54
+
55
+ ## Publishing an index
56
+
57
+ Install publisher dependencies from `requirements-publish.txt`, set
58
+ `TURBOX_PUBLISH_ACCESS_KEY_ID` and `TURBOX_PUBLISH_ACCESS_KEY_SECRET` in the release
59
+ job environment, then run:
60
+
61
+ ```sh
62
+ python publish_oss_index.py --output releases.json --publish
63
+ ```
64
+
65
+ The script lists `release/dist/agent/` using a slash delimiter and an additional
66
+ relative-path check. It ignores all subdirectories. New or changed wheels are
67
+ streamed, hashed and checked for ZIP integrity and matching metadata. An anonymous
68
+ HEAD verifies the download address. Unchanged ETag/size/LastModified entries reuse
69
+ previous hashes after public availability is checked. It publishes a versioned
70
+ snapshot and updates releases.json, explicitly sets object public-read ACLs, and
71
+ verifies anonymous readback. It never changes bucket or original wheel ACLs.
72
+
73
+ `approved` means these index checks passed; it does not claim every runtime
74
+ combination has been qualified. Missing CUDA information is not guessed: special
75
+ filenames remain in the inventory with review_required until explicit mappings
76
+ are supplied through release-policy.json. Dependency constraints and disabled
77
+ artifacts are also policy data, not hardcoded installer version branches.
78
+
79
+ Run one publishing job at a time. Transport/integrity errors fail the publication
80
+ instead of replacing the working index with a transient partial result. Index
81
+ writes happen only after generation finishes; retain snapshots for rollback.
82
+ Credentials are environment-only and are absent from JSON, tar and normal logs.
83
+
84
+ To publish a new installation entry after building it:
85
+
86
+ ```sh
87
+ python -m build --sdist
88
+ python publish_oss_index.py --output releases.json --publish \
89
+ --installer dist/pai_turbox-VERSION.tar.gz
90
+ ```
91
+
92
+ The publisher checks name/version, the embedded default index and absence of
93
+ credentials or business payloads before uploading the source entry. Immutable
94
+ artifact names cannot be overwritten with different bytes. It also writes the
95
+ trailing-slash Simple Index project object that pip actually requests from OSS.
96
+ Increment VERSION for a changed installer release; index-only updates do not need
97
+ a new installer artifact.
98
+
99
+ ## Selection, updates and cache
100
+
101
+ Match Python/ABI/platform, Torch/CUDA and device; select the highest compatible
102
+ approved stable version, then the latest OSS LastModified within that version.
103
+ Times are normalized to UTC. Pin a version with
104
+ `--config-settings turbox-version=0.2.11`, or an exact build with
105
+ `--config-settings wheel-filename=EXACT_WHEEL_FILENAME`.
106
+
107
+ Keep `--no-cache-dir` when reselecting an environment-dependent wheel. Standard
108
+ wheel tags do not encode Torch/CUDA/device. A runtime guard rejects a mismatched
109
+ cached binary, but rejection is not automatic cache invalidation. An existing
110
+ same-version pai-turbox is not automatically updated by bare `pip install` merely
111
+ because the JSON changed. Publish new installer release versions for normal
112
+ package-name upgrades. Explicit same-version rebuilds can download/extract the
113
+ source tar and pip-install the local tar, as covered by development verification;
114
+ there is no separate installer script.
115
+
116
+ References: [OSS object ACL](https://www.alibabacloud.com/help/en/oss/developer-reference/manage-object-acls-1),
117
+ [pip build interface](https://pip.pypa.io/en/stable/reference/build-system/),
118
+ [pip caching](https://pip.pypa.io/en/stable/topics/caching/).
@@ -0,0 +1,102 @@
1
+ # pai-turbox pip installation and OSS release index
2
+
3
+ The package name is `pai-turbox`; usage remains `import turbox`, existing console
4
+ commands and `ENABLE_TURBOX=1`. Installation finishes through pip, without a
5
+ second script or first-import binary download. Supported GPU PyTorch must already
6
+ be installed in the active environment.
7
+
8
+ ## Customer installation
9
+
10
+ ```sh
11
+ python -m pip install --no-cache-dir pai-turbox
12
+ ```
13
+
14
+ Install into the active GPU PyTorch environment. The archive contains installation
15
+ code only. Customers do not supply wheel-index or OSS credentials. Its embedded
16
+ public index is:
17
+
18
+ https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
19
+
20
+ The installer version is independent of the selected TurboX binary version.
21
+ Existing same-version installations are not automatically replaced when the
22
+ external index changes; see the update and cache rules below.
23
+
24
+ ## Build and installation
25
+
26
+ The PEP 517 backend inspects the target interpreter's Torch/CUDA/device, reads the
27
+ public index and selects an existing wheel. It verifies the original archive's
28
+ SHA256 and metadata, then builds a pai-turbox wheel owning that binary payload.
29
+ The compiled files stay unchanged under `_pai_turbox_payload/`. A lightweight
30
+ loader exposes them as `turbox` and preserves original startup and license
31
+ checks, without sharing files with an existing native turbox distribution.
32
+
33
+ Public METADATA, dist-info and wheel filename use pai-turbox and VERSION.
34
+ Original dependencies are resolved alongside exact target Torch and index policy
35
+ constraints. There is no separate turbox distribution dependency. Source metadata
36
+ marks Requires-Dist dynamic. RECORD is rebuilt for all owned files. Selection is
37
+ frozen between metadata generation and wheel generation.
38
+
39
+ ## Publishing an index
40
+
41
+ Install publisher dependencies from `requirements-publish.txt`, set
42
+ `TURBOX_PUBLISH_ACCESS_KEY_ID` and `TURBOX_PUBLISH_ACCESS_KEY_SECRET` in the release
43
+ job environment, then run:
44
+
45
+ ```sh
46
+ python publish_oss_index.py --output releases.json --publish
47
+ ```
48
+
49
+ The script lists `release/dist/agent/` using a slash delimiter and an additional
50
+ relative-path check. It ignores all subdirectories. New or changed wheels are
51
+ streamed, hashed and checked for ZIP integrity and matching metadata. An anonymous
52
+ HEAD verifies the download address. Unchanged ETag/size/LastModified entries reuse
53
+ previous hashes after public availability is checked. It publishes a versioned
54
+ snapshot and updates releases.json, explicitly sets object public-read ACLs, and
55
+ verifies anonymous readback. It never changes bucket or original wheel ACLs.
56
+
57
+ `approved` means these index checks passed; it does not claim every runtime
58
+ combination has been qualified. Missing CUDA information is not guessed: special
59
+ filenames remain in the inventory with review_required until explicit mappings
60
+ are supplied through release-policy.json. Dependency constraints and disabled
61
+ artifacts are also policy data, not hardcoded installer version branches.
62
+
63
+ Run one publishing job at a time. Transport/integrity errors fail the publication
64
+ instead of replacing the working index with a transient partial result. Index
65
+ writes happen only after generation finishes; retain snapshots for rollback.
66
+ Credentials are environment-only and are absent from JSON, tar and normal logs.
67
+
68
+ To publish a new installation entry after building it:
69
+
70
+ ```sh
71
+ python -m build --sdist
72
+ python publish_oss_index.py --output releases.json --publish \
73
+ --installer dist/pai_turbox-VERSION.tar.gz
74
+ ```
75
+
76
+ The publisher checks name/version, the embedded default index and absence of
77
+ credentials or business payloads before uploading the source entry. Immutable
78
+ artifact names cannot be overwritten with different bytes. It also writes the
79
+ trailing-slash Simple Index project object that pip actually requests from OSS.
80
+ Increment VERSION for a changed installer release; index-only updates do not need
81
+ a new installer artifact.
82
+
83
+ ## Selection, updates and cache
84
+
85
+ Match Python/ABI/platform, Torch/CUDA and device; select the highest compatible
86
+ approved stable version, then the latest OSS LastModified within that version.
87
+ Times are normalized to UTC. Pin a version with
88
+ `--config-settings turbox-version=0.2.11`, or an exact build with
89
+ `--config-settings wheel-filename=EXACT_WHEEL_FILENAME`.
90
+
91
+ Keep `--no-cache-dir` when reselecting an environment-dependent wheel. Standard
92
+ wheel tags do not encode Torch/CUDA/device. A runtime guard rejects a mismatched
93
+ cached binary, but rejection is not automatic cache invalidation. An existing
94
+ same-version pai-turbox is not automatically updated by bare `pip install` merely
95
+ because the JSON changed. Publish new installer release versions for normal
96
+ package-name upgrades. Explicit same-version rebuilds can download/extract the
97
+ source tar and pip-install the local tar, as covered by development verification;
98
+ there is no separate installer script.
99
+
100
+ References: [OSS object ACL](https://www.alibabacloud.com/help/en/oss/developer-reference/manage-object-acls-1),
101
+ [pip build interface](https://pip.pypa.io/en/stable/reference/build-system/),
102
+ [pip caching](https://pip.pypa.io/en/stable/topics/caching/).
@@ -0,0 +1 @@
1
+ 0.1.1
@@ -0,0 +1,43 @@
1
+ """Expose the binary payload as turbox without sharing another wheel's files."""
2
+
3
+ import importlib.machinery
4
+ import json
5
+ import os
6
+ from pathlib import Path
7
+ import sys
8
+
9
+ PAYLOAD = Path(__file__).resolve().parent / "_pai_turbox_payload"
10
+
11
+
12
+ class _TurboXFinder:
13
+ def find_spec(self, fullname, path=None, target=None):
14
+ if fullname != "turbox" or path is not None:
15
+ return None
16
+ # Relocated extensions must have Torch's shared libraries loaded first.
17
+ # This is deferred until TurboX is actually imported.
18
+ import torch
19
+ entry = json.loads((PAYLOAD / "selection.json").read_text())["wheels"][0]
20
+ device_mismatch = False
21
+ if torch.cuda.is_available():
22
+ name = torch.cuda.get_device_name(0)
23
+ device = "ppu" if name.startswith(("PPU", "ZW-M")) else "nv" if name.startswith(
24
+ ("NVIDIA", "Tesla", "Quadro", "GeForce")
25
+ ) else "unknown"
26
+ device_mismatch = device != entry["device"]
27
+ if (str(torch.__version__) != entry["torch"]
28
+ or torch.version.cuda != entry["cuda"] or device_mismatch):
29
+ raise ImportError(
30
+ "The installed pai-turbox binary belongs to a different Torch/CUDA/device environment. "
31
+ "Reinstall the source entry with --no-cache-dir to select a matching binary."
32
+ )
33
+ return importlib.machinery.PathFinder.find_spec(fullname, [str(PAYLOAD)])
34
+
35
+
36
+ def install():
37
+ if not any(isinstance(finder, _TurboXFinder) for finder in sys.meta_path):
38
+ sys.meta_path.insert(0, _TurboXFinder())
39
+ if str(PAYLOAD) not in sys.path:
40
+ sys.path.insert(0, str(PAYLOAD))
41
+ if os.environ.get("ENABLE_TURBOX") == "1":
42
+ # Preserve the original wheel's startup behavior and license checks.
43
+ import _turbox_bootstrap # noqa: F401
@@ -0,0 +1 @@
1
+ https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
@@ -0,0 +1,316 @@
1
+ """PEP 517 adapter selecting original, hash-pinned TurboX binary wheels."""
2
+
3
+ import json
4
+ import base64
5
+ import csv
6
+ import hashlib
7
+ import io
8
+ import tempfile
9
+ import tarfile
10
+ import zipfile
11
+ from email.parser import BytesParser
12
+ import os
13
+ from pathlib import Path
14
+ import subprocess
15
+ import sys
16
+ import urllib.parse
17
+ import urllib.request
18
+ from datetime import datetime, timezone
19
+
20
+ from packaging.tags import sys_tags
21
+ from packaging.utils import parse_wheel_filename
22
+ from packaging.version import Version
23
+ from setuptools import build_meta as _backend
24
+
25
+ ROOT = Path(__file__).resolve().parent
26
+ PROBE = r'''
27
+ import json, platform, sys, torch
28
+ if not torch.cuda.is_available():
29
+ raise RuntimeError("No available GPU in the target PyTorch environment")
30
+ name = torch.cuda.get_device_name(0)
31
+ if name.startswith(("PPU", "ZW-M")):
32
+ device = "ppu"
33
+ elif name.startswith(("NVIDIA", "Tesla", "Quadro", "GeForce")):
34
+ device = "nv"
35
+ else:
36
+ raise RuntimeError("Unknown GPU family: " + name)
37
+ print("PAI_TURBOX_ENV=" + json.dumps({
38
+ "torch": str(torch.__version__), "cuda": torch.version.cuda,
39
+ "device": device, "device_name": name,
40
+ "python": "cp" + str(sys.version_info.major) + str(sys.version_info.minor),
41
+ "system": platform.system(), "machine": platform.machine(),
42
+ }))
43
+ '''
44
+
45
+
46
+ def detect_environment():
47
+ # pip's build isolation is sys.path/PYTHONPATH based. -I starts the same
48
+ # target interpreter without that overlay, so it sees preinstalled torch.
49
+ env = dict(os.environ, ENABLE_TURBOX="0")
50
+ env.pop("PYTHONPATH", None)
51
+ result = subprocess.run(
52
+ [sys.executable, "-I", "-c", PROBE], env=env, text=True,
53
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=90,
54
+ )
55
+ if result.returncode:
56
+ raise RuntimeError(
57
+ "Cannot inspect the target PyTorch environment. Install supported "
58
+ "GPU PyTorch first. Probe output:\n" + result.stderr[-4000:]
59
+ )
60
+ for line in reversed(result.stdout.splitlines()):
61
+ if line.startswith("PAI_TURBOX_ENV="):
62
+ return json.loads(line.split("=", 1)[1])
63
+ raise RuntimeError("Target environment probe returned no environment data")
64
+
65
+
66
+ def load_index(location):
67
+ if not location:
68
+ raise RuntimeError(
69
+ "Set TURBOX_WHEEL_INDEX to a release index JSON path or HTTPS URL. "
70
+ "The installation archive does not contain a frozen support matrix."
71
+ )
72
+ if location.startswith("https://"):
73
+ with urllib.request.urlopen(location, timeout=30) as response:
74
+ raw = response.read(8 * 1024 * 1024 + 1)
75
+ if len(raw) > 8 * 1024 * 1024:
76
+ raise RuntimeError("Release index exceeds 8 MiB")
77
+ manifest = json.loads(raw)
78
+ else:
79
+ if "://" in location:
80
+ raise RuntimeError("Remote indexes must use HTTPS")
81
+ manifest = json.loads(Path(location).read_text())
82
+ if manifest.get("schema_version") != 1:
83
+ raise RuntimeError("Unsupported release index schema; update the installer")
84
+ return manifest
85
+
86
+
87
+ def modified_time(entry):
88
+ value = entry.get("last_modified", "")
89
+ try:
90
+ result = datetime.fromisoformat(value.replace("Z", "+00:00"))
91
+ if result.tzinfo is None or result.utcoffset() is None:
92
+ raise ValueError("timezone missing")
93
+ return result.astimezone(timezone.utc)
94
+ except (ValueError, TypeError, AttributeError) as exc:
95
+ raise RuntimeError(
96
+ "A timezone-aware last_modified is required to order builds: "
97
+ + entry["filename"]
98
+ ) from exc
99
+
100
+
101
+ def select_release(manifest, environment, supported_tags=None, requested_version=None,
102
+ requested_build=None):
103
+ tags = set(sys_tags() if supported_tags is None else supported_tags)
104
+ if environment["system"] != "Linux" or environment["machine"] != "x86_64":
105
+ raise RuntimeError("Only Linux x86_64 is currently supported")
106
+ keys = ("cuda", "device", "python")
107
+ matches = []
108
+ for entry in manifest["wheels"]:
109
+ if not entry.get("approved", False):
110
+ continue
111
+ if requested_build and entry["filename"] != requested_build:
112
+ continue
113
+ if requested_version and Version(Version(entry["version"]).public) != Version(requested_version):
114
+ continue
115
+ expected_torch = Version(entry["torch"])
116
+ actual_torch = Version(environment["torch"])
117
+ if expected_torch.public != actual_torch.public:
118
+ continue
119
+ if expected_torch.local is not None and expected_torch != actual_torch:
120
+ continue
121
+ if not all(entry[key] == environment[key] for key in keys):
122
+ continue
123
+ name, version, _, wheel_tags = parse_wheel_filename(entry["filename"])
124
+ if name != "turbox" or str(version) != entry["version"]:
125
+ raise RuntimeError("Manifest filename and package metadata disagree")
126
+ if not tags.intersection(wheel_tags):
127
+ continue
128
+ digest = entry["sha256"]
129
+ if len(digest) != 64 or any(c not in "0123456789abcdef" for c in digest):
130
+ raise RuntimeError("A valid SHA256 is required for every release")
131
+ url = urllib.parse.urlsplit(entry["url"])
132
+ if (url.scheme != "https" or not url.hostname or url.username
133
+ or url.password or url.fragment
134
+ or urllib.parse.unquote(url.path.rsplit("/", 1)[-1]) != entry["filename"]):
135
+ raise RuntimeError("A matching HTTPS wheel URL is required")
136
+ matches.append(entry)
137
+ # Consider only the highest stable public TurboX release for this exact
138
+ # environment. A prerelease must be explicitly opted in through the index.
139
+ matches = [e for e in matches if not Version(e["version"]).is_prerelease
140
+ or e.get("allow_prerelease", False)]
141
+ if matches:
142
+ latest = max(
143
+ Version(Version(e["version"]).public) for e in matches
144
+ )
145
+ matches = [e for e in matches if Version(Version(e["version"]).public) == latest]
146
+ if len(matches) > 1:
147
+ newest = max(modified_time(entry) for entry in matches)
148
+ matches = [entry for entry in matches if modified_time(entry) == newest]
149
+ if len(matches) != 1:
150
+ raise RuntimeError(
151
+ f"Expected one approved TurboX wheel, found {len(matches)} for "
152
+ + json.dumps(environment, sort_keys=True)
153
+ )
154
+ return matches[0]
155
+
156
+
157
+ def selected_plan(config_settings=None, prepared=None):
158
+ settings = config_settings or {}
159
+ location = (settings.get("wheel-index") or os.environ.get("TURBOX_WHEEL_INDEX")
160
+ or (ROOT / "index-url.txt").read_text().strip())
161
+ manifest = prepared if prepared is not None else load_index(location)
162
+ environment = detect_environment()
163
+ entry = select_release(
164
+ manifest, environment,
165
+ requested_version=settings.get("turbox-version") or os.environ.get("TURBOX_VERSION"),
166
+ requested_build=settings.get("wheel-filename") or os.environ.get("TURBOX_WHEEL_FILENAME"),
167
+ )
168
+ print("[pai-turbox] Selected " + entry["filename"], flush=True)
169
+ return {"schema_version": 1, "wheels": [dict(entry, torch=environment["torch"])]}
170
+
171
+
172
+ def download_wheel(entry):
173
+ cache = ROOT / ".wheel-cache"
174
+ cache.mkdir(exist_ok=True)
175
+ path = cache / entry["filename"]
176
+ if path.is_file() and hashlib.sha256(path.read_bytes()).hexdigest() == entry["sha256"]:
177
+ return path
178
+ with tempfile.NamedTemporaryFile(dir=cache, delete=False) as output:
179
+ temporary = Path(output.name)
180
+ try:
181
+ digest = hashlib.sha256()
182
+ with urllib.request.urlopen(entry["url"], timeout=60) as response:
183
+ while True:
184
+ chunk = response.read(1024 * 1024)
185
+ if not chunk:
186
+ break
187
+ digest.update(chunk)
188
+ output.write(chunk)
189
+ output.close()
190
+ if digest.hexdigest() != entry["sha256"]:
191
+ raise RuntimeError("Downloaded TurboX wheel SHA256 mismatch")
192
+ temporary.replace(path)
193
+ finally:
194
+ temporary.unlink(missing_ok=True)
195
+ return path
196
+
197
+
198
+ def public_metadata():
199
+ with tempfile.TemporaryDirectory() as directory:
200
+ name = _backend.prepare_metadata_for_build_wheel(directory)
201
+ return (Path(directory) / name / "METADATA").read_bytes()
202
+
203
+
204
+ def package_files(plan):
205
+ entry = plan["wheels"][0]
206
+ archive = download_wheel(entry)
207
+ public_version = (ROOT / "VERSION").read_text().strip()
208
+ destination = "pai_turbox-" + public_version + ".dist-info"
209
+ output = {}
210
+ with zipfile.ZipFile(archive) as wheel:
211
+ metadata_paths = [n for n in wheel.namelist() if n.endswith(".dist-info/METADATA")]
212
+ if len(metadata_paths) != 1:
213
+ raise RuntimeError("Expected one original wheel METADATA")
214
+ origin = metadata_paths[0].rsplit("/", 1)[0]
215
+ original = BytesParser().parsebytes(wheel.read(metadata_paths[0]))
216
+ if original["Name"] != "turbox" or original["Version"] != entry["version"]:
217
+ raise RuntimeError("Downloaded wheel metadata differs from selected release")
218
+ message = BytesParser().parsebytes(public_metadata())
219
+ if message["Name"] != "pai-turbox" or message["Version"] != public_version:
220
+ raise RuntimeError("Source entry metadata differs from public package name/version")
221
+ # The public distribution owns the binary payload; no turbox dependency.
222
+ for requirement in original.get_all("Requires-Dist", []):
223
+ message["Requires-Dist"] = requirement
224
+ message["Requires-Dist"] = "torch==" + entry["torch"]
225
+ for requirement in entry.get("extra_requires", []):
226
+ message["Requires-Dist"] = requirement
227
+ for name in wheel.namelist():
228
+ if name.endswith("/"):
229
+ continue
230
+ parts = Path(name).parts
231
+ if name.startswith("/") or ".." in parts or ".data" in parts[0]:
232
+ raise RuntimeError("Unsupported original wheel installation path: " + name)
233
+ if name.startswith(origin + "/"):
234
+ relative = name[len(origin) + 1:]
235
+ if relative in ("RECORD", "RECORD.jws", "RECORD.p7s"):
236
+ continue
237
+ output[destination + "/" + relative] = wheel.read(name)
238
+ else:
239
+ output["_pai_turbox_payload/" + name] = wheel.read(name)
240
+ output[destination + "/METADATA"] = message.as_bytes()
241
+ snapshot = json.dumps(plan, sort_keys=True).encode()
242
+ output[destination + "/turbox-selection.json"] = snapshot
243
+ output["_pai_turbox_payload/selection.json"] = snapshot
244
+ output["_pai_turbox_loader.py"] = (ROOT / "_pai_turbox_loader.py").read_bytes()
245
+ output["000_pai_turbox.pth"] = b"import _pai_turbox_loader; _pai_turbox_loader.install()\n"
246
+ return destination, output
247
+
248
+
249
+ def get_requires_for_build_wheel(config_settings=None):
250
+ return []
251
+
252
+
253
+ def get_requires_for_build_sdist(config_settings=None):
254
+ return []
255
+
256
+
257
+ def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
258
+ plan = selected_plan(config_settings)
259
+ destination, files = package_files(plan)
260
+ root = Path(metadata_directory)
261
+ for name, data in files.items():
262
+ if name.startswith(destination + "/"):
263
+ path = root / name
264
+ path.parent.mkdir(parents=True, exist_ok=True)
265
+ path.write_bytes(data)
266
+ return destination
267
+
268
+
269
+ def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
270
+ snapshot = Path(metadata_directory) / "turbox-selection.json" if metadata_directory else None
271
+ prepared = json.loads(snapshot.read_text()) if snapshot and snapshot.is_file() else None
272
+ plan = selected_plan(config_settings, prepared)
273
+ destination, files = package_files(plan)
274
+ if metadata_directory:
275
+ # Preserve every metadata byte prepared for pip's dependency resolver.
276
+ for path in Path(metadata_directory).rglob("*"):
277
+ if path.is_file() and path.name != "RECORD":
278
+ files[destination + "/" + path.relative_to(metadata_directory).as_posix()] = path.read_bytes()
279
+ rows = []
280
+ for name, data in sorted(files.items()):
281
+ digest = base64.urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b"=").decode()
282
+ rows.append((name, "sha256=" + digest, str(len(data))))
283
+ record = destination + "/RECORD"
284
+ rows.append((record, "", ""))
285
+ stream = io.StringIO(newline="")
286
+ csv.writer(stream).writerows(rows)
287
+ files[record] = stream.getvalue().encode()
288
+ tags = plan["wheels"][0]["filename"].rsplit("-", 3)[-3:]
289
+ version = (ROOT / "VERSION").read_text().strip()
290
+ filename = "pai_turbox-" + version + "-" + "-".join(tags)
291
+ Path(wheel_directory).mkdir(parents=True, exist_ok=True)
292
+ with zipfile.ZipFile(Path(wheel_directory) / filename, "w", zipfile.ZIP_DEFLATED) as wheel:
293
+ for name, data in files.items():
294
+ wheel.writestr(name, data)
295
+ return filename
296
+
297
+
298
+ def build_sdist(sdist_directory, config_settings=None):
299
+ name = _backend.build_sdist(sdist_directory, config_settings)
300
+ archive = Path(sdist_directory) / name
301
+ temporary = archive.with_suffix(".tmp")
302
+ # PEP 643: dependencies are generated from the selected binary on the target.
303
+ # All other public metadata remains the source entry's static metadata.
304
+ with tarfile.open(archive, "r:gz") as source, tarfile.open(temporary, "w:gz") as target:
305
+ for member in source:
306
+ data = source.extractfile(member) if member.isfile() else None
307
+ if data is not None and member.name.endswith("/PKG-INFO"):
308
+ message = BytesParser().parsebytes(data.read())
309
+ if "requires-dist" not in [x.lower() for x in message.get_all("Dynamic", [])]:
310
+ message["Dynamic"] = "Requires-Dist"
311
+ raw = message.as_bytes()
312
+ member.size = len(raw)
313
+ data = io.BytesIO(raw)
314
+ target.addfile(member, data)
315
+ temporary.replace(archive)
316
+ return name
@@ -0,0 +1,118 @@
1
+ Metadata-Version: 2.4
2
+ Name: pai-turbox
3
+ Version: 0.1.1
4
+ Summary: Environment-aware installation entry for TurboX binary wheels
5
+ Author: TurboX Team
6
+ Classifier: Operating System :: POSIX :: Linux
7
+ Requires-Python: >=3.8
8
+ Description-Content-Type: text/markdown
9
+ Dynamic: author
10
+ Dynamic: classifier
11
+ Dynamic: description
12
+ Dynamic: description-content-type
13
+ Dynamic: requires-python
14
+ Dynamic: summary
15
+ Dynamic: Requires-Dist
16
+
17
+ # pai-turbox pip installation and OSS release index
18
+
19
+ The package name is `pai-turbox`; usage remains `import turbox`, existing console
20
+ commands and `ENABLE_TURBOX=1`. Installation finishes through pip, without a
21
+ second script or first-import binary download. Supported GPU PyTorch must already
22
+ be installed in the active environment.
23
+
24
+ ## Customer installation
25
+
26
+ ```sh
27
+ python -m pip install --no-cache-dir pai-turbox
28
+ ```
29
+
30
+ Install into the active GPU PyTorch environment. The archive contains installation
31
+ code only. Customers do not supply wheel-index or OSS credentials. Its embedded
32
+ public index is:
33
+
34
+ https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
35
+
36
+ The installer version is independent of the selected TurboX binary version.
37
+ Existing same-version installations are not automatically replaced when the
38
+ external index changes; see the update and cache rules below.
39
+
40
+ ## Build and installation
41
+
42
+ The PEP 517 backend inspects the target interpreter's Torch/CUDA/device, reads the
43
+ public index and selects an existing wheel. It verifies the original archive's
44
+ SHA256 and metadata, then builds a pai-turbox wheel owning that binary payload.
45
+ The compiled files stay unchanged under `_pai_turbox_payload/`. A lightweight
46
+ loader exposes them as `turbox` and preserves original startup and license
47
+ checks, without sharing files with an existing native turbox distribution.
48
+
49
+ Public METADATA, dist-info and wheel filename use pai-turbox and VERSION.
50
+ Original dependencies are resolved alongside exact target Torch and index policy
51
+ constraints. There is no separate turbox distribution dependency. Source metadata
52
+ marks Requires-Dist dynamic. RECORD is rebuilt for all owned files. Selection is
53
+ frozen between metadata generation and wheel generation.
54
+
55
+ ## Publishing an index
56
+
57
+ Install publisher dependencies from `requirements-publish.txt`, set
58
+ `TURBOX_PUBLISH_ACCESS_KEY_ID` and `TURBOX_PUBLISH_ACCESS_KEY_SECRET` in the release
59
+ job environment, then run:
60
+
61
+ ```sh
62
+ python publish_oss_index.py --output releases.json --publish
63
+ ```
64
+
65
+ The script lists `release/dist/agent/` using a slash delimiter and an additional
66
+ relative-path check. It ignores all subdirectories. New or changed wheels are
67
+ streamed, hashed and checked for ZIP integrity and matching metadata. An anonymous
68
+ HEAD verifies the download address. Unchanged ETag/size/LastModified entries reuse
69
+ previous hashes after public availability is checked. It publishes a versioned
70
+ snapshot and updates releases.json, explicitly sets object public-read ACLs, and
71
+ verifies anonymous readback. It never changes bucket or original wheel ACLs.
72
+
73
+ `approved` means these index checks passed; it does not claim every runtime
74
+ combination has been qualified. Missing CUDA information is not guessed: special
75
+ filenames remain in the inventory with review_required until explicit mappings
76
+ are supplied through release-policy.json. Dependency constraints and disabled
77
+ artifacts are also policy data, not hardcoded installer version branches.
78
+
79
+ Run one publishing job at a time. Transport/integrity errors fail the publication
80
+ instead of replacing the working index with a transient partial result. Index
81
+ writes happen only after generation finishes; retain snapshots for rollback.
82
+ Credentials are environment-only and are absent from JSON, tar and normal logs.
83
+
84
+ To publish a new installation entry after building it:
85
+
86
+ ```sh
87
+ python -m build --sdist
88
+ python publish_oss_index.py --output releases.json --publish \
89
+ --installer dist/pai_turbox-VERSION.tar.gz
90
+ ```
91
+
92
+ The publisher checks name/version, the embedded default index and absence of
93
+ credentials or business payloads before uploading the source entry. Immutable
94
+ artifact names cannot be overwritten with different bytes. It also writes the
95
+ trailing-slash Simple Index project object that pip actually requests from OSS.
96
+ Increment VERSION for a changed installer release; index-only updates do not need
97
+ a new installer artifact.
98
+
99
+ ## Selection, updates and cache
100
+
101
+ Match Python/ABI/platform, Torch/CUDA and device; select the highest compatible
102
+ approved stable version, then the latest OSS LastModified within that version.
103
+ Times are normalized to UTC. Pin a version with
104
+ `--config-settings turbox-version=0.2.11`, or an exact build with
105
+ `--config-settings wheel-filename=EXACT_WHEEL_FILENAME`.
106
+
107
+ Keep `--no-cache-dir` when reselecting an environment-dependent wheel. Standard
108
+ wheel tags do not encode Torch/CUDA/device. A runtime guard rejects a mismatched
109
+ cached binary, but rejection is not automatic cache invalidation. An existing
110
+ same-version pai-turbox is not automatically updated by bare `pip install` merely
111
+ because the JSON changed. Publish new installer release versions for normal
112
+ package-name upgrades. Explicit same-version rebuilds can download/extract the
113
+ source tar and pip-install the local tar, as covered by development verification;
114
+ there is no separate installer script.
115
+
116
+ References: [OSS object ACL](https://www.alibabacloud.com/help/en/oss/developer-reference/manage-object-acls-1),
117
+ [pip build interface](https://pip.pypa.io/en/stable/reference/build-system/),
118
+ [pip caching](https://pip.pypa.io/en/stable/topics/caching/).
@@ -0,0 +1,12 @@
1
+ MANIFEST.in
2
+ README.md
3
+ VERSION
4
+ _pai_turbox_loader.py
5
+ index-url.txt
6
+ installer_backend.py
7
+ pyproject.toml
8
+ setup.py
9
+ pai_turbox.egg-info/PKG-INFO
10
+ pai_turbox.egg-info/SOURCES.txt
11
+ pai_turbox.egg-info/dependency_links.txt
12
+ pai_turbox.egg-info/top_level.txt
@@ -0,0 +1,4 @@
1
+ [build-system]
2
+ requires = ["setuptools>=68", "wheel", "packaging>=24"]
3
+ build-backend = "installer_backend"
4
+ backend-path = ["."]
@@ -0,0 +1,4 @@
1
+ [egg_info]
2
+ tag_build =
3
+ tag_date = 0
4
+
@@ -0,0 +1,19 @@
1
+ """Build the public installation entry, never TurboX business sources."""
2
+ from pathlib import Path
3
+
4
+ from setuptools import setup
5
+
6
+ ROOT = Path(__file__).resolve().parent
7
+ setup(
8
+ name="pai-turbox",
9
+ version=(ROOT / "VERSION").read_text().strip(),
10
+ description="Environment-aware installation entry for TurboX binary wheels",
11
+ long_description=(ROOT / "README.md").read_text(),
12
+ long_description_content_type="text/markdown",
13
+ author="TurboX Team",
14
+ python_requires=">=3.8",
15
+ packages=[],
16
+ py_modules=[],
17
+ install_requires=[],
18
+ classifiers=["Operating System :: POSIX :: Linux"],
19
+ )