pai-turbox 0.1.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of pai-turbox might be problematic. Click here for more details.
- pai_turbox-0.1.1/MANIFEST.in +6 -0
- pai_turbox-0.1.1/PKG-INFO +118 -0
- pai_turbox-0.1.1/README.md +102 -0
- pai_turbox-0.1.1/VERSION +1 -0
- pai_turbox-0.1.1/_pai_turbox_loader.py +43 -0
- pai_turbox-0.1.1/index-url.txt +1 -0
- pai_turbox-0.1.1/installer_backend.py +316 -0
- pai_turbox-0.1.1/pai_turbox.egg-info/PKG-INFO +118 -0
- pai_turbox-0.1.1/pai_turbox.egg-info/SOURCES.txt +12 -0
- pai_turbox-0.1.1/pai_turbox.egg-info/dependency_links.txt +1 -0
- pai_turbox-0.1.1/pai_turbox.egg-info/top_level.txt +1 -0
- pai_turbox-0.1.1/pyproject.toml +4 -0
- pai_turbox-0.1.1/setup.cfg +4 -0
- pai_turbox-0.1.1/setup.py +19 -0
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
include installer_backend.py _pai_turbox_loader.py index-url.txt VERSION README.md pyproject.toml setup.py
|
|
2
|
+
exclude releases.json make_index.py
|
|
3
|
+
exclude publish_oss_index.py release-policy.json requirements-publish.txt
|
|
4
|
+
exclude install.py
|
|
5
|
+
recursive-exclude tests *
|
|
6
|
+
recursive-exclude .wheel-cache *
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pai-turbox
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Environment-aware installation entry for TurboX binary wheels
|
|
5
|
+
Author: TurboX Team
|
|
6
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
7
|
+
Requires-Python: >=3.8
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
Dynamic: author
|
|
10
|
+
Dynamic: classifier
|
|
11
|
+
Dynamic: description
|
|
12
|
+
Dynamic: description-content-type
|
|
13
|
+
Dynamic: requires-python
|
|
14
|
+
Dynamic: summary
|
|
15
|
+
Dynamic: Requires-Dist
|
|
16
|
+
|
|
17
|
+
# pai-turbox pip installation and OSS release index
|
|
18
|
+
|
|
19
|
+
The package name is `pai-turbox`; usage remains `import turbox`, existing console
|
|
20
|
+
commands and `ENABLE_TURBOX=1`. Installation finishes through pip, without a
|
|
21
|
+
second script or first-import binary download. Supported GPU PyTorch must already
|
|
22
|
+
be installed in the active environment.
|
|
23
|
+
|
|
24
|
+
## Customer installation
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
python -m pip install --no-cache-dir pai-turbox
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Install into the active GPU PyTorch environment. The archive contains installation
|
|
31
|
+
code only. Customers do not supply wheel-index or OSS credentials. Its embedded
|
|
32
|
+
public index is:
|
|
33
|
+
|
|
34
|
+
https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
|
|
35
|
+
|
|
36
|
+
The installer version is independent of the selected TurboX binary version.
|
|
37
|
+
Existing same-version installations are not automatically replaced when the
|
|
38
|
+
external index changes; see the update and cache rules below.
|
|
39
|
+
|
|
40
|
+
## Build and installation
|
|
41
|
+
|
|
42
|
+
The PEP 517 backend inspects the target interpreter's Torch/CUDA/device, reads the
|
|
43
|
+
public index and selects an existing wheel. It verifies the original archive's
|
|
44
|
+
SHA256 and metadata, then builds a pai-turbox wheel owning that binary payload.
|
|
45
|
+
The compiled files stay unchanged under `_pai_turbox_payload/`. A lightweight
|
|
46
|
+
loader exposes them as `turbox` and preserves original startup and license
|
|
47
|
+
checks, without sharing files with an existing native turbox distribution.
|
|
48
|
+
|
|
49
|
+
Public METADATA, dist-info and wheel filename use pai-turbox and VERSION.
|
|
50
|
+
Original dependencies are resolved alongside exact target Torch and index policy
|
|
51
|
+
constraints. There is no separate turbox distribution dependency. Source metadata
|
|
52
|
+
marks Requires-Dist dynamic. RECORD is rebuilt for all owned files. Selection is
|
|
53
|
+
frozen between metadata generation and wheel generation.
|
|
54
|
+
|
|
55
|
+
## Publishing an index
|
|
56
|
+
|
|
57
|
+
Install publisher dependencies from `requirements-publish.txt`, set
|
|
58
|
+
`TURBOX_PUBLISH_ACCESS_KEY_ID` and `TURBOX_PUBLISH_ACCESS_KEY_SECRET` in the release
|
|
59
|
+
job environment, then run:
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
python publish_oss_index.py --output releases.json --publish
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
The script lists `release/dist/agent/` using a slash delimiter and an additional
|
|
66
|
+
relative-path check. It ignores all subdirectories. New or changed wheels are
|
|
67
|
+
streamed, hashed and checked for ZIP integrity and matching metadata. An anonymous
|
|
68
|
+
HEAD verifies the download address. Unchanged ETag/size/LastModified entries reuse
|
|
69
|
+
previous hashes after public availability is checked. It publishes a versioned
|
|
70
|
+
snapshot and updates releases.json, explicitly sets object public-read ACLs, and
|
|
71
|
+
verifies anonymous readback. It never changes bucket or original wheel ACLs.
|
|
72
|
+
|
|
73
|
+
`approved` means these index checks passed; it does not claim every runtime
|
|
74
|
+
combination has been qualified. Missing CUDA information is not guessed: special
|
|
75
|
+
filenames remain in the inventory with review_required until explicit mappings
|
|
76
|
+
are supplied through release-policy.json. Dependency constraints and disabled
|
|
77
|
+
artifacts are also policy data, not hardcoded installer version branches.
|
|
78
|
+
|
|
79
|
+
Run one publishing job at a time. Transport/integrity errors fail the publication
|
|
80
|
+
instead of replacing the working index with a transient partial result. Index
|
|
81
|
+
writes happen only after generation finishes; retain snapshots for rollback.
|
|
82
|
+
Credentials are environment-only and are absent from JSON, tar and normal logs.
|
|
83
|
+
|
|
84
|
+
To publish a new installation entry after building it:
|
|
85
|
+
|
|
86
|
+
```sh
|
|
87
|
+
python -m build --sdist
|
|
88
|
+
python publish_oss_index.py --output releases.json --publish \
|
|
89
|
+
--installer dist/pai_turbox-VERSION.tar.gz
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
The publisher checks name/version, the embedded default index and absence of
|
|
93
|
+
credentials or business payloads before uploading the source entry. Immutable
|
|
94
|
+
artifact names cannot be overwritten with different bytes. It also writes the
|
|
95
|
+
trailing-slash Simple Index project object that pip actually requests from OSS.
|
|
96
|
+
Increment VERSION for a changed installer release; index-only updates do not need
|
|
97
|
+
a new installer artifact.
|
|
98
|
+
|
|
99
|
+
## Selection, updates and cache
|
|
100
|
+
|
|
101
|
+
Match Python/ABI/platform, Torch/CUDA and device; select the highest compatible
|
|
102
|
+
approved stable version, then the latest OSS LastModified within that version.
|
|
103
|
+
Times are normalized to UTC. Pin a version with
|
|
104
|
+
`--config-settings turbox-version=0.2.11`, or an exact build with
|
|
105
|
+
`--config-settings wheel-filename=EXACT_WHEEL_FILENAME`.
|
|
106
|
+
|
|
107
|
+
Keep `--no-cache-dir` when reselecting an environment-dependent wheel. Standard
|
|
108
|
+
wheel tags do not encode Torch/CUDA/device. A runtime guard rejects a mismatched
|
|
109
|
+
cached binary, but rejection is not automatic cache invalidation. An existing
|
|
110
|
+
same-version pai-turbox is not automatically updated by bare `pip install` merely
|
|
111
|
+
because the JSON changed. Publish new installer release versions for normal
|
|
112
|
+
package-name upgrades. Explicit same-version rebuilds can download/extract the
|
|
113
|
+
source tar and pip-install the local tar, as covered by development verification;
|
|
114
|
+
there is no separate installer script.
|
|
115
|
+
|
|
116
|
+
References: [OSS object ACL](https://www.alibabacloud.com/help/en/oss/developer-reference/manage-object-acls-1),
|
|
117
|
+
[pip build interface](https://pip.pypa.io/en/stable/reference/build-system/),
|
|
118
|
+
[pip caching](https://pip.pypa.io/en/stable/topics/caching/).
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
# pai-turbox pip installation and OSS release index
|
|
2
|
+
|
|
3
|
+
The package name is `pai-turbox`; usage remains `import turbox`, existing console
|
|
4
|
+
commands and `ENABLE_TURBOX=1`. Installation finishes through pip, without a
|
|
5
|
+
second script or first-import binary download. Supported GPU PyTorch must already
|
|
6
|
+
be installed in the active environment.
|
|
7
|
+
|
|
8
|
+
## Customer installation
|
|
9
|
+
|
|
10
|
+
```sh
|
|
11
|
+
python -m pip install --no-cache-dir pai-turbox
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Install into the active GPU PyTorch environment. The archive contains installation
|
|
15
|
+
code only. Customers do not supply wheel-index or OSS credentials. Its embedded
|
|
16
|
+
public index is:
|
|
17
|
+
|
|
18
|
+
https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
|
|
19
|
+
|
|
20
|
+
The installer version is independent of the selected TurboX binary version.
|
|
21
|
+
Existing same-version installations are not automatically replaced when the
|
|
22
|
+
external index changes; see the update and cache rules below.
|
|
23
|
+
|
|
24
|
+
## Build and installation
|
|
25
|
+
|
|
26
|
+
The PEP 517 backend inspects the target interpreter's Torch/CUDA/device, reads the
|
|
27
|
+
public index and selects an existing wheel. It verifies the original archive's
|
|
28
|
+
SHA256 and metadata, then builds a pai-turbox wheel owning that binary payload.
|
|
29
|
+
The compiled files stay unchanged under `_pai_turbox_payload/`. A lightweight
|
|
30
|
+
loader exposes them as `turbox` and preserves original startup and license
|
|
31
|
+
checks, without sharing files with an existing native turbox distribution.
|
|
32
|
+
|
|
33
|
+
Public METADATA, dist-info and wheel filename use pai-turbox and VERSION.
|
|
34
|
+
Original dependencies are resolved alongside exact target Torch and index policy
|
|
35
|
+
constraints. There is no separate turbox distribution dependency. Source metadata
|
|
36
|
+
marks Requires-Dist dynamic. RECORD is rebuilt for all owned files. Selection is
|
|
37
|
+
frozen between metadata generation and wheel generation.
|
|
38
|
+
|
|
39
|
+
## Publishing an index
|
|
40
|
+
|
|
41
|
+
Install publisher dependencies from `requirements-publish.txt`, set
|
|
42
|
+
`TURBOX_PUBLISH_ACCESS_KEY_ID` and `TURBOX_PUBLISH_ACCESS_KEY_SECRET` in the release
|
|
43
|
+
job environment, then run:
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
python publish_oss_index.py --output releases.json --publish
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The script lists `release/dist/agent/` using a slash delimiter and an additional
|
|
50
|
+
relative-path check. It ignores all subdirectories. New or changed wheels are
|
|
51
|
+
streamed, hashed and checked for ZIP integrity and matching metadata. An anonymous
|
|
52
|
+
HEAD verifies the download address. Unchanged ETag/size/LastModified entries reuse
|
|
53
|
+
previous hashes after public availability is checked. It publishes a versioned
|
|
54
|
+
snapshot and updates releases.json, explicitly sets object public-read ACLs, and
|
|
55
|
+
verifies anonymous readback. It never changes bucket or original wheel ACLs.
|
|
56
|
+
|
|
57
|
+
`approved` means these index checks passed; it does not claim every runtime
|
|
58
|
+
combination has been qualified. Missing CUDA information is not guessed: special
|
|
59
|
+
filenames remain in the inventory with review_required until explicit mappings
|
|
60
|
+
are supplied through release-policy.json. Dependency constraints and disabled
|
|
61
|
+
artifacts are also policy data, not hardcoded installer version branches.
|
|
62
|
+
|
|
63
|
+
Run one publishing job at a time. Transport/integrity errors fail the publication
|
|
64
|
+
instead of replacing the working index with a transient partial result. Index
|
|
65
|
+
writes happen only after generation finishes; retain snapshots for rollback.
|
|
66
|
+
Credentials are environment-only and are absent from JSON, tar and normal logs.
|
|
67
|
+
|
|
68
|
+
To publish a new installation entry after building it:
|
|
69
|
+
|
|
70
|
+
```sh
|
|
71
|
+
python -m build --sdist
|
|
72
|
+
python publish_oss_index.py --output releases.json --publish \
|
|
73
|
+
--installer dist/pai_turbox-VERSION.tar.gz
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
The publisher checks name/version, the embedded default index and absence of
|
|
77
|
+
credentials or business payloads before uploading the source entry. Immutable
|
|
78
|
+
artifact names cannot be overwritten with different bytes. It also writes the
|
|
79
|
+
trailing-slash Simple Index project object that pip actually requests from OSS.
|
|
80
|
+
Increment VERSION for a changed installer release; index-only updates do not need
|
|
81
|
+
a new installer artifact.
|
|
82
|
+
|
|
83
|
+
## Selection, updates and cache
|
|
84
|
+
|
|
85
|
+
Match Python/ABI/platform, Torch/CUDA and device; select the highest compatible
|
|
86
|
+
approved stable version, then the latest OSS LastModified within that version.
|
|
87
|
+
Times are normalized to UTC. Pin a version with
|
|
88
|
+
`--config-settings turbox-version=0.2.11`, or an exact build with
|
|
89
|
+
`--config-settings wheel-filename=EXACT_WHEEL_FILENAME`.
|
|
90
|
+
|
|
91
|
+
Keep `--no-cache-dir` when reselecting an environment-dependent wheel. Standard
|
|
92
|
+
wheel tags do not encode Torch/CUDA/device. A runtime guard rejects a mismatched
|
|
93
|
+
cached binary, but rejection is not automatic cache invalidation. An existing
|
|
94
|
+
same-version pai-turbox is not automatically updated by bare `pip install` merely
|
|
95
|
+
because the JSON changed. Publish new installer release versions for normal
|
|
96
|
+
package-name upgrades. Explicit same-version rebuilds can download/extract the
|
|
97
|
+
source tar and pip-install the local tar, as covered by development verification;
|
|
98
|
+
there is no separate installer script.
|
|
99
|
+
|
|
100
|
+
References: [OSS object ACL](https://www.alibabacloud.com/help/en/oss/developer-reference/manage-object-acls-1),
|
|
101
|
+
[pip build interface](https://pip.pypa.io/en/stable/reference/build-system/),
|
|
102
|
+
[pip caching](https://pip.pypa.io/en/stable/topics/caching/).
|
pai_turbox-0.1.1/VERSION
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
0.1.1
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
"""Expose the binary payload as turbox without sharing another wheel's files."""
|
|
2
|
+
|
|
3
|
+
import importlib.machinery
|
|
4
|
+
import json
|
|
5
|
+
import os
|
|
6
|
+
from pathlib import Path
|
|
7
|
+
import sys
|
|
8
|
+
|
|
9
|
+
PAYLOAD = Path(__file__).resolve().parent / "_pai_turbox_payload"
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
class _TurboXFinder:
|
|
13
|
+
def find_spec(self, fullname, path=None, target=None):
|
|
14
|
+
if fullname != "turbox" or path is not None:
|
|
15
|
+
return None
|
|
16
|
+
# Relocated extensions must have Torch's shared libraries loaded first.
|
|
17
|
+
# This is deferred until TurboX is actually imported.
|
|
18
|
+
import torch
|
|
19
|
+
entry = json.loads((PAYLOAD / "selection.json").read_text())["wheels"][0]
|
|
20
|
+
device_mismatch = False
|
|
21
|
+
if torch.cuda.is_available():
|
|
22
|
+
name = torch.cuda.get_device_name(0)
|
|
23
|
+
device = "ppu" if name.startswith(("PPU", "ZW-M")) else "nv" if name.startswith(
|
|
24
|
+
("NVIDIA", "Tesla", "Quadro", "GeForce")
|
|
25
|
+
) else "unknown"
|
|
26
|
+
device_mismatch = device != entry["device"]
|
|
27
|
+
if (str(torch.__version__) != entry["torch"]
|
|
28
|
+
or torch.version.cuda != entry["cuda"] or device_mismatch):
|
|
29
|
+
raise ImportError(
|
|
30
|
+
"The installed pai-turbox binary belongs to a different Torch/CUDA/device environment. "
|
|
31
|
+
"Reinstall the source entry with --no-cache-dir to select a matching binary."
|
|
32
|
+
)
|
|
33
|
+
return importlib.machinery.PathFinder.find_spec(fullname, [str(PAYLOAD)])
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
def install():
|
|
37
|
+
if not any(isinstance(finder, _TurboXFinder) for finder in sys.meta_path):
|
|
38
|
+
sys.meta_path.insert(0, _TurboXFinder())
|
|
39
|
+
if str(PAYLOAD) not in sys.path:
|
|
40
|
+
sys.path.insert(0, str(PAYLOAD))
|
|
41
|
+
if os.environ.get("ENABLE_TURBOX") == "1":
|
|
42
|
+
# Preserve the original wheel's startup behavior and license checks.
|
|
43
|
+
import _turbox_bootstrap # noqa: F401
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
|
|
@@ -0,0 +1,316 @@
|
|
|
1
|
+
"""PEP 517 adapter selecting original, hash-pinned TurboX binary wheels."""
|
|
2
|
+
|
|
3
|
+
import json
|
|
4
|
+
import base64
|
|
5
|
+
import csv
|
|
6
|
+
import hashlib
|
|
7
|
+
import io
|
|
8
|
+
import tempfile
|
|
9
|
+
import tarfile
|
|
10
|
+
import zipfile
|
|
11
|
+
from email.parser import BytesParser
|
|
12
|
+
import os
|
|
13
|
+
from pathlib import Path
|
|
14
|
+
import subprocess
|
|
15
|
+
import sys
|
|
16
|
+
import urllib.parse
|
|
17
|
+
import urllib.request
|
|
18
|
+
from datetime import datetime, timezone
|
|
19
|
+
|
|
20
|
+
from packaging.tags import sys_tags
|
|
21
|
+
from packaging.utils import parse_wheel_filename
|
|
22
|
+
from packaging.version import Version
|
|
23
|
+
from setuptools import build_meta as _backend
|
|
24
|
+
|
|
25
|
+
ROOT = Path(__file__).resolve().parent
|
|
26
|
+
PROBE = r'''
|
|
27
|
+
import json, platform, sys, torch
|
|
28
|
+
if not torch.cuda.is_available():
|
|
29
|
+
raise RuntimeError("No available GPU in the target PyTorch environment")
|
|
30
|
+
name = torch.cuda.get_device_name(0)
|
|
31
|
+
if name.startswith(("PPU", "ZW-M")):
|
|
32
|
+
device = "ppu"
|
|
33
|
+
elif name.startswith(("NVIDIA", "Tesla", "Quadro", "GeForce")):
|
|
34
|
+
device = "nv"
|
|
35
|
+
else:
|
|
36
|
+
raise RuntimeError("Unknown GPU family: " + name)
|
|
37
|
+
print("PAI_TURBOX_ENV=" + json.dumps({
|
|
38
|
+
"torch": str(torch.__version__), "cuda": torch.version.cuda,
|
|
39
|
+
"device": device, "device_name": name,
|
|
40
|
+
"python": "cp" + str(sys.version_info.major) + str(sys.version_info.minor),
|
|
41
|
+
"system": platform.system(), "machine": platform.machine(),
|
|
42
|
+
}))
|
|
43
|
+
'''
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def detect_environment():
|
|
47
|
+
# pip's build isolation is sys.path/PYTHONPATH based. -I starts the same
|
|
48
|
+
# target interpreter without that overlay, so it sees preinstalled torch.
|
|
49
|
+
env = dict(os.environ, ENABLE_TURBOX="0")
|
|
50
|
+
env.pop("PYTHONPATH", None)
|
|
51
|
+
result = subprocess.run(
|
|
52
|
+
[sys.executable, "-I", "-c", PROBE], env=env, text=True,
|
|
53
|
+
stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=90,
|
|
54
|
+
)
|
|
55
|
+
if result.returncode:
|
|
56
|
+
raise RuntimeError(
|
|
57
|
+
"Cannot inspect the target PyTorch environment. Install supported "
|
|
58
|
+
"GPU PyTorch first. Probe output:\n" + result.stderr[-4000:]
|
|
59
|
+
)
|
|
60
|
+
for line in reversed(result.stdout.splitlines()):
|
|
61
|
+
if line.startswith("PAI_TURBOX_ENV="):
|
|
62
|
+
return json.loads(line.split("=", 1)[1])
|
|
63
|
+
raise RuntimeError("Target environment probe returned no environment data")
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def load_index(location):
|
|
67
|
+
if not location:
|
|
68
|
+
raise RuntimeError(
|
|
69
|
+
"Set TURBOX_WHEEL_INDEX to a release index JSON path or HTTPS URL. "
|
|
70
|
+
"The installation archive does not contain a frozen support matrix."
|
|
71
|
+
)
|
|
72
|
+
if location.startswith("https://"):
|
|
73
|
+
with urllib.request.urlopen(location, timeout=30) as response:
|
|
74
|
+
raw = response.read(8 * 1024 * 1024 + 1)
|
|
75
|
+
if len(raw) > 8 * 1024 * 1024:
|
|
76
|
+
raise RuntimeError("Release index exceeds 8 MiB")
|
|
77
|
+
manifest = json.loads(raw)
|
|
78
|
+
else:
|
|
79
|
+
if "://" in location:
|
|
80
|
+
raise RuntimeError("Remote indexes must use HTTPS")
|
|
81
|
+
manifest = json.loads(Path(location).read_text())
|
|
82
|
+
if manifest.get("schema_version") != 1:
|
|
83
|
+
raise RuntimeError("Unsupported release index schema; update the installer")
|
|
84
|
+
return manifest
|
|
85
|
+
|
|
86
|
+
|
|
87
|
+
def modified_time(entry):
|
|
88
|
+
value = entry.get("last_modified", "")
|
|
89
|
+
try:
|
|
90
|
+
result = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
|
91
|
+
if result.tzinfo is None or result.utcoffset() is None:
|
|
92
|
+
raise ValueError("timezone missing")
|
|
93
|
+
return result.astimezone(timezone.utc)
|
|
94
|
+
except (ValueError, TypeError, AttributeError) as exc:
|
|
95
|
+
raise RuntimeError(
|
|
96
|
+
"A timezone-aware last_modified is required to order builds: "
|
|
97
|
+
+ entry["filename"]
|
|
98
|
+
) from exc
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
def select_release(manifest, environment, supported_tags=None, requested_version=None,
|
|
102
|
+
requested_build=None):
|
|
103
|
+
tags = set(sys_tags() if supported_tags is None else supported_tags)
|
|
104
|
+
if environment["system"] != "Linux" or environment["machine"] != "x86_64":
|
|
105
|
+
raise RuntimeError("Only Linux x86_64 is currently supported")
|
|
106
|
+
keys = ("cuda", "device", "python")
|
|
107
|
+
matches = []
|
|
108
|
+
for entry in manifest["wheels"]:
|
|
109
|
+
if not entry.get("approved", False):
|
|
110
|
+
continue
|
|
111
|
+
if requested_build and entry["filename"] != requested_build:
|
|
112
|
+
continue
|
|
113
|
+
if requested_version and Version(Version(entry["version"]).public) != Version(requested_version):
|
|
114
|
+
continue
|
|
115
|
+
expected_torch = Version(entry["torch"])
|
|
116
|
+
actual_torch = Version(environment["torch"])
|
|
117
|
+
if expected_torch.public != actual_torch.public:
|
|
118
|
+
continue
|
|
119
|
+
if expected_torch.local is not None and expected_torch != actual_torch:
|
|
120
|
+
continue
|
|
121
|
+
if not all(entry[key] == environment[key] for key in keys):
|
|
122
|
+
continue
|
|
123
|
+
name, version, _, wheel_tags = parse_wheel_filename(entry["filename"])
|
|
124
|
+
if name != "turbox" or str(version) != entry["version"]:
|
|
125
|
+
raise RuntimeError("Manifest filename and package metadata disagree")
|
|
126
|
+
if not tags.intersection(wheel_tags):
|
|
127
|
+
continue
|
|
128
|
+
digest = entry["sha256"]
|
|
129
|
+
if len(digest) != 64 or any(c not in "0123456789abcdef" for c in digest):
|
|
130
|
+
raise RuntimeError("A valid SHA256 is required for every release")
|
|
131
|
+
url = urllib.parse.urlsplit(entry["url"])
|
|
132
|
+
if (url.scheme != "https" or not url.hostname or url.username
|
|
133
|
+
or url.password or url.fragment
|
|
134
|
+
or urllib.parse.unquote(url.path.rsplit("/", 1)[-1]) != entry["filename"]):
|
|
135
|
+
raise RuntimeError("A matching HTTPS wheel URL is required")
|
|
136
|
+
matches.append(entry)
|
|
137
|
+
# Consider only the highest stable public TurboX release for this exact
|
|
138
|
+
# environment. A prerelease must be explicitly opted in through the index.
|
|
139
|
+
matches = [e for e in matches if not Version(e["version"]).is_prerelease
|
|
140
|
+
or e.get("allow_prerelease", False)]
|
|
141
|
+
if matches:
|
|
142
|
+
latest = max(
|
|
143
|
+
Version(Version(e["version"]).public) for e in matches
|
|
144
|
+
)
|
|
145
|
+
matches = [e for e in matches if Version(Version(e["version"]).public) == latest]
|
|
146
|
+
if len(matches) > 1:
|
|
147
|
+
newest = max(modified_time(entry) for entry in matches)
|
|
148
|
+
matches = [entry for entry in matches if modified_time(entry) == newest]
|
|
149
|
+
if len(matches) != 1:
|
|
150
|
+
raise RuntimeError(
|
|
151
|
+
f"Expected one approved TurboX wheel, found {len(matches)} for "
|
|
152
|
+
+ json.dumps(environment, sort_keys=True)
|
|
153
|
+
)
|
|
154
|
+
return matches[0]
|
|
155
|
+
|
|
156
|
+
|
|
157
|
+
def selected_plan(config_settings=None, prepared=None):
|
|
158
|
+
settings = config_settings or {}
|
|
159
|
+
location = (settings.get("wheel-index") or os.environ.get("TURBOX_WHEEL_INDEX")
|
|
160
|
+
or (ROOT / "index-url.txt").read_text().strip())
|
|
161
|
+
manifest = prepared if prepared is not None else load_index(location)
|
|
162
|
+
environment = detect_environment()
|
|
163
|
+
entry = select_release(
|
|
164
|
+
manifest, environment,
|
|
165
|
+
requested_version=settings.get("turbox-version") or os.environ.get("TURBOX_VERSION"),
|
|
166
|
+
requested_build=settings.get("wheel-filename") or os.environ.get("TURBOX_WHEEL_FILENAME"),
|
|
167
|
+
)
|
|
168
|
+
print("[pai-turbox] Selected " + entry["filename"], flush=True)
|
|
169
|
+
return {"schema_version": 1, "wheels": [dict(entry, torch=environment["torch"])]}
|
|
170
|
+
|
|
171
|
+
|
|
172
|
+
def download_wheel(entry):
|
|
173
|
+
cache = ROOT / ".wheel-cache"
|
|
174
|
+
cache.mkdir(exist_ok=True)
|
|
175
|
+
path = cache / entry["filename"]
|
|
176
|
+
if path.is_file() and hashlib.sha256(path.read_bytes()).hexdigest() == entry["sha256"]:
|
|
177
|
+
return path
|
|
178
|
+
with tempfile.NamedTemporaryFile(dir=cache, delete=False) as output:
|
|
179
|
+
temporary = Path(output.name)
|
|
180
|
+
try:
|
|
181
|
+
digest = hashlib.sha256()
|
|
182
|
+
with urllib.request.urlopen(entry["url"], timeout=60) as response:
|
|
183
|
+
while True:
|
|
184
|
+
chunk = response.read(1024 * 1024)
|
|
185
|
+
if not chunk:
|
|
186
|
+
break
|
|
187
|
+
digest.update(chunk)
|
|
188
|
+
output.write(chunk)
|
|
189
|
+
output.close()
|
|
190
|
+
if digest.hexdigest() != entry["sha256"]:
|
|
191
|
+
raise RuntimeError("Downloaded TurboX wheel SHA256 mismatch")
|
|
192
|
+
temporary.replace(path)
|
|
193
|
+
finally:
|
|
194
|
+
temporary.unlink(missing_ok=True)
|
|
195
|
+
return path
|
|
196
|
+
|
|
197
|
+
|
|
198
|
+
def public_metadata():
|
|
199
|
+
with tempfile.TemporaryDirectory() as directory:
|
|
200
|
+
name = _backend.prepare_metadata_for_build_wheel(directory)
|
|
201
|
+
return (Path(directory) / name / "METADATA").read_bytes()
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
def package_files(plan):
|
|
205
|
+
entry = plan["wheels"][0]
|
|
206
|
+
archive = download_wheel(entry)
|
|
207
|
+
public_version = (ROOT / "VERSION").read_text().strip()
|
|
208
|
+
destination = "pai_turbox-" + public_version + ".dist-info"
|
|
209
|
+
output = {}
|
|
210
|
+
with zipfile.ZipFile(archive) as wheel:
|
|
211
|
+
metadata_paths = [n for n in wheel.namelist() if n.endswith(".dist-info/METADATA")]
|
|
212
|
+
if len(metadata_paths) != 1:
|
|
213
|
+
raise RuntimeError("Expected one original wheel METADATA")
|
|
214
|
+
origin = metadata_paths[0].rsplit("/", 1)[0]
|
|
215
|
+
original = BytesParser().parsebytes(wheel.read(metadata_paths[0]))
|
|
216
|
+
if original["Name"] != "turbox" or original["Version"] != entry["version"]:
|
|
217
|
+
raise RuntimeError("Downloaded wheel metadata differs from selected release")
|
|
218
|
+
message = BytesParser().parsebytes(public_metadata())
|
|
219
|
+
if message["Name"] != "pai-turbox" or message["Version"] != public_version:
|
|
220
|
+
raise RuntimeError("Source entry metadata differs from public package name/version")
|
|
221
|
+
# The public distribution owns the binary payload; no turbox dependency.
|
|
222
|
+
for requirement in original.get_all("Requires-Dist", []):
|
|
223
|
+
message["Requires-Dist"] = requirement
|
|
224
|
+
message["Requires-Dist"] = "torch==" + entry["torch"]
|
|
225
|
+
for requirement in entry.get("extra_requires", []):
|
|
226
|
+
message["Requires-Dist"] = requirement
|
|
227
|
+
for name in wheel.namelist():
|
|
228
|
+
if name.endswith("/"):
|
|
229
|
+
continue
|
|
230
|
+
parts = Path(name).parts
|
|
231
|
+
if name.startswith("/") or ".." in parts or ".data" in parts[0]:
|
|
232
|
+
raise RuntimeError("Unsupported original wheel installation path: " + name)
|
|
233
|
+
if name.startswith(origin + "/"):
|
|
234
|
+
relative = name[len(origin) + 1:]
|
|
235
|
+
if relative in ("RECORD", "RECORD.jws", "RECORD.p7s"):
|
|
236
|
+
continue
|
|
237
|
+
output[destination + "/" + relative] = wheel.read(name)
|
|
238
|
+
else:
|
|
239
|
+
output["_pai_turbox_payload/" + name] = wheel.read(name)
|
|
240
|
+
output[destination + "/METADATA"] = message.as_bytes()
|
|
241
|
+
snapshot = json.dumps(plan, sort_keys=True).encode()
|
|
242
|
+
output[destination + "/turbox-selection.json"] = snapshot
|
|
243
|
+
output["_pai_turbox_payload/selection.json"] = snapshot
|
|
244
|
+
output["_pai_turbox_loader.py"] = (ROOT / "_pai_turbox_loader.py").read_bytes()
|
|
245
|
+
output["000_pai_turbox.pth"] = b"import _pai_turbox_loader; _pai_turbox_loader.install()\n"
|
|
246
|
+
return destination, output
|
|
247
|
+
|
|
248
|
+
|
|
249
|
+
def get_requires_for_build_wheel(config_settings=None):
|
|
250
|
+
return []
|
|
251
|
+
|
|
252
|
+
|
|
253
|
+
def get_requires_for_build_sdist(config_settings=None):
|
|
254
|
+
return []
|
|
255
|
+
|
|
256
|
+
|
|
257
|
+
def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
|
|
258
|
+
plan = selected_plan(config_settings)
|
|
259
|
+
destination, files = package_files(plan)
|
|
260
|
+
root = Path(metadata_directory)
|
|
261
|
+
for name, data in files.items():
|
|
262
|
+
if name.startswith(destination + "/"):
|
|
263
|
+
path = root / name
|
|
264
|
+
path.parent.mkdir(parents=True, exist_ok=True)
|
|
265
|
+
path.write_bytes(data)
|
|
266
|
+
return destination
|
|
267
|
+
|
|
268
|
+
|
|
269
|
+
def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
|
|
270
|
+
snapshot = Path(metadata_directory) / "turbox-selection.json" if metadata_directory else None
|
|
271
|
+
prepared = json.loads(snapshot.read_text()) if snapshot and snapshot.is_file() else None
|
|
272
|
+
plan = selected_plan(config_settings, prepared)
|
|
273
|
+
destination, files = package_files(plan)
|
|
274
|
+
if metadata_directory:
|
|
275
|
+
# Preserve every metadata byte prepared for pip's dependency resolver.
|
|
276
|
+
for path in Path(metadata_directory).rglob("*"):
|
|
277
|
+
if path.is_file() and path.name != "RECORD":
|
|
278
|
+
files[destination + "/" + path.relative_to(metadata_directory).as_posix()] = path.read_bytes()
|
|
279
|
+
rows = []
|
|
280
|
+
for name, data in sorted(files.items()):
|
|
281
|
+
digest = base64.urlsafe_b64encode(hashlib.sha256(data).digest()).rstrip(b"=").decode()
|
|
282
|
+
rows.append((name, "sha256=" + digest, str(len(data))))
|
|
283
|
+
record = destination + "/RECORD"
|
|
284
|
+
rows.append((record, "", ""))
|
|
285
|
+
stream = io.StringIO(newline="")
|
|
286
|
+
csv.writer(stream).writerows(rows)
|
|
287
|
+
files[record] = stream.getvalue().encode()
|
|
288
|
+
tags = plan["wheels"][0]["filename"].rsplit("-", 3)[-3:]
|
|
289
|
+
version = (ROOT / "VERSION").read_text().strip()
|
|
290
|
+
filename = "pai_turbox-" + version + "-" + "-".join(tags)
|
|
291
|
+
Path(wheel_directory).mkdir(parents=True, exist_ok=True)
|
|
292
|
+
with zipfile.ZipFile(Path(wheel_directory) / filename, "w", zipfile.ZIP_DEFLATED) as wheel:
|
|
293
|
+
for name, data in files.items():
|
|
294
|
+
wheel.writestr(name, data)
|
|
295
|
+
return filename
|
|
296
|
+
|
|
297
|
+
|
|
298
|
+
def build_sdist(sdist_directory, config_settings=None):
|
|
299
|
+
name = _backend.build_sdist(sdist_directory, config_settings)
|
|
300
|
+
archive = Path(sdist_directory) / name
|
|
301
|
+
temporary = archive.with_suffix(".tmp")
|
|
302
|
+
# PEP 643: dependencies are generated from the selected binary on the target.
|
|
303
|
+
# All other public metadata remains the source entry's static metadata.
|
|
304
|
+
with tarfile.open(archive, "r:gz") as source, tarfile.open(temporary, "w:gz") as target:
|
|
305
|
+
for member in source:
|
|
306
|
+
data = source.extractfile(member) if member.isfile() else None
|
|
307
|
+
if data is not None and member.name.endswith("/PKG-INFO"):
|
|
308
|
+
message = BytesParser().parsebytes(data.read())
|
|
309
|
+
if "requires-dist" not in [x.lower() for x in message.get_all("Dynamic", [])]:
|
|
310
|
+
message["Dynamic"] = "Requires-Dist"
|
|
311
|
+
raw = message.as_bytes()
|
|
312
|
+
member.size = len(raw)
|
|
313
|
+
data = io.BytesIO(raw)
|
|
314
|
+
target.addfile(member, data)
|
|
315
|
+
temporary.replace(archive)
|
|
316
|
+
return name
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: pai-turbox
|
|
3
|
+
Version: 0.1.1
|
|
4
|
+
Summary: Environment-aware installation entry for TurboX binary wheels
|
|
5
|
+
Author: TurboX Team
|
|
6
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
7
|
+
Requires-Python: >=3.8
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
Dynamic: author
|
|
10
|
+
Dynamic: classifier
|
|
11
|
+
Dynamic: description
|
|
12
|
+
Dynamic: description-content-type
|
|
13
|
+
Dynamic: requires-python
|
|
14
|
+
Dynamic: summary
|
|
15
|
+
Dynamic: Requires-Dist
|
|
16
|
+
|
|
17
|
+
# pai-turbox pip installation and OSS release index
|
|
18
|
+
|
|
19
|
+
The package name is `pai-turbox`; usage remains `import turbox`, existing console
|
|
20
|
+
commands and `ENABLE_TURBOX=1`. Installation finishes through pip, without a
|
|
21
|
+
second script or first-import binary download. Supported GPU PyTorch must already
|
|
22
|
+
be installed in the active environment.
|
|
23
|
+
|
|
24
|
+
## Customer installation
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
python -m pip install --no-cache-dir pai-turbox
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Install into the active GPU PyTorch environment. The archive contains installation
|
|
31
|
+
code only. Customers do not supply wheel-index or OSS credentials. Its embedded
|
|
32
|
+
public index is:
|
|
33
|
+
|
|
34
|
+
https://turbox-wlcb.oss-cn-wulanchabu.aliyuncs.com/release/dist/agent/index/releases.json
|
|
35
|
+
|
|
36
|
+
The installer version is independent of the selected TurboX binary version.
|
|
37
|
+
Existing same-version installations are not automatically replaced when the
|
|
38
|
+
external index changes; see the update and cache rules below.
|
|
39
|
+
|
|
40
|
+
## Build and installation
|
|
41
|
+
|
|
42
|
+
The PEP 517 backend inspects the target interpreter's Torch/CUDA/device, reads the
|
|
43
|
+
public index and selects an existing wheel. It verifies the original archive's
|
|
44
|
+
SHA256 and metadata, then builds a pai-turbox wheel owning that binary payload.
|
|
45
|
+
The compiled files stay unchanged under `_pai_turbox_payload/`. A lightweight
|
|
46
|
+
loader exposes them as `turbox` and preserves original startup and license
|
|
47
|
+
checks, without sharing files with an existing native turbox distribution.
|
|
48
|
+
|
|
49
|
+
Public METADATA, dist-info and wheel filename use pai-turbox and VERSION.
|
|
50
|
+
Original dependencies are resolved alongside exact target Torch and index policy
|
|
51
|
+
constraints. There is no separate turbox distribution dependency. Source metadata
|
|
52
|
+
marks Requires-Dist dynamic. RECORD is rebuilt for all owned files. Selection is
|
|
53
|
+
frozen between metadata generation and wheel generation.
|
|
54
|
+
|
|
55
|
+
## Publishing an index
|
|
56
|
+
|
|
57
|
+
Install publisher dependencies from `requirements-publish.txt`, set
|
|
58
|
+
`TURBOX_PUBLISH_ACCESS_KEY_ID` and `TURBOX_PUBLISH_ACCESS_KEY_SECRET` in the release
|
|
59
|
+
job environment, then run:
|
|
60
|
+
|
|
61
|
+
```sh
|
|
62
|
+
python publish_oss_index.py --output releases.json --publish
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
The script lists `release/dist/agent/` using a slash delimiter and an additional
|
|
66
|
+
relative-path check. It ignores all subdirectories. New or changed wheels are
|
|
67
|
+
streamed, hashed and checked for ZIP integrity and matching metadata. An anonymous
|
|
68
|
+
HEAD verifies the download address. Unchanged ETag/size/LastModified entries reuse
|
|
69
|
+
previous hashes after public availability is checked. It publishes a versioned
|
|
70
|
+
snapshot and updates releases.json, explicitly sets object public-read ACLs, and
|
|
71
|
+
verifies anonymous readback. It never changes bucket or original wheel ACLs.
|
|
72
|
+
|
|
73
|
+
`approved` means these index checks passed; it does not claim every runtime
|
|
74
|
+
combination has been qualified. Missing CUDA information is not guessed: special
|
|
75
|
+
filenames remain in the inventory with review_required until explicit mappings
|
|
76
|
+
are supplied through release-policy.json. Dependency constraints and disabled
|
|
77
|
+
artifacts are also policy data, not hardcoded installer version branches.
|
|
78
|
+
|
|
79
|
+
Run one publishing job at a time. Transport/integrity errors fail the publication
|
|
80
|
+
instead of replacing the working index with a transient partial result. Index
|
|
81
|
+
writes happen only after generation finishes; retain snapshots for rollback.
|
|
82
|
+
Credentials are environment-only and are absent from JSON, tar and normal logs.
|
|
83
|
+
|
|
84
|
+
To publish a new installation entry after building it:
|
|
85
|
+
|
|
86
|
+
```sh
|
|
87
|
+
python -m build --sdist
|
|
88
|
+
python publish_oss_index.py --output releases.json --publish \
|
|
89
|
+
--installer dist/pai_turbox-VERSION.tar.gz
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
The publisher checks name/version, the embedded default index and absence of
|
|
93
|
+
credentials or business payloads before uploading the source entry. Immutable
|
|
94
|
+
artifact names cannot be overwritten with different bytes. It also writes the
|
|
95
|
+
trailing-slash Simple Index project object that pip actually requests from OSS.
|
|
96
|
+
Increment VERSION for a changed installer release; index-only updates do not need
|
|
97
|
+
a new installer artifact.
|
|
98
|
+
|
|
99
|
+
## Selection, updates and cache
|
|
100
|
+
|
|
101
|
+
Match Python/ABI/platform, Torch/CUDA and device; select the highest compatible
|
|
102
|
+
approved stable version, then the latest OSS LastModified within that version.
|
|
103
|
+
Times are normalized to UTC. Pin a version with
|
|
104
|
+
`--config-settings turbox-version=0.2.11`, or an exact build with
|
|
105
|
+
`--config-settings wheel-filename=EXACT_WHEEL_FILENAME`.
|
|
106
|
+
|
|
107
|
+
Keep `--no-cache-dir` when reselecting an environment-dependent wheel. Standard
|
|
108
|
+
wheel tags do not encode Torch/CUDA/device. A runtime guard rejects a mismatched
|
|
109
|
+
cached binary, but rejection is not automatic cache invalidation. An existing
|
|
110
|
+
same-version pai-turbox is not automatically updated by bare `pip install` merely
|
|
111
|
+
because the JSON changed. Publish new installer release versions for normal
|
|
112
|
+
package-name upgrades. Explicit same-version rebuilds can download/extract the
|
|
113
|
+
source tar and pip-install the local tar, as covered by development verification;
|
|
114
|
+
there is no separate installer script.
|
|
115
|
+
|
|
116
|
+
References: [OSS object ACL](https://www.alibabacloud.com/help/en/oss/developer-reference/manage-object-acls-1),
|
|
117
|
+
[pip build interface](https://pip.pypa.io/en/stable/reference/build-system/),
|
|
118
|
+
[pip caching](https://pip.pypa.io/en/stable/topics/caching/).
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
MANIFEST.in
|
|
2
|
+
README.md
|
|
3
|
+
VERSION
|
|
4
|
+
_pai_turbox_loader.py
|
|
5
|
+
index-url.txt
|
|
6
|
+
installer_backend.py
|
|
7
|
+
pyproject.toml
|
|
8
|
+
setup.py
|
|
9
|
+
pai_turbox.egg-info/PKG-INFO
|
|
10
|
+
pai_turbox.egg-info/SOURCES.txt
|
|
11
|
+
pai_turbox.egg-info/dependency_links.txt
|
|
12
|
+
pai_turbox.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
"""Build the public installation entry, never TurboX business sources."""
|
|
2
|
+
from pathlib import Path
|
|
3
|
+
|
|
4
|
+
from setuptools import setup
|
|
5
|
+
|
|
6
|
+
ROOT = Path(__file__).resolve().parent
|
|
7
|
+
setup(
|
|
8
|
+
name="pai-turbox",
|
|
9
|
+
version=(ROOT / "VERSION").read_text().strip(),
|
|
10
|
+
description="Environment-aware installation entry for TurboX binary wheels",
|
|
11
|
+
long_description=(ROOT / "README.md").read_text(),
|
|
12
|
+
long_description_content_type="text/markdown",
|
|
13
|
+
author="TurboX Team",
|
|
14
|
+
python_requires=">=3.8",
|
|
15
|
+
packages=[],
|
|
16
|
+
py_modules=[],
|
|
17
|
+
install_requires=[],
|
|
18
|
+
classifiers=["Operating System :: POSIX :: Linux"],
|
|
19
|
+
)
|