gowalk-cicd 1.0.60 → 1.0.61
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +8 -1
- package/README.md +12 -5
- package/action/.daemux-version +1 -1
- package/action/action.yml +3 -3
- package/action/scripts/cert_factory.py +12 -111
- package/action/scripts/creds_store.py +1 -1
- package/action/scripts/test_certificate_cap_policy.py +16 -18
- package/android-action/.daemux-version +1 -1
- package/android-action/action.yml +2 -1
- package/android-action/scripts/jvm_proxy_relay.py +119 -0
- package/android-action/scripts/play_store_proxy.py +4 -1
- package/android-action/scripts/test_jvm_proxy_relay.py +99 -0
- package/android-action/scripts/test_play_store_proxy.py +15 -0
- package/backend-action/.daemux-version +1 -1
- package/package.json +1 -1
- package/templates/deploy.yml +2 -1
package/CLAUDE.md
CHANGED
|
@@ -115,6 +115,13 @@ node /path/to/gowalk-cicd/bin/cli.mjs
|
|
|
115
115
|
`IOS_DISTRIBUTION_CERT_REGISTRY_BASE64`), and the Android keystore, properties
|
|
116
116
|
and Play service account into an ephemeral runner checkout.
|
|
117
117
|
Tracked `creds/` files remain a backward-compatible fallback.
|
|
118
|
+
- Never revoke a distribution certificate automatically, including legacy caches.
|
|
119
|
+
`certificate-cap-policy` accepts only `fail`; a full cap requires registry reconciliation.
|
|
120
|
+
- Keep preinstalled Android SDK/NDKs. Every Gradle test/build and SDK download runs
|
|
121
|
+
under `play_store_proxy.py --`, whose owned loopback relay gives Java an explicit
|
|
122
|
+
authenticated account exit without credentials in JVM options. HTTPS is tunneled
|
|
123
|
+
without TLS interception; no upstream failure may fall back to the destination.
|
|
124
|
+
The wrapper disables Gradle daemons and closes its relay when the command ends.
|
|
118
125
|
- Backend runtime credentials use the single encrypted `BACKEND_RUNTIME_ENV`
|
|
119
126
|
secret. The action writes it to host-side `.runtime.env` with mode 0600 and
|
|
120
127
|
supplies it to Compose after the generated `.env`; never print its content.
|
|
@@ -168,7 +175,7 @@ node /path/to/gowalk-cicd/bin/cli.mjs
|
|
|
168
175
|
can be raised to cover a release's life.
|
|
169
176
|
- Firebase Crashlytics traffic follows the Google account proxy. Android builds and
|
|
170
177
|
the pinned Firebase CLI receive the explicit proxy environment; Crashlytics Gradle
|
|
171
|
-
2.9.2+ and the pinned buildtools support it
|
|
178
|
+
2.9.2+ and the pinned buildtools support it; generic JVM networking uses the relay.
|
|
172
179
|
Native iOS upload-symbols uses NSURLSession, so it is never executed. Before
|
|
173
180
|
archiving, dedicated Crashlytics upload phases in the ephemeral Xcode projects
|
|
174
181
|
are deferred; mixed/unidentified upload phases refuse the archive for repair.
|
package/README.md
CHANGED
|
@@ -327,12 +327,19 @@ zip I/O error: No space left on device
|
|
|
327
327
|
```
|
|
328
328
|
|
|
329
329
|
The Android job removes the preinstalled toolchains a Flutter build never uses
|
|
330
|
-
(.NET,
|
|
330
|
+
(.NET, GHC, PowerShell, Swift, Chromium) and prunes Docker
|
|
331
331
|
images, reclaiming roughly 25 GB in a few seconds. It prints `df -h /` before
|
|
332
332
|
and after. Linux only; skipped in Bitrise mode. npm survives on purpose: the
|
|
333
333
|
[Crashlytics symbol upload](#crashlytics-symbol-delivery-firebase_app_id) runs
|
|
334
334
|
the Firebase CLI through `npx`.
|
|
335
335
|
|
|
336
|
+
The preinstalled Android SDK and NDKs are retained. Gradle and SDK-manager Java
|
|
337
|
+
networking runs through an owned loopback relay to `GOOGLE_STORE_PROXY_URL`.
|
|
338
|
+
The relay authenticates to that exit without putting credentials in JVM options;
|
|
339
|
+
HTTPS remains an end-to-end TLS tunnel. An unavailable proxy fails the request,
|
|
340
|
+
and the relay shuts down with its child command. Gradle daemons are disabled for
|
|
341
|
+
these commands so they cannot retain an expired relay port.
|
|
342
|
+
|
|
336
343
|
### Private git dependencies (Flutter)
|
|
337
344
|
|
|
338
345
|
A Flutter app can depend on private git packages:
|
|
@@ -441,7 +448,8 @@ identity and create per-app provisioning profiles, but it never creates,
|
|
|
441
448
|
replaces, or revokes a registry-managed distribution certificate. Missing,
|
|
442
449
|
partially written, corrupt, expired, Apple-revoked, or resource-ID/P12-mismatched
|
|
443
450
|
managed material aborts with a reconciliation error. Legacy repos without the
|
|
444
|
-
marker
|
|
451
|
+
marker may create a certificate in an available slot, but never revoke another
|
|
452
|
+
identity. A full certificate cap stops for account-registry reconciliation.
|
|
445
453
|
|
|
446
454
|
## How it works
|
|
447
455
|
|
|
@@ -773,7 +781,7 @@ common ones:
|
|
|
773
781
|
| `bundle-id` | Override the auto-detected bundle identifier |
|
|
774
782
|
| `team-id` | Override the auto-detected team ID |
|
|
775
783
|
| `app-store-apple-id` | Numeric ASC app ID (override auto-lookup) |
|
|
776
|
-
| `certificate-cap-policy` |
|
|
784
|
+
| `certificate-cap-policy` | Only `fail` is accepted (the default); existing identities are always preserved. |
|
|
777
785
|
| `run-tests` | `false` to skip the simulator test stage |
|
|
778
786
|
| `uses-non-exempt-encryption` | Value for `ITSAppUsesNonExemptEncryption` |
|
|
779
787
|
| `archive` | `false` to build-only (PR runs without secrets) |
|
|
@@ -793,8 +801,7 @@ inside GitHub Actions.
|
|
|
793
801
|
Once the app exists, all subsequent builds and uploads are fully automated via
|
|
794
802
|
the ASC API key.
|
|
795
803
|
|
|
796
|
-
|
|
797
|
-
`certificate-cap-policy: 'fail'`. The action may add a certificate when Apple has a free
|
|
804
|
+
Automatic certificate revocation is disabled for every account. The action may add a certificate when Apple has a free
|
|
798
805
|
slot, but a full-cap response aborts without listing or revoking existing identities. If
|
|
799
806
|
signing preparation fails after creating a certificate, the default-branch workflow first
|
|
800
807
|
commits any completed cache files, then re-raises the failure so the private key is not lost.
|
package/action/.daemux-version
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
1.0.
|
|
1
|
+
1.0.61
|
package/action/action.yml
CHANGED
|
@@ -53,10 +53,10 @@ inputs:
|
|
|
53
53
|
certificate-cap-policy:
|
|
54
54
|
description: >
|
|
55
55
|
Behavior when Apple reports the two-certificate Distribution cap.
|
|
56
|
-
'
|
|
57
|
-
|
|
56
|
+
Only 'fail' is supported: abort without revoking any existing certificate.
|
|
57
|
+
Reconcile the account registry when no live signing identity is available.
|
|
58
58
|
required: false
|
|
59
|
-
default: "
|
|
59
|
+
default: "fail"
|
|
60
60
|
persist-signing-cache:
|
|
61
61
|
description: >
|
|
62
62
|
Commit refreshed signing files back to the default branch. Set to
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
Cert-creation primitives for the iOS native TestFlight action.
|
|
4
4
|
|
|
5
5
|
Owns the cryptographic legwork (RSA key + CSR), Apple's per-team cert
|
|
6
|
-
cap
|
|
6
|
+
cap refusal (never automatic revocation), and PKCS12 serialisation. Split
|
|
7
7
|
from ``prepare_signing.py`` so the orchestrator there stays focused on
|
|
8
8
|
the load-or-regen control flow and the file count stays under the
|
|
9
9
|
project's per-file limits.
|
|
@@ -16,35 +16,20 @@ from __future__ import annotations
|
|
|
16
16
|
|
|
17
17
|
import base64
|
|
18
18
|
import os
|
|
19
|
-
import time
|
|
20
19
|
|
|
21
|
-
from asc_common import
|
|
20
|
+
from asc_common import request
|
|
22
21
|
from cryptography import x509
|
|
23
22
|
from cryptography.hazmat.primitives import hashes, serialization
|
|
24
23
|
from cryptography.hazmat.primitives.asymmetric import rsa
|
|
25
24
|
from cryptography.hazmat.primitives.serialization import pkcs12
|
|
26
25
|
from cryptography.x509.oid import NameOID
|
|
27
26
|
|
|
28
|
-
# Apple's cert revoke -> create pipeline is eventually consistent: a
|
|
29
|
-
# DELETE on the per-team cap-blocking cert sometimes still shows up as
|
|
30
|
-
# "active" to a follow-up POST for a second or two, returning a fresh
|
|
31
|
-
# 409 even though we just freed a slot. Sleep then retry-with-backoff
|
|
32
|
-
# rather than failing the whole CI run on a known-transient race.
|
|
33
|
-
CERT_REVOKE_PROPAGATION_DELAY_SEC = float(
|
|
34
|
-
os.getenv("CERT_REVOKE_PROPAGATION_DELAY_SEC", "2.0")
|
|
35
|
-
)
|
|
36
|
-
CERT_POST_RETRIES_AFTER_REVOKE = int(
|
|
37
|
-
os.getenv("CERT_POST_RETRIES_AFTER_REVOKE", "2")
|
|
38
|
-
)
|
|
39
|
-
CERTIFICATE_CAP_POLICIES = frozenset({"revoke-oldest", "fail"})
|
|
40
|
-
|
|
41
27
|
|
|
42
28
|
def certificate_cap_policy() -> str:
|
|
43
|
-
"""
|
|
44
|
-
policy = os.getenv("CERTIFICATE_CAP_POLICY", "
|
|
45
|
-
if policy
|
|
46
|
-
|
|
47
|
-
raise SystemExit(f"invalid CERTIFICATE_CAP_POLICY {policy!r}; use {allowed}")
|
|
29
|
+
"""Refuse destructive legacy policy before any provider call."""
|
|
30
|
+
policy = os.getenv("CERTIFICATE_CAP_POLICY", "fail").strip().lower()
|
|
31
|
+
if policy != "fail":
|
|
32
|
+
raise SystemExit("invalid CERTIFICATE_CAP_POLICY; use fail. Automatic certificate revocation is disabled")
|
|
48
33
|
return policy
|
|
49
34
|
|
|
50
35
|
|
|
@@ -75,79 +60,9 @@ def generate_key_and_csr() -> tuple[rsa.RSAPrivateKey, bytes]:
|
|
|
75
60
|
return private_key, csr_payload
|
|
76
61
|
|
|
77
62
|
|
|
78
|
-
def oldest_distribution_cert_id(token: str) -> str | None:
|
|
79
|
-
"""Return the DISTRIBUTION cert with the EARLIEST expirationDate.
|
|
80
|
-
|
|
81
|
-
Apple caps each team at 2 distribution certs; when CI hits the cap
|
|
82
|
-
we must revoke one. Picking the OLDEST is the safest choice — the
|
|
83
|
-
NEWEST cert signed the most recent build that may still be in ASC
|
|
84
|
-
processing, and revoking that mid-processing produces ITMS-90035
|
|
85
|
-
("signed with an ad-hoc certificate, not a distribution
|
|
86
|
-
certificate") on the prior run. Older certs have long since cleared
|
|
87
|
-
processing.
|
|
88
|
-
"""
|
|
89
|
-
data = get_json(
|
|
90
|
-
"/certificates",
|
|
91
|
-
token,
|
|
92
|
-
params={
|
|
93
|
-
"limit": "200",
|
|
94
|
-
"sort": "-id",
|
|
95
|
-
"filter[certificateType]": "DISTRIBUTION",
|
|
96
|
-
},
|
|
97
|
-
)
|
|
98
|
-
oldest_id = None
|
|
99
|
-
oldest_exp: str | None = None
|
|
100
|
-
for cert in data.get("data", []):
|
|
101
|
-
attrs = cert.get("attributes") or {}
|
|
102
|
-
exp = attrs.get("expirationDate") or ""
|
|
103
|
-
if not exp:
|
|
104
|
-
continue
|
|
105
|
-
if oldest_exp is None or exp < oldest_exp:
|
|
106
|
-
oldest_exp = exp
|
|
107
|
-
oldest_id = cert["id"]
|
|
108
|
-
return oldest_id
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
def _post_with_revoke_backoff(token: str, body: dict):
|
|
112
|
-
"""POST a cert create after a revoke, tolerating ASC propagation lag.
|
|
113
|
-
|
|
114
|
-
Apple's DELETE -> POST pipeline is eventually consistent: a freshly
|
|
115
|
-
revoked cert may still count against the per-team cap for a brief
|
|
116
|
-
window, producing a spurious 409 on the immediate re-POST. Sleep
|
|
117
|
-
once for ``CERT_REVOKE_PROPAGATION_DELAY_SEC`` then retry up to
|
|
118
|
-
``CERT_POST_RETRIES_AFTER_REVOKE`` times with exponential backoff.
|
|
119
|
-
The final attempt drops ``allow_status`` so a real 409 surfaces as
|
|
120
|
-
Apple's full body via ``request``'s SystemExit.
|
|
121
|
-
"""
|
|
122
|
-
time.sleep(CERT_REVOKE_PROPAGATION_DELAY_SEC)
|
|
123
|
-
delay = CERT_REVOKE_PROPAGATION_DELAY_SEC
|
|
124
|
-
for attempt in range(CERT_POST_RETRIES_AFTER_REVOKE):
|
|
125
|
-
resp = request(
|
|
126
|
-
"POST", "/certificates", token, json_body=body, allow_status={409}
|
|
127
|
-
)
|
|
128
|
-
if resp.status_code != 409:
|
|
129
|
-
return resp
|
|
130
|
-
print(
|
|
131
|
-
f"Post-revoke 409 on attempt {attempt + 1}/"
|
|
132
|
-
f"{CERT_POST_RETRIES_AFTER_REVOKE}; "
|
|
133
|
-
f"sleeping {delay}s before final retry"
|
|
134
|
-
)
|
|
135
|
-
time.sleep(delay)
|
|
136
|
-
delay *= 2
|
|
137
|
-
# Final attempt without allow_status — let Apple's body surface via
|
|
138
|
-
# request()'s SystemExit if the cap is genuinely still hit.
|
|
139
|
-
return request("POST", "/certificates", token, json_body=body)
|
|
140
|
-
|
|
141
|
-
|
|
142
63
|
def create_distribution_cert(token: str, csr_b64: str) -> tuple[str, bytes]:
|
|
143
|
-
"""POST
|
|
144
|
-
|
|
145
|
-
On 409 (per-team cap of 2 hit) revoke the OLDEST existing cert
|
|
146
|
-
(see :func:`oldest_distribution_cert_id` for why) and retry with
|
|
147
|
-
backoff to absorb Apple's revoke -> create propagation lag (see
|
|
148
|
-
:func:`_post_with_revoke_backoff`).
|
|
149
|
-
"""
|
|
150
|
-
cap_policy = certificate_cap_policy()
|
|
64
|
+
"""POST one CSR; a full team cap preserves every existing certificate."""
|
|
65
|
+
certificate_cap_policy()
|
|
151
66
|
body = {
|
|
152
67
|
"data": {
|
|
153
68
|
"type": "certificates",
|
|
@@ -161,24 +76,10 @@ def create_distribution_cert(token: str, csr_b64: str) -> tuple[str, bytes]:
|
|
|
161
76
|
"POST", "/certificates", token, json_body=body, allow_status={409}
|
|
162
77
|
)
|
|
163
78
|
if resp.status_code == 409:
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
)
|
|
169
|
-
# Revoke the OLDEST cert, NOT the newest. The newest cert signed
|
|
170
|
-
# the previous build that may still be in ASC processing — revoking
|
|
171
|
-
# it during processing yields ITMS-90035 on the prior build.
|
|
172
|
-
# See oldest_distribution_cert_id() for the full rationale.
|
|
173
|
-
print("Distribution cert cap hit; revoking oldest existing cert")
|
|
174
|
-
target = oldest_distribution_cert_id(token)
|
|
175
|
-
if not target:
|
|
176
|
-
raise SystemExit(
|
|
177
|
-
"409 from cert create but no existing DISTRIBUTION cert "
|
|
178
|
-
"found to revoke"
|
|
179
|
-
)
|
|
180
|
-
request("DELETE", f"/certificates/{target}", token)
|
|
181
|
-
resp = _post_with_revoke_backoff(token, body)
|
|
79
|
+
raise SystemExit(
|
|
80
|
+
"distribution certificate cap reached; policy=fail preserves all "
|
|
81
|
+
"existing certificates and refuses automatic revocation; reconcile the account registry"
|
|
82
|
+
)
|
|
182
83
|
data = resp.json()["data"]
|
|
183
84
|
cert_id = data["id"]
|
|
184
85
|
cert_der = base64.b64decode(data["attributes"]["certificateContent"])
|
|
@@ -220,7 +220,7 @@ def verify_cert_alive(token: str, cert_id: str,
|
|
|
220
220
|
"""Return True iff GET /certificates/{cert_id} returns 200.
|
|
221
221
|
|
|
222
222
|
A 404 means Apple revoked it (manually or via the cap-rotation done
|
|
223
|
-
by
|
|
223
|
+
by another signing client); any other non-200 is treated
|
|
224
224
|
conservatively as not-alive so we regenerate.
|
|
225
225
|
"""
|
|
226
226
|
resp = request(
|
|
@@ -28,27 +28,25 @@ class CertificateCapPolicyTests(unittest.TestCase):
|
|
|
28
28
|
response = mock.MagicMock(status_code=409)
|
|
29
29
|
with mock.patch.dict(os.environ, {"CERTIFICATE_CAP_POLICY": "fail"}, clear=True):
|
|
30
30
|
with mock.patch.object(cert_factory, "request", return_value=response) as request:
|
|
31
|
-
with
|
|
32
|
-
|
|
33
|
-
cert_factory.create_distribution_cert("token", "csr")
|
|
31
|
+
with self.assertRaisesRegex(SystemExit, "refuses automatic revocation"):
|
|
32
|
+
cert_factory.create_distribution_cert("token", "csr")
|
|
34
33
|
request.assert_called_once()
|
|
35
|
-
|
|
34
|
+
self.assertEqual(request.call_args.args[:2], ("POST", "/certificates"))
|
|
36
35
|
|
|
37
|
-
def
|
|
36
|
+
def test_default_policy_never_revokes_or_retries_a_full_cap(self):
|
|
38
37
|
capped = mock.MagicMock(status_code=409)
|
|
39
|
-
created = mock.MagicMock()
|
|
40
|
-
created.json.return_value = {
|
|
41
|
-
"data": {"id": "NEW", "attributes": {"certificateContent": "Y2VydA=="}}
|
|
42
|
-
}
|
|
43
38
|
with mock.patch.dict(os.environ, {}, clear=True):
|
|
44
|
-
with mock.patch.object(
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
39
|
+
with mock.patch.object(cert_factory, "request", return_value=capped) as request:
|
|
40
|
+
with self.assertRaisesRegex(SystemExit, "refuses automatic revocation"):
|
|
41
|
+
cert_factory.create_distribution_cert("token", "csr")
|
|
42
|
+
request.assert_called_once()
|
|
43
|
+
|
|
44
|
+
def test_historical_destructive_setting_is_refused_before_contacting_apple(self):
|
|
45
|
+
with mock.patch.dict(os.environ, {"CERTIFICATE_CAP_POLICY": "revoke-oldest"}, clear=True):
|
|
46
|
+
with mock.patch.object(cert_factory, "request") as request:
|
|
47
|
+
with self.assertRaisesRegex(SystemExit, "Automatic certificate revocation is disabled"):
|
|
48
|
+
cert_factory.create_distribution_cert("token", "csr")
|
|
49
|
+
request.assert_not_called()
|
|
52
50
|
|
|
53
51
|
|
|
54
52
|
class CertificateCapActionWiringTests(unittest.TestCase):
|
|
@@ -56,7 +54,7 @@ class CertificateCapActionWiringTests(unittest.TestCase):
|
|
|
56
54
|
source = ACTION_YAML.read_text(encoding="utf-8")
|
|
57
55
|
self.assertRegex(
|
|
58
56
|
source,
|
|
59
|
-
r"(?m)^ certificate-cap-policy:\n(?: .*\n)*? default: \"
|
|
57
|
+
r"(?m)^ certificate-cap-policy:\n(?: .*\n)*? default: \"fail\"$",
|
|
60
58
|
)
|
|
61
59
|
self.assertIn("CERTIFICATE_CAP_POLICY: ${{ inputs.certificate-cap-policy }}", source)
|
|
62
60
|
self.assertIn("SIGNING_PREP_EXIT=$signing_exit", source)
|
|
@@ -1 +1 @@
|
|
|
1
|
-
1.0.
|
|
1
|
+
1.0.61
|
|
@@ -222,7 +222,8 @@ runs:
|
|
|
222
222
|
working-directory: ${{ steps.config.outputs.gradle_root || '.' }}
|
|
223
223
|
run: |
|
|
224
224
|
chmod +x ./gradlew
|
|
225
|
-
|
|
225
|
+
python3 "${{ github.action_path }}/scripts/play_store_proxy.py" -- \
|
|
226
|
+
./gradlew test --console=plain --stacktrace
|
|
226
227
|
|
|
227
228
|
# ANDROID_BUILD_NUMBER is already in the environment: resolve_android.py
|
|
228
229
|
# exported it through GITHUB_ENV. ANDROID_BUILD_NAME is only set when the
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
"""An owned loopback relay for JVMs that cannot authenticate to the assigned HTTP proxy.
|
|
2
|
+
|
|
3
|
+
Only the configured upstream is ever dialled. HTTPS remains an opaque CONNECT
|
|
4
|
+
tunnel; Java still verifies the provider's TLS certificate end to end.
|
|
5
|
+
"""
|
|
6
|
+
from __future__ import annotations
|
|
7
|
+
|
|
8
|
+
import asyncio
|
|
9
|
+
import base64
|
|
10
|
+
from contextlib import contextmanager
|
|
11
|
+
import ssl
|
|
12
|
+
import threading
|
|
13
|
+
from urllib.parse import unquote, urlsplit
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
class Relay:
|
|
17
|
+
def __init__(self, proxy: str):
|
|
18
|
+
self.proxy = urlsplit(proxy)
|
|
19
|
+
self.loop = asyncio.new_event_loop()
|
|
20
|
+
self.ready = threading.Event()
|
|
21
|
+
self.thread = threading.Thread(target=self._serve, daemon=True)
|
|
22
|
+
self.server = None
|
|
23
|
+
self.port = None
|
|
24
|
+
|
|
25
|
+
def _serve(self):
|
|
26
|
+
asyncio.set_event_loop(self.loop)
|
|
27
|
+
try:
|
|
28
|
+
self.server = self.loop.run_until_complete(
|
|
29
|
+
asyncio.start_server(self._handle, "127.0.0.1", 0, limit=65536))
|
|
30
|
+
self.port = self.server.sockets[0].getsockname()[1]
|
|
31
|
+
self.ready.set()
|
|
32
|
+
self.loop.run_forever()
|
|
33
|
+
finally:
|
|
34
|
+
self.ready.set()
|
|
35
|
+
if self.server:
|
|
36
|
+
self.server.close()
|
|
37
|
+
self.loop.run_until_complete(self.server.wait_closed())
|
|
38
|
+
tasks = asyncio.all_tasks(self.loop)
|
|
39
|
+
for task in tasks:
|
|
40
|
+
task.cancel()
|
|
41
|
+
self.loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
|
|
42
|
+
self.loop.close()
|
|
43
|
+
|
|
44
|
+
async def _upstream(self, request: bytes):
|
|
45
|
+
secure = ssl.create_default_context() if self.proxy.scheme == "https" else None
|
|
46
|
+
reader, writer = await asyncio.wait_for(asyncio.open_connection(
|
|
47
|
+
self.proxy.hostname, self.proxy.port, ssl=secure), timeout=30)
|
|
48
|
+
lines = request.decode("iso-8859-1").split("\r\n")
|
|
49
|
+
lines = [line for line in lines if line and not line.lower().startswith("proxy-authorization:")]
|
|
50
|
+
if self.proxy.username is not None:
|
|
51
|
+
login = f"{unquote(self.proxy.username)}:{unquote(self.proxy.password or '')}"
|
|
52
|
+
encoded = base64.b64encode(login.encode()).decode("ascii")
|
|
53
|
+
lines.append("Proxy-Authorization: Basic " + encoded)
|
|
54
|
+
writer.write(("\r\n".join(lines) + "\r\n\r\n").encode("iso-8859-1"))
|
|
55
|
+
await writer.drain()
|
|
56
|
+
return reader, writer
|
|
57
|
+
|
|
58
|
+
async def _pump(self, reader, writer):
|
|
59
|
+
while data := await asyncio.wait_for(reader.read(65536), timeout=120):
|
|
60
|
+
writer.write(data)
|
|
61
|
+
await writer.drain()
|
|
62
|
+
|
|
63
|
+
async def _handle(self, reader, writer):
|
|
64
|
+
upstream = None
|
|
65
|
+
pumps = []
|
|
66
|
+
try:
|
|
67
|
+
request = await asyncio.wait_for(reader.readuntil(b"\r\n\r\n"), timeout=15)
|
|
68
|
+
upstream_reader, upstream = await self._upstream(request)
|
|
69
|
+
if request.startswith(b"CONNECT "):
|
|
70
|
+
answer = await asyncio.wait_for(upstream_reader.readuntil(b"\r\n\r\n"), timeout=30)
|
|
71
|
+
if answer.split(b" ", 2)[1] != b"200":
|
|
72
|
+
raise ConnectionError("upstream tunnel refused")
|
|
73
|
+
writer.write(b"HTTP/1.1 200 Connection established\r\n\r\n")
|
|
74
|
+
await writer.drain()
|
|
75
|
+
pumps = [asyncio.create_task(self._pump(reader, upstream)),
|
|
76
|
+
asyncio.create_task(self._pump(upstream_reader, writer))]
|
|
77
|
+
await asyncio.wait(pumps, return_when=asyncio.FIRST_COMPLETED)
|
|
78
|
+
except (OSError, ValueError, IndexError, asyncio.TimeoutError,
|
|
79
|
+
asyncio.IncompleteReadError, asyncio.LimitOverrunError):
|
|
80
|
+
writer.write(b"HTTP/1.1 502 Proxy unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n")
|
|
81
|
+
finally:
|
|
82
|
+
for task in pumps:
|
|
83
|
+
task.cancel()
|
|
84
|
+
if pumps:
|
|
85
|
+
await asyncio.gather(*pumps, return_exceptions=True)
|
|
86
|
+
for connection in (writer, upstream):
|
|
87
|
+
if connection:
|
|
88
|
+
connection.close()
|
|
89
|
+
try:
|
|
90
|
+
await connection.wait_closed()
|
|
91
|
+
except OSError:
|
|
92
|
+
pass
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
@contextmanager
|
|
96
|
+
def running(proxy: str):
|
|
97
|
+
relay = Relay(proxy)
|
|
98
|
+
relay.thread.start()
|
|
99
|
+
try:
|
|
100
|
+
if not relay.ready.wait(10) or relay.port is None:
|
|
101
|
+
raise RuntimeError("Could not start the owned JVM proxy relay")
|
|
102
|
+
yield relay.port
|
|
103
|
+
finally:
|
|
104
|
+
if relay.loop.is_running():
|
|
105
|
+
relay.loop.call_soon_threadsafe(relay.loop.stop)
|
|
106
|
+
relay.thread.join(timeout=10)
|
|
107
|
+
if relay.thread.is_alive():
|
|
108
|
+
raise RuntimeError("Owned JVM proxy relay did not finish cleanup")
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
def environment(env: dict, port: int) -> dict:
|
|
112
|
+
options = (f"-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort={port} "
|
|
113
|
+
f"-Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort={port} "
|
|
114
|
+
"-Dhttp.nonProxyHosts= -Dhttps.nonProxyHosts= -Djava.net.useSystemProxies=false")
|
|
115
|
+
# Java prints these options at startup; they contain no proxy credentials.
|
|
116
|
+
for name in ("JAVA_TOOL_OPTIONS", "JDK_JAVA_OPTIONS", "_JAVA_OPTIONS", "GRADLE_OPTS"):
|
|
117
|
+
env[name] = (env.get(name, "") + " " + options).strip()
|
|
118
|
+
env["GRADLE_OPTS"] += " -Dorg.gradle.daemon=false"
|
|
119
|
+
return env
|
|
@@ -53,8 +53,11 @@ def run(argv: list[str]) -> int:
|
|
|
53
53
|
"""Run ``argv`` with the explicit proxy environment; the parent process is untouched."""
|
|
54
54
|
if not argv:
|
|
55
55
|
raise SystemExit("usage: play_store_proxy.py -- <command …>")
|
|
56
|
+
from jvm_proxy_relay import environment as jvm_environment, running
|
|
57
|
+
|
|
56
58
|
env = environment()
|
|
57
|
-
|
|
59
|
+
with running(proxy_url()) as port:
|
|
60
|
+
return subprocess.run(argv, env=jvm_environment(env, port), check=False).returncode
|
|
58
61
|
|
|
59
62
|
|
|
60
63
|
def clear_github_env_bypass(path: str | None = None) -> list[str]:
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
"""Real socket and Java proofs: authenticated upstream only, no target fallback."""
|
|
2
|
+
import base64
|
|
3
|
+
from contextlib import closing, contextmanager
|
|
4
|
+
import http.client
|
|
5
|
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
6
|
+
import os
|
|
7
|
+
from pathlib import Path
|
|
8
|
+
import shutil
|
|
9
|
+
import socket
|
|
10
|
+
import subprocess
|
|
11
|
+
import tempfile
|
|
12
|
+
import threading
|
|
13
|
+
import unittest
|
|
14
|
+
|
|
15
|
+
import jvm_proxy_relay as relay
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
@contextmanager
|
|
19
|
+
def upstream():
|
|
20
|
+
seen = []
|
|
21
|
+
|
|
22
|
+
class Proxy(BaseHTTPRequestHandler):
|
|
23
|
+
def do_GET(self):
|
|
24
|
+
seen.append((self.command, self.path, self.headers.get("Proxy-Authorization")))
|
|
25
|
+
self.send_response(200)
|
|
26
|
+
self.end_headers()
|
|
27
|
+
self.wfile.write(b"through pinned proxy")
|
|
28
|
+
|
|
29
|
+
def do_CONNECT(self):
|
|
30
|
+
seen.append((self.command, self.path, self.headers.get("Proxy-Authorization")))
|
|
31
|
+
self.send_response(200)
|
|
32
|
+
self.end_headers()
|
|
33
|
+
value = self.rfile.read(4)
|
|
34
|
+
self.wfile.write(value)
|
|
35
|
+
|
|
36
|
+
def log_message(self, *_):
|
|
37
|
+
pass
|
|
38
|
+
|
|
39
|
+
server = ThreadingHTTPServer(("127.0.0.1", 0), Proxy)
|
|
40
|
+
server.daemon_threads = True
|
|
41
|
+
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
|
42
|
+
thread.start()
|
|
43
|
+
try:
|
|
44
|
+
yield f"http://fixture-user:fixture-pass@127.0.0.1:{server.server_port}", seen
|
|
45
|
+
finally:
|
|
46
|
+
server.shutdown()
|
|
47
|
+
server.server_close()
|
|
48
|
+
thread.join(timeout=5)
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
class JvmProxyTests(unittest.TestCase):
|
|
52
|
+
def test_http_and_connect_only_reach_the_authenticated_upstream_and_cleanup(self):
|
|
53
|
+
with upstream() as (proxy, seen):
|
|
54
|
+
with relay.running(proxy) as port:
|
|
55
|
+
with closing(http.client.HTTPConnection("127.0.0.1", port, timeout=5)) as client:
|
|
56
|
+
client.request("GET", "http://unresolvable.invalid/sdk")
|
|
57
|
+
self.assertEqual(client.getresponse().read(), b"through pinned proxy")
|
|
58
|
+
with socket.create_connection(("127.0.0.1", port), timeout=5) as client:
|
|
59
|
+
client.sendall(b"CONNECT unresolvable.invalid:443 HTTP/1.1\r\nHost: ignored\r\n\r\n")
|
|
60
|
+
self.assertIn(b"200", client.recv(4096))
|
|
61
|
+
client.sendall(b"test")
|
|
62
|
+
self.assertEqual(client.recv(4), b"test")
|
|
63
|
+
with self.assertRaises(OSError):
|
|
64
|
+
socket.create_connection(("127.0.0.1", port), timeout=0.2)
|
|
65
|
+
auth = "Basic " + base64.b64encode(b"fixture-user:fixture-pass").decode()
|
|
66
|
+
self.assertEqual(seen, [("GET", "http://unresolvable.invalid/sdk", auth),
|
|
67
|
+
("CONNECT", "unresolvable.invalid:443", auth)])
|
|
68
|
+
|
|
69
|
+
def test_upstream_failure_returns_bounded_502_without_a_direct_request(self):
|
|
70
|
+
with socket.socket() as unused:
|
|
71
|
+
unused.bind(("127.0.0.1", 0))
|
|
72
|
+
closed_port = unused.getsockname()[1]
|
|
73
|
+
with relay.running(f"http://127.0.0.1:{closed_port}") as port:
|
|
74
|
+
with closing(http.client.HTTPConnection("127.0.0.1", port, timeout=5)) as client:
|
|
75
|
+
client.request("CONNECT", "unresolvable.invalid:443")
|
|
76
|
+
response = client.getresponse()
|
|
77
|
+
self.assertEqual((response.status, response.read()), (502, b""))
|
|
78
|
+
|
|
79
|
+
@unittest.skipUnless(shutil.which("java"), "Java runtime unavailable")
|
|
80
|
+
def test_actual_java_urlconnection_uses_explicit_proxy_with_no_credentials_in_options(self):
|
|
81
|
+
source = ('import java.net.*; public class Probe { public static void main(String[] args) '
|
|
82
|
+
'throws Exception { var c = new URL("http://unresolvable.invalid/sdk").openConnection(); '
|
|
83
|
+
'c.setConnectTimeout(3000); c.setReadTimeout(3000); '
|
|
84
|
+
'System.out.print(new String(c.getInputStream().readAllBytes())); }}')
|
|
85
|
+
with tempfile.TemporaryDirectory() as directory, upstream() as (proxy, seen):
|
|
86
|
+
target = Path(directory) / "Probe.java"
|
|
87
|
+
target.write_text(source)
|
|
88
|
+
with relay.running(proxy) as port:
|
|
89
|
+
env = relay.environment({**os.environ, "JAVA_TOOL_OPTIONS": "-Dhttp.proxyHost=wrong.invalid"}, port)
|
|
90
|
+
result = subprocess.run(["java", str(target)], env=env, capture_output=True, text=True, timeout=20)
|
|
91
|
+
self.assertEqual(result.returncode, 0, result.stderr)
|
|
92
|
+
self.assertEqual(result.stdout, "through pinned proxy")
|
|
93
|
+
self.assertEqual(seen[0][1], "http://unresolvable.invalid/sdk")
|
|
94
|
+
self.assertNotIn("fixture-pass", result.stderr)
|
|
95
|
+
self.assertIn("-Dorg.gradle.daemon=false", env["GRADLE_OPTS"])
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
if __name__ == "__main__":
|
|
99
|
+
unittest.main()
|
|
@@ -57,6 +57,21 @@ class PlayProxyTests(unittest.TestCase):
|
|
|
57
57
|
play_store_proxy.run(["true"])
|
|
58
58
|
run.assert_not_called()
|
|
59
59
|
|
|
60
|
+
def test_failed_child_preserves_exit_code_and_closes_its_relay(self):
|
|
61
|
+
import json
|
|
62
|
+
import socket
|
|
63
|
+
import tempfile
|
|
64
|
+
with tempfile.TemporaryDirectory() as directory:
|
|
65
|
+
path = Path(directory) / "jvm-options.json"
|
|
66
|
+
probe = ("import json, os, sys; "
|
|
67
|
+
"open(sys.argv[1], 'w').write(json.dumps(os.environ['JAVA_TOOL_OPTIONS'])); sys.exit(7)")
|
|
68
|
+
with mock.patch.dict(os.environ, {"GOOGLE_STORE_PROXY_URL": "http://127.0.0.1:1"}):
|
|
69
|
+
self.assertEqual(play_store_proxy.run([sys.executable, "-c", probe, str(path)]), 7)
|
|
70
|
+
options = json.loads(path.read_text())
|
|
71
|
+
port = int(options.split("-Dhttp.proxyPort=", 1)[1].split()[0])
|
|
72
|
+
with self.assertRaises(OSError):
|
|
73
|
+
socket.create_connection(("127.0.0.1", port), timeout=0.2)
|
|
74
|
+
|
|
60
75
|
def test_github_env_mode_clears_only_the_lowercase_bypass_for_later_steps(self):
|
|
61
76
|
import tempfile
|
|
62
77
|
value = "http://assigned.proxy.test:1234"
|
|
@@ -1 +1 @@
|
|
|
1
|
-
1.0.
|
|
1
|
+
1.0.61
|
package/package.json
CHANGED
package/templates/deploy.yml
CHANGED
|
@@ -286,7 +286,8 @@ jobs:
|
|
|
286
286
|
echo "Before:"; df -h / | tail -1
|
|
287
287
|
# NOT /usr/local/lib/node_modules: npm and npx live there, and the
|
|
288
288
|
# Android action's Crashlytics symbol upload shells out to npx.
|
|
289
|
-
|
|
289
|
+
# Keep installed Android SDK/NDKs: removing them makes AGP download them again.
|
|
290
|
+
sudo rm -rf /usr/share/dotnet \
|
|
290
291
|
/opt/ghc /usr/local/.ghcup /usr/local/share/powershell \
|
|
291
292
|
/usr/share/swift /usr/local/share/chromium 2>/dev/null || true
|
|
292
293
|
sudo docker image prune --all --force >/dev/null 2>&1 || true
|