@myelinbridge/cli 0.12.2 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -0
- package/bin/myelin.js +429 -4
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -71,6 +71,11 @@ npx @myelinbridge/cli push ./run_042 --dataset onco1-wes --submit
|
|
|
71
71
|
too (S3 multipart). Uploads go direct to storage over short-lived presigned
|
|
72
72
|
URLs — no credential is stored on your machine, and revoking the API key cuts
|
|
73
73
|
off signing immediately.
|
|
74
|
+
- **A corporate proxy is not our problem to ignore (0.13.0).** `HTTPS_PROXY`,
|
|
75
|
+
`HTTP_PROXY` and `NO_PROXY` are honoured on every call — the API and the part
|
|
76
|
+
uploads alike. Before this the CLI dialled straight out and a pipeline host
|
|
77
|
+
whose only egress is a proxy could not deliver at all. The hosts and ports to
|
|
78
|
+
allow are written down under **For your IT department** below.
|
|
74
79
|
- **Transient failures retry themselves.** Since 0.8.0 a rate limit (`429`)
|
|
75
80
|
waits out `Retry-After` and retries on every command, and a part upload that
|
|
76
81
|
hits a storage hiccup (5xx, an edge timeout, an expired URL) re-signs, backs
|
|
@@ -193,6 +198,99 @@ version.
|
|
|
193
198
|
| `MYELIN_API_KEY` | Required. Created by your bridge owner in **Bridge → API**. |
|
|
194
199
|
| `MYELIN_API_URL` | Optional. Defaults to `https://myelinbridge.com/api/v1`. |
|
|
195
200
|
| `MYELIN_API_HEADER` | Optional. Extra headers sent with every API call, one `Name: value` per line — for a Myelin deployment fronted by something that authenticates before Myelin does (a corporate gateway, an SSO-protected preview). It can never override `Authorization`. |
|
|
201
|
+
| `HTTPS_PROXY` / `HTTP_PROXY` / `NO_PROXY` | Optional, honoured since **0.13.0** — see below. Lowercase spellings read too. |
|
|
202
|
+
|
|
203
|
+
## For your IT department
|
|
204
|
+
|
|
205
|
+
Everything below is derived from what the code actually connects to. If you need
|
|
206
|
+
to hand one page to a network team, hand them this one.
|
|
207
|
+
|
|
208
|
+
### What the CLI needs to reach
|
|
209
|
+
|
|
210
|
+
Outbound **TCP 443 only**, to two hosts:
|
|
211
|
+
|
|
212
|
+
| Host | Why |
|
|
213
|
+
|---|---|
|
|
214
|
+
| `myelinbridge.com` | the API — auth, the quality contract, preflight, part signing, submit, status |
|
|
215
|
+
| `cdjlhyyrbogzkoaujzmj.storage.supabase.co` | the bytes — uploads go **straight to storage** over short-lived presigned URLs, never through the API host |
|
|
216
|
+
|
|
217
|
+
The second one surprises people, and it is the single commonest way a delivery
|
|
218
|
+
half-works: `myelin ping` and `myelin check` succeed, `myelin push` stalls on
|
|
219
|
+
part 1. Since 0.13.0 that failure names the host it could not reach, so the
|
|
220
|
+
allowlist gap is legible from the error.
|
|
221
|
+
|
|
222
|
+
If your client runs Myelin at their own address, `MYELIN_API_URL` names the
|
|
223
|
+
first host and the second is that deployment's storage host. You never have to
|
|
224
|
+
guess it — every Myelin deployment publishes the authoritative list in its own
|
|
225
|
+
`content-security-policy` header:
|
|
226
|
+
|
|
227
|
+
```bash
|
|
228
|
+
curl -sI https://myelinbridge.com | tr ';' '\n' | grep connect-src
|
|
229
|
+
# connect-src 'self' https://<ref>.supabase.co wss://<ref>.supabase.co https://<ref>.storage.supabase.co
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
A wildcard rule works if your proxy prefers one: `*.supabase.co`.
|
|
233
|
+
|
|
234
|
+
### What a person using the web portal needs
|
|
235
|
+
|
|
236
|
+
Someone reviewing a delivery in a browser needs the two hosts above **plus** the
|
|
237
|
+
Supabase API host, because sign-in and the session live there:
|
|
238
|
+
|
|
239
|
+
| Host | Port | Why |
|
|
240
|
+
|---|---|---|
|
|
241
|
+
| `myelinbridge.com` | 443 | the application itself |
|
|
242
|
+
| `cdjlhyyrbogzkoaujzmj.supabase.co` | 443 | sign-in and session |
|
|
243
|
+
| `cdjlhyyrbogzkoaujzmj.storage.supabase.co` | 443 | browser uploads, the same presigned PUTs the CLI makes |
|
|
244
|
+
|
|
245
|
+
Plain HTTPS is enough. The policy also permits `wss:` to the Supabase host, but
|
|
246
|
+
the portal opens no WebSocket today — if your proxy distinguishes the two, you
|
|
247
|
+
do not need to allow WebSocket upgrades for the portal to work.
|
|
248
|
+
|
|
249
|
+
Nothing else is contacted: scripts and fonts are served by the application
|
|
250
|
+
itself and no analytics or tracker host is involved. That is not a promise in a
|
|
251
|
+
README — it is the `content-security-policy` header above, which the browser
|
|
252
|
+
enforces on every response.
|
|
253
|
+
|
|
254
|
+
### What you do *not* need to open
|
|
255
|
+
|
|
256
|
+
- **No inbound anything.** Nothing connects to your network. The one exception
|
|
257
|
+
is opt-in and yours to choose: if you register a webhook endpoint, Myelin
|
|
258
|
+
makes an outbound HTTPS call to an address you publish.
|
|
259
|
+
- **No access to the destination bucket.** The client's cloud bucket is written
|
|
260
|
+
by Myelin, server-side, after review. A partner never touches it and never
|
|
261
|
+
needs a credential for it.
|
|
262
|
+
- **No FTP, SSH, rsync or fixed source IPs.** One protocol, one port.
|
|
263
|
+
|
|
264
|
+
### Behind a proxy
|
|
265
|
+
|
|
266
|
+
Set the standard variables — the CLI honours them on every call, the API and
|
|
267
|
+
the part uploads alike:
|
|
268
|
+
|
|
269
|
+
```bash
|
|
270
|
+
export HTTPS_PROXY=http://proxy.corp.example:8080
|
|
271
|
+
export NO_PROXY=.corp.example,10.0.0.0/8
|
|
272
|
+
myelin ping # prints "via proxy http://proxy.corp.example:8080" when one is in use
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
- `HTTPS_PROXY` for https targets, `HTTP_PROXY` for http, `NO_PROXY` to exclude.
|
|
276
|
+
Lowercase spellings (`https_proxy`, …) are read too — both are in the wild.
|
|
277
|
+
- Credentials go in the URL: `http://user:pass@proxy:8080` (Basic). They are
|
|
278
|
+
never printed, logged, or echoed in an error.
|
|
279
|
+
- `NO_PROXY` accepts a host, a suffix (`.corp.example`, `*.corp.example`),
|
|
280
|
+
`host:port`, an IPv4 CIDR (`10.0.0.0/8`), or `*` for everything.
|
|
281
|
+
- **HTTP CONNECT proxies only.** A `socks5://` value is refused with a message
|
|
282
|
+
rather than silently bypassed — a CLI that quietly dials direct while you
|
|
283
|
+
believe it is proxied is worse than one that stops.
|
|
284
|
+
- A typo in a proxy variable fails immediately, before any command runs.
|
|
285
|
+
- **TLS terminates at Myelin, not at your proxy.** Through a CONNECT tunnel the
|
|
286
|
+
proxy sees the hostname and nothing else. If your proxy performs TLS
|
|
287
|
+
interception, point Node at its CA:
|
|
288
|
+
`export NODE_EXTRA_CA_CERTS=/etc/ssl/corp-root.pem`.
|
|
289
|
+
|
|
290
|
+
Before 0.13.0 the CLI ignored all three variables — Node's `fetch` does — so on
|
|
291
|
+
a host whose only egress is a proxy every command failed with `fetch failed`.
|
|
292
|
+
If you are running an older version behind a proxy, upgrade rather than work
|
|
293
|
+
around it.
|
|
196
294
|
|
|
197
295
|
## Webhooks instead of polling
|
|
198
296
|
|
package/bin/myelin.js
CHANGED
|
@@ -40,10 +40,27 @@
|
|
|
40
40
|
// for), even though `check` correctly warned about it via preflight. The API
|
|
41
41
|
// now includes required-but-empty fields in `metadata` too
|
|
42
42
|
// (`required: true`, `value: null`), and `contract` renders them.
|
|
43
|
+
//
|
|
44
|
+
// 0.13.0 — the CLI reaches the network the way a corporate pipeline host
|
|
45
|
+
// actually reaches it. HTTPS_PROXY / HTTP_PROXY / NO_PROXY (either case) are
|
|
46
|
+
// honoured on every call, API and part upload alike; before this, Node's
|
|
47
|
+
// global fetch ignored all three and a partner whose only egress is a proxy
|
|
48
|
+
// could not deliver at all — the failure read as Myelin being down. The
|
|
49
|
+
// transport is now ours (node:http/node:https, still zero dependencies), so
|
|
50
|
+
// there is one path whether or not a proxy is in the picture. Two smaller
|
|
51
|
+
// consequences: a part upload that cannot reach storage now names the host it
|
|
52
|
+
// could not reach (never the presigned URL — its query string is the
|
|
53
|
+
// credential), and `ping` prints the proxy it used. The hosts and ports a
|
|
54
|
+
// partner's firewall must allow are written down for the first time, in the
|
|
55
|
+
// README's "For your IT department" section.
|
|
43
56
|
|
|
44
57
|
import { readdirSync, statSync, createReadStream, readFileSync } from 'node:fs'
|
|
45
58
|
import { resolve, join, relative, sep, basename } from 'node:path'
|
|
46
59
|
import { createHash } from 'node:crypto'
|
|
60
|
+
import http from 'node:http'
|
|
61
|
+
import https from 'node:https'
|
|
62
|
+
import tls from 'node:tls'
|
|
63
|
+
import zlib from 'node:zlib'
|
|
47
64
|
import process from 'node:process'
|
|
48
65
|
|
|
49
66
|
const API_URL = (process.env.MYELIN_API_URL ?? 'https://myelinbridge.com/api/v1').replace(/\/$/, '')
|
|
@@ -154,13 +171,404 @@ function opt(name) {
|
|
|
154
171
|
return i >= 0 && args[i + 1] && !args[i + 1].startsWith('--') ? args[i + 1] : null
|
|
155
172
|
}
|
|
156
173
|
|
|
174
|
+
// ── Egress: the corporate network this CLI actually runs on ────────────────
|
|
175
|
+
//
|
|
176
|
+
// A CRO's pipeline host usually has no route to the internet of its own. Its
|
|
177
|
+
// only way out is an HTTP CONNECT proxy named by HTTPS_PROXY / HTTP_PROXY,
|
|
178
|
+
// with NO_PROXY listing what must stay inside. Node's global `fetch` ignores
|
|
179
|
+
// all three — Node 22+ honours them only if NODE_USE_ENV_PROXY=1 was set
|
|
180
|
+
// before the process started, Node 20 not at all — so until 0.13.0 every
|
|
181
|
+
// command dialled straight out and died with `fetch failed`. A partner whose
|
|
182
|
+
// only egress is a proxy could not deliver a single byte, and it read as
|
|
183
|
+
// Myelin being down rather than as a variable nobody had honoured.
|
|
184
|
+
//
|
|
185
|
+
// So the CLI speaks the proxy protocol itself, over node:http / node:https,
|
|
186
|
+
// and adds no dependency to do it: this is a binary a customer's IT
|
|
187
|
+
// department installs, and an empty `dependencies` is worth more than the
|
|
188
|
+
// ~200 lines below. It is also the only fix that reaches every Node in
|
|
189
|
+
// `engines` — the undici workaround needs a NODE_OPTIONS preload and a
|
|
190
|
+
// package we would have to ship.
|
|
191
|
+
//
|
|
192
|
+
// One transport for every call, proxied or not. Keeping `fetch` for the
|
|
193
|
+
// direct case would have left the proxy path running only at customer sites,
|
|
194
|
+
// which is the definition of a path that rots.
|
|
195
|
+
//
|
|
196
|
+
// ⚠ Every request carries an explicit agent, always. Node ≥24.5 can install a
|
|
197
|
+
// proxy-aware global agent of its own; if it ever did so under us, a request
|
|
198
|
+
// we had already tunnelled would be tunnelled twice. Our own agent means one
|
|
199
|
+
// behaviour on every Node we support.
|
|
200
|
+
|
|
201
|
+
const UA = `myelin-cli/${VERSION} (node ${process.versions.node})`
|
|
202
|
+
|
|
203
|
+
// Lowercase first — curl's precedence, and the spelling most pipelines set.
|
|
204
|
+
// Both are read because both are in the wild: on Windows they are the same
|
|
205
|
+
// variable, on Linux they are not, and which spelling we happened to prefer
|
|
206
|
+
// must never become the partner's problem.
|
|
207
|
+
const envPair = (...names) => {
|
|
208
|
+
for (const name of names) {
|
|
209
|
+
const value = process.env[name]
|
|
210
|
+
if (typeof value === 'string' && value.trim()) return { name, value: value.trim() }
|
|
211
|
+
}
|
|
212
|
+
return null
|
|
213
|
+
}
|
|
214
|
+
const httpsProxyEnv = () => envPair('https_proxy', 'HTTPS_PROXY')
|
|
215
|
+
const httpProxyEnv = () => envPair('http_proxy', 'HTTP_PROXY')
|
|
216
|
+
const noProxyEnv = () => envPair('no_proxy', 'NO_PROXY')
|
|
217
|
+
const anyProxyEnv = () => Boolean(httpsProxyEnv() || httpProxyEnv())
|
|
218
|
+
|
|
219
|
+
const ipv4ToInt = (ip) => {
|
|
220
|
+
const parts = ip.split('.')
|
|
221
|
+
if (parts.length !== 4) return null
|
|
222
|
+
let n = 0
|
|
223
|
+
for (const part of parts) {
|
|
224
|
+
if (!/^\d{1,3}$/.test(part)) return null
|
|
225
|
+
const v = Number(part)
|
|
226
|
+
if (v > 255) return null
|
|
227
|
+
n = n * 256 + v
|
|
228
|
+
}
|
|
229
|
+
return n >>> 0
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
const cidrContains = (host, entry) => {
|
|
233
|
+
const [network, bitsRaw] = entry.split('/')
|
|
234
|
+
const bits = Number(bitsRaw)
|
|
235
|
+
const a = ipv4ToInt(host)
|
|
236
|
+
const b = ipv4ToInt(network)
|
|
237
|
+
if (a === null || b === null || !Number.isInteger(bits) || bits < 0 || bits > 32) return false
|
|
238
|
+
if (bits === 0) return true
|
|
239
|
+
const mask = (0xffffffff << (32 - bits)) >>> 0
|
|
240
|
+
return ((a & mask) >>> 0) === ((b & mask) >>> 0)
|
|
241
|
+
}
|
|
242
|
+
|
|
243
|
+
// NO_PROXY entries are hostnames, suffixes (`.corp.example`, `*.corp.example`),
|
|
244
|
+
// `host:port`, an IPv4 CIDR, or `*` for everything. The CIDR form is here
|
|
245
|
+
// because a corporate NO_PROXY nearly always carries one (10.0.0.0/8), and
|
|
246
|
+
// tunnelling an internal host fails exactly as hard as not tunnelling an
|
|
247
|
+
// external one.
|
|
248
|
+
function noProxyMatches(hostname, port) {
|
|
249
|
+
const raw = noProxyEnv()?.value
|
|
250
|
+
if (!raw) return false
|
|
251
|
+
const host = hostname.toLowerCase().replace(/^\[/, '').replace(/\]$/, '')
|
|
252
|
+
for (const token of raw.split(/[,\s]+/)) {
|
|
253
|
+
if (!token) continue
|
|
254
|
+
if (token === '*') return true
|
|
255
|
+
let entry = token.toLowerCase()
|
|
256
|
+
// `host:port` — but never split a bare IPv6 literal on its own colons.
|
|
257
|
+
const withPort = /^(.+):(\d+)$/.exec(entry)
|
|
258
|
+
if (withPort && !withPort[1].includes(':')) {
|
|
259
|
+
if (withPort[2] !== String(port)) continue
|
|
260
|
+
entry = withPort[1]
|
|
261
|
+
}
|
|
262
|
+
if (entry.includes('/')) {
|
|
263
|
+
if (cidrContains(host, entry)) return true
|
|
264
|
+
continue
|
|
265
|
+
}
|
|
266
|
+
entry = entry.replace(/^\*?\./, '')
|
|
267
|
+
if (!entry) continue
|
|
268
|
+
if (host === entry || host.endsWith(`.${entry}`)) return true
|
|
269
|
+
}
|
|
270
|
+
return false
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
// A proxy variable we cannot parse is refused out loud. Ignoring it would put
|
|
274
|
+
// us back exactly where this release started: a CLI dialling direct while the
|
|
275
|
+
// partner believes their proxy is carrying the traffic.
|
|
276
|
+
//
|
|
277
|
+
// It throws rather than die()s. die() prints, and this runs inside the request
|
|
278
|
+
// path AND inside egressHint() — so a bad variable printed its message three
|
|
279
|
+
// times before the error it caused. One config problem, one line: the throw is
|
|
280
|
+
// rendered once by whoever was making the call, and checked up front by
|
|
281
|
+
// assertProxyEnvUsable() before any command runs.
|
|
282
|
+
const proxyCache = new Map()
|
|
283
|
+
function parseProxy(varName, raw) {
|
|
284
|
+
const cacheKey = `${varName} ${raw}`
|
|
285
|
+
if (proxyCache.has(cacheKey)) return proxyCache.get(cacheKey)
|
|
286
|
+
// `proxy.corp:8080`, with no scheme at all, is common enough to accept.
|
|
287
|
+
const withScheme = /^[a-z][a-z0-9+.-]*:\/\//i.test(raw) ? raw : `http://${raw}`
|
|
288
|
+
let u = null
|
|
289
|
+
try { u = new URL(withScheme) } catch { u = null }
|
|
290
|
+
if (!u || !u.hostname) {
|
|
291
|
+
throw new Error(`${varName} is not a usable proxy URL ("${raw}") — expected http://host:port`)
|
|
292
|
+
}
|
|
293
|
+
if (u.protocol !== 'http:' && u.protocol !== 'https:') {
|
|
294
|
+
throw new Error(
|
|
295
|
+
`${varName} names a ${u.protocol.replace(':', '')} proxy ("${raw}"), which this CLI cannot use. ` +
|
|
296
|
+
`Myelin speaks to an HTTP CONNECT proxy (http://host:port). Ask your network team for one, or for the ` +
|
|
297
|
+
`allowlist in the CLI README's "For your IT department" section.`,
|
|
298
|
+
)
|
|
299
|
+
}
|
|
300
|
+
const port = Number(u.port || (u.protocol === 'https:' ? 443 : 80))
|
|
301
|
+
const proxy = {
|
|
302
|
+
protocol: u.protocol,
|
|
303
|
+
hostname: u.hostname,
|
|
304
|
+
port,
|
|
305
|
+
// Credentials go on the wire and nowhere else: `display` is the only form
|
|
306
|
+
// a message, a log line or an error is allowed to carry.
|
|
307
|
+
auth: u.username
|
|
308
|
+
? Buffer.from(`${decodeURIComponent(u.username)}:${decodeURIComponent(u.password)}`).toString('base64')
|
|
309
|
+
: null,
|
|
310
|
+
display: `${u.protocol}//${u.hostname}:${port}`,
|
|
311
|
+
}
|
|
312
|
+
proxyCache.set(cacheKey, proxy)
|
|
313
|
+
return proxy
|
|
314
|
+
}
|
|
315
|
+
|
|
316
|
+
function proxyForUrl(url) {
|
|
317
|
+
const port = Number(url.port || (url.protocol === 'https:' ? 443 : 80))
|
|
318
|
+
if (noProxyMatches(url.hostname, port)) return null
|
|
319
|
+
const pair = url.protocol === 'https:' ? httpsProxyEnv() : httpProxyEnv()
|
|
320
|
+
return pair ? parseProxy(pair.name, pair.value) : null
|
|
321
|
+
}
|
|
322
|
+
|
|
323
|
+
// Checked before the first request rather than at the moment one fails: a typo
|
|
324
|
+
// in HTTPS_PROXY deserves one clear line at the top, not a connection error
|
|
325
|
+
// wearing a configuration error's clothes.
|
|
326
|
+
function assertProxyEnvUsable() {
|
|
327
|
+
for (const pair of [httpsProxyEnv(), httpProxyEnv()]) {
|
|
328
|
+
if (!pair) continue
|
|
329
|
+
try { parseProxy(pair.name, pair.value) } catch (err) { die(err.message) }
|
|
330
|
+
}
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
const hostOf = (rawUrl) => {
|
|
334
|
+
try { return new URL(rawUrl).host } catch { return 'the storage host' }
|
|
335
|
+
}
|
|
336
|
+
|
|
337
|
+
// Which egress a failed call took, in one clause appended to the error. The
|
|
338
|
+
// old message ("fetch failed") sent a proxied partner to us; naming the proxy
|
|
339
|
+
// sends them to their network team, which is where the answer lives.
|
|
340
|
+
function egressHint(rawUrl, message = '') {
|
|
341
|
+
let url = null
|
|
342
|
+
try { url = new URL(rawUrl) } catch { return '' }
|
|
343
|
+
let proxy = null
|
|
344
|
+
try { proxy = proxyForUrl(url) } catch { return '' }
|
|
345
|
+
if (proxy) return message.includes(proxy.display) ? '' : ` — via proxy ${proxy.display}`
|
|
346
|
+
const port = Number(url.port || (url.protocol === 'https:' ? 443 : 80))
|
|
347
|
+
if (anyProxyEnv() && noProxyMatches(url.hostname, port)) {
|
|
348
|
+
return ` — connected directly: NO_PROXY excludes ${url.hostname}`
|
|
349
|
+
}
|
|
350
|
+
return ''
|
|
351
|
+
}
|
|
352
|
+
|
|
353
|
+
const directAgents = {
|
|
354
|
+
'http:': new http.Agent({ keepAlive: true }),
|
|
355
|
+
'https:': new https.Agent({ keepAlive: true }),
|
|
356
|
+
}
|
|
357
|
+
|
|
358
|
+
// CONNECT is where a misconfigured proxy hangs for ever — a blackholed SYN, or
|
|
359
|
+
// a proxy that accepts the socket and then says nothing. Every other wait in
|
|
360
|
+
// this file deliberately inherits fetch's "no timeout" (a 10 000-file
|
|
361
|
+
// preflight legitimately takes minutes); a tunnel that has not opened in 30 s
|
|
362
|
+
// is not opening.
|
|
363
|
+
const TUNNEL_TIMEOUT_MS = 30_000
|
|
364
|
+
|
|
365
|
+
function openTunnel(proxy, host, port) {
|
|
366
|
+
return new Promise((resolveSocket, reject) => {
|
|
367
|
+
const transport = proxy.protocol === 'https:' ? https : http
|
|
368
|
+
const req = transport.request({
|
|
369
|
+
host: proxy.hostname,
|
|
370
|
+
port: proxy.port,
|
|
371
|
+
method: 'CONNECT',
|
|
372
|
+
path: `${host}:${port}`,
|
|
373
|
+
agent: false,
|
|
374
|
+
headers: {
|
|
375
|
+
Host: `${host}:${port}`,
|
|
376
|
+
'User-Agent': UA,
|
|
377
|
+
...(proxy.auth ? { 'Proxy-Authorization': `Basic ${proxy.auth}` } : {}),
|
|
378
|
+
},
|
|
379
|
+
...(proxy.protocol === 'https:' ? { servername: proxy.hostname } : {}),
|
|
380
|
+
})
|
|
381
|
+
let settled = false
|
|
382
|
+
const timer = setTimeout(() => {
|
|
383
|
+
if (settled) return
|
|
384
|
+
settled = true
|
|
385
|
+
req.destroy()
|
|
386
|
+
reject(new Error(`proxy ${proxy.display} did not open a tunnel to ${host}:${port} within ${TUNNEL_TIMEOUT_MS / 1000}s`))
|
|
387
|
+
}, TUNNEL_TIMEOUT_MS)
|
|
388
|
+
timer.unref?.()
|
|
389
|
+
const settle = (fn) => (...a) => {
|
|
390
|
+
if (settled) return
|
|
391
|
+
settled = true
|
|
392
|
+
clearTimeout(timer)
|
|
393
|
+
fn(...a)
|
|
394
|
+
}
|
|
395
|
+
req.once('connect', settle((res, socket) => {
|
|
396
|
+
if (res.statusCode === 200) { resolveSocket(socket); return }
|
|
397
|
+
socket.destroy()
|
|
398
|
+
reject(new Error(
|
|
399
|
+
res.statusCode === 407
|
|
400
|
+
? `proxy ${proxy.display} requires authentication (407) — put the credentials in the proxy URL (http://user:pass@host:port)`
|
|
401
|
+
: `proxy ${proxy.display} refused a tunnel to ${host}:${port} (HTTP ${res.statusCode})`,
|
|
402
|
+
))
|
|
403
|
+
}))
|
|
404
|
+
// A proxy that answers CONNECT with an ordinary response — a captive
|
|
405
|
+
// portal, or an explicit proxy expecting absolute-form requests — is not
|
|
406
|
+
// going to tunnel anything. Say so rather than wait for the timeout.
|
|
407
|
+
req.once('response', settle((res) => {
|
|
408
|
+
res.resume()
|
|
409
|
+
reject(new Error(`proxy ${proxy.display} answered CONNECT with HTTP ${res.statusCode} instead of opening a tunnel to ${host}:${port}`))
|
|
410
|
+
}))
|
|
411
|
+
req.once('error', settle((err) => reject(new Error(`proxy ${proxy.display}: ${err.message}`))))
|
|
412
|
+
req.end()
|
|
413
|
+
})
|
|
414
|
+
}
|
|
415
|
+
|
|
416
|
+
// One tunnelling agent per proxy. keepAlive is OFF on this path on purpose: a
|
|
417
|
+
// pooled socket here is a CONNECT tunnel the proxy may have torn down while it
|
|
418
|
+
// sat idle, and the next part upload would then fail on a socket that still
|
|
419
|
+
// looked healthy. One handshake per request, against a 64 MB part, is noise.
|
|
420
|
+
class TunnelAgent extends https.Agent {
|
|
421
|
+
constructor(proxy) {
|
|
422
|
+
super({ keepAlive: false })
|
|
423
|
+
this.proxy = proxy
|
|
424
|
+
}
|
|
425
|
+
createConnection(options, cb) {
|
|
426
|
+
openTunnel(this.proxy, options.host, options.port).then(
|
|
427
|
+
(socket) => cb(null, tls.connect({ socket, servername: options.servername || options.host })),
|
|
428
|
+
(err) => cb(err),
|
|
429
|
+
)
|
|
430
|
+
}
|
|
431
|
+
}
|
|
432
|
+
const tunnelAgents = new Map()
|
|
433
|
+
const tunnelAgentFor = (proxy) => {
|
|
434
|
+
const key = `${proxy.display} ${proxy.auth ?? ''}`
|
|
435
|
+
let agent = tunnelAgents.get(key)
|
|
436
|
+
if (!agent) {
|
|
437
|
+
agent = new TunnelAgent(proxy)
|
|
438
|
+
tunnelAgents.set(key, agent)
|
|
439
|
+
}
|
|
440
|
+
return agent
|
|
441
|
+
}
|
|
442
|
+
|
|
443
|
+
// Bodies: a string, a Buffer, or a FormData (the sandbox upload). Multipart is
|
|
444
|
+
// not hand-rolled — `Response` is a global, so letting the platform encode the
|
|
445
|
+
// FormData costs no dependency and cannot disagree with what a server expects.
|
|
446
|
+
async function encodeBody(body) {
|
|
447
|
+
if (body == null) return { bytes: null, contentType: null }
|
|
448
|
+
if (typeof body === 'string') return { bytes: Buffer.from(body, 'utf8'), contentType: null }
|
|
449
|
+
if (Buffer.isBuffer(body)) return { bytes: body, contentType: null }
|
|
450
|
+
if (ArrayBuffer.isView(body)) return { bytes: Buffer.from(body.buffer, body.byteOffset, body.byteLength), contentType: null }
|
|
451
|
+
const encoded = new Response(body)
|
|
452
|
+
return { bytes: Buffer.from(await encoded.arrayBuffer()), contentType: encoded.headers.get('content-type') }
|
|
453
|
+
}
|
|
454
|
+
|
|
455
|
+
function decodeBody(raw, encoding) {
|
|
456
|
+
const enc = String(encoding ?? '').trim().toLowerCase()
|
|
457
|
+
try {
|
|
458
|
+
if (enc === 'gzip' || enc === 'x-gzip') return zlib.gunzipSync(raw).toString('utf8')
|
|
459
|
+
if (enc === 'br') return zlib.brotliDecompressSync(raw).toString('utf8')
|
|
460
|
+
if (enc === 'deflate') {
|
|
461
|
+
try { return zlib.inflateSync(raw).toString('utf8') } catch { return zlib.inflateRawSync(raw).toString('utf8') }
|
|
462
|
+
}
|
|
463
|
+
} catch {
|
|
464
|
+
// A body we cannot inflate is still worth showing raw — half an error
|
|
465
|
+
// message beats none.
|
|
466
|
+
}
|
|
467
|
+
return raw.toString('utf8')
|
|
468
|
+
}
|
|
469
|
+
|
|
470
|
+
// The whole HTTP surface of this CLI, in one function. It returns the small
|
|
471
|
+
// slice of the fetch Response shape the rest of the file uses — `status`,
|
|
472
|
+
// `headers.get()`, `json()`, `text()` — so the call sites read as they did.
|
|
473
|
+
async function httpRequest(rawUrl, { method = 'GET', headers = {}, body } = {}, redirectsLeft = 5) {
|
|
474
|
+
const url = new URL(rawUrl)
|
|
475
|
+
const proxy = proxyForUrl(url)
|
|
476
|
+
const { bytes, contentType } = await encodeBody(body)
|
|
477
|
+
|
|
478
|
+
const sent = {}
|
|
479
|
+
for (const [k, v] of Object.entries(headers)) if (v != null) sent[k.toLowerCase()] = String(v)
|
|
480
|
+
sent['user-agent'] ??= UA
|
|
481
|
+
sent['accept-encoding'] ??= 'gzip, deflate, br'
|
|
482
|
+
if (bytes) {
|
|
483
|
+
// ⚠ Always a Content-Length, never chunked: S3 rejects a chunked PUT, and
|
|
484
|
+
// a presigned part upload is the one request here that must never be
|
|
485
|
+
// refused for a transport reason.
|
|
486
|
+
sent['content-length'] = String(bytes.length)
|
|
487
|
+
if (contentType && !sent['content-type']) sent['content-type'] = contentType
|
|
488
|
+
}
|
|
489
|
+
|
|
490
|
+
// https through a proxy is a CONNECT tunnel; plain http through a proxy is
|
|
491
|
+
// an absolute-form request line addressed to the proxy itself.
|
|
492
|
+
const tunnelled = Boolean(proxy) && url.protocol === 'https:'
|
|
493
|
+
const forwarded = Boolean(proxy) && url.protocol === 'http:'
|
|
494
|
+
if (forwarded) {
|
|
495
|
+
sent.host = url.host
|
|
496
|
+
if (proxy.auth) sent['proxy-authorization'] = `Basic ${proxy.auth}`
|
|
497
|
+
}
|
|
498
|
+
const transport = (forwarded ? proxy.protocol : url.protocol) === 'https:' ? https : http
|
|
499
|
+
const options = forwarded
|
|
500
|
+
? { host: proxy.hostname, port: proxy.port, method, path: url.href, headers: sent, agent: directAgents[proxy.protocol] }
|
|
501
|
+
: {
|
|
502
|
+
host: url.hostname,
|
|
503
|
+
port: Number(url.port || (url.protocol === 'https:' ? 443 : 80)),
|
|
504
|
+
method,
|
|
505
|
+
path: `${url.pathname}${url.search}`,
|
|
506
|
+
headers: sent,
|
|
507
|
+
agent: tunnelled ? tunnelAgentFor(proxy) : directAgents[url.protocol],
|
|
508
|
+
}
|
|
509
|
+
|
|
510
|
+
return new Promise((settle, reject) => {
|
|
511
|
+
const req = transport.request(options, (res) => {
|
|
512
|
+
// 407 is a proxy's status, never an origin server's. On the forwarded
|
|
513
|
+
// path it arrives as an ordinary response, so without this the CLI
|
|
514
|
+
// reported it as though Myelin had answered — "[error] HTTP 407" and a
|
|
515
|
+
// partner reading it as our problem. The tunnel path already says this
|
|
516
|
+
// properly; both now do.
|
|
517
|
+
if (forwarded && res.statusCode === 407) {
|
|
518
|
+
res.resume()
|
|
519
|
+
reject(new Error(`proxy ${proxy.display} requires authentication (407) — put the credentials in the proxy URL (http://user:pass@host:port)`))
|
|
520
|
+
return
|
|
521
|
+
}
|
|
522
|
+
const location = res.headers.location
|
|
523
|
+
if (location && [301, 302, 303, 307, 308].includes(res.statusCode) && redirectsLeft > 0) {
|
|
524
|
+
res.resume()
|
|
525
|
+
let next = null
|
|
526
|
+
try { next = new URL(location, url) } catch { next = null }
|
|
527
|
+
if (next) {
|
|
528
|
+
const toGet = res.statusCode === 303 || ([301, 302].includes(res.statusCode) && method !== 'GET' && method !== 'HEAD')
|
|
529
|
+
const forward = {}
|
|
530
|
+
for (const [k, v] of Object.entries(headers)) {
|
|
531
|
+
// The rule fetch follows, and the one that matters here: a bearer
|
|
532
|
+
// token does not cross an origin. A front door that bounces us to
|
|
533
|
+
// an identity provider must never be handed the API key.
|
|
534
|
+
if (next.origin !== url.origin && k.toLowerCase() === 'authorization') continue
|
|
535
|
+
forward[k] = v
|
|
536
|
+
}
|
|
537
|
+
settle(httpRequest(next.href, { method: toGet ? 'GET' : method, headers: forward, body: toGet ? undefined : body }, redirectsLeft - 1))
|
|
538
|
+
return
|
|
539
|
+
}
|
|
540
|
+
}
|
|
541
|
+
const chunks = []
|
|
542
|
+
res.on('data', (c) => chunks.push(c))
|
|
543
|
+
res.on('error', reject)
|
|
544
|
+
res.on('end', () => {
|
|
545
|
+
const text = decodeBody(Buffer.concat(chunks), res.headers['content-encoding'])
|
|
546
|
+
settle({
|
|
547
|
+
status: res.statusCode,
|
|
548
|
+
headers: {
|
|
549
|
+
get: (name) => {
|
|
550
|
+
const v = res.headers[String(name).toLowerCase()]
|
|
551
|
+
return v == null ? null : Array.isArray(v) ? v.join(', ') : v
|
|
552
|
+
},
|
|
553
|
+
},
|
|
554
|
+
text: async () => text,
|
|
555
|
+
json: async () => JSON.parse(text),
|
|
556
|
+
})
|
|
557
|
+
})
|
|
558
|
+
})
|
|
559
|
+
req.once('error', reject)
|
|
560
|
+
if (bytes) req.end(bytes)
|
|
561
|
+
else req.end()
|
|
562
|
+
})
|
|
563
|
+
}
|
|
564
|
+
|
|
157
565
|
// 429s are handled HERE so every command inherits the platform's documented
|
|
158
566
|
// contract (honour Retry-After): bounded — 4 waits, then the error surfaces
|
|
159
567
|
// like any other. The wait is capped at 60 s so a pathological header cannot
|
|
160
568
|
// stall a pipeline for an hour.
|
|
161
569
|
async function api(method, path, body, raw) {
|
|
162
570
|
for (let attempt = 1; ; attempt++) {
|
|
163
|
-
const res = await
|
|
571
|
+
const res = await httpRequest(API_URL + path, {
|
|
164
572
|
method,
|
|
165
573
|
headers: {
|
|
166
574
|
...EXTRA_HEADERS,
|
|
@@ -168,7 +576,7 @@ async function api(method, path, body, raw) {
|
|
|
168
576
|
...(body && !raw ? { 'Content-Type': 'application/json' } : {}),
|
|
169
577
|
},
|
|
170
578
|
body: raw ? body : body ? JSON.stringify(body) : undefined,
|
|
171
|
-
}).catch((err) => die(`Cannot reach ${API_URL}: ${err.message}`))
|
|
579
|
+
}).catch((err) => die(`Cannot reach ${API_URL}: ${err.message}${egressHint(API_URL, err.message)}`))
|
|
172
580
|
let json = null
|
|
173
581
|
try { json = await res.json() } catch { /* non-JSON body */ }
|
|
174
582
|
const requestId = res.headers.get('myelin-request-id') ?? json?.request_id ?? null
|
|
@@ -452,10 +860,19 @@ async function uploadFileParts(absPath, size, fileId, partSize, adoptLandedGeome
|
|
|
452
860
|
if (url === undefined) url = await resign(partNo, bytes)
|
|
453
861
|
if (url === null) break // part landed server-side — resign() recorded its etag
|
|
454
862
|
const startedAt = Date.now()
|
|
455
|
-
|
|
863
|
+
let netErr = null
|
|
864
|
+
put = await httpRequest(url, { method: 'PUT', body: bytes }).catch((err) => { netErr = err; return null })
|
|
456
865
|
if (put && put.status === 200) break
|
|
457
866
|
const status = put ? put.status : 0
|
|
458
|
-
|
|
867
|
+
// ⚠ Never print `url` itself — a presigned URL carries its
|
|
868
|
+
// authorization in the query string. The HOST is the part a partner
|
|
869
|
+
// needs: a firewall or proxy that allows the API host and not the
|
|
870
|
+
// storage host is the commonest way a delivery dies, and the old
|
|
871
|
+
// bare "network error" sent them to look at Myelin instead of at
|
|
872
|
+
// their own egress.
|
|
873
|
+
const reason = put
|
|
874
|
+
? `HTTP ${put.status}`
|
|
875
|
+
: `network error reaching ${hostOf(url)}${netErr ? ` (${netErr.message})` : ''}${egressHint(url, netErr?.message ?? '')}`
|
|
459
876
|
if (isTimeoutClassFailure(status, Date.now() - startedAt)) {
|
|
460
877
|
const strikes = (timeoutStrikes.get(partNo) ?? 0) + 1
|
|
461
878
|
timeoutStrikes.set(partNo, strikes)
|
|
@@ -496,6 +913,11 @@ async function cmdPing() {
|
|
|
496
913
|
const j = expectOk(await api('GET', '/ping'), 'ping')
|
|
497
914
|
emit(j)
|
|
498
915
|
out(`✓ key "${j.key.name}" (${j.key.prefix}…) · bridge ${j.bridge.name} · ${j.projects.length} project(s)`)
|
|
916
|
+
// The first question an IT department asks once a proxy is in play: is the
|
|
917
|
+
// CLI actually going through it? Printed only when one applies, so the
|
|
918
|
+
// ordinary output is unchanged.
|
|
919
|
+
const via = proxyForUrl(new URL(API_URL))
|
|
920
|
+
if (via) out(` via proxy ${via.display}`)
|
|
499
921
|
}
|
|
500
922
|
|
|
501
923
|
async function cmdProjects() {
|
|
@@ -1041,6 +1463,8 @@ Setup:
|
|
|
1041
1463
|
export MYELIN_API_URL=… (optional; default https://myelinbridge.com/api/v1)
|
|
1042
1464
|
export MYELIN_API_HEADER="Name: value" (optional; extra headers, one per line —
|
|
1043
1465
|
for a gateway that authenticates ahead of Myelin)
|
|
1466
|
+
export HTTPS_PROXY=http://proxy:8080 (optional; HTTP_PROXY / NO_PROXY honoured too,
|
|
1467
|
+
either case — see README "For your IT department")
|
|
1044
1468
|
|
|
1045
1469
|
Commands:
|
|
1046
1470
|
ping verify the key; show bridge + projects
|
|
@@ -1165,6 +1589,7 @@ const commands = {
|
|
|
1165
1589
|
}
|
|
1166
1590
|
try {
|
|
1167
1591
|
if (!KEY) die('Set MYELIN_API_KEY (create a key in Bridge → API).')
|
|
1592
|
+
assertProxyEnvUsable()
|
|
1168
1593
|
const fn = commands[command]
|
|
1169
1594
|
if (!fn) die(`Unknown command "${command}" — run: myelin help`)
|
|
1170
1595
|
await fn()
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@myelinbridge/cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "Myelin Partner Ingestion CLI — push R&D data deliveries from a pipeline: preflight against the client's quality rules, resumable upload, submit, track review outcomes.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|