@pikku/core 0.12.78 → 0.12.79
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +73 -0
- package/dist/services/in-memory-workflow-service.d.ts +10 -0
- package/dist/services/in-memory-workflow-service.js +23 -0
- package/dist/utils/node-host-resolver.d.ts +12 -0
- package/dist/utils/node-host-resolver.js +16 -0
- package/dist/utils/safe-fetch.d.ts +18 -0
- package/dist/utils/safe-fetch.js +167 -29
- package/dist/wirings/gateway/gateway-runner.js +32 -3
- package/dist/wirings/secret/validate-secret-definitions.js +2 -0
- package/dist/wirings/workflow/pikku-workflow-service.d.ts +52 -0
- package/dist/wirings/workflow/pikku-workflow-service.js +104 -0
- package/knowledge/decisions/security/gateway-handlers-run-through-the-function-runner-gate.md +13 -3
- package/package.json +2 -1
- package/src/services/in-memory-workflow-service.ts +23 -0
- package/src/utils/node-host-resolver.ts +20 -0
- package/src/utils/safe-fetch.test.ts +143 -1
- package/src/utils/safe-fetch.ts +200 -25
- package/src/wirings/gateway/gateway-authorization.test.ts +131 -0
- package/src/wirings/gateway/gateway-runner.ts +35 -3
- package/src/wirings/secret/validate-secret-definitions.test.ts +47 -0
- package/src/wirings/secret/validate-secret-definitions.ts +2 -0
- package/src/wirings/workflow/pikku-workflow-service.ts +131 -0
- package/src/wirings/workflow/workflow-dispatch-relay.test.ts +128 -0
- package/tsconfig.tsbuildinfo +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,76 @@
|
|
|
1
|
+
## 0.12.79
|
|
2
|
+
|
|
3
|
+
### Patch Changes
|
|
4
|
+
|
|
5
|
+
- e848eb2: Add `relayUndispatchedSteps()`, which re-drives steps whose queue or scheduler
|
|
6
|
+
dispatch was lost.
|
|
7
|
+
|
|
8
|
+
Arming a step is two writes to two systems — the step row lands `pending`, then
|
|
9
|
+
a job is published — and nothing spans both, so a crash in between leaves a
|
|
10
|
+
durable row nothing will ever pick up. The run then neither finishes nor fails.
|
|
11
|
+
|
|
12
|
+
The step row is the outbox record and this is the relay. It is safe to
|
|
13
|
+
re-dispatch a step that already has a live job because `executeWorkflowStepInner`
|
|
14
|
+
claims the step under `withStepLock` before invoking anything: the loser reads
|
|
15
|
+
`running` and returns. Stores opt in by overriding `findUndispatchedSteps`; the
|
|
16
|
+
default returns nothing, so a store without an atomic step lock gains no
|
|
17
|
+
re-dispatches. Opted in: `kysely-postgres` and `kysely-mysql` (real locks) and
|
|
18
|
+
in-memory (inline, single-process, no queues). Not opted in: `mongodb` and
|
|
19
|
+
`kysely-sqlite`, whose `withStepLock` is a pass-through.
|
|
20
|
+
|
|
21
|
+
Not self-starting — call it from a scheduled task.
|
|
22
|
+
|
|
23
|
+
- b170489: Gateway handlers now run under the metadata their author actually wrote, instead
|
|
24
|
+
of a fabricated meta object.
|
|
25
|
+
|
|
26
|
+
`registerGatewayHandler` used to synthesize `{ sessionless: true,
|
|
27
|
+
inputSchemaName: null, outputSchemaName: null }` for every gateway handler, so a
|
|
28
|
+
function declared with `pikkuFunc` — which records "session required" in meta
|
|
29
|
+
rather than through an `auth` property — was silently made sessionless, and its
|
|
30
|
+
input schema was never validated. It now inherits `sessionless`, the schema
|
|
31
|
+
names, `scopes` and tag middleware from the function the gateway was wired with,
|
|
32
|
+
falling back to the old sessionless default only when nothing was declared (a
|
|
33
|
+
gateway wired by hand rather than through codegen).
|
|
34
|
+
|
|
35
|
+
Tag middleware also reaches gateways for the first time. `addTagMiddleware('x',
|
|
36
|
+
…)` combined with `wireGateway({ tags: ['x'] })` previously resolved to nothing
|
|
37
|
+
at all — the inspector computed the middleware and no runtime path read it — so
|
|
38
|
+
a tag that read like a gate applied none.
|
|
39
|
+
|
|
40
|
+
**Both are tightening changes.** A gateway whose handler declared a session
|
|
41
|
+
requirement, or whose tags name registered middleware, now enforces what it
|
|
42
|
+
already said it enforced.
|
|
43
|
+
|
|
44
|
+
- ae4e898: Carry a secret's `allowedHosts` through code generation, and close three gaps in
|
|
45
|
+
the SSRF guard.
|
|
46
|
+
|
|
47
|
+
`allowedHosts` was declared and enforced but never survived codegen: the
|
|
48
|
+
inspector did not read the property off the `defineSecret` literal, and the meta
|
|
49
|
+
builder rebuilt its objects without it. Enforcement in `assertSecretAllowedForHost`
|
|
50
|
+
then always saw `undefined`, so the egress restriction was a no-op by default —
|
|
51
|
+
and, with `secrets.requireAllowedHosts` set, threw for every secret including the
|
|
52
|
+
ones that correctly declared hosts. Both stages now carry the field, and the
|
|
53
|
+
secrets verifier asserts it against the generated JSON rather than a hand-written
|
|
54
|
+
meta literal, which is why the existing tests stayed green.
|
|
55
|
+
|
|
56
|
+
`isPrivateHost` now checks an explicit CIDR table instead of ad-hoc octet
|
|
57
|
+
comparisons. It previously missed `100.64.0.0/10` — which contains Alibaba
|
|
58
|
+
Cloud's `100.100.100.200` metadata endpoint — along with `192.0.0.0/24`,
|
|
59
|
+
`198.18.0.0/15`, `192.88.99.0/24`, the TEST-NETs, multicast and reserved space.
|
|
60
|
+
IPv6 gains a real parser, so `fec0::/10`, `ff00::/8`, and NAT64 (`64:ff9b::/96`)
|
|
61
|
+
and 6to4 (`2002::/16`) forms wrapping an internal IPv4 address are caught.
|
|
62
|
+
|
|
63
|
+
`safeFetch` takes an optional `resolveHost`, checked on the initial URL and every
|
|
64
|
+
redirect hop, so a _public_ hostname pointing at a private address is refused —
|
|
65
|
+
the `169-254-169-254.nip.io` shape a literal-only check cannot see. Core cannot
|
|
66
|
+
resolve DNS itself (Workers has no DNS API), so the Node resolver ships as
|
|
67
|
+
`@pikku/core/node-host-resolver` and the Node server runtimes install it during
|
|
68
|
+
`init()`. The connection is not pinned to the address that was checked, so a
|
|
69
|
+
rebind between check and connect is still possible.
|
|
70
|
+
|
|
71
|
+
The graph addon's `httpRequest` node called bare `fetch`, bypassing the guard
|
|
72
|
+
entirely; it now goes through `safeFetch`.
|
|
73
|
+
|
|
1
74
|
## 0.12.78
|
|
2
75
|
|
|
3
76
|
### Patch Changes
|
|
@@ -29,6 +29,16 @@ export declare class InMemoryWorkflowService extends PikkuWorkflowService implem
|
|
|
29
29
|
protected setStepErrorImpl(stepId: string, error: Error): Promise<void>;
|
|
30
30
|
protected setStepChildRunIdImpl(stepId: string, childRunId: string): Promise<void>;
|
|
31
31
|
protected createRetryAttemptImpl(failedStepId: string, status: 'pending' | 'running'): Promise<StepState>;
|
|
32
|
+
/**
|
|
33
|
+
* Safe to opt in despite the pass-through `withStepLock`: this service wires
|
|
34
|
+
* no queues and runs every step inline in one process, so the relay's
|
|
35
|
+
* redundant dispatch has no second holder to race with. See
|
|
36
|
+
* knowledge: decisions/internals/the-in-memory-workflow-service-is-inline-only-and-single-process.md
|
|
37
|
+
*/
|
|
38
|
+
protected findUndispatchedSteps(before: Date, limit: number): Promise<Array<{
|
|
39
|
+
runId: string;
|
|
40
|
+
stepId: string;
|
|
41
|
+
}>>;
|
|
32
42
|
protected findStalledRunIds(before: Date, limit: number): Promise<string[]>;
|
|
33
43
|
listRuns(options?: {
|
|
34
44
|
workflowName?: string;
|
|
@@ -199,6 +199,29 @@ export class InMemoryWorkflowService extends PikkuWorkflowService {
|
|
|
199
199
|
this.stepHistory.set(runId, history);
|
|
200
200
|
return newStep;
|
|
201
201
|
}
|
|
202
|
+
/**
|
|
203
|
+
* Safe to opt in despite the pass-through `withStepLock`: this service wires
|
|
204
|
+
* no queues and runs every step inline in one process, so the relay's
|
|
205
|
+
* redundant dispatch has no second holder to race with. See
|
|
206
|
+
* knowledge: decisions/internals/the-in-memory-workflow-service-is-inline-only-and-single-process.md
|
|
207
|
+
*/
|
|
208
|
+
async findUndispatchedSteps(before, limit) {
|
|
209
|
+
const undispatched = [];
|
|
210
|
+
for (const [runId, run] of this.runs) {
|
|
211
|
+
if (run.status !== 'running')
|
|
212
|
+
continue;
|
|
213
|
+
for (const step of this.stepHistory.get(runId) ?? []) {
|
|
214
|
+
if (step.status !== 'pending')
|
|
215
|
+
continue;
|
|
216
|
+
if (step.updatedAt >= before)
|
|
217
|
+
continue;
|
|
218
|
+
undispatched.push({ runId, stepId: step.stepId });
|
|
219
|
+
if (undispatched.length >= limit)
|
|
220
|
+
return undispatched;
|
|
221
|
+
}
|
|
222
|
+
}
|
|
223
|
+
return undispatched;
|
|
224
|
+
}
|
|
202
225
|
async findStalledRunIds(before, limit) {
|
|
203
226
|
const stalled = [];
|
|
204
227
|
for (const [runId, run] of this.runs) {
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import { type HostResolver } from './safe-fetch.js';
|
|
2
|
+
/**
|
|
3
|
+
* Resolves a hostname through the platform resolver, returning every address it
|
|
4
|
+
* points at so `safeFetch` can reject a public name aimed at an internal one.
|
|
5
|
+
*
|
|
6
|
+
* This module is Node-only and is never imported by core itself — Workers has no
|
|
7
|
+
* DNS API, and a static `node:dns` import in the shared path would break that
|
|
8
|
+
* build.
|
|
9
|
+
*/
|
|
10
|
+
export declare const nodeHostResolver: HostResolver;
|
|
11
|
+
/** Installs {@link nodeHostResolver} as the default for every `safeFetch`. */
|
|
12
|
+
export declare const installNodeHostResolver: () => void;
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
import { lookup } from 'node:dns/promises';
|
|
2
|
+
import { setDefaultHostResolver } from './safe-fetch.js';
|
|
3
|
+
/**
|
|
4
|
+
* Resolves a hostname through the platform resolver, returning every address it
|
|
5
|
+
* points at so `safeFetch` can reject a public name aimed at an internal one.
|
|
6
|
+
*
|
|
7
|
+
* This module is Node-only and is never imported by core itself — Workers has no
|
|
8
|
+
* DNS API, and a static `node:dns` import in the shared path would break that
|
|
9
|
+
* build.
|
|
10
|
+
*/
|
|
11
|
+
export const nodeHostResolver = async (hostname) => {
|
|
12
|
+
const results = await lookup(hostname, { all: true, verbatim: true });
|
|
13
|
+
return results.map(({ address }) => address);
|
|
14
|
+
};
|
|
15
|
+
/** Installs {@link nodeHostResolver} as the default for every `safeFetch`. */
|
|
16
|
+
export const installNodeHostResolver = () => setDefaultHostResolver(nodeHostResolver);
|
|
@@ -4,6 +4,18 @@
|
|
|
4
4
|
* private IP.
|
|
5
5
|
*/
|
|
6
6
|
export declare function isPrivateHost(hostname: string): boolean;
|
|
7
|
+
/** Resolves a hostname to the IP addresses it points at. */
|
|
8
|
+
export type HostResolver = (hostname: string) => Promise<string[]>;
|
|
9
|
+
/**
|
|
10
|
+
* Installs the resolver {@link safeFetch} uses when a call passes no
|
|
11
|
+
* `resolveHost` of its own.
|
|
12
|
+
*
|
|
13
|
+
* Core cannot resolve DNS itself — Workers has no DNS API — so without a
|
|
14
|
+
* resolver the guard is literal-only and a public name pointing at
|
|
15
|
+
* `169.254.169.254` passes. Node runtimes install
|
|
16
|
+
* `nodeHostResolver` from `@pikku/core/node-host-resolver` at startup.
|
|
17
|
+
*/
|
|
18
|
+
export declare function setDefaultHostResolver(resolver: HostResolver | undefined): void;
|
|
7
19
|
export interface SafeFetchOptions {
|
|
8
20
|
/**
|
|
9
21
|
* When set, the host of every hop must appear in this allowlist. When omitted,
|
|
@@ -12,6 +24,12 @@ export interface SafeFetchOptions {
|
|
|
12
24
|
allowedHosts?: string[];
|
|
13
25
|
/** Maximum redirect hops to follow (each re-validated). Defaults to 3. */
|
|
14
26
|
maxRedirects?: number;
|
|
27
|
+
/**
|
|
28
|
+
* Resolves a hostname so a *public* name pointing at a private address is
|
|
29
|
+
* refused. Defaults to whatever {@link setDefaultHostResolver} installed;
|
|
30
|
+
* pass `null` to opt a call out of resolution entirely.
|
|
31
|
+
*/
|
|
32
|
+
resolveHost?: HostResolver | null;
|
|
15
33
|
}
|
|
16
34
|
export declare function assertFetchableUrl(url: string, options?: SafeFetchOptions): URL;
|
|
17
35
|
/**
|
package/dist/utils/safe-fetch.js
CHANGED
|
@@ -34,6 +34,120 @@ function parseIPv4Octets(host) {
|
|
|
34
34
|
}
|
|
35
35
|
return octets;
|
|
36
36
|
}
|
|
37
|
+
/**
|
|
38
|
+
* IPv4 blocks that must never be reachable from user-supplied URLs: private,
|
|
39
|
+
* loopback, link-local (cloud metadata), carrier-grade NAT (Alibaba's
|
|
40
|
+
* `100.100.100.200` metadata endpoint), IETF protocol assignments, benchmarking,
|
|
41
|
+
* 6to4 anycast, the documentation TEST-NETs, multicast and reserved space.
|
|
42
|
+
*/
|
|
43
|
+
const PRIVATE_IPV4_BLOCKS = [
|
|
44
|
+
['0.0.0.0', 8],
|
|
45
|
+
['10.0.0.0', 8],
|
|
46
|
+
['100.64.0.0', 10],
|
|
47
|
+
['127.0.0.0', 8],
|
|
48
|
+
['169.254.0.0', 16],
|
|
49
|
+
['172.16.0.0', 12],
|
|
50
|
+
['192.0.0.0', 24],
|
|
51
|
+
['192.0.2.0', 24],
|
|
52
|
+
['192.88.99.0', 24],
|
|
53
|
+
['192.168.0.0', 16],
|
|
54
|
+
['198.18.0.0', 15],
|
|
55
|
+
['198.51.100.0', 24],
|
|
56
|
+
['203.0.113.0', 24],
|
|
57
|
+
['224.0.0.0', 4],
|
|
58
|
+
['240.0.0.0', 4],
|
|
59
|
+
];
|
|
60
|
+
const toUint32 = (octets) => ((octets[0] << 24) | (octets[1] << 16) | (octets[2] << 8) | octets[3]) >>> 0;
|
|
61
|
+
const PRIVATE_IPV4_RANGES = PRIVATE_IPV4_BLOCKS.map(([base, prefix]) => {
|
|
62
|
+
const mask = prefix === 0 ? 0 : (0xffffffff << (32 - prefix)) >>> 0;
|
|
63
|
+
return [(toUint32(parseIPv4Octets(base)) & mask) >>> 0, mask];
|
|
64
|
+
});
|
|
65
|
+
const isPrivateIPv4 = (octets) => {
|
|
66
|
+
const addr = toUint32(octets);
|
|
67
|
+
return PRIVATE_IPV4_RANGES.some(([base, mask]) => (addr & mask) >>> 0 === base);
|
|
68
|
+
};
|
|
69
|
+
/**
|
|
70
|
+
* Expands an IPv6 literal into its eight 16-bit groups, handling `::` elision
|
|
71
|
+
* and a trailing dotted-quad. `null` when not a well-formed IPv6 literal.
|
|
72
|
+
*/
|
|
73
|
+
function parseIPv6Groups(host) {
|
|
74
|
+
const zoneless = host.split('%')[0];
|
|
75
|
+
const halves = zoneless.split('::');
|
|
76
|
+
if (halves.length > 2)
|
|
77
|
+
return null;
|
|
78
|
+
const parseSide = (side) => {
|
|
79
|
+
if (side === '')
|
|
80
|
+
return [];
|
|
81
|
+
const parts = side.split(':');
|
|
82
|
+
const groups = [];
|
|
83
|
+
for (let i = 0; i < parts.length; i++) {
|
|
84
|
+
const part = parts[i];
|
|
85
|
+
if (i === parts.length - 1 && part.includes('.')) {
|
|
86
|
+
const v4 = parseIPv4Octets(part);
|
|
87
|
+
if (!v4)
|
|
88
|
+
return null;
|
|
89
|
+
groups.push((v4[0] << 8) | v4[1], (v4[2] << 8) | v4[3]);
|
|
90
|
+
continue;
|
|
91
|
+
}
|
|
92
|
+
if (!/^[0-9a-f]{1,4}$/.test(part))
|
|
93
|
+
return null;
|
|
94
|
+
groups.push(parseInt(part, 16));
|
|
95
|
+
}
|
|
96
|
+
return groups;
|
|
97
|
+
};
|
|
98
|
+
const head = parseSide(halves[0]);
|
|
99
|
+
if (head === null)
|
|
100
|
+
return null;
|
|
101
|
+
if (halves.length === 1) {
|
|
102
|
+
return head.length === 8 ? head : null;
|
|
103
|
+
}
|
|
104
|
+
const tail = parseSide(halves[1]);
|
|
105
|
+
if (tail === null)
|
|
106
|
+
return null;
|
|
107
|
+
if (head.length + tail.length > 7)
|
|
108
|
+
return null;
|
|
109
|
+
return [...head, ...new Array(8 - head.length - tail.length).fill(0), ...tail];
|
|
110
|
+
}
|
|
111
|
+
const embeddedIPv4 = (hi, lo) => [
|
|
112
|
+
(hi >> 8) & 0xff,
|
|
113
|
+
hi & 0xff,
|
|
114
|
+
(lo >> 8) & 0xff,
|
|
115
|
+
lo & 0xff,
|
|
116
|
+
];
|
|
117
|
+
function isPrivateIPv6(groups) {
|
|
118
|
+
const [g0, g1, g2, g3, g4, g5, g6, g7] = groups;
|
|
119
|
+
if (groups.every((g) => g === 0))
|
|
120
|
+
return true; // :: unspecified
|
|
121
|
+
if (groups.slice(0, 7).every((g) => g === 0) && g7 === 1)
|
|
122
|
+
return true; // ::1
|
|
123
|
+
// IPv4-mapped ::ffff:0:0/96 and IPv4-compatible ::/96
|
|
124
|
+
if (g0 === 0 && g1 === 0 && g2 === 0 && g3 === 0 && g4 === 0) {
|
|
125
|
+
if (g5 === 0xffff || g5 === 0)
|
|
126
|
+
return isPrivateIPv4(embeddedIPv4(g6, g7));
|
|
127
|
+
}
|
|
128
|
+
// NAT64 well-known prefix 64:ff9b::/96
|
|
129
|
+
if (g0 === 0x64 &&
|
|
130
|
+
g1 === 0xff9b &&
|
|
131
|
+
g2 === 0 &&
|
|
132
|
+
g3 === 0 &&
|
|
133
|
+
g4 === 0 &&
|
|
134
|
+
g5 === 0)
|
|
135
|
+
return isPrivateIPv4(embeddedIPv4(g6, g7));
|
|
136
|
+
// 6to4 2002::/16 carries the IPv4 address in the next 32 bits
|
|
137
|
+
if (g0 === 0x2002)
|
|
138
|
+
return isPrivateIPv4(embeddedIPv4(g1, g2));
|
|
139
|
+
if (g0 === 0x100 && g1 === 0 && g2 === 0 && g3 === 0)
|
|
140
|
+
return true; // discard-only 100::/64
|
|
141
|
+
if ((g0 & 0xffc0) === 0xfe80)
|
|
142
|
+
return true; // link-local fe80::/10
|
|
143
|
+
if ((g0 & 0xfe00) === 0xfc00)
|
|
144
|
+
return true; // unique-local fc00::/7
|
|
145
|
+
if ((g0 & 0xffc0) === 0xfec0)
|
|
146
|
+
return true; // deprecated site-local fec0::/10
|
|
147
|
+
if ((g0 & 0xff00) === 0xff00)
|
|
148
|
+
return true; // multicast ff00::/8
|
|
149
|
+
return false;
|
|
150
|
+
}
|
|
37
151
|
/**
|
|
38
152
|
* Whether a hostname is an obvious internal target. Best-effort literal
|
|
39
153
|
* matching only: it cannot catch a public hostname that *resolves* to a
|
|
@@ -47,36 +161,56 @@ export function isPrivateHost(hostname) {
|
|
|
47
161
|
if (host === '' || host === 'localhost' || host.endsWith('.localhost'))
|
|
48
162
|
return true;
|
|
49
163
|
if (host.includes(':')) {
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
const mappedV4 = host.match(/^::ffff:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})$/);
|
|
53
|
-
if (mappedV4)
|
|
54
|
-
return isPrivateHost(mappedV4[1]);
|
|
55
|
-
const mappedHex = host.match(/^::ffff:([0-9a-f]{1,4}):([0-9a-f]{1,4})$/);
|
|
56
|
-
if (mappedHex) {
|
|
57
|
-
const hi = parseInt(mappedHex[1], 16);
|
|
58
|
-
const lo = parseInt(mappedHex[2], 16);
|
|
59
|
-
return isPrivateHost(`${(hi >> 8) & 0xff}.${hi & 0xff}.${(lo >> 8) & 0xff}.${lo & 0xff}`);
|
|
60
|
-
}
|
|
61
|
-
if (/^fe[89ab]/.test(host))
|
|
62
|
-
return true; // link-local fe80::/10
|
|
63
|
-
if (host.startsWith('fc') || host.startsWith('fd'))
|
|
64
|
-
return true; // unique-local fc00::/7
|
|
65
|
-
return false;
|
|
164
|
+
const groups = parseIPv6Groups(host);
|
|
165
|
+
return groups ? isPrivateIPv6(groups) : false;
|
|
66
166
|
}
|
|
67
167
|
const v4 = parseIPv4Octets(host);
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
168
|
+
return v4 ? isPrivateIPv4(v4) : false;
|
|
169
|
+
}
|
|
170
|
+
let defaultHostResolver;
|
|
171
|
+
/**
|
|
172
|
+
* Installs the resolver {@link safeFetch} uses when a call passes no
|
|
173
|
+
* `resolveHost` of its own.
|
|
174
|
+
*
|
|
175
|
+
* Core cannot resolve DNS itself — Workers has no DNS API — so without a
|
|
176
|
+
* resolver the guard is literal-only and a public name pointing at
|
|
177
|
+
* `169.254.169.254` passes. Node runtimes install
|
|
178
|
+
* `nodeHostResolver` from `@pikku/core/node-host-resolver` at startup.
|
|
179
|
+
*/
|
|
180
|
+
export function setDefaultHostResolver(resolver) {
|
|
181
|
+
defaultHostResolver = resolver;
|
|
182
|
+
}
|
|
183
|
+
/** Whether a hostname is already an IP literal, which the sync check covers. */
|
|
184
|
+
function isIpLiteral(hostname) {
|
|
185
|
+
const host = hostname.replace(/^\[|\]$/g, '').replace(/\.$/, '');
|
|
186
|
+
if (host.includes(':'))
|
|
187
|
+
return parseIPv6Groups(host.toLowerCase()) !== null;
|
|
188
|
+
return parseIPv4Octets(host) !== null;
|
|
189
|
+
}
|
|
190
|
+
/**
|
|
191
|
+
* Rejects a hostname that resolves to an internal address.
|
|
192
|
+
*
|
|
193
|
+
* Resolution happens once per hop and the connection is not pinned to the
|
|
194
|
+
* address checked, so a rebind between this check and the socket connecting is
|
|
195
|
+
* still possible; catching that needs a runtime-level connect hook.
|
|
196
|
+
*/
|
|
197
|
+
async function assertResolvedHostAllowed(hostname, options) {
|
|
198
|
+
if (options.allowedHosts)
|
|
199
|
+
return;
|
|
200
|
+
const resolver = options.resolveHost === null
|
|
201
|
+
? undefined
|
|
202
|
+
: (options.resolveHost ?? defaultHostResolver);
|
|
203
|
+
if (!resolver || isIpLiteral(hostname))
|
|
204
|
+
return;
|
|
205
|
+
const addresses = await resolver(hostname);
|
|
206
|
+
if (addresses.length === 0) {
|
|
207
|
+
throw new Error(`Refusing to fetch: '${hostname}' resolved to no addresses`);
|
|
208
|
+
}
|
|
209
|
+
for (const address of addresses) {
|
|
210
|
+
if (isPrivateHost(address)) {
|
|
211
|
+
throw new Error(`Refusing to fetch from a private/internal host: '${hostname}' resolves to ${address}`);
|
|
212
|
+
}
|
|
78
213
|
}
|
|
79
|
-
return false;
|
|
80
214
|
}
|
|
81
215
|
export function assertFetchableUrl(url, options = {}) {
|
|
82
216
|
const parsed = new URL(url);
|
|
@@ -126,7 +260,9 @@ function redirectInit(status, init) {
|
|
|
126
260
|
*/
|
|
127
261
|
export async function safeFetch(url, init = {}, options = {}) {
|
|
128
262
|
const maxRedirects = options.maxRedirects ?? 3;
|
|
129
|
-
|
|
263
|
+
const initial = assertFetchableUrl(url, options);
|
|
264
|
+
await assertResolvedHostAllowed(initial.hostname, options);
|
|
265
|
+
let currentUrl = initial.toString();
|
|
130
266
|
let currentInit = init;
|
|
131
267
|
for (let hop = 0;; hop++) {
|
|
132
268
|
const response = await fetch(currentUrl, {
|
|
@@ -140,7 +276,9 @@ export async function safeFetch(url, init = {}, options = {}) {
|
|
|
140
276
|
if (!location || hop >= maxRedirects) {
|
|
141
277
|
return response;
|
|
142
278
|
}
|
|
143
|
-
const
|
|
279
|
+
const next = assertFetchableUrl(new URL(location, currentUrl).toString(), options);
|
|
280
|
+
await assertResolvedHostAllowed(next.hostname, options);
|
|
281
|
+
const nextUrl = next.toString();
|
|
144
282
|
await response.body?.cancel();
|
|
145
283
|
let nextInit = redirectInit(response.status, currentInit);
|
|
146
284
|
if (new URL(nextUrl).origin !== new URL(currentUrl).origin) {
|
|
@@ -14,19 +14,42 @@ const bridgeMiddlewareSession = async (wire) => {
|
|
|
14
14
|
wire.session = session;
|
|
15
15
|
}
|
|
16
16
|
};
|
|
17
|
+
/**
|
|
18
|
+
* Metadata the inspector recorded for the function the gateway was wired with.
|
|
19
|
+
* The bootstrap loads every meta file before any wiring file, so this is
|
|
20
|
+
* already populated by the time a gateway wires itself. A gateway wired by
|
|
21
|
+
* hand rather than through codegen has no entry, and falls back to the
|
|
22
|
+
* sessionless default below.
|
|
23
|
+
*/
|
|
24
|
+
const declaredHandlerMeta = (config) => {
|
|
25
|
+
const declaredFuncId = pikkuState(null, 'gateway', 'meta')[config.name]
|
|
26
|
+
?.pikkuFuncId;
|
|
27
|
+
return declaredFuncId
|
|
28
|
+
? pikkuState(null, 'function', 'meta')[declaredFuncId]
|
|
29
|
+
: undefined;
|
|
30
|
+
};
|
|
17
31
|
// knowledge: decisions/security/gateway-handlers-run-through-the-function-runner-gate.md
|
|
18
32
|
const registerGatewayHandler = (config) => {
|
|
19
33
|
const funcId = gatewayHandlerFuncId(config.name);
|
|
20
34
|
const funcMeta = pikkuState(null, 'function', 'meta');
|
|
35
|
+
const declared = declaredHandlerMeta(config);
|
|
21
36
|
funcMeta[funcId] = {
|
|
37
|
+
...declared,
|
|
22
38
|
pikkuFuncId: funcId,
|
|
23
|
-
inputSchemaName: null,
|
|
24
|
-
outputSchemaName: null,
|
|
25
|
-
sessionless: true,
|
|
39
|
+
inputSchemaName: declared?.inputSchemaName ?? null,
|
|
40
|
+
outputSchemaName: declared?.outputSchemaName ?? null,
|
|
41
|
+
sessionless: declared?.sessionless ?? true,
|
|
26
42
|
};
|
|
27
43
|
addFunction(funcId, config.func);
|
|
28
44
|
return funcId;
|
|
29
45
|
};
|
|
46
|
+
/**
|
|
47
|
+
* Tag middleware the inspector resolved for this gateway. It is keyed by
|
|
48
|
+
* gateway name rather than reachable from `config`, because `tags` is a
|
|
49
|
+
* compile-time input everywhere — nothing at runtime maps a tag to its
|
|
50
|
+
* middleware group.
|
|
51
|
+
*/
|
|
52
|
+
const gatewayInheritedMiddleware = (config) => pikkuState(null, 'gateway', 'meta')[config.name]?.middleware;
|
|
30
53
|
export const resolveGatewayAdapter = (config, services) => {
|
|
31
54
|
let resolved = resolvedAdapters.get(config);
|
|
32
55
|
if (!resolved) {
|
|
@@ -100,6 +123,7 @@ const wireWebhookGateway = (config) => {
|
|
|
100
123
|
const createWebhookPostHandler = (config) => {
|
|
101
124
|
const { name, middleware: userMiddleware } = config;
|
|
102
125
|
const handlerFuncId = registerGatewayHandler(config);
|
|
126
|
+
const inheritedMiddleware = gatewayInheritedMiddleware(config);
|
|
103
127
|
return async (services, data, wire) => {
|
|
104
128
|
const adapter = await resolveGatewayAdapter(config, services);
|
|
105
129
|
if (adapter.verifyWebhook) {
|
|
@@ -126,6 +150,7 @@ const createWebhookPostHandler = (config) => {
|
|
|
126
150
|
singletonServices: services,
|
|
127
151
|
data: () => parsed,
|
|
128
152
|
auth: config.auth,
|
|
153
|
+
inheritedMiddleware,
|
|
129
154
|
wire: wire,
|
|
130
155
|
});
|
|
131
156
|
};
|
|
@@ -191,6 +216,7 @@ const wireWebsocketGateway = (config) => {
|
|
|
191
216
|
};
|
|
192
217
|
const userMiddleware = config.middleware;
|
|
193
218
|
const handlerFuncId = registerGatewayHandler(config);
|
|
219
|
+
const inheritedMiddleware = gatewayInheritedMiddleware(config);
|
|
194
220
|
addFunction(connectFuncId, {
|
|
195
221
|
auth: false,
|
|
196
222
|
func: async (services, _data, wire) => {
|
|
@@ -227,6 +253,7 @@ const wireWebsocketGateway = (config) => {
|
|
|
227
253
|
singletonServices: services,
|
|
228
254
|
data: () => parsed,
|
|
229
255
|
auth: config.auth,
|
|
256
|
+
inheritedMiddleware,
|
|
230
257
|
wire: wire,
|
|
231
258
|
});
|
|
232
259
|
};
|
|
@@ -256,6 +283,7 @@ const wireListenerGateway = (config) => {
|
|
|
256
283
|
export const createListenerMessageHandler = (name, config, singletonServices) => {
|
|
257
284
|
const userMiddleware = config.middleware;
|
|
258
285
|
const handlerFuncId = registerGatewayHandler(config);
|
|
286
|
+
const inheritedMiddleware = gatewayInheritedMiddleware(config);
|
|
259
287
|
return async (rawData) => {
|
|
260
288
|
const adapter = await resolveGatewayAdapter(config, singletonServices);
|
|
261
289
|
const parsed = adapter.parse(rawData);
|
|
@@ -275,6 +303,7 @@ export const createListenerMessageHandler = (name, config, singletonServices) =>
|
|
|
275
303
|
singletonServices,
|
|
276
304
|
data: () => parsed,
|
|
277
305
|
auth: config.auth,
|
|
306
|
+
inheritedMiddleware,
|
|
278
307
|
wire,
|
|
279
308
|
});
|
|
280
309
|
};
|
|
@@ -35,6 +35,7 @@ export function validateAndBuildSecretDefinitionsMeta(definitions, schemaLookup)
|
|
|
35
35
|
oauth2: def.oauth2,
|
|
36
36
|
rotationPeriod: def.rotationPeriod,
|
|
37
37
|
docsUrl: def.docsUrl,
|
|
38
|
+
allowedHosts: def.allowedHosts,
|
|
38
39
|
sourceFile: def.sourceFile,
|
|
39
40
|
};
|
|
40
41
|
}
|
|
@@ -51,6 +52,7 @@ export function validateAndBuildSecretDefinitionsMeta(definitions, schemaLookup)
|
|
|
51
52
|
oauth2: def.oauth2,
|
|
52
53
|
rotationPeriod: def.rotationPeriod,
|
|
53
54
|
docsUrl: def.docsUrl,
|
|
55
|
+
allowedHosts: def.allowedHosts,
|
|
54
56
|
sourceFile: def.sourceFile,
|
|
55
57
|
};
|
|
56
58
|
}
|
|
@@ -203,6 +203,58 @@ export declare abstract class PikkuWorkflowService implements WorkflowService {
|
|
|
203
203
|
}): Promise<{
|
|
204
204
|
resumed: string[];
|
|
205
205
|
}>;
|
|
206
|
+
/**
|
|
207
|
+
* Runs holding a step that has sat `pending` since before `before`, paired
|
|
208
|
+
* with the step that flagged them.
|
|
209
|
+
*
|
|
210
|
+
* Returns nothing by default so a store that cannot express the query keeps
|
|
211
|
+
* working unchanged — and, because it does not opt in, gains no re-dispatches
|
|
212
|
+
* either. A store must have an atomic `withStepLock` before overriding this,
|
|
213
|
+
* or no concurrency for one to exclude: the relay makes duplicate dispatch
|
|
214
|
+
* routine, and the claim in `executeWorkflowStepInner` is what keeps a
|
|
215
|
+
* duplicate from becoming a second execution. `kysely-postgres` and
|
|
216
|
+
* `kysely-mysql` qualify on the lock, `in-memory` on being inline and
|
|
217
|
+
* single-process; `mongodb` and `kysely-sqlite` qualify on neither.
|
|
218
|
+
*/
|
|
219
|
+
protected findUndispatchedSteps(_before: Date, _limit: number): Promise<Array<{
|
|
220
|
+
runId: string;
|
|
221
|
+
stepId: string;
|
|
222
|
+
}>>;
|
|
223
|
+
/**
|
|
224
|
+
* Re-drive steps whose dispatch was lost, and report which runs were nudged.
|
|
225
|
+
*
|
|
226
|
+
* Arming a step is two writes to two systems: the step row lands `pending`,
|
|
227
|
+
* then a queue or scheduler job is published. Nothing spans both, so a crash
|
|
228
|
+
* in between leaves a durable row that nothing will ever pick up — the run
|
|
229
|
+
* neither finishes nor fails. (Seen on a `workflow.sleep()`: a deploy restart
|
|
230
|
+
* landed between the sleep step's insert and its timer.)
|
|
231
|
+
*
|
|
232
|
+
* The row is the outbox record and this is the relay. Age is the only signal
|
|
233
|
+
* available — a step `pending` because its dispatch was lost is
|
|
234
|
+
* indistinguishable from one whose job is merely still queued — so a step
|
|
235
|
+
* past `undispatchedAfterMs` is re-dispatched regardless, and correctness
|
|
236
|
+
* rests on the claim rather than on the guess being right. A redundant
|
|
237
|
+
* dispatch costs one queue message: the loser reads `running` and returns
|
|
238
|
+
* without invoking anything.
|
|
239
|
+
*
|
|
240
|
+
* Re-dispatches back off per step (doubling from 30s, capped at 10m) so a
|
|
241
|
+
* genuine queue backlog is not amplified by a tick that keeps firing at the
|
|
242
|
+
* steps the backlog is already delaying. The backoff is per process and
|
|
243
|
+
* advisory — losing it on restart costs extra dispatches, never correctness.
|
|
244
|
+
*
|
|
245
|
+
* This is not self-starting. Call it from a scheduled task; ~30s suits a
|
|
246
|
+
* queue whose `pending`→`running` latency is well under that.
|
|
247
|
+
*/
|
|
248
|
+
relayUndispatchedSteps(options?: {
|
|
249
|
+
undispatchedAfterMs?: number;
|
|
250
|
+
limit?: number;
|
|
251
|
+
}): Promise<{
|
|
252
|
+
redispatched: string[];
|
|
253
|
+
}>;
|
|
254
|
+
/** Advisory, per-process record of when a run may next be re-dispatched. */
|
|
255
|
+
private readonly redispatchBackoff;
|
|
256
|
+
private readonly redispatchDelays;
|
|
257
|
+
private noteRedispatch;
|
|
206
258
|
protected resolveStepJobOptions(stepOptions?: WorkflowStepOptions): JobOptions;
|
|
207
259
|
queueStepWorker(runId: string, stepName: string, rpcName: string, data: any, stepOptions?: WorkflowStepOptions, fromStepName?: string): Promise<void>;
|
|
208
260
|
executeWorkflowSleepCompleted(runId: string, stepId: string): Promise<void>;
|