hazo_collect 0.2.8 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGE_LOG.md +69 -0
- package/README.md +70 -3
- package/config/hazo_collect_config.ini.sample +44 -0
- package/db_setup_postgres.sql +189 -0
- package/db_setup_sqlite.sql +87 -0
- package/dist/ddl/postgres.sql +141 -6
- package/dist/ddl/sqlite.sql +23 -6
- package/dist/server/index.d.ts +108 -1
- package/dist/server/index.js +283 -26
- package/package.json +12 -9
package/CHANGE_LOG.md
CHANGED
|
@@ -1,5 +1,74 @@
|
|
|
1
1
|
# hazo_collect — Change Log
|
|
2
2
|
|
|
3
|
+
## 0.3.0 — 2026-11-21 (faraday data-foundation ticket 05: shared per-host rate limit)
|
|
4
|
+
|
|
5
|
+
### ⚠️ DDL required
|
|
6
|
+
|
|
7
|
+
New table `hazo_collect_rate_limits` (one token-bucket row per host) plus two
|
|
8
|
+
new Postgres RPC functions, `hazo_collect_take_token_rpc` and
|
|
9
|
+
`hazo_collect_apply_backoff_rpc`, and one new column on
|
|
10
|
+
`hazo_collect_plugin_health` (`last_rate_limit_wait_ms`, default 0).
|
|
11
|
+
Re-run `db_setup_postgres.sql` / `db_setup_sqlite.sql` (idempotent) or apply
|
|
12
|
+
`src/ddl/postgres.sql` / `src/ddl/sqlite.sql` directly. SQLite has no
|
|
13
|
+
RPC/plpgsql, so its atomic "take a token" step is an optimistic
|
|
14
|
+
compare-and-swap retry loop over the same table instead of a stored
|
|
15
|
+
function (see `src/server/rate-limit.ts`, `createSqliteRateLimitStore`) —
|
|
16
|
+
same technique hazo_jobs 0.25.0 used for its SQLite group-completion CAS.
|
|
17
|
+
PostgREST consumers must `GRANT EXECUTE` on both new functions to the role
|
|
18
|
+
PostgREST connects as — see README's "PostgREST setup" note.
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- **Shared, DB-backed per-host token-bucket rate limiting inside `ctx.http`.**
|
|
23
|
+
Consumers configure `requestsPerSecond` + `burst` per host via
|
|
24
|
+
`createManager({ rateLimits: { 'api.example.com': { requestsPerSecond: 5,
|
|
25
|
+
burst: 10 } } })`. A host absent from that map keeps today's unthrottled
|
|
26
|
+
behaviour (no limiting), so this is strictly additive/opt-in. The bucket
|
|
27
|
+
state lives in the database (`hazo_collect_rate_limits`) and is shared by
|
|
28
|
+
every worker/process collecting through it — not just this process's view
|
|
29
|
+
of the limit.
|
|
30
|
+
- **Postgres**: taking a token is one atomic statement,
|
|
31
|
+
`hazo_collect_take_token_rpc` — a single `UPDATE ... FROM (SELECT ...
|
|
32
|
+
FOR UPDATE) ... RETURNING`, so no lock is ever held across requests
|
|
33
|
+
(safe to call via PostgREST's `POST /rpc/hazo_collect_take_token_rpc`,
|
|
34
|
+
not just a raw SQL connection).
|
|
35
|
+
- **SQLite**: the same refill-on-read token-bucket math
|
|
36
|
+
(`computeTokenBucket` in `src/server/rate-limit.ts`) runs in an
|
|
37
|
+
optimistic CAS retry loop over the raw adapter, since SQLite has no
|
|
38
|
+
RPC/plpgsql equivalent.
|
|
39
|
+
- **429 + `Retry-After`** on a rate-limited host trigger backoff: the
|
|
40
|
+
shared bucket is drained and `backoff_until` is raised (monotonically)
|
|
41
|
+
via `hazo_collect_apply_backoff_rpc` / the SQLite equivalent, so every
|
|
42
|
+
caller sharing that host's bucket backs off — not just the process that
|
|
43
|
+
saw the 429.
|
|
44
|
+
- **Collector health** (`hazo_collect_plugin_health.last_rate_limit_wait_ms`)
|
|
45
|
+
records how long the most recent run waited on the limit.
|
|
46
|
+
- **`ManagerOptions.fetch`** — an injected fetch implementation used by
|
|
47
|
+
`ctx.http` in every run the manager dispatches. Lets consumers (and this
|
|
48
|
+
package's own tests) point collectors at a fake host while per-host rate
|
|
49
|
+
limiting still applies, without touching real network code.
|
|
50
|
+
`ManagerOptions.rateLimits` is the per-host config described above; both
|
|
51
|
+
are threaded through a new `createNodeRuntime({ fetch, rateLimits })`
|
|
52
|
+
factory (`nodeRuntime` stays the pre-0.3.0 default instance — no
|
|
53
|
+
behaviour change when neither option is passed).
|
|
54
|
+
- New `hazo_collect/server` exports: `createNodeRuntime`,
|
|
55
|
+
`createRateLimitManager`, `createRateLimitStore`, `computeTokenBucket`,
|
|
56
|
+
`parseRetryAfterMs`, and types `HostRateLimitConfig`, `RateLimitManager`,
|
|
57
|
+
`RateLimitStore`, `TakeTokenResult`, `FetchFn`, `WaitTracker`,
|
|
58
|
+
`NodeRuntimeOptions`.
|
|
59
|
+
- `config/hazo_collect_config.ini.sample` documents the per-host rate-limit
|
|
60
|
+
config shape (`[rate_limit:<host>]`) consumers should map into
|
|
61
|
+
`ManagerOptions.rateLimits` — this package has no ini-loading code of its
|
|
62
|
+
own, so the section is documentation, not a parsed config file.
|
|
63
|
+
|
|
64
|
+
### Scope notes
|
|
65
|
+
|
|
66
|
+
- `RuntimeAdapter.run()`'s return type gained a required `rateLimitWaitMs`
|
|
67
|
+
field (present on both the `ok: true` and `ok: false` branches). This is
|
|
68
|
+
additive for any existing `RuntimeAdapter` implementation that already
|
|
69
|
+
destructures `{ ok, outcome }` / `{ ok, error }` — it simply won't read
|
|
70
|
+
the new field.
|
|
71
|
+
|
|
3
72
|
## 0.2.8 — 2026-09-19
|
|
4
73
|
|
|
5
74
|
### Added
|
package/README.md
CHANGED
|
@@ -75,12 +75,17 @@ const result = await manager.runNow({ plugin: 'my_source' });
|
|
|
75
75
|
|
|
76
76
|
| Export | Description |
|
|
77
77
|
|---|---|
|
|
78
|
-
| `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist |
|
|
79
|
-
| `nodeRuntime` |
|
|
78
|
+
| `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist. Accepts `fetch` + `rateLimits` (see "Per-host rate limiting" above) |
|
|
79
|
+
| `nodeRuntime` | Default execution adapter for in-process Node.js collectors (no fetch override, no rate limits) |
|
|
80
|
+
| `createNodeRuntime({ fetch, rateLimits })` | Build a `RuntimeAdapter` with an injected fetch and/or per-host rate limits |
|
|
80
81
|
| `createInMemoryRegistry()` | In-memory registry backed by the SDK Map |
|
|
81
82
|
| `persistRegistry(adapter, manifests)` | Upsert manifests into `hazo_collect_plugin_registry`; quarantines invalid entries |
|
|
82
83
|
| `discover(opts)` | Scan directories for `manifest.json` files and return parsed entries |
|
|
83
84
|
| `createWriteAdapter(adapter, runId)` | Create a `ctx.write` function bound to a specific run |
|
|
85
|
+
| `createRateLimitManager(opts)` | Build the `ctx.http` rate-limiting wrapper directly (advanced — `createManager`'s `rateLimits`/`fetch` cover the common case) |
|
|
86
|
+
| `createRateLimitStore(adapter)` | Pick the dialect-appropriate (`pg` / `postgrest` / `sqlite`) atomic token-bucket store for an adapter |
|
|
87
|
+
| `computeTokenBucket(state, cfg, cost, nowMs)` | Pure refill-on-read token-bucket math, shared by the SQLite CAS loop |
|
|
88
|
+
| `parseRetryAfterMs(header, nowMs)` | Parse a `Retry-After` header (seconds or HTTP-date) into a millisecond delay |
|
|
84
89
|
|
|
85
90
|
### `hazo_collect/sdk`
|
|
86
91
|
|
|
@@ -109,8 +114,70 @@ Run migrations from `hazo_collect/ddl/sqlite.sql` (SQLite) or `hazo_collect/ddl/
|
|
|
109
114
|
|
|
110
115
|
- `hazo_collect_plugin_registry` — discovered manifest catalogue with validity + quarantine info
|
|
111
116
|
- `hazo_collect_plugin_runs` — per-run scorecard (status, records, contract_version, watermark)
|
|
112
|
-
- `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status)
|
|
117
|
+
- `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status, last rate-limit wait)
|
|
113
118
|
- `hazo_collect_landing` — idempotent landing zone for collector output rows
|
|
119
|
+
- `hazo_collect_rate_limits` — shared per-host token-bucket state (0.3.0+; see below)
|
|
120
|
+
|
|
121
|
+
## Per-host rate limiting (`ctx.http`)
|
|
122
|
+
|
|
123
|
+
Every collector's `ctx.http` can enforce a shared, database-backed per-host
|
|
124
|
+
token bucket. Configure `requestsPerSecond` + `burst` per host on the
|
|
125
|
+
manager; a host with no entry keeps today's unthrottled behaviour:
|
|
126
|
+
|
|
127
|
+
```ts
|
|
128
|
+
import { createManager } from 'hazo_collect/server';
|
|
129
|
+
|
|
130
|
+
const manager = createManager({
|
|
131
|
+
adapter,
|
|
132
|
+
registry,
|
|
133
|
+
rateLimits: {
|
|
134
|
+
'api.example.com': { requestsPerSecond: 5, burst: 10 },
|
|
135
|
+
},
|
|
136
|
+
});
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Because the bucket lives in `hazo_collect_rate_limits` (not in process
|
|
140
|
+
memory), the limit is shared across every worker/process collecting through
|
|
141
|
+
the same database — not just this process's view of it. A `429` response
|
|
142
|
+
(honouring `Retry-After`, in seconds or as an HTTP-date) drains the shared
|
|
143
|
+
bucket and raises its backoff window, so every caller sharing that host
|
|
144
|
+
backs off, not just the one that saw the 429. `hazo_collect_plugin_health`
|
|
145
|
+
records how long the most recent run waited on the limit
|
|
146
|
+
(`last_rate_limit_wait_ms`).
|
|
147
|
+
|
|
148
|
+
### Injected fetch
|
|
149
|
+
|
|
150
|
+
`createManager({ fetch })` lets you point `ctx.http` at a different fetch
|
|
151
|
+
implementation for every run the manager dispatches — the rate limiter still
|
|
152
|
+
applies on top of it. This is how both this package's own tests and a
|
|
153
|
+
consumer's tests can exercise `ctx.http` against a fake host without any
|
|
154
|
+
real network traffic:
|
|
155
|
+
|
|
156
|
+
```ts
|
|
157
|
+
const manager = createManager({
|
|
158
|
+
adapter,
|
|
159
|
+
registry,
|
|
160
|
+
fetch: fakeFetch, // e.g. a jest.fn() resolving a canned Response
|
|
161
|
+
rateLimits: { 'fake-host.test': { requestsPerSecond: 10, burst: 10 } },
|
|
162
|
+
});
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
`fetch` and `rateLimits` are ignored if you also pass your own `runtime`
|
|
166
|
+
(that runtime owns its own fetch wiring) — use `createNodeRuntime({ fetch,
|
|
167
|
+
rateLimits })` directly in that case.
|
|
168
|
+
|
|
169
|
+
### PostgREST setup
|
|
170
|
+
|
|
171
|
+
If you run `createHazoConnect({ type: 'postgrest', ... })`, the rate limiter
|
|
172
|
+
calls the two new functions via `POST /rpc/<fn>`. Grant execute to the role
|
|
173
|
+
PostgREST connects as:
|
|
174
|
+
|
|
175
|
+
```sql
|
|
176
|
+
GRANT EXECUTE ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) TO api_user;
|
|
177
|
+
GRANT EXECUTE ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) TO api_user;
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
(substitute the role PostgREST actually connects as for `api_user`).
|
|
114
181
|
|
|
115
182
|
## Tailwind v4 (`@source` required)
|
|
116
183
|
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
; hazo_collect configuration
|
|
2
|
+
; Copy to hazo_collect_config.ini and fill in values for your environment.
|
|
3
|
+
|
|
4
|
+
; [env]
|
|
5
|
+
; Uncomment to hard-code the environment name (overrides HAZO_ENV / NODE_ENV).
|
|
6
|
+
; env = development
|
|
7
|
+
|
|
8
|
+
; [secrets]
|
|
9
|
+
; Prefix for environment variables that hold plugin secrets.
|
|
10
|
+
; A manifest declaring secrets = ["API_KEY"] will read env var ${env_prefix}_API_KEY.
|
|
11
|
+
; env_prefix = HAZO_COLLECT_SECRET
|
|
12
|
+
|
|
13
|
+
; [plugins]
|
|
14
|
+
; Directory to scan for plugin manifests (future folder-scan discovery provider).
|
|
15
|
+
; Each sub-directory should contain a manifest.json.
|
|
16
|
+
; dir = ./plugins
|
|
17
|
+
|
|
18
|
+
; [general]
|
|
19
|
+
; Example setting
|
|
20
|
+
; some_key = some_value
|
|
21
|
+
|
|
22
|
+
; [rate_limit:<host>]
|
|
23
|
+
; Per-host token-bucket rate limit enforced inside ctx.http (0.3.0+). One
|
|
24
|
+
; section per host you want throttled — a host with no section here keeps
|
|
25
|
+
; today's unthrottled behaviour. The bucket itself lives in the database
|
|
26
|
+
; (hazo_collect_rate_limits), shared by every worker/process collecting
|
|
27
|
+
; through it, so this config only needs to be loaded wherever createManager()
|
|
28
|
+
; is called — it does not need to agree across processes.
|
|
29
|
+
;
|
|
30
|
+
; This file isn't parsed by hazo_collect itself (there's no ini-loading code
|
|
31
|
+
; in this package yet) — map whatever ini/env/JSON config your app already
|
|
32
|
+
; loads into the shape createManager() expects:
|
|
33
|
+
;
|
|
34
|
+
; createManager({
|
|
35
|
+
; adapter,
|
|
36
|
+
; registry,
|
|
37
|
+
; rateLimits: {
|
|
38
|
+
; 'api.example.com': { requestsPerSecond: 5, burst: 10 },
|
|
39
|
+
; },
|
|
40
|
+
; });
|
|
41
|
+
;
|
|
42
|
+
; [rate_limit:api.example.com]
|
|
43
|
+
; requests_per_second = 5
|
|
44
|
+
; burst = 10
|
|
@@ -0,0 +1,189 @@
|
|
|
1
|
+
-- hazo_collect — PostgreSQL DDL
|
|
2
|
+
-- Consumers MAY copy this verbatim or wrap with their own role grants.
|
|
3
|
+
-- Idempotent — safe to re-run.
|
|
4
|
+
--
|
|
5
|
+
-- Tables:
|
|
6
|
+
-- hazo_collect_plugin_runs — one row per plugin run (RunResult scorecard)
|
|
7
|
+
-- hazo_collect_plugin_health — latest health snapshot per plugin
|
|
8
|
+
-- hazo_collect_plugin_registry — discovered manifest catalogue
|
|
9
|
+
-- hazo_collect_watermarks — per-plugin incremental watermark
|
|
10
|
+
-- hazo_collect_landing — raw as-fetched payloads (idempotent on plugin+key)
|
|
11
|
+
-- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+)
|
|
12
|
+
|
|
13
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
|
|
14
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
15
|
+
plugin TEXT NOT NULL,
|
|
16
|
+
run_id TEXT NOT NULL UNIQUE,
|
|
17
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
18
|
+
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
19
|
+
completed_at TIMESTAMPTZ,
|
|
20
|
+
status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
|
|
21
|
+
records_fetched INT NOT NULL DEFAULT 0,
|
|
22
|
+
records_written INT NOT NULL DEFAULT 0,
|
|
23
|
+
watermark TEXT,
|
|
24
|
+
errors JSONB NOT NULL DEFAULT '[]'::jsonb,
|
|
25
|
+
metrics JSONB,
|
|
26
|
+
correlation_id TEXT,
|
|
27
|
+
env TEXT
|
|
28
|
+
);
|
|
29
|
+
|
|
30
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
|
|
31
|
+
ON hazo_collect_plugin_runs (plugin, started_at DESC);
|
|
32
|
+
|
|
33
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_run_id_idx
|
|
34
|
+
ON hazo_collect_plugin_runs (run_id);
|
|
35
|
+
|
|
36
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
|
|
37
|
+
plugin TEXT PRIMARY KEY,
|
|
38
|
+
last_success_at TIMESTAMPTZ,
|
|
39
|
+
last_run_at TIMESTAMPTZ,
|
|
40
|
+
last_status TEXT,
|
|
41
|
+
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
|
|
42
|
+
consecutive_failures INT NOT NULL DEFAULT 0,
|
|
43
|
+
last_rate_limit_wait_ms INT NOT NULL DEFAULT 0
|
|
44
|
+
);
|
|
45
|
+
|
|
46
|
+
-- 0.3.0: existing deployments — add the new column if upgrading in place.
|
|
47
|
+
ALTER TABLE hazo_collect_plugin_health
|
|
48
|
+
ADD COLUMN IF NOT EXISTS last_rate_limit_wait_ms INT NOT NULL DEFAULT 0;
|
|
49
|
+
|
|
50
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
|
|
51
|
+
name TEXT PRIMARY KEY,
|
|
52
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
53
|
+
version TEXT NOT NULL,
|
|
54
|
+
runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
|
|
55
|
+
manifest JSONB NOT NULL,
|
|
56
|
+
source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
|
|
57
|
+
discovered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
58
|
+
valid BOOLEAN NOT NULL DEFAULT TRUE,
|
|
59
|
+
quarantine_reason TEXT
|
|
60
|
+
);
|
|
61
|
+
|
|
62
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
|
|
63
|
+
plugin TEXT PRIMARY KEY,
|
|
64
|
+
watermark TEXT NOT NULL,
|
|
65
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
66
|
+
);
|
|
67
|
+
|
|
68
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_landing (
|
|
69
|
+
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
70
|
+
plugin TEXT NOT NULL,
|
|
71
|
+
run_id TEXT NOT NULL,
|
|
72
|
+
idempotency_key TEXT NOT NULL,
|
|
73
|
+
payload JSONB NOT NULL,
|
|
74
|
+
fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
75
|
+
window_since TEXT,
|
|
76
|
+
window_until TEXT,
|
|
77
|
+
CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
|
|
78
|
+
);
|
|
79
|
+
|
|
80
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
|
|
81
|
+
ON hazo_collect_landing (plugin, fetched_at DESC);
|
|
82
|
+
|
|
83
|
+
-- ===========================================================================
|
|
84
|
+
-- Shared per-host rate limiting (0.3.0+) — see src/ddl/postgres.sql for the
|
|
85
|
+
-- full design notes. Mirrored here per repo convention.
|
|
86
|
+
-- ===========================================================================
|
|
87
|
+
|
|
88
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
|
|
89
|
+
host TEXT PRIMARY KEY,
|
|
90
|
+
requests_per_second DOUBLE PRECISION NOT NULL,
|
|
91
|
+
burst INT NOT NULL,
|
|
92
|
+
tokens_available DOUBLE PRECISION NOT NULL,
|
|
93
|
+
last_refill_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
94
|
+
backoff_until TIMESTAMPTZ,
|
|
95
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
96
|
+
);
|
|
97
|
+
|
|
98
|
+
DROP FUNCTION IF EXISTS hazo_collect_take_token_rpc(text, double precision, int, int);
|
|
99
|
+
|
|
100
|
+
CREATE OR REPLACE FUNCTION hazo_collect_take_token_rpc(
|
|
101
|
+
p_host text,
|
|
102
|
+
p_requests_per_second double precision,
|
|
103
|
+
p_burst int,
|
|
104
|
+
p_cost int DEFAULT 1
|
|
105
|
+
) RETURNS TABLE(allowed boolean, wait_ms int)
|
|
106
|
+
LANGUAGE plpgsql
|
|
107
|
+
SECURITY DEFINER
|
|
108
|
+
AS $$
|
|
109
|
+
DECLARE
|
|
110
|
+
now_ts timestamptz := clock_timestamp();
|
|
111
|
+
BEGIN
|
|
112
|
+
INSERT INTO hazo_collect_rate_limits
|
|
113
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
|
|
114
|
+
VALUES (p_host, p_requests_per_second, p_burst, p_burst, now_ts, now_ts)
|
|
115
|
+
ON CONFLICT (host) DO NOTHING;
|
|
116
|
+
|
|
117
|
+
RETURN QUERY
|
|
118
|
+
UPDATE hazo_collect_rate_limits r
|
|
119
|
+
SET tokens_available = CASE
|
|
120
|
+
WHEN old.is_allowed THEN old.refilled - p_cost
|
|
121
|
+
ELSE old.refilled
|
|
122
|
+
END,
|
|
123
|
+
last_refill_at = now_ts,
|
|
124
|
+
requests_per_second = p_requests_per_second,
|
|
125
|
+
burst = p_burst,
|
|
126
|
+
updated_at = now_ts
|
|
127
|
+
FROM (
|
|
128
|
+
SELECT
|
|
129
|
+
host,
|
|
130
|
+
LEAST(
|
|
131
|
+
p_burst::double precision,
|
|
132
|
+
tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
|
|
133
|
+
) AS refilled,
|
|
134
|
+
backoff_until,
|
|
135
|
+
(backoff_until IS NULL OR backoff_until <= now_ts)
|
|
136
|
+
AND LEAST(
|
|
137
|
+
p_burst::double precision,
|
|
138
|
+
tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
|
|
139
|
+
) >= p_cost AS is_allowed
|
|
140
|
+
FROM hazo_collect_rate_limits
|
|
141
|
+
WHERE host = p_host
|
|
142
|
+
FOR UPDATE
|
|
143
|
+
) AS old
|
|
144
|
+
WHERE r.host = old.host
|
|
145
|
+
RETURNING
|
|
146
|
+
old.is_allowed AS allowed,
|
|
147
|
+
CASE
|
|
148
|
+
WHEN old.is_allowed THEN 0
|
|
149
|
+
WHEN old.backoff_until IS NOT NULL AND old.backoff_until > now_ts
|
|
150
|
+
THEN CEIL(EXTRACT(EPOCH FROM (old.backoff_until - now_ts)) * 1000)::int
|
|
151
|
+
WHEN p_requests_per_second > 0
|
|
152
|
+
THEN CEIL((p_cost - old.refilled) / p_requests_per_second * 1000)::int
|
|
153
|
+
ELSE 2147483647
|
|
154
|
+
END AS wait_ms;
|
|
155
|
+
END;
|
|
156
|
+
$$;
|
|
157
|
+
|
|
158
|
+
COMMENT ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) IS
|
|
159
|
+
'hazo_collect: atomically take a rate-limit token for a host (refill-on-read token bucket). '
|
|
160
|
+
'Callable via PostgREST POST /rpc/hazo_collect_take_token_rpc. GRANT EXECUTE to the PostgREST role.';
|
|
161
|
+
|
|
162
|
+
DROP FUNCTION IF EXISTS hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int);
|
|
163
|
+
|
|
164
|
+
CREATE OR REPLACE FUNCTION hazo_collect_apply_backoff_rpc(
|
|
165
|
+
p_host text,
|
|
166
|
+
p_backoff_until timestamptz,
|
|
167
|
+
p_requests_per_second double precision,
|
|
168
|
+
p_burst int
|
|
169
|
+
) RETURNS void
|
|
170
|
+
LANGUAGE plpgsql
|
|
171
|
+
SECURITY DEFINER
|
|
172
|
+
AS $$
|
|
173
|
+
BEGIN
|
|
174
|
+
INSERT INTO hazo_collect_rate_limits
|
|
175
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, backoff_until, updated_at)
|
|
176
|
+
VALUES (p_host, p_requests_per_second, p_burst, 0, clock_timestamp(), p_backoff_until, clock_timestamp())
|
|
177
|
+
ON CONFLICT (host) DO UPDATE
|
|
178
|
+
SET tokens_available = 0,
|
|
179
|
+
backoff_until = GREATEST(hazo_collect_rate_limits.backoff_until, EXCLUDED.backoff_until),
|
|
180
|
+
updated_at = clock_timestamp();
|
|
181
|
+
END;
|
|
182
|
+
$$;
|
|
183
|
+
|
|
184
|
+
COMMENT ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) IS
|
|
185
|
+
'hazo_collect: record a 429 + Retry-After for a host — drains the shared bucket and raises '
|
|
186
|
+
'backoff_until so every caller sharing this host backs off. Callable via PostgREST POST '
|
|
187
|
+
'/rpc/hazo_collect_apply_backoff_rpc. GRANT EXECUTE to the PostgREST role.';
|
|
188
|
+
|
|
189
|
+
NOTIFY pgrst, 'reload schema';
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
-- hazo_collect — SQLite DDL
|
|
2
|
+
-- Idempotent — safe to re-run.
|
|
3
|
+
--
|
|
4
|
+
-- SQLite companion to postgres.sql.
|
|
5
|
+
-- Type mapping: UUID→TEXT, BOOLEAN→INTEGER(0/1), TIMESTAMPTZ→TEXT, JSONB→TEXT.
|
|
6
|
+
--
|
|
7
|
+
-- Tables:
|
|
8
|
+
-- hazo_collect_plugin_runs
|
|
9
|
+
-- hazo_collect_plugin_health
|
|
10
|
+
-- hazo_collect_plugin_registry
|
|
11
|
+
-- hazo_collect_watermarks
|
|
12
|
+
-- hazo_collect_landing
|
|
13
|
+
-- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+);
|
|
14
|
+
-- see src/ddl/sqlite.sql for the CAS-loop design note.
|
|
15
|
+
|
|
16
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
|
|
17
|
+
id TEXT PRIMARY KEY,
|
|
18
|
+
plugin TEXT NOT NULL,
|
|
19
|
+
run_id TEXT NOT NULL UNIQUE,
|
|
20
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
21
|
+
started_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
22
|
+
completed_at TEXT,
|
|
23
|
+
status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
|
|
24
|
+
records_fetched INTEGER NOT NULL DEFAULT 0,
|
|
25
|
+
records_written INTEGER NOT NULL DEFAULT 0,
|
|
26
|
+
watermark TEXT,
|
|
27
|
+
errors TEXT NOT NULL DEFAULT '[]',
|
|
28
|
+
metrics TEXT,
|
|
29
|
+
correlation_id TEXT,
|
|
30
|
+
env TEXT
|
|
31
|
+
);
|
|
32
|
+
|
|
33
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
|
|
34
|
+
ON hazo_collect_plugin_runs (plugin, started_at DESC);
|
|
35
|
+
|
|
36
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
|
|
37
|
+
plugin TEXT PRIMARY KEY,
|
|
38
|
+
last_success_at TEXT,
|
|
39
|
+
last_run_at TEXT,
|
|
40
|
+
last_status TEXT,
|
|
41
|
+
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
|
|
42
|
+
consecutive_failures INTEGER NOT NULL DEFAULT 0,
|
|
43
|
+
last_rate_limit_wait_ms INTEGER NOT NULL DEFAULT 0
|
|
44
|
+
);
|
|
45
|
+
|
|
46
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
|
|
47
|
+
name TEXT PRIMARY KEY,
|
|
48
|
+
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
49
|
+
version TEXT NOT NULL,
|
|
50
|
+
runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
|
|
51
|
+
manifest TEXT NOT NULL,
|
|
52
|
+
source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
|
|
53
|
+
discovered_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
54
|
+
valid INTEGER NOT NULL DEFAULT 1,
|
|
55
|
+
quarantine_reason TEXT
|
|
56
|
+
);
|
|
57
|
+
|
|
58
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
|
|
59
|
+
plugin TEXT PRIMARY KEY,
|
|
60
|
+
watermark TEXT NOT NULL,
|
|
61
|
+
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
|
62
|
+
);
|
|
63
|
+
|
|
64
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_landing (
|
|
65
|
+
id TEXT PRIMARY KEY,
|
|
66
|
+
plugin TEXT NOT NULL,
|
|
67
|
+
run_id TEXT NOT NULL,
|
|
68
|
+
idempotency_key TEXT NOT NULL,
|
|
69
|
+
payload TEXT NOT NULL,
|
|
70
|
+
fetched_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
71
|
+
window_since TEXT,
|
|
72
|
+
window_until TEXT,
|
|
73
|
+
CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
|
|
74
|
+
);
|
|
75
|
+
|
|
76
|
+
CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
|
|
77
|
+
ON hazo_collect_landing (plugin, fetched_at DESC);
|
|
78
|
+
|
|
79
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
|
|
80
|
+
host TEXT PRIMARY KEY,
|
|
81
|
+
requests_per_second REAL NOT NULL,
|
|
82
|
+
burst INTEGER NOT NULL,
|
|
83
|
+
tokens_available REAL NOT NULL,
|
|
84
|
+
last_refill_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
85
|
+
backoff_until TEXT,
|
|
86
|
+
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
|
87
|
+
);
|
package/dist/ddl/postgres.sql
CHANGED
|
@@ -8,6 +8,7 @@
|
|
|
8
8
|
-- hazo_collect_plugin_registry — discovered manifest catalogue
|
|
9
9
|
-- hazo_collect_watermarks — per-plugin incremental watermark
|
|
10
10
|
-- hazo_collect_landing — raw as-fetched payloads (idempotent on plugin+key)
|
|
11
|
+
-- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+)
|
|
11
12
|
|
|
12
13
|
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
|
|
13
14
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
|
@@ -34,14 +35,19 @@ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_run_id_idx
|
|
|
34
35
|
ON hazo_collect_plugin_runs (run_id);
|
|
35
36
|
|
|
36
37
|
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
|
|
37
|
-
plugin
|
|
38
|
-
last_success_at
|
|
39
|
-
last_run_at
|
|
40
|
-
last_status
|
|
41
|
-
state
|
|
42
|
-
consecutive_failures
|
|
38
|
+
plugin TEXT PRIMARY KEY,
|
|
39
|
+
last_success_at TIMESTAMPTZ,
|
|
40
|
+
last_run_at TIMESTAMPTZ,
|
|
41
|
+
last_status TEXT,
|
|
42
|
+
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
|
|
43
|
+
consecutive_failures INT NOT NULL DEFAULT 0,
|
|
44
|
+
last_rate_limit_wait_ms INT NOT NULL DEFAULT 0
|
|
43
45
|
);
|
|
44
46
|
|
|
47
|
+
-- 0.3.0: existing deployments — add the new column if upgrading in place.
|
|
48
|
+
ALTER TABLE hazo_collect_plugin_health
|
|
49
|
+
ADD COLUMN IF NOT EXISTS last_rate_limit_wait_ms INT NOT NULL DEFAULT 0;
|
|
50
|
+
|
|
45
51
|
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
|
|
46
52
|
name TEXT PRIMARY KEY,
|
|
47
53
|
kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
|
|
@@ -75,4 +81,133 @@ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
|
|
|
75
81
|
CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
|
|
76
82
|
ON hazo_collect_landing (plugin, fetched_at DESC);
|
|
77
83
|
|
|
84
|
+
-- ===========================================================================
|
|
85
|
+
-- Shared per-host rate limiting (0.3.0+)
|
|
86
|
+
--
|
|
87
|
+
-- hazo_collect_rate_limits holds one token-bucket row per host, shared by
|
|
88
|
+
-- every worker/process that collects through this database — ctx.http
|
|
89
|
+
-- enforces it transparently for hosts configured via ManagerOptions.rateLimits
|
|
90
|
+
-- (see createNodeRuntime / createRateLimitManager in src/server/rate-limit.ts).
|
|
91
|
+
--
|
|
92
|
+
-- hazo_collect_take_token_rpc is the atomic "take a token" operation. It is a
|
|
93
|
+
-- SINGLE UPDATE statement (the UPDATE ... FROM (... FOR UPDATE) subquery below
|
|
94
|
+
-- is one statement, not a separate SELECT then UPDATE over two round trips) so
|
|
95
|
+
-- no lock is ever held across requests — safe to call from PostgREST via
|
|
96
|
+
-- POST /rpc/hazo_collect_take_token_rpc, and safe under concurrent callers.
|
|
97
|
+
--
|
|
98
|
+
-- hazo_collect_apply_backoff_rpc records a 429 + Retry-After: it drains the
|
|
99
|
+
-- bucket and raises backoff_until so every caller sharing this host's bucket
|
|
100
|
+
-- (any process, not just the one that saw the 429) backs off too.
|
|
101
|
+
--
|
|
102
|
+
-- Consumers using the PostgREST path must GRANT EXECUTE on both functions to
|
|
103
|
+
-- the role PostgREST connects as (see README's "PostgREST setup" note).
|
|
104
|
+
-- ===========================================================================
|
|
105
|
+
|
|
106
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
|
|
107
|
+
host TEXT PRIMARY KEY,
|
|
108
|
+
requests_per_second DOUBLE PRECISION NOT NULL,
|
|
109
|
+
burst INT NOT NULL,
|
|
110
|
+
tokens_available DOUBLE PRECISION NOT NULL,
|
|
111
|
+
last_refill_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
|
112
|
+
backoff_until TIMESTAMPTZ,
|
|
113
|
+
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
114
|
+
);
|
|
115
|
+
|
|
116
|
+
DROP FUNCTION IF EXISTS hazo_collect_take_token_rpc(text, double precision, int, int);
|
|
117
|
+
|
|
118
|
+
CREATE OR REPLACE FUNCTION hazo_collect_take_token_rpc(
|
|
119
|
+
p_host text,
|
|
120
|
+
p_requests_per_second double precision,
|
|
121
|
+
p_burst int,
|
|
122
|
+
p_cost int DEFAULT 1
|
|
123
|
+
) RETURNS TABLE(allowed boolean, wait_ms int)
|
|
124
|
+
LANGUAGE plpgsql
|
|
125
|
+
SECURITY DEFINER
|
|
126
|
+
AS $$
|
|
127
|
+
DECLARE
|
|
128
|
+
now_ts timestamptz := clock_timestamp();
|
|
129
|
+
BEGIN
|
|
130
|
+
-- First use: seed the bucket at full burst. Race-safe — a concurrent
|
|
131
|
+
-- racer's insert simply loses (ON CONFLICT DO NOTHING) and the UPDATE
|
|
132
|
+
-- below re-reads whichever row won.
|
|
133
|
+
INSERT INTO hazo_collect_rate_limits
|
|
134
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
|
|
135
|
+
VALUES (p_host, p_requests_per_second, p_burst, p_burst, now_ts, now_ts)
|
|
136
|
+
ON CONFLICT (host) DO NOTHING;
|
|
137
|
+
|
|
138
|
+
RETURN QUERY
|
|
139
|
+
UPDATE hazo_collect_rate_limits r
|
|
140
|
+
SET tokens_available = CASE
|
|
141
|
+
WHEN old.is_allowed THEN old.refilled - p_cost
|
|
142
|
+
ELSE old.refilled
|
|
143
|
+
END,
|
|
144
|
+
last_refill_at = now_ts,
|
|
145
|
+
requests_per_second = p_requests_per_second,
|
|
146
|
+
burst = p_burst,
|
|
147
|
+
updated_at = now_ts
|
|
148
|
+
FROM (
|
|
149
|
+
-- FOR UPDATE locks the row before computing refill/allowed, so a
|
|
150
|
+
-- concurrent caller blocks here (not just at the outer UPDATE) and,
|
|
151
|
+
-- once unblocked, re-reads the fresh committed row — no lost updates.
|
|
152
|
+
SELECT
|
|
153
|
+
host,
|
|
154
|
+
LEAST(
|
|
155
|
+
p_burst::double precision,
|
|
156
|
+
tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
|
|
157
|
+
) AS refilled,
|
|
158
|
+
backoff_until,
|
|
159
|
+
(backoff_until IS NULL OR backoff_until <= now_ts)
|
|
160
|
+
AND LEAST(
|
|
161
|
+
p_burst::double precision,
|
|
162
|
+
tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
|
|
163
|
+
) >= p_cost AS is_allowed
|
|
164
|
+
FROM hazo_collect_rate_limits
|
|
165
|
+
WHERE host = p_host
|
|
166
|
+
FOR UPDATE
|
|
167
|
+
) AS old
|
|
168
|
+
WHERE r.host = old.host
|
|
169
|
+
RETURNING
|
|
170
|
+
old.is_allowed AS allowed,
|
|
171
|
+
CASE
|
|
172
|
+
WHEN old.is_allowed THEN 0
|
|
173
|
+
WHEN old.backoff_until IS NOT NULL AND old.backoff_until > now_ts
|
|
174
|
+
THEN CEIL(EXTRACT(EPOCH FROM (old.backoff_until - now_ts)) * 1000)::int
|
|
175
|
+
WHEN p_requests_per_second > 0
|
|
176
|
+
THEN CEIL((p_cost - old.refilled) / p_requests_per_second * 1000)::int
|
|
177
|
+
ELSE 2147483647
|
|
178
|
+
END AS wait_ms;
|
|
179
|
+
END;
|
|
180
|
+
$$;
|
|
181
|
+
|
|
182
|
+
COMMENT ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) IS
|
|
183
|
+
'hazo_collect: atomically take a rate-limit token for a host (refill-on-read token bucket). '
|
|
184
|
+
'Callable via PostgREST POST /rpc/hazo_collect_take_token_rpc. GRANT EXECUTE to the PostgREST role.';
|
|
185
|
+
|
|
186
|
+
DROP FUNCTION IF EXISTS hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int);
|
|
187
|
+
|
|
188
|
+
CREATE OR REPLACE FUNCTION hazo_collect_apply_backoff_rpc(
|
|
189
|
+
p_host text,
|
|
190
|
+
p_backoff_until timestamptz,
|
|
191
|
+
p_requests_per_second double precision,
|
|
192
|
+
p_burst int
|
|
193
|
+
) RETURNS void
|
|
194
|
+
LANGUAGE plpgsql
|
|
195
|
+
SECURITY DEFINER
|
|
196
|
+
AS $$
|
|
197
|
+
BEGIN
|
|
198
|
+
INSERT INTO hazo_collect_rate_limits
|
|
199
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, backoff_until, updated_at)
|
|
200
|
+
VALUES (p_host, p_requests_per_second, p_burst, 0, clock_timestamp(), p_backoff_until, clock_timestamp())
|
|
201
|
+
ON CONFLICT (host) DO UPDATE
|
|
202
|
+
SET tokens_available = 0,
|
|
203
|
+
backoff_until = GREATEST(hazo_collect_rate_limits.backoff_until, EXCLUDED.backoff_until),
|
|
204
|
+
updated_at = clock_timestamp();
|
|
205
|
+
END;
|
|
206
|
+
$$;
|
|
207
|
+
|
|
208
|
+
COMMENT ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) IS
|
|
209
|
+
'hazo_collect: record a 429 + Retry-After for a host — drains the shared bucket and raises '
|
|
210
|
+
'backoff_until so every caller sharing this host backs off. Callable via PostgREST POST '
|
|
211
|
+
'/rpc/hazo_collect_apply_backoff_rpc. GRANT EXECUTE to the PostgREST role.';
|
|
212
|
+
|
|
78
213
|
NOTIFY pgrst, 'reload schema';
|
package/dist/ddl/sqlite.sql
CHANGED
|
@@ -10,6 +10,12 @@
|
|
|
10
10
|
-- hazo_collect_plugin_registry
|
|
11
11
|
-- hazo_collect_watermarks
|
|
12
12
|
-- hazo_collect_landing
|
|
13
|
+
-- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+).
|
|
14
|
+
-- SQLite has no RPC/plpgsql, so the atomic "take a token" operation is
|
|
15
|
+
-- an optimistic compare-and-swap retry loop over this table (see
|
|
16
|
+
-- src/server/rate-limit.ts, createSqliteRateLimitStore) instead of a
|
|
17
|
+
-- stored function — same technique hazo_jobs uses for its SQLite
|
|
18
|
+
-- group-completion CAS (src/server/group-tracker.ts).
|
|
13
19
|
|
|
14
20
|
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
|
|
15
21
|
id TEXT PRIMARY KEY,
|
|
@@ -33,12 +39,13 @@ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
|
|
|
33
39
|
ON hazo_collect_plugin_runs (plugin, started_at DESC);
|
|
34
40
|
|
|
35
41
|
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
|
|
36
|
-
plugin
|
|
37
|
-
last_success_at
|
|
38
|
-
last_run_at
|
|
39
|
-
last_status
|
|
40
|
-
state
|
|
41
|
-
consecutive_failures
|
|
42
|
+
plugin TEXT PRIMARY KEY,
|
|
43
|
+
last_success_at TEXT,
|
|
44
|
+
last_run_at TEXT,
|
|
45
|
+
last_status TEXT,
|
|
46
|
+
state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
|
|
47
|
+
consecutive_failures INTEGER NOT NULL DEFAULT 0,
|
|
48
|
+
last_rate_limit_wait_ms INTEGER NOT NULL DEFAULT 0
|
|
42
49
|
);
|
|
43
50
|
|
|
44
51
|
CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
|
|
@@ -73,3 +80,13 @@ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
|
|
|
73
80
|
|
|
74
81
|
CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
|
|
75
82
|
ON hazo_collect_landing (plugin, fetched_at DESC);
|
|
83
|
+
|
|
84
|
+
CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
|
|
85
|
+
host TEXT PRIMARY KEY,
|
|
86
|
+
requests_per_second REAL NOT NULL,
|
|
87
|
+
burst INTEGER NOT NULL,
|
|
88
|
+
tokens_available REAL NOT NULL,
|
|
89
|
+
last_refill_at TEXT NOT NULL DEFAULT (datetime('now')),
|
|
90
|
+
backoff_until TEXT,
|
|
91
|
+
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
|
92
|
+
);
|
package/dist/server/index.d.ts
CHANGED
|
@@ -34,15 +34,108 @@ declare function persistRegistry(adapter: HazoConnectAdapter, entries: Array<{
|
|
|
34
34
|
reseed?: boolean;
|
|
35
35
|
}): Promise<DbResult<void>>;
|
|
36
36
|
|
|
37
|
+
/** Per-host token-bucket config. Hosts not present in the manager's `hosts`
|
|
38
|
+
* map keep today's unthrottled behaviour — this feature is strictly opt-in
|
|
39
|
+
* per host. */
|
|
40
|
+
interface HostRateLimitConfig {
|
|
41
|
+
requestsPerSecond: number;
|
|
42
|
+
burst: number;
|
|
43
|
+
}
|
|
44
|
+
type FetchFn = (url: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
|
|
45
|
+
interface TakeTokenResult {
|
|
46
|
+
allowed: boolean;
|
|
47
|
+
/** Milliseconds the caller should wait before trying again when `allowed` is false. 0 when allowed. */
|
|
48
|
+
waitMs: number;
|
|
49
|
+
}
|
|
50
|
+
/**
|
|
51
|
+
* The DB-backed, atomic per-host token bucket. One implementation per
|
|
52
|
+
* hazo_connect adapter dialect — see `createRateLimitStore`.
|
|
53
|
+
*/
|
|
54
|
+
interface RateLimitStore {
|
|
55
|
+
takeToken(host: string, cfg: HostRateLimitConfig, cost: number): Promise<TakeTokenResult>;
|
|
56
|
+
applyBackoff(host: string, backoffUntilIso: string, cfg: HostRateLimitConfig): Promise<void>;
|
|
57
|
+
}
|
|
58
|
+
interface WaitTracker {
|
|
59
|
+
totalWaitMs: number;
|
|
60
|
+
}
|
|
61
|
+
interface TokenBucketState {
|
|
62
|
+
tokensAvailable: number;
|
|
63
|
+
lastRefillAtMs: number;
|
|
64
|
+
backoffUntilMs: number | null;
|
|
65
|
+
}
|
|
66
|
+
interface ComputedTokenBucket {
|
|
67
|
+
allowed: boolean;
|
|
68
|
+
tokensAvailable: number;
|
|
69
|
+
waitMs: number;
|
|
70
|
+
}
|
|
71
|
+
/**
|
|
72
|
+
* Refill-on-read token bucket. Always advances the notion of "last refill"
|
|
73
|
+
* to `nowMs` and lets tokens accrue (capped at `burst`) even while a backoff
|
|
74
|
+
* is active — a backoff means "don't send requests", not "stop the clock" —
|
|
75
|
+
* but `allowed` is gated by the backoff window regardless of token count.
|
|
76
|
+
*/
|
|
77
|
+
declare function computeTokenBucket(state: TokenBucketState, cfg: HostRateLimitConfig, cost: number, nowMs: number): ComputedTokenBucket;
|
|
78
|
+
/** Parses a `Retry-After` header value (seconds, or an HTTP-date) into a
|
|
79
|
+
* millisecond delay relative to `nowMs`. Returns null when missing/unparsable. */
|
|
80
|
+
declare function parseRetryAfterMs(headerValue: string | null | undefined, nowMs: number): number | null;
|
|
81
|
+
declare function createRateLimitStore(adapter: HazoConnectAdapter): Promise<RateLimitStore>;
|
|
82
|
+
interface CreateRateLimitManagerOptions {
|
|
83
|
+
adapter: HazoConnectAdapter;
|
|
84
|
+
/** Per-host config. A host absent from this map is never throttled. */
|
|
85
|
+
hosts: Record<string, HostRateLimitConfig>;
|
|
86
|
+
/** The real network fetch — injectable so consumers (and this package's
|
|
87
|
+
* own tests) can point `ctx.http` at a fake host while the limit still
|
|
88
|
+
* applies. Defaults to hazo_core's `fetchWithRequestId`. */
|
|
89
|
+
fetchFn?: FetchFn;
|
|
90
|
+
/** Testing seam: bypass dialect detection with a store directly (used by
|
|
91
|
+
* this package's own unit tests; consumers normally never pass this). */
|
|
92
|
+
store?: RateLimitStore;
|
|
93
|
+
now?: () => number;
|
|
94
|
+
sleep?: (ms: number) => Promise<void>;
|
|
95
|
+
/** Safety valve so a pathological config can't hang a run forever waiting
|
|
96
|
+
* on tokens. Default 60s. */
|
|
97
|
+
maxWaitMs?: number;
|
|
98
|
+
}
|
|
99
|
+
interface RateLimitManager {
|
|
100
|
+
/** Returns a fetch-compatible function enforcing the shared per-host
|
|
101
|
+
* bucket. Pass a `waitTracker` to accumulate how long this call spent
|
|
102
|
+
* waiting on the limit — used to populate collector health. */
|
|
103
|
+
createScopedFetch(waitTracker?: WaitTracker): FetchFn;
|
|
104
|
+
}
|
|
105
|
+
declare function createRateLimitManager(opts: CreateRateLimitManagerOptions): RateLimitManager;
|
|
106
|
+
|
|
37
107
|
interface RuntimeAdapter {
|
|
38
108
|
run(def: CollectorDefinition, envelope: RunEnvelope, adapter: HazoConnectAdapter): Promise<{
|
|
39
109
|
ok: true;
|
|
40
110
|
outcome: CollectorOutcome;
|
|
111
|
+
rateLimitWaitMs: number;
|
|
41
112
|
} | {
|
|
42
113
|
ok: false;
|
|
43
114
|
error: unknown;
|
|
115
|
+
rateLimitWaitMs: number;
|
|
44
116
|
}>;
|
|
45
117
|
}
|
|
118
|
+
interface NodeRuntimeOptions {
|
|
119
|
+
/**
|
|
120
|
+
* Injected fetch implementation used by every collector's `ctx.http`.
|
|
121
|
+
* Lets consumers (and this package's own tests) point `ctx.http` at a
|
|
122
|
+
* fake host while per-host rate limiting still applies. Defaults to
|
|
123
|
+
* hazo_core's `fetchWithRequestId`.
|
|
124
|
+
*/
|
|
125
|
+
fetch?: FetchFn;
|
|
126
|
+
/**
|
|
127
|
+
* Per-host token-bucket config (`requestsPerSecond` + `burst`). A host
|
|
128
|
+
* absent from this map keeps today's unthrottled behaviour.
|
|
129
|
+
*/
|
|
130
|
+
rateLimits?: Record<string, HostRateLimitConfig>;
|
|
131
|
+
}
|
|
132
|
+
/**
|
|
133
|
+
* Builds a `RuntimeAdapter`. `nodeRuntime` below is the default instance
|
|
134
|
+
* (no fetch override, no rate limits — identical to pre-0.3.0 behaviour).
|
|
135
|
+
* `createManager({ fetch, rateLimits })` calls this factory so the manager
|
|
136
|
+
* is the seam consumers configure instead of touching the runtime directly.
|
|
137
|
+
*/
|
|
138
|
+
declare function createNodeRuntime(options?: NodeRuntimeOptions): RuntimeAdapter;
|
|
46
139
|
declare const nodeRuntime: RuntimeAdapter;
|
|
47
140
|
|
|
48
141
|
interface SecretsProvider {
|
|
@@ -54,6 +147,20 @@ interface ManagerOptions {
|
|
|
54
147
|
registry: Registry;
|
|
55
148
|
secrets?: SecretsProvider;
|
|
56
149
|
runtime?: RuntimeAdapter;
|
|
150
|
+
/**
|
|
151
|
+
* Injected fetch implementation used by `ctx.http` in every run this
|
|
152
|
+
* manager dispatches — lets consumers (and tests) point collectors at a
|
|
153
|
+
* fake host while per-host rate limiting still applies. Ignored when
|
|
154
|
+
* `runtime` is also supplied (that runtime owns its own fetch wiring).
|
|
155
|
+
* Defaults to hazo_core's `fetchWithRequestId`.
|
|
156
|
+
*/
|
|
157
|
+
fetch?: FetchFn;
|
|
158
|
+
/**
|
|
159
|
+
* Per-host token-bucket rate limits (`requestsPerSecond` + `burst`)
|
|
160
|
+
* enforced inside `ctx.http`. A host absent from this map keeps today's
|
|
161
|
+
* unthrottled behaviour. Ignored when `runtime` is also supplied.
|
|
162
|
+
*/
|
|
163
|
+
rateLimits?: Record<string, HostRateLimitConfig>;
|
|
57
164
|
}
|
|
58
165
|
interface Manager {
|
|
59
166
|
runNow(pluginName: string, opts?: {
|
|
@@ -63,4 +170,4 @@ interface Manager {
|
|
|
63
170
|
}
|
|
64
171
|
declare function createManager(opts: ManagerOptions): Manager;
|
|
65
172
|
|
|
66
|
-
export { type DiscoverOptions, type Manager, type ManagerOptions, type Registry, type RegistryEntry, type RuntimeAdapter, createInMemoryRegistry, createManager, discover, nodeRuntime, persistRegistry };
|
|
173
|
+
export { type DiscoverOptions, type FetchFn, type HostRateLimitConfig, type Manager, type ManagerOptions, type NodeRuntimeOptions, type RateLimitManager, type RateLimitStore, type Registry, type RegistryEntry, type RuntimeAdapter, type TakeTokenResult, type WaitTracker, computeTokenBucket, createInMemoryRegistry, createManager, createNodeRuntime, createRateLimitManager, createRateLimitStore, discover, nodeRuntime, parseRetryAfterMs, persistRegistry };
|
package/dist/server/index.js
CHANGED
|
@@ -54,7 +54,7 @@ import { QueryBuilder as QueryBuilder2, wrapResult as wrapResult2 } from "hazo_c
|
|
|
54
54
|
import { generateRequestId as generateRequestId2, getCorrelationId, withContext } from "hazo_core";
|
|
55
55
|
|
|
56
56
|
// src/runtime/node.ts
|
|
57
|
-
import { createLogger } from "hazo_core";
|
|
57
|
+
import { createLogger, fetchWithRequestId as fetchWithRequestId3 } from "hazo_core";
|
|
58
58
|
|
|
59
59
|
// src/sdk/http.ts
|
|
60
60
|
import { fetchWithRequestId, sleep } from "hazo_core";
|
|
@@ -88,29 +88,278 @@ function createHttpHelper(_requestId, fetchFn = fetchWithRequestId) {
|
|
|
88
88
|
};
|
|
89
89
|
}
|
|
90
90
|
|
|
91
|
-
// src/
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
91
|
+
// src/server/rate-limit.ts
|
|
92
|
+
import { fetchWithRequestId as fetchWithRequestId2, sleep as coreSleep } from "hazo_core";
|
|
93
|
+
function computeTokenBucket(state, cfg, cost, nowMs) {
|
|
94
|
+
const elapsedSec = Math.max(0, (nowMs - state.lastRefillAtMs) / 1e3);
|
|
95
|
+
const refilled = Math.min(cfg.burst, state.tokensAvailable + elapsedSec * cfg.requestsPerSecond);
|
|
96
|
+
const inBackoff = state.backoffUntilMs != null && state.backoffUntilMs > nowMs;
|
|
97
|
+
const allowed = !inBackoff && refilled >= cost;
|
|
98
|
+
const tokensAvailable = allowed ? refilled - cost : refilled;
|
|
99
|
+
let waitMs;
|
|
100
|
+
if (allowed) {
|
|
101
|
+
waitMs = 0;
|
|
102
|
+
} else if (inBackoff) {
|
|
103
|
+
waitMs = Math.max(0, state.backoffUntilMs - nowMs);
|
|
104
|
+
} else if (cfg.requestsPerSecond > 0) {
|
|
105
|
+
const deficit = cost - refilled;
|
|
106
|
+
waitMs = Math.ceil(deficit / cfg.requestsPerSecond * 1e3);
|
|
107
|
+
} else {
|
|
108
|
+
waitMs = Number.MAX_SAFE_INTEGER;
|
|
109
|
+
}
|
|
110
|
+
return { allowed, tokensAvailable, waitMs };
|
|
111
|
+
}
|
|
112
|
+
var DEFAULT_BACKOFF_MS = 1e3;
|
|
113
|
+
function parseRetryAfterMs(headerValue, nowMs) {
|
|
114
|
+
if (!headerValue) return null;
|
|
115
|
+
const asSeconds = Number(headerValue);
|
|
116
|
+
if (Number.isFinite(asSeconds)) return Math.max(0, asSeconds * 1e3);
|
|
117
|
+
const asDateMs = Date.parse(headerValue);
|
|
118
|
+
if (!Number.isNaN(asDateMs)) return Math.max(0, asDateMs - nowMs);
|
|
119
|
+
return null;
|
|
120
|
+
}
|
|
121
|
+
function hasRaw(adapter) {
|
|
122
|
+
return typeof adapter.raw === "function";
|
|
123
|
+
}
|
|
124
|
+
async function detectDialect(adapter) {
|
|
125
|
+
if (hasRaw(adapter)) return "pg";
|
|
126
|
+
let cfg;
|
|
127
|
+
try {
|
|
128
|
+
cfg = await adapter.getConfig();
|
|
129
|
+
} catch {
|
|
130
|
+
cfg = void 0;
|
|
131
|
+
}
|
|
132
|
+
if (cfg && typeof cfg.base_url === "string") return "postgrest";
|
|
133
|
+
return "sqlite";
|
|
134
|
+
}
|
|
135
|
+
function createPgRateLimitStore(adapter) {
|
|
136
|
+
return {
|
|
137
|
+
async takeToken(host, cfg, cost) {
|
|
138
|
+
const rows = await adapter.raw(
|
|
139
|
+
"SELECT allowed, wait_ms FROM hazo_collect_take_token_rpc($1, $2, $3, $4)",
|
|
140
|
+
[host, cfg.requestsPerSecond, cfg.burst, cost]
|
|
141
|
+
);
|
|
142
|
+
const row = rows[0];
|
|
143
|
+
return { allowed: Boolean(row.allowed), waitMs: Number(row.wait_ms) };
|
|
144
|
+
},
|
|
145
|
+
async applyBackoff(host, backoffUntilIso, cfg) {
|
|
146
|
+
await adapter.raw("SELECT hazo_collect_apply_backoff_rpc($1, $2, $3, $4)", [
|
|
147
|
+
host,
|
|
148
|
+
backoffUntilIso,
|
|
149
|
+
cfg.requestsPerSecond,
|
|
150
|
+
cfg.burst
|
|
151
|
+
]);
|
|
152
|
+
}
|
|
153
|
+
};
|
|
154
|
+
}
|
|
155
|
+
function createPostgrestRateLimitStore(adapter) {
|
|
156
|
+
return {
|
|
157
|
+
async takeToken(host, cfg, cost) {
|
|
158
|
+
const result = await adapter.rawQuery("/rpc/hazo_collect_take_token_rpc", {
|
|
159
|
+
method: "POST",
|
|
160
|
+
headers: { "Content-Type": "application/json" },
|
|
161
|
+
body: JSON.stringify({
|
|
162
|
+
p_host: host,
|
|
163
|
+
p_requests_per_second: cfg.requestsPerSecond,
|
|
164
|
+
p_burst: cfg.burst,
|
|
165
|
+
p_cost: cost
|
|
166
|
+
})
|
|
167
|
+
});
|
|
168
|
+
const row = Array.isArray(result) ? result[0] : result;
|
|
169
|
+
return { allowed: Boolean(row.allowed), waitMs: Number(row.wait_ms) };
|
|
170
|
+
},
|
|
171
|
+
async applyBackoff(host, backoffUntilIso, cfg) {
|
|
172
|
+
await adapter.rawQuery("/rpc/hazo_collect_apply_backoff_rpc", {
|
|
173
|
+
method: "POST",
|
|
174
|
+
headers: { "Content-Type": "application/json" },
|
|
175
|
+
body: JSON.stringify({
|
|
176
|
+
p_host: host,
|
|
177
|
+
p_backoff_until: backoffUntilIso,
|
|
178
|
+
p_requests_per_second: cfg.requestsPerSecond,
|
|
179
|
+
p_burst: cfg.burst
|
|
180
|
+
})
|
|
181
|
+
});
|
|
182
|
+
}
|
|
183
|
+
};
|
|
184
|
+
}
|
|
185
|
+
var RATE_LIMIT_TABLE = "hazo_collect_rate_limits";
|
|
186
|
+
function createSqliteRateLimitStore(adapter, maxAttempts = 50) {
|
|
187
|
+
async function readRow(host) {
|
|
188
|
+
const rows = await adapter.rawQuery(`SELECT * FROM ${RATE_LIMIT_TABLE} WHERE host = ?`, {
|
|
189
|
+
params: [host]
|
|
98
190
|
});
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
191
|
+
return rows[0];
|
|
192
|
+
}
|
|
193
|
+
async function seedRow(host, cfg, tokens, nowIso) {
|
|
194
|
+
await adapter.rawQuery(
|
|
195
|
+
`INSERT OR IGNORE INTO ${RATE_LIMIT_TABLE}
|
|
196
|
+
(host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
|
|
197
|
+
VALUES (?, ?, ?, ?, ?, ?)`,
|
|
198
|
+
{ params: [host, cfg.requestsPerSecond, cfg.burst, tokens, nowIso, nowIso] }
|
|
199
|
+
);
|
|
200
|
+
}
|
|
201
|
+
return {
|
|
202
|
+
async takeToken(host, cfg, cost) {
|
|
203
|
+
for (let attempt = 0; attempt < maxAttempts; attempt++) {
|
|
204
|
+
const existing = await readRow(host);
|
|
205
|
+
const nowMs = Date.now();
|
|
206
|
+
const nowIso = new Date(nowMs).toISOString();
|
|
207
|
+
if (!existing) {
|
|
208
|
+
await seedRow(host, cfg, cfg.burst, nowIso);
|
|
209
|
+
continue;
|
|
210
|
+
}
|
|
211
|
+
const computed = computeTokenBucket(
|
|
212
|
+
{
|
|
213
|
+
tokensAvailable: Number(existing.tokens_available),
|
|
214
|
+
lastRefillAtMs: Date.parse(existing.last_refill_at),
|
|
215
|
+
backoffUntilMs: existing.backoff_until ? Date.parse(existing.backoff_until) : null
|
|
216
|
+
},
|
|
217
|
+
cfg,
|
|
218
|
+
cost,
|
|
219
|
+
nowMs
|
|
220
|
+
);
|
|
221
|
+
const updated = await adapter.rawQuery(
|
|
222
|
+
`UPDATE ${RATE_LIMIT_TABLE}
|
|
223
|
+
SET tokens_available = ?, last_refill_at = ?, requests_per_second = ?, burst = ?, updated_at = ?
|
|
224
|
+
WHERE host = ?
|
|
225
|
+
AND last_refill_at = ?
|
|
226
|
+
AND IFNULL(backoff_until, '') = IFNULL(?, '')
|
|
227
|
+
RETURNING tokens_available`,
|
|
228
|
+
{
|
|
229
|
+
params: [
|
|
230
|
+
computed.tokensAvailable,
|
|
231
|
+
nowIso,
|
|
232
|
+
cfg.requestsPerSecond,
|
|
233
|
+
cfg.burst,
|
|
234
|
+
nowIso,
|
|
235
|
+
host,
|
|
236
|
+
existing.last_refill_at,
|
|
237
|
+
existing.backoff_until
|
|
238
|
+
]
|
|
239
|
+
}
|
|
240
|
+
);
|
|
241
|
+
if (!updated.length) continue;
|
|
242
|
+
return { allowed: computed.allowed, waitMs: computed.waitMs };
|
|
243
|
+
}
|
|
244
|
+
throw new Error(`hazo_collect: exhausted retries taking a rate-limit token for host "${host}"`);
|
|
245
|
+
},
|
|
246
|
+
async applyBackoff(host, backoffUntilIso, cfg) {
|
|
247
|
+
const nowIso = (/* @__PURE__ */ new Date()).toISOString();
|
|
248
|
+
await seedRow(host, cfg, 0, nowIso);
|
|
249
|
+
await adapter.rawQuery(
|
|
250
|
+
`UPDATE ${RATE_LIMIT_TABLE}
|
|
251
|
+
SET tokens_available = 0,
|
|
252
|
+
backoff_until = CASE
|
|
253
|
+
WHEN backoff_until IS NULL OR backoff_until < ? THEN ?
|
|
254
|
+
ELSE backoff_until
|
|
255
|
+
END,
|
|
256
|
+
updated_at = ?
|
|
257
|
+
WHERE host = ?`,
|
|
258
|
+
{ params: [backoffUntilIso, backoffUntilIso, nowIso, host] }
|
|
259
|
+
);
|
|
111
260
|
}
|
|
261
|
+
};
|
|
262
|
+
}
|
|
263
|
+
async function createRateLimitStore(adapter) {
|
|
264
|
+
const dialect = await detectDialect(adapter);
|
|
265
|
+
switch (dialect) {
|
|
266
|
+
case "pg":
|
|
267
|
+
return createPgRateLimitStore(adapter);
|
|
268
|
+
case "postgrest":
|
|
269
|
+
return createPostgrestRateLimitStore(adapter);
|
|
270
|
+
case "sqlite":
|
|
271
|
+
return createSqliteRateLimitStore(adapter);
|
|
112
272
|
}
|
|
113
|
-
}
|
|
273
|
+
}
|
|
274
|
+
function hostnameOf(url) {
|
|
275
|
+
try {
|
|
276
|
+
const raw = url instanceof Request ? url.url : url;
|
|
277
|
+
return (typeof raw === "string" ? new URL(raw) : raw).hostname;
|
|
278
|
+
} catch {
|
|
279
|
+
return String(url);
|
|
280
|
+
}
|
|
281
|
+
}
|
|
282
|
+
function createRateLimitManager(opts) {
|
|
283
|
+
const hosts = opts.hosts ?? {};
|
|
284
|
+
const fetchFn = opts.fetchFn ?? fetchWithRequestId2;
|
|
285
|
+
const now = opts.now ?? (() => Date.now());
|
|
286
|
+
const sleepFn = opts.sleep ?? coreSleep;
|
|
287
|
+
const maxWaitMs = opts.maxWaitMs ?? 6e4;
|
|
288
|
+
let storePromise;
|
|
289
|
+
function getStore() {
|
|
290
|
+
if (opts.store) return Promise.resolve(opts.store);
|
|
291
|
+
if (!storePromise) storePromise = createRateLimitStore(opts.adapter);
|
|
292
|
+
return storePromise;
|
|
293
|
+
}
|
|
294
|
+
return {
|
|
295
|
+
createScopedFetch(waitTracker) {
|
|
296
|
+
return async (url, init) => {
|
|
297
|
+
const host = hostnameOf(url);
|
|
298
|
+
const cfg = hosts[host];
|
|
299
|
+
if (!cfg) return fetchFn(url, init);
|
|
300
|
+
const store = await getStore();
|
|
301
|
+
let waitedMs = 0;
|
|
302
|
+
while (true) {
|
|
303
|
+
const { allowed, waitMs } = await store.takeToken(host, cfg, 1);
|
|
304
|
+
if (allowed) break;
|
|
305
|
+
const sleepMs = Math.max(1, waitMs);
|
|
306
|
+
waitedMs += sleepMs;
|
|
307
|
+
if (waitedMs > maxWaitMs) break;
|
|
308
|
+
await sleepFn(sleepMs);
|
|
309
|
+
}
|
|
310
|
+
if (waitTracker) waitTracker.totalWaitMs += waitedMs;
|
|
311
|
+
const response = await fetchFn(url, init);
|
|
312
|
+
if (response.status === 429) {
|
|
313
|
+
const retryAfter = parseRetryAfterMs(response.headers?.get?.("Retry-After") ?? null, now());
|
|
314
|
+
const backoffUntilIso = new Date(now() + (retryAfter ?? DEFAULT_BACKOFF_MS)).toISOString();
|
|
315
|
+
await store.applyBackoff(host, backoffUntilIso, cfg);
|
|
316
|
+
}
|
|
317
|
+
return response;
|
|
318
|
+
};
|
|
319
|
+
}
|
|
320
|
+
};
|
|
321
|
+
}
|
|
322
|
+
|
|
323
|
+
// src/runtime/node.ts
|
|
324
|
+
function createNodeRuntime(options = {}) {
|
|
325
|
+
const baseFetch = options.fetch ?? fetchWithRequestId3;
|
|
326
|
+
const hosts = options.rateLimits ?? {};
|
|
327
|
+
const rateLimitingEnabled = Object.keys(hosts).length > 0;
|
|
328
|
+
const managers = /* @__PURE__ */ new WeakMap();
|
|
329
|
+
function managerFor(adapter) {
|
|
330
|
+
let manager = managers.get(adapter);
|
|
331
|
+
if (!manager) {
|
|
332
|
+
manager = createRateLimitManager({ adapter, hosts, fetchFn: baseFetch });
|
|
333
|
+
managers.set(adapter, manager);
|
|
334
|
+
}
|
|
335
|
+
return manager;
|
|
336
|
+
}
|
|
337
|
+
return {
|
|
338
|
+
async run(def, envelope, adapter) {
|
|
339
|
+
const writeAdapter = createWriteAdapter(adapter);
|
|
340
|
+
const log = createLogger("hazo_collect").child({
|
|
341
|
+
collector: def.manifest.name,
|
|
342
|
+
runId: envelope.run_id
|
|
343
|
+
});
|
|
344
|
+
const waitTracker = { totalWaitMs: 0 };
|
|
345
|
+
const scopedFetch = rateLimitingEnabled ? managerFor(adapter).createScopedFetch(waitTracker) : baseFetch;
|
|
346
|
+
const http = createHttpHelper(envelope.run_id, scopedFetch);
|
|
347
|
+
const ctx = {
|
|
348
|
+
envelope,
|
|
349
|
+
write: writeAdapter.write.bind(writeAdapter),
|
|
350
|
+
http,
|
|
351
|
+
log
|
|
352
|
+
};
|
|
353
|
+
try {
|
|
354
|
+
const outcome = await def.run(ctx);
|
|
355
|
+
return { ok: true, outcome, rateLimitWaitMs: waitTracker.totalWaitMs };
|
|
356
|
+
} catch (error) {
|
|
357
|
+
return { ok: false, error, rateLimitWaitMs: waitTracker.totalWaitMs };
|
|
358
|
+
}
|
|
359
|
+
}
|
|
360
|
+
};
|
|
361
|
+
}
|
|
362
|
+
var nodeRuntime = createNodeRuntime();
|
|
114
363
|
|
|
115
364
|
// src/manager/index.ts
|
|
116
365
|
async function buildDefaultSecretsProvider() {
|
|
@@ -193,7 +442,7 @@ async function upsertWatermark(adapter, plugin, watermark) {
|
|
|
193
442
|
updated_at: nowISO()
|
|
194
443
|
});
|
|
195
444
|
}
|
|
196
|
-
async function upsertHealth(adapter, plugin, status, now) {
|
|
445
|
+
async function upsertHealth(adapter, plugin, status, now, rateLimitWaitMs) {
|
|
197
446
|
if (status === "success") {
|
|
198
447
|
const qb = new QueryBuilder2().from("hazo_collect_plugin_health").onConflict(["plugin"]).doUpdate();
|
|
199
448
|
await adapter.query(qb, "POST", {
|
|
@@ -202,7 +451,8 @@ async function upsertHealth(adapter, plugin, status, now) {
|
|
|
202
451
|
last_run_at: now,
|
|
203
452
|
last_status: "success",
|
|
204
453
|
consecutive_failures: 0,
|
|
205
|
-
state: "ok"
|
|
454
|
+
state: "ok",
|
|
455
|
+
last_rate_limit_wait_ms: rateLimitWaitMs
|
|
206
456
|
});
|
|
207
457
|
return;
|
|
208
458
|
}
|
|
@@ -213,7 +463,8 @@ async function upsertHealth(adapter, plugin, status, now) {
|
|
|
213
463
|
plugin,
|
|
214
464
|
last_run_at: now,
|
|
215
465
|
last_status: status,
|
|
216
|
-
consecutive_failures: current + 1
|
|
466
|
+
consecutive_failures: current + 1,
|
|
467
|
+
last_rate_limit_wait_ms: rateLimitWaitMs
|
|
217
468
|
};
|
|
218
469
|
if (status === "failed") {
|
|
219
470
|
upsertRow.state = "failed";
|
|
@@ -223,7 +474,7 @@ async function upsertHealth(adapter, plugin, status, now) {
|
|
|
223
474
|
}
|
|
224
475
|
function createManager(opts) {
|
|
225
476
|
const { adapter, registry } = opts;
|
|
226
|
-
const runtime = opts.runtime ?? nodeRuntime;
|
|
477
|
+
const runtime = opts.runtime ?? (opts.fetch || opts.rateLimits ? createNodeRuntime({ fetch: opts.fetch, rateLimits: opts.rateLimits }) : nodeRuntime);
|
|
227
478
|
let _secretsProviderPromise;
|
|
228
479
|
function getSecretsProvider() {
|
|
229
480
|
if (opts.secrets) return Promise.resolve(opts.secrets);
|
|
@@ -243,6 +494,7 @@ function createManager(opts) {
|
|
|
243
494
|
started_at
|
|
244
495
|
});
|
|
245
496
|
const dispatchResult = await runtime.run(entry.worker, envelope, adapter);
|
|
497
|
+
const rateLimitWaitMs = dispatchResult.rateLimitWaitMs ?? 0;
|
|
246
498
|
let result;
|
|
247
499
|
if (dispatchResult.ok) {
|
|
248
500
|
const outcome = dispatchResult.outcome;
|
|
@@ -283,7 +535,7 @@ function createManager(opts) {
|
|
|
283
535
|
if (result.status === "success" && result.watermark) {
|
|
284
536
|
await upsertWatermark(adapter, envelope.plugin, result.watermark);
|
|
285
537
|
}
|
|
286
|
-
await upsertHealth(adapter, envelope.plugin, result.status, nowISO());
|
|
538
|
+
await upsertHealth(adapter, envelope.plugin, result.status, nowISO(), rateLimitWaitMs);
|
|
287
539
|
return result;
|
|
288
540
|
});
|
|
289
541
|
}
|
|
@@ -470,10 +722,15 @@ async function persistRegistry(adapter, entries, opts) {
|
|
|
470
722
|
});
|
|
471
723
|
}
|
|
472
724
|
export {
|
|
725
|
+
computeTokenBucket,
|
|
473
726
|
createInMemoryRegistry,
|
|
474
727
|
createManager,
|
|
728
|
+
createNodeRuntime,
|
|
729
|
+
createRateLimitManager,
|
|
730
|
+
createRateLimitStore,
|
|
475
731
|
createWriteAdapter,
|
|
476
732
|
discover,
|
|
477
733
|
nodeRuntime,
|
|
734
|
+
parseRetryAfterMs,
|
|
478
735
|
persistRegistry
|
|
479
736
|
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "hazo_collect",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Collector-manager engine for the Ocdata platform",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"module": "./dist/index.js",
|
|
@@ -27,6 +27,9 @@
|
|
|
27
27
|
},
|
|
28
28
|
"files": [
|
|
29
29
|
"dist",
|
|
30
|
+
"config/hazo_collect_config.ini.sample",
|
|
31
|
+
"db_setup_postgres.sql",
|
|
32
|
+
"db_setup_sqlite.sql",
|
|
30
33
|
"README.md",
|
|
31
34
|
"CHANGE_LOG.md",
|
|
32
35
|
"SETUP_CHECKLIST.md"
|
|
@@ -43,13 +46,13 @@
|
|
|
43
46
|
"ajv": "^8.17.1"
|
|
44
47
|
},
|
|
45
48
|
"peerDependencies": {
|
|
46
|
-
"hazo_core": "^2.
|
|
47
|
-
"hazo_connect": "^4.3.
|
|
48
|
-
"hazo_secure": "^1.
|
|
49
|
+
"hazo_core": "^2.5.0",
|
|
50
|
+
"hazo_connect": "^4.3.5",
|
|
51
|
+
"hazo_secure": "^1.9.1",
|
|
49
52
|
"react": "^18.0.0 || ^19.0.0",
|
|
50
53
|
"react-dom": "^18.0.0 || ^19.0.0",
|
|
51
|
-
"hazo_ui": "^6.
|
|
52
|
-
"hazo_i18n": "^0.
|
|
54
|
+
"hazo_ui": "^6.21.0",
|
|
55
|
+
"hazo_i18n": "^0.17.0"
|
|
53
56
|
},
|
|
54
57
|
"peerDependenciesMeta": {
|
|
55
58
|
"hazo_secure": {
|
|
@@ -78,9 +81,9 @@
|
|
|
78
81
|
"@types/node": "^22.10.0",
|
|
79
82
|
"@types/react": "^19.0.0",
|
|
80
83
|
"@types/react-dom": "^19.0.0",
|
|
81
|
-
"hazo_core": "^2.
|
|
82
|
-
"hazo_connect": "^4.3.
|
|
83
|
-
"hazo_secure": "^1.
|
|
84
|
+
"hazo_core": "^2.5.0",
|
|
85
|
+
"hazo_connect": "^4.3.5",
|
|
86
|
+
"hazo_secure": "^1.9.1"
|
|
84
87
|
},
|
|
85
88
|
"keywords": [],
|
|
86
89
|
"author": "Pubs Abayasiri",
|