hazo_collect 0.2.8 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGE_LOG.md CHANGED
@@ -1,5 +1,74 @@
1
1
  # hazo_collect — Change Log
2
2
 
3
+ ## 0.3.0 — 2026-11-21 (faraday data-foundation ticket 05: shared per-host rate limit)
4
+
5
+ ### ⚠️ DDL required
6
+
7
+ New table `hazo_collect_rate_limits` (one token-bucket row per host) plus two
8
+ new Postgres RPC functions, `hazo_collect_take_token_rpc` and
9
+ `hazo_collect_apply_backoff_rpc`, and one new column on
10
+ `hazo_collect_plugin_health` (`last_rate_limit_wait_ms`, default 0).
11
+ Re-run `db_setup_postgres.sql` / `db_setup_sqlite.sql` (idempotent) or apply
12
+ `src/ddl/postgres.sql` / `src/ddl/sqlite.sql` directly. SQLite has no
13
+ RPC/plpgsql, so its atomic "take a token" step is an optimistic
14
+ compare-and-swap retry loop over the same table instead of a stored
15
+ function (see `src/server/rate-limit.ts`, `createSqliteRateLimitStore`) —
16
+ same technique hazo_jobs 0.25.0 used for its SQLite group-completion CAS.
17
+ PostgREST consumers must `GRANT EXECUTE` on both new functions to the role
18
+ PostgREST connects as — see README's "PostgREST setup" note.
19
+
20
+ ### Added
21
+
22
+ - **Shared, DB-backed per-host token-bucket rate limiting inside `ctx.http`.**
23
+ Consumers configure `requestsPerSecond` + `burst` per host via
24
+ `createManager({ rateLimits: { 'api.example.com': { requestsPerSecond: 5,
25
+ burst: 10 } } })`. A host absent from that map keeps today's unthrottled
26
+ behaviour (no limiting), so this is strictly additive/opt-in. The bucket
27
+ state lives in the database (`hazo_collect_rate_limits`) and is shared by
28
+ every worker/process collecting through it — not just this process's view
29
+ of the limit.
30
+ - **Postgres**: taking a token is one atomic statement,
31
+ `hazo_collect_take_token_rpc` — a single `UPDATE ... FROM (SELECT ...
32
+ FOR UPDATE) ... RETURNING`, so no lock is ever held across requests
33
+ (safe to call via PostgREST's `POST /rpc/hazo_collect_take_token_rpc`,
34
+ not just a raw SQL connection).
35
+ - **SQLite**: the same refill-on-read token-bucket math
36
+ (`computeTokenBucket` in `src/server/rate-limit.ts`) runs in an
37
+ optimistic CAS retry loop over the raw adapter, since SQLite has no
38
+ RPC/plpgsql equivalent.
39
+ - **429 + `Retry-After`** on a rate-limited host trigger backoff: the
40
+ shared bucket is drained and `backoff_until` is raised (monotonically)
41
+ via `hazo_collect_apply_backoff_rpc` / the SQLite equivalent, so every
42
+ caller sharing that host's bucket backs off — not just the process that
43
+ saw the 429.
44
+ - **Collector health** (`hazo_collect_plugin_health.last_rate_limit_wait_ms`)
45
+ records how long the most recent run waited on the limit.
46
+ - **`ManagerOptions.fetch`** — an injected fetch implementation used by
47
+ `ctx.http` in every run the manager dispatches. Lets consumers (and this
48
+ package's own tests) point collectors at a fake host while per-host rate
49
+ limiting still applies, without touching real network code.
50
+ `ManagerOptions.rateLimits` is the per-host config described above; both
51
+ are threaded through a new `createNodeRuntime({ fetch, rateLimits })`
52
+ factory (`nodeRuntime` stays the pre-0.3.0 default instance — no
53
+ behaviour change when neither option is passed).
54
+ - New `hazo_collect/server` exports: `createNodeRuntime`,
55
+ `createRateLimitManager`, `createRateLimitStore`, `computeTokenBucket`,
56
+ `parseRetryAfterMs`, and types `HostRateLimitConfig`, `RateLimitManager`,
57
+ `RateLimitStore`, `TakeTokenResult`, `FetchFn`, `WaitTracker`,
58
+ `NodeRuntimeOptions`.
59
+ - `config/hazo_collect_config.ini.sample` documents the per-host rate-limit
60
+ config shape (`[rate_limit:<host>]`) consumers should map into
61
+ `ManagerOptions.rateLimits` — this package has no ini-loading code of its
62
+ own, so the section is documentation, not a parsed config file.
63
+
64
+ ### Scope notes
65
+
66
+ - `RuntimeAdapter.run()`'s return type gained a required `rateLimitWaitMs`
67
+ field (present on both the `ok: true` and `ok: false` branches). This is
68
+ additive for any existing `RuntimeAdapter` implementation that already
69
+ destructures `{ ok, outcome }` / `{ ok, error }` — it simply won't read
70
+ the new field.
71
+
3
72
  ## 0.2.8 — 2026-09-19
4
73
 
5
74
  ### Added
package/README.md CHANGED
@@ -75,12 +75,17 @@ const result = await manager.runNow({ plugin: 'my_source' });
75
75
 
76
76
  | Export | Description |
77
77
  |---|---|
78
- | `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist |
79
- | `nodeRuntime` | Execution adapter for in-process Node.js collectors |
78
+ | `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist. Accepts `fetch` + `rateLimits` (see "Per-host rate limiting" above) |
79
+ | `nodeRuntime` | Default execution adapter for in-process Node.js collectors (no fetch override, no rate limits) |
80
+ | `createNodeRuntime({ fetch, rateLimits })` | Build a `RuntimeAdapter` with an injected fetch and/or per-host rate limits |
80
81
  | `createInMemoryRegistry()` | In-memory registry backed by the SDK Map |
81
82
  | `persistRegistry(adapter, manifests)` | Upsert manifests into `hazo_collect_plugin_registry`; quarantines invalid entries |
82
83
  | `discover(opts)` | Scan directories for `manifest.json` files and return parsed entries |
83
84
  | `createWriteAdapter(adapter, runId)` | Create a `ctx.write` function bound to a specific run |
85
+ | `createRateLimitManager(opts)` | Build the `ctx.http` rate-limiting wrapper directly (advanced — `createManager`'s `rateLimits`/`fetch` cover the common case) |
86
+ | `createRateLimitStore(adapter)` | Pick the dialect-appropriate (`pg` / `postgrest` / `sqlite`) atomic token-bucket store for an adapter |
87
+ | `computeTokenBucket(state, cfg, cost, nowMs)` | Pure refill-on-read token-bucket math, shared by the SQLite CAS loop |
88
+ | `parseRetryAfterMs(header, nowMs)` | Parse a `Retry-After` header (seconds or HTTP-date) into a millisecond delay |
84
89
 
85
90
  ### `hazo_collect/sdk`
86
91
 
@@ -109,8 +114,70 @@ Run migrations from `hazo_collect/ddl/sqlite.sql` (SQLite) or `hazo_collect/ddl/
109
114
 
110
115
  - `hazo_collect_plugin_registry` — discovered manifest catalogue with validity + quarantine info
111
116
  - `hazo_collect_plugin_runs` — per-run scorecard (status, records, contract_version, watermark)
112
- - `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status)
117
+ - `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status, last rate-limit wait)
113
118
  - `hazo_collect_landing` — idempotent landing zone for collector output rows
119
+ - `hazo_collect_rate_limits` — shared per-host token-bucket state (0.3.0+; see below)
120
+
121
+ ## Per-host rate limiting (`ctx.http`)
122
+
123
+ Every collector's `ctx.http` can enforce a shared, database-backed per-host
124
+ token bucket. Configure `requestsPerSecond` + `burst` per host on the
125
+ manager; a host with no entry keeps today's unthrottled behaviour:
126
+
127
+ ```ts
128
+ import { createManager } from 'hazo_collect/server';
129
+
130
+ const manager = createManager({
131
+ adapter,
132
+ registry,
133
+ rateLimits: {
134
+ 'api.example.com': { requestsPerSecond: 5, burst: 10 },
135
+ },
136
+ });
137
+ ```
138
+
139
+ Because the bucket lives in `hazo_collect_rate_limits` (not in process
140
+ memory), the limit is shared across every worker/process collecting through
141
+ the same database — not just this process's view of it. A `429` response
142
+ (honouring `Retry-After`, in seconds or as an HTTP-date) drains the shared
143
+ bucket and raises its backoff window, so every caller sharing that host
144
+ backs off, not just the one that saw the 429. `hazo_collect_plugin_health`
145
+ records how long the most recent run waited on the limit
146
+ (`last_rate_limit_wait_ms`).
147
+
148
+ ### Injected fetch
149
+
150
+ `createManager({ fetch })` lets you point `ctx.http` at a different fetch
151
+ implementation for every run the manager dispatches — the rate limiter still
152
+ applies on top of it. This is how both this package's own tests and a
153
+ consumer's tests can exercise `ctx.http` against a fake host without any
154
+ real network traffic:
155
+
156
+ ```ts
157
+ const manager = createManager({
158
+ adapter,
159
+ registry,
160
+ fetch: fakeFetch, // e.g. a jest.fn() resolving a canned Response
161
+ rateLimits: { 'fake-host.test': { requestsPerSecond: 10, burst: 10 } },
162
+ });
163
+ ```
164
+
165
+ `fetch` and `rateLimits` are ignored if you also pass your own `runtime`
166
+ (that runtime owns its own fetch wiring) — use `createNodeRuntime({ fetch,
167
+ rateLimits })` directly in that case.
168
+
169
+ ### PostgREST setup
170
+
171
+ If you run `createHazoConnect({ type: 'postgrest', ... })`, the rate limiter
172
+ calls the two new functions via `POST /rpc/<fn>`. Grant execute to the role
173
+ PostgREST connects as:
174
+
175
+ ```sql
176
+ GRANT EXECUTE ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) TO api_user;
177
+ GRANT EXECUTE ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) TO api_user;
178
+ ```
179
+
180
+ (substitute the role PostgREST actually connects as for `api_user`).
114
181
 
115
182
  ## Tailwind v4 (`@source` required)
116
183
 
@@ -0,0 +1,44 @@
1
+ ; hazo_collect configuration
2
+ ; Copy to hazo_collect_config.ini and fill in values for your environment.
3
+
4
+ ; [env]
5
+ ; Uncomment to hard-code the environment name (overrides HAZO_ENV / NODE_ENV).
6
+ ; env = development
7
+
8
+ ; [secrets]
9
+ ; Prefix for environment variables that hold plugin secrets.
10
+ ; A manifest declaring secrets = ["API_KEY"] will read env var ${env_prefix}_API_KEY.
11
+ ; env_prefix = HAZO_COLLECT_SECRET
12
+
13
+ ; [plugins]
14
+ ; Directory to scan for plugin manifests (future folder-scan discovery provider).
15
+ ; Each sub-directory should contain a manifest.json.
16
+ ; dir = ./plugins
17
+
18
+ ; [general]
19
+ ; Example setting
20
+ ; some_key = some_value
21
+
22
+ ; [rate_limit:<host>]
23
+ ; Per-host token-bucket rate limit enforced inside ctx.http (0.3.0+). One
24
+ ; section per host you want throttled — a host with no section here keeps
25
+ ; today's unthrottled behaviour. The bucket itself lives in the database
26
+ ; (hazo_collect_rate_limits), shared by every worker/process collecting
27
+ ; through it, so this config only needs to be loaded wherever createManager()
28
+ ; is called — it does not need to agree across processes.
29
+ ;
30
+ ; This file isn't parsed by hazo_collect itself (there's no ini-loading code
31
+ ; in this package yet) — map whatever ini/env/JSON config your app already
32
+ ; loads into the shape createManager() expects:
33
+ ;
34
+ ; createManager({
35
+ ; adapter,
36
+ ; registry,
37
+ ; rateLimits: {
38
+ ; 'api.example.com': { requestsPerSecond: 5, burst: 10 },
39
+ ; },
40
+ ; });
41
+ ;
42
+ ; [rate_limit:api.example.com]
43
+ ; requests_per_second = 5
44
+ ; burst = 10
@@ -0,0 +1,189 @@
1
+ -- hazo_collect — PostgreSQL DDL
2
+ -- Consumers MAY copy this verbatim or wrap with their own role grants.
3
+ -- Idempotent — safe to re-run.
4
+ --
5
+ -- Tables:
6
+ -- hazo_collect_plugin_runs — one row per plugin run (RunResult scorecard)
7
+ -- hazo_collect_plugin_health — latest health snapshot per plugin
8
+ -- hazo_collect_plugin_registry — discovered manifest catalogue
9
+ -- hazo_collect_watermarks — per-plugin incremental watermark
10
+ -- hazo_collect_landing — raw as-fetched payloads (idempotent on plugin+key)
11
+ -- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+)
12
+
13
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
14
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
15
+ plugin TEXT NOT NULL,
16
+ run_id TEXT NOT NULL UNIQUE,
17
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
18
+ started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
19
+ completed_at TIMESTAMPTZ,
20
+ status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
21
+ records_fetched INT NOT NULL DEFAULT 0,
22
+ records_written INT NOT NULL DEFAULT 0,
23
+ watermark TEXT,
24
+ errors JSONB NOT NULL DEFAULT '[]'::jsonb,
25
+ metrics JSONB,
26
+ correlation_id TEXT,
27
+ env TEXT
28
+ );
29
+
30
+ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
31
+ ON hazo_collect_plugin_runs (plugin, started_at DESC);
32
+
33
+ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_run_id_idx
34
+ ON hazo_collect_plugin_runs (run_id);
35
+
36
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
37
+ plugin TEXT PRIMARY KEY,
38
+ last_success_at TIMESTAMPTZ,
39
+ last_run_at TIMESTAMPTZ,
40
+ last_status TEXT,
41
+ state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
42
+ consecutive_failures INT NOT NULL DEFAULT 0,
43
+ last_rate_limit_wait_ms INT NOT NULL DEFAULT 0
44
+ );
45
+
46
+ -- 0.3.0: existing deployments — add the new column if upgrading in place.
47
+ ALTER TABLE hazo_collect_plugin_health
48
+ ADD COLUMN IF NOT EXISTS last_rate_limit_wait_ms INT NOT NULL DEFAULT 0;
49
+
50
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
51
+ name TEXT PRIMARY KEY,
52
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
53
+ version TEXT NOT NULL,
54
+ runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
55
+ manifest JSONB NOT NULL,
56
+ source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
57
+ discovered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
58
+ valid BOOLEAN NOT NULL DEFAULT TRUE,
59
+ quarantine_reason TEXT
60
+ );
61
+
62
+ CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
63
+ plugin TEXT PRIMARY KEY,
64
+ watermark TEXT NOT NULL,
65
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
66
+ );
67
+
68
+ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
69
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
70
+ plugin TEXT NOT NULL,
71
+ run_id TEXT NOT NULL,
72
+ idempotency_key TEXT NOT NULL,
73
+ payload JSONB NOT NULL,
74
+ fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
75
+ window_since TEXT,
76
+ window_until TEXT,
77
+ CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
78
+ );
79
+
80
+ CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
81
+ ON hazo_collect_landing (plugin, fetched_at DESC);
82
+
83
+ -- ===========================================================================
84
+ -- Shared per-host rate limiting (0.3.0+) — see src/ddl/postgres.sql for the
85
+ -- full design notes. Mirrored here per repo convention.
86
+ -- ===========================================================================
87
+
88
+ CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
89
+ host TEXT PRIMARY KEY,
90
+ requests_per_second DOUBLE PRECISION NOT NULL,
91
+ burst INT NOT NULL,
92
+ tokens_available DOUBLE PRECISION NOT NULL,
93
+ last_refill_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
94
+ backoff_until TIMESTAMPTZ,
95
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
96
+ );
97
+
98
+ DROP FUNCTION IF EXISTS hazo_collect_take_token_rpc(text, double precision, int, int);
99
+
100
+ CREATE OR REPLACE FUNCTION hazo_collect_take_token_rpc(
101
+ p_host text,
102
+ p_requests_per_second double precision,
103
+ p_burst int,
104
+ p_cost int DEFAULT 1
105
+ ) RETURNS TABLE(allowed boolean, wait_ms int)
106
+ LANGUAGE plpgsql
107
+ SECURITY DEFINER
108
+ AS $$
109
+ DECLARE
110
+ now_ts timestamptz := clock_timestamp();
111
+ BEGIN
112
+ INSERT INTO hazo_collect_rate_limits
113
+ (host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
114
+ VALUES (p_host, p_requests_per_second, p_burst, p_burst, now_ts, now_ts)
115
+ ON CONFLICT (host) DO NOTHING;
116
+
117
+ RETURN QUERY
118
+ UPDATE hazo_collect_rate_limits r
119
+ SET tokens_available = CASE
120
+ WHEN old.is_allowed THEN old.refilled - p_cost
121
+ ELSE old.refilled
122
+ END,
123
+ last_refill_at = now_ts,
124
+ requests_per_second = p_requests_per_second,
125
+ burst = p_burst,
126
+ updated_at = now_ts
127
+ FROM (
128
+ SELECT
129
+ host,
130
+ LEAST(
131
+ p_burst::double precision,
132
+ tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
133
+ ) AS refilled,
134
+ backoff_until,
135
+ (backoff_until IS NULL OR backoff_until <= now_ts)
136
+ AND LEAST(
137
+ p_burst::double precision,
138
+ tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
139
+ ) >= p_cost AS is_allowed
140
+ FROM hazo_collect_rate_limits
141
+ WHERE host = p_host
142
+ FOR UPDATE
143
+ ) AS old
144
+ WHERE r.host = old.host
145
+ RETURNING
146
+ old.is_allowed AS allowed,
147
+ CASE
148
+ WHEN old.is_allowed THEN 0
149
+ WHEN old.backoff_until IS NOT NULL AND old.backoff_until > now_ts
150
+ THEN CEIL(EXTRACT(EPOCH FROM (old.backoff_until - now_ts)) * 1000)::int
151
+ WHEN p_requests_per_second > 0
152
+ THEN CEIL((p_cost - old.refilled) / p_requests_per_second * 1000)::int
153
+ ELSE 2147483647
154
+ END AS wait_ms;
155
+ END;
156
+ $$;
157
+
158
+ COMMENT ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) IS
159
+ 'hazo_collect: atomically take a rate-limit token for a host (refill-on-read token bucket). '
160
+ 'Callable via PostgREST POST /rpc/hazo_collect_take_token_rpc. GRANT EXECUTE to the PostgREST role.';
161
+
162
+ DROP FUNCTION IF EXISTS hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int);
163
+
164
+ CREATE OR REPLACE FUNCTION hazo_collect_apply_backoff_rpc(
165
+ p_host text,
166
+ p_backoff_until timestamptz,
167
+ p_requests_per_second double precision,
168
+ p_burst int
169
+ ) RETURNS void
170
+ LANGUAGE plpgsql
171
+ SECURITY DEFINER
172
+ AS $$
173
+ BEGIN
174
+ INSERT INTO hazo_collect_rate_limits
175
+ (host, requests_per_second, burst, tokens_available, last_refill_at, backoff_until, updated_at)
176
+ VALUES (p_host, p_requests_per_second, p_burst, 0, clock_timestamp(), p_backoff_until, clock_timestamp())
177
+ ON CONFLICT (host) DO UPDATE
178
+ SET tokens_available = 0,
179
+ backoff_until = GREATEST(hazo_collect_rate_limits.backoff_until, EXCLUDED.backoff_until),
180
+ updated_at = clock_timestamp();
181
+ END;
182
+ $$;
183
+
184
+ COMMENT ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) IS
185
+ 'hazo_collect: record a 429 + Retry-After for a host — drains the shared bucket and raises '
186
+ 'backoff_until so every caller sharing this host backs off. Callable via PostgREST POST '
187
+ '/rpc/hazo_collect_apply_backoff_rpc. GRANT EXECUTE to the PostgREST role.';
188
+
189
+ NOTIFY pgrst, 'reload schema';
@@ -0,0 +1,87 @@
1
+ -- hazo_collect — SQLite DDL
2
+ -- Idempotent — safe to re-run.
3
+ --
4
+ -- SQLite companion to postgres.sql.
5
+ -- Type mapping: UUID→TEXT, BOOLEAN→INTEGER(0/1), TIMESTAMPTZ→TEXT, JSONB→TEXT.
6
+ --
7
+ -- Tables:
8
+ -- hazo_collect_plugin_runs
9
+ -- hazo_collect_plugin_health
10
+ -- hazo_collect_plugin_registry
11
+ -- hazo_collect_watermarks
12
+ -- hazo_collect_landing
13
+ -- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+);
14
+ -- see src/ddl/sqlite.sql for the CAS-loop design note.
15
+
16
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
17
+ id TEXT PRIMARY KEY,
18
+ plugin TEXT NOT NULL,
19
+ run_id TEXT NOT NULL UNIQUE,
20
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
21
+ started_at TEXT NOT NULL DEFAULT (datetime('now')),
22
+ completed_at TEXT,
23
+ status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
24
+ records_fetched INTEGER NOT NULL DEFAULT 0,
25
+ records_written INTEGER NOT NULL DEFAULT 0,
26
+ watermark TEXT,
27
+ errors TEXT NOT NULL DEFAULT '[]',
28
+ metrics TEXT,
29
+ correlation_id TEXT,
30
+ env TEXT
31
+ );
32
+
33
+ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
34
+ ON hazo_collect_plugin_runs (plugin, started_at DESC);
35
+
36
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
37
+ plugin TEXT PRIMARY KEY,
38
+ last_success_at TEXT,
39
+ last_run_at TEXT,
40
+ last_status TEXT,
41
+ state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
42
+ consecutive_failures INTEGER NOT NULL DEFAULT 0,
43
+ last_rate_limit_wait_ms INTEGER NOT NULL DEFAULT 0
44
+ );
45
+
46
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
47
+ name TEXT PRIMARY KEY,
48
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
49
+ version TEXT NOT NULL,
50
+ runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
51
+ manifest TEXT NOT NULL,
52
+ source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
53
+ discovered_at TEXT NOT NULL DEFAULT (datetime('now')),
54
+ valid INTEGER NOT NULL DEFAULT 1,
55
+ quarantine_reason TEXT
56
+ );
57
+
58
+ CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
59
+ plugin TEXT PRIMARY KEY,
60
+ watermark TEXT NOT NULL,
61
+ updated_at TEXT NOT NULL DEFAULT (datetime('now'))
62
+ );
63
+
64
+ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
65
+ id TEXT PRIMARY KEY,
66
+ plugin TEXT NOT NULL,
67
+ run_id TEXT NOT NULL,
68
+ idempotency_key TEXT NOT NULL,
69
+ payload TEXT NOT NULL,
70
+ fetched_at TEXT NOT NULL DEFAULT (datetime('now')),
71
+ window_since TEXT,
72
+ window_until TEXT,
73
+ CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
74
+ );
75
+
76
+ CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
77
+ ON hazo_collect_landing (plugin, fetched_at DESC);
78
+
79
+ CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
80
+ host TEXT PRIMARY KEY,
81
+ requests_per_second REAL NOT NULL,
82
+ burst INTEGER NOT NULL,
83
+ tokens_available REAL NOT NULL,
84
+ last_refill_at TEXT NOT NULL DEFAULT (datetime('now')),
85
+ backoff_until TEXT,
86
+ updated_at TEXT NOT NULL DEFAULT (datetime('now'))
87
+ );
@@ -8,6 +8,7 @@
8
8
  -- hazo_collect_plugin_registry — discovered manifest catalogue
9
9
  -- hazo_collect_watermarks — per-plugin incremental watermark
10
10
  -- hazo_collect_landing — raw as-fetched payloads (idempotent on plugin+key)
11
+ -- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+)
11
12
 
12
13
  CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
13
14
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
@@ -34,14 +35,19 @@ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_run_id_idx
34
35
  ON hazo_collect_plugin_runs (run_id);
35
36
 
36
37
  CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
37
- plugin TEXT PRIMARY KEY,
38
- last_success_at TIMESTAMPTZ,
39
- last_run_at TIMESTAMPTZ,
40
- last_status TEXT,
41
- state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
42
- consecutive_failures INT NOT NULL DEFAULT 0
38
+ plugin TEXT PRIMARY KEY,
39
+ last_success_at TIMESTAMPTZ,
40
+ last_run_at TIMESTAMPTZ,
41
+ last_status TEXT,
42
+ state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
43
+ consecutive_failures INT NOT NULL DEFAULT 0,
44
+ last_rate_limit_wait_ms INT NOT NULL DEFAULT 0
43
45
  );
44
46
 
47
+ -- 0.3.0: existing deployments — add the new column if upgrading in place.
48
+ ALTER TABLE hazo_collect_plugin_health
49
+ ADD COLUMN IF NOT EXISTS last_rate_limit_wait_ms INT NOT NULL DEFAULT 0;
50
+
45
51
  CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
46
52
  name TEXT PRIMARY KEY,
47
53
  kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
@@ -75,4 +81,133 @@ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
75
81
  CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
76
82
  ON hazo_collect_landing (plugin, fetched_at DESC);
77
83
 
84
+ -- ===========================================================================
85
+ -- Shared per-host rate limiting (0.3.0+)
86
+ --
87
+ -- hazo_collect_rate_limits holds one token-bucket row per host, shared by
88
+ -- every worker/process that collects through this database — ctx.http
89
+ -- enforces it transparently for hosts configured via ManagerOptions.rateLimits
90
+ -- (see createNodeRuntime / createRateLimitManager in src/server/rate-limit.ts).
91
+ --
92
+ -- hazo_collect_take_token_rpc is the atomic "take a token" operation. It is a
93
+ -- SINGLE UPDATE statement (the UPDATE ... FROM (... FOR UPDATE) subquery below
94
+ -- is one statement, not a separate SELECT then UPDATE over two round trips) so
95
+ -- no lock is ever held across requests — safe to call from PostgREST via
96
+ -- POST /rpc/hazo_collect_take_token_rpc, and safe under concurrent callers.
97
+ --
98
+ -- hazo_collect_apply_backoff_rpc records a 429 + Retry-After: it drains the
99
+ -- bucket and raises backoff_until so every caller sharing this host's bucket
100
+ -- (any process, not just the one that saw the 429) backs off too.
101
+ --
102
+ -- Consumers using the PostgREST path must GRANT EXECUTE on both functions to
103
+ -- the role PostgREST connects as (see README's "PostgREST setup" note).
104
+ -- ===========================================================================
105
+
106
+ CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
107
+ host TEXT PRIMARY KEY,
108
+ requests_per_second DOUBLE PRECISION NOT NULL,
109
+ burst INT NOT NULL,
110
+ tokens_available DOUBLE PRECISION NOT NULL,
111
+ last_refill_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
112
+ backoff_until TIMESTAMPTZ,
113
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
114
+ );
115
+
116
+ DROP FUNCTION IF EXISTS hazo_collect_take_token_rpc(text, double precision, int, int);
117
+
118
+ CREATE OR REPLACE FUNCTION hazo_collect_take_token_rpc(
119
+ p_host text,
120
+ p_requests_per_second double precision,
121
+ p_burst int,
122
+ p_cost int DEFAULT 1
123
+ ) RETURNS TABLE(allowed boolean, wait_ms int)
124
+ LANGUAGE plpgsql
125
+ SECURITY DEFINER
126
+ AS $$
127
+ DECLARE
128
+ now_ts timestamptz := clock_timestamp();
129
+ BEGIN
130
+ -- First use: seed the bucket at full burst. Race-safe — a concurrent
131
+ -- racer's insert simply loses (ON CONFLICT DO NOTHING) and the UPDATE
132
+ -- below re-reads whichever row won.
133
+ INSERT INTO hazo_collect_rate_limits
134
+ (host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
135
+ VALUES (p_host, p_requests_per_second, p_burst, p_burst, now_ts, now_ts)
136
+ ON CONFLICT (host) DO NOTHING;
137
+
138
+ RETURN QUERY
139
+ UPDATE hazo_collect_rate_limits r
140
+ SET tokens_available = CASE
141
+ WHEN old.is_allowed THEN old.refilled - p_cost
142
+ ELSE old.refilled
143
+ END,
144
+ last_refill_at = now_ts,
145
+ requests_per_second = p_requests_per_second,
146
+ burst = p_burst,
147
+ updated_at = now_ts
148
+ FROM (
149
+ -- FOR UPDATE locks the row before computing refill/allowed, so a
150
+ -- concurrent caller blocks here (not just at the outer UPDATE) and,
151
+ -- once unblocked, re-reads the fresh committed row — no lost updates.
152
+ SELECT
153
+ host,
154
+ LEAST(
155
+ p_burst::double precision,
156
+ tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
157
+ ) AS refilled,
158
+ backoff_until,
159
+ (backoff_until IS NULL OR backoff_until <= now_ts)
160
+ AND LEAST(
161
+ p_burst::double precision,
162
+ tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
163
+ ) >= p_cost AS is_allowed
164
+ FROM hazo_collect_rate_limits
165
+ WHERE host = p_host
166
+ FOR UPDATE
167
+ ) AS old
168
+ WHERE r.host = old.host
169
+ RETURNING
170
+ old.is_allowed AS allowed,
171
+ CASE
172
+ WHEN old.is_allowed THEN 0
173
+ WHEN old.backoff_until IS NOT NULL AND old.backoff_until > now_ts
174
+ THEN CEIL(EXTRACT(EPOCH FROM (old.backoff_until - now_ts)) * 1000)::int
175
+ WHEN p_requests_per_second > 0
176
+ THEN CEIL((p_cost - old.refilled) / p_requests_per_second * 1000)::int
177
+ ELSE 2147483647
178
+ END AS wait_ms;
179
+ END;
180
+ $$;
181
+
182
+ COMMENT ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) IS
183
+ 'hazo_collect: atomically take a rate-limit token for a host (refill-on-read token bucket). '
184
+ 'Callable via PostgREST POST /rpc/hazo_collect_take_token_rpc. GRANT EXECUTE to the PostgREST role.';
185
+
186
+ DROP FUNCTION IF EXISTS hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int);
187
+
188
+ CREATE OR REPLACE FUNCTION hazo_collect_apply_backoff_rpc(
189
+ p_host text,
190
+ p_backoff_until timestamptz,
191
+ p_requests_per_second double precision,
192
+ p_burst int
193
+ ) RETURNS void
194
+ LANGUAGE plpgsql
195
+ SECURITY DEFINER
196
+ AS $$
197
+ BEGIN
198
+ INSERT INTO hazo_collect_rate_limits
199
+ (host, requests_per_second, burst, tokens_available, last_refill_at, backoff_until, updated_at)
200
+ VALUES (p_host, p_requests_per_second, p_burst, 0, clock_timestamp(), p_backoff_until, clock_timestamp())
201
+ ON CONFLICT (host) DO UPDATE
202
+ SET tokens_available = 0,
203
+ backoff_until = GREATEST(hazo_collect_rate_limits.backoff_until, EXCLUDED.backoff_until),
204
+ updated_at = clock_timestamp();
205
+ END;
206
+ $$;
207
+
208
+ COMMENT ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) IS
209
+ 'hazo_collect: record a 429 + Retry-After for a host — drains the shared bucket and raises '
210
+ 'backoff_until so every caller sharing this host backs off. Callable via PostgREST POST '
211
+ '/rpc/hazo_collect_apply_backoff_rpc. GRANT EXECUTE to the PostgREST role.';
212
+
78
213
  NOTIFY pgrst, 'reload schema';
@@ -10,6 +10,12 @@
10
10
  -- hazo_collect_plugin_registry
11
11
  -- hazo_collect_watermarks
12
12
  -- hazo_collect_landing
13
+ -- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+).
14
+ -- SQLite has no RPC/plpgsql, so the atomic "take a token" operation is
15
+ -- an optimistic compare-and-swap retry loop over this table (see
16
+ -- src/server/rate-limit.ts, createSqliteRateLimitStore) instead of a
17
+ -- stored function — same technique hazo_jobs uses for its SQLite
18
+ -- group-completion CAS (src/server/group-tracker.ts).
13
19
 
14
20
  CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
15
21
  id TEXT PRIMARY KEY,
@@ -33,12 +39,13 @@ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
33
39
  ON hazo_collect_plugin_runs (plugin, started_at DESC);
34
40
 
35
41
  CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
36
- plugin TEXT PRIMARY KEY,
37
- last_success_at TEXT,
38
- last_run_at TEXT,
39
- last_status TEXT,
40
- state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
41
- consecutive_failures INTEGER NOT NULL DEFAULT 0
42
+ plugin TEXT PRIMARY KEY,
43
+ last_success_at TEXT,
44
+ last_run_at TEXT,
45
+ last_status TEXT,
46
+ state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
47
+ consecutive_failures INTEGER NOT NULL DEFAULT 0,
48
+ last_rate_limit_wait_ms INTEGER NOT NULL DEFAULT 0
42
49
  );
43
50
 
44
51
  CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
@@ -73,3 +80,13 @@ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
73
80
 
74
81
  CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
75
82
  ON hazo_collect_landing (plugin, fetched_at DESC);
83
+
84
+ CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
85
+ host TEXT PRIMARY KEY,
86
+ requests_per_second REAL NOT NULL,
87
+ burst INTEGER NOT NULL,
88
+ tokens_available REAL NOT NULL,
89
+ last_refill_at TEXT NOT NULL DEFAULT (datetime('now')),
90
+ backoff_until TEXT,
91
+ updated_at TEXT NOT NULL DEFAULT (datetime('now'))
92
+ );
@@ -34,15 +34,108 @@ declare function persistRegistry(adapter: HazoConnectAdapter, entries: Array<{
34
34
  reseed?: boolean;
35
35
  }): Promise<DbResult<void>>;
36
36
 
37
+ /** Per-host token-bucket config. Hosts not present in the manager's `hosts`
38
+ * map keep today's unthrottled behaviour — this feature is strictly opt-in
39
+ * per host. */
40
+ interface HostRateLimitConfig {
41
+ requestsPerSecond: number;
42
+ burst: number;
43
+ }
44
+ type FetchFn = (url: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
45
+ interface TakeTokenResult {
46
+ allowed: boolean;
47
+ /** Milliseconds the caller should wait before trying again when `allowed` is false. 0 when allowed. */
48
+ waitMs: number;
49
+ }
50
+ /**
51
+ * The DB-backed, atomic per-host token bucket. One implementation per
52
+ * hazo_connect adapter dialect — see `createRateLimitStore`.
53
+ */
54
+ interface RateLimitStore {
55
+ takeToken(host: string, cfg: HostRateLimitConfig, cost: number): Promise<TakeTokenResult>;
56
+ applyBackoff(host: string, backoffUntilIso: string, cfg: HostRateLimitConfig): Promise<void>;
57
+ }
58
+ interface WaitTracker {
59
+ totalWaitMs: number;
60
+ }
61
+ interface TokenBucketState {
62
+ tokensAvailable: number;
63
+ lastRefillAtMs: number;
64
+ backoffUntilMs: number | null;
65
+ }
66
+ interface ComputedTokenBucket {
67
+ allowed: boolean;
68
+ tokensAvailable: number;
69
+ waitMs: number;
70
+ }
71
+ /**
72
+ * Refill-on-read token bucket. Always advances the notion of "last refill"
73
+ * to `nowMs` and lets tokens accrue (capped at `burst`) even while a backoff
74
+ * is active — a backoff means "don't send requests", not "stop the clock" —
75
+ * but `allowed` is gated by the backoff window regardless of token count.
76
+ */
77
+ declare function computeTokenBucket(state: TokenBucketState, cfg: HostRateLimitConfig, cost: number, nowMs: number): ComputedTokenBucket;
78
+ /** Parses a `Retry-After` header value (seconds, or an HTTP-date) into a
79
+ * millisecond delay relative to `nowMs`. Returns null when missing/unparsable. */
80
+ declare function parseRetryAfterMs(headerValue: string | null | undefined, nowMs: number): number | null;
81
+ declare function createRateLimitStore(adapter: HazoConnectAdapter): Promise<RateLimitStore>;
82
+ interface CreateRateLimitManagerOptions {
83
+ adapter: HazoConnectAdapter;
84
+ /** Per-host config. A host absent from this map is never throttled. */
85
+ hosts: Record<string, HostRateLimitConfig>;
86
+ /** The real network fetch — injectable so consumers (and this package's
87
+ * own tests) can point `ctx.http` at a fake host while the limit still
88
+ * applies. Defaults to hazo_core's `fetchWithRequestId`. */
89
+ fetchFn?: FetchFn;
90
+ /** Testing seam: bypass dialect detection with a store directly (used by
91
+ * this package's own unit tests; consumers normally never pass this). */
92
+ store?: RateLimitStore;
93
+ now?: () => number;
94
+ sleep?: (ms: number) => Promise<void>;
95
+ /** Safety valve so a pathological config can't hang a run forever waiting
96
+ * on tokens. Default 60s. */
97
+ maxWaitMs?: number;
98
+ }
99
+ interface RateLimitManager {
100
+ /** Returns a fetch-compatible function enforcing the shared per-host
101
+ * bucket. Pass a `waitTracker` to accumulate how long this call spent
102
+ * waiting on the limit — used to populate collector health. */
103
+ createScopedFetch(waitTracker?: WaitTracker): FetchFn;
104
+ }
105
+ declare function createRateLimitManager(opts: CreateRateLimitManagerOptions): RateLimitManager;
106
+
37
107
  interface RuntimeAdapter {
38
108
  run(def: CollectorDefinition, envelope: RunEnvelope, adapter: HazoConnectAdapter): Promise<{
39
109
  ok: true;
40
110
  outcome: CollectorOutcome;
111
+ rateLimitWaitMs: number;
41
112
  } | {
42
113
  ok: false;
43
114
  error: unknown;
115
+ rateLimitWaitMs: number;
44
116
  }>;
45
117
  }
118
+ interface NodeRuntimeOptions {
119
+ /**
120
+ * Injected fetch implementation used by every collector's `ctx.http`.
121
+ * Lets consumers (and this package's own tests) point `ctx.http` at a
122
+ * fake host while per-host rate limiting still applies. Defaults to
123
+ * hazo_core's `fetchWithRequestId`.
124
+ */
125
+ fetch?: FetchFn;
126
+ /**
127
+ * Per-host token-bucket config (`requestsPerSecond` + `burst`). A host
128
+ * absent from this map keeps today's unthrottled behaviour.
129
+ */
130
+ rateLimits?: Record<string, HostRateLimitConfig>;
131
+ }
132
+ /**
133
+ * Builds a `RuntimeAdapter`. `nodeRuntime` below is the default instance
134
+ * (no fetch override, no rate limits — identical to pre-0.3.0 behaviour).
135
+ * `createManager({ fetch, rateLimits })` calls this factory so the manager
136
+ * is the seam consumers configure instead of touching the runtime directly.
137
+ */
138
+ declare function createNodeRuntime(options?: NodeRuntimeOptions): RuntimeAdapter;
46
139
  declare const nodeRuntime: RuntimeAdapter;
47
140
 
48
141
  interface SecretsProvider {
@@ -54,6 +147,20 @@ interface ManagerOptions {
54
147
  registry: Registry;
55
148
  secrets?: SecretsProvider;
56
149
  runtime?: RuntimeAdapter;
150
+ /**
151
+ * Injected fetch implementation used by `ctx.http` in every run this
152
+ * manager dispatches — lets consumers (and tests) point collectors at a
153
+ * fake host while per-host rate limiting still applies. Ignored when
154
+ * `runtime` is also supplied (that runtime owns its own fetch wiring).
155
+ * Defaults to hazo_core's `fetchWithRequestId`.
156
+ */
157
+ fetch?: FetchFn;
158
+ /**
159
+ * Per-host token-bucket rate limits (`requestsPerSecond` + `burst`)
160
+ * enforced inside `ctx.http`. A host absent from this map keeps today's
161
+ * unthrottled behaviour. Ignored when `runtime` is also supplied.
162
+ */
163
+ rateLimits?: Record<string, HostRateLimitConfig>;
57
164
  }
58
165
  interface Manager {
59
166
  runNow(pluginName: string, opts?: {
@@ -63,4 +170,4 @@ interface Manager {
63
170
  }
64
171
  declare function createManager(opts: ManagerOptions): Manager;
65
172
 
66
- export { type DiscoverOptions, type Manager, type ManagerOptions, type Registry, type RegistryEntry, type RuntimeAdapter, createInMemoryRegistry, createManager, discover, nodeRuntime, persistRegistry };
173
+ export { type DiscoverOptions, type FetchFn, type HostRateLimitConfig, type Manager, type ManagerOptions, type NodeRuntimeOptions, type RateLimitManager, type RateLimitStore, type Registry, type RegistryEntry, type RuntimeAdapter, type TakeTokenResult, type WaitTracker, computeTokenBucket, createInMemoryRegistry, createManager, createNodeRuntime, createRateLimitManager, createRateLimitStore, discover, nodeRuntime, parseRetryAfterMs, persistRegistry };
@@ -54,7 +54,7 @@ import { QueryBuilder as QueryBuilder2, wrapResult as wrapResult2 } from "hazo_c
54
54
  import { generateRequestId as generateRequestId2, getCorrelationId, withContext } from "hazo_core";
55
55
 
56
56
  // src/runtime/node.ts
57
- import { createLogger } from "hazo_core";
57
+ import { createLogger, fetchWithRequestId as fetchWithRequestId3 } from "hazo_core";
58
58
 
59
59
  // src/sdk/http.ts
60
60
  import { fetchWithRequestId, sleep } from "hazo_core";
@@ -88,29 +88,278 @@ function createHttpHelper(_requestId, fetchFn = fetchWithRequestId) {
88
88
  };
89
89
  }
90
90
 
91
- // src/runtime/node.ts
92
- var nodeRuntime = {
93
- async run(def, envelope, adapter) {
94
- const writeAdapter = createWriteAdapter(adapter);
95
- const log = createLogger("hazo_collect").child({
96
- collector: def.manifest.name,
97
- runId: envelope.run_id
91
+ // src/server/rate-limit.ts
92
+ import { fetchWithRequestId as fetchWithRequestId2, sleep as coreSleep } from "hazo_core";
93
+ function computeTokenBucket(state, cfg, cost, nowMs) {
94
+ const elapsedSec = Math.max(0, (nowMs - state.lastRefillAtMs) / 1e3);
95
+ const refilled = Math.min(cfg.burst, state.tokensAvailable + elapsedSec * cfg.requestsPerSecond);
96
+ const inBackoff = state.backoffUntilMs != null && state.backoffUntilMs > nowMs;
97
+ const allowed = !inBackoff && refilled >= cost;
98
+ const tokensAvailable = allowed ? refilled - cost : refilled;
99
+ let waitMs;
100
+ if (allowed) {
101
+ waitMs = 0;
102
+ } else if (inBackoff) {
103
+ waitMs = Math.max(0, state.backoffUntilMs - nowMs);
104
+ } else if (cfg.requestsPerSecond > 0) {
105
+ const deficit = cost - refilled;
106
+ waitMs = Math.ceil(deficit / cfg.requestsPerSecond * 1e3);
107
+ } else {
108
+ waitMs = Number.MAX_SAFE_INTEGER;
109
+ }
110
+ return { allowed, tokensAvailable, waitMs };
111
+ }
112
+ var DEFAULT_BACKOFF_MS = 1e3;
113
+ function parseRetryAfterMs(headerValue, nowMs) {
114
+ if (!headerValue) return null;
115
+ const asSeconds = Number(headerValue);
116
+ if (Number.isFinite(asSeconds)) return Math.max(0, asSeconds * 1e3);
117
+ const asDateMs = Date.parse(headerValue);
118
+ if (!Number.isNaN(asDateMs)) return Math.max(0, asDateMs - nowMs);
119
+ return null;
120
+ }
121
+ function hasRaw(adapter) {
122
+ return typeof adapter.raw === "function";
123
+ }
124
+ async function detectDialect(adapter) {
125
+ if (hasRaw(adapter)) return "pg";
126
+ let cfg;
127
+ try {
128
+ cfg = await adapter.getConfig();
129
+ } catch {
130
+ cfg = void 0;
131
+ }
132
+ if (cfg && typeof cfg.base_url === "string") return "postgrest";
133
+ return "sqlite";
134
+ }
135
+ function createPgRateLimitStore(adapter) {
136
+ return {
137
+ async takeToken(host, cfg, cost) {
138
+ const rows = await adapter.raw(
139
+ "SELECT allowed, wait_ms FROM hazo_collect_take_token_rpc($1, $2, $3, $4)",
140
+ [host, cfg.requestsPerSecond, cfg.burst, cost]
141
+ );
142
+ const row = rows[0];
143
+ return { allowed: Boolean(row.allowed), waitMs: Number(row.wait_ms) };
144
+ },
145
+ async applyBackoff(host, backoffUntilIso, cfg) {
146
+ await adapter.raw("SELECT hazo_collect_apply_backoff_rpc($1, $2, $3, $4)", [
147
+ host,
148
+ backoffUntilIso,
149
+ cfg.requestsPerSecond,
150
+ cfg.burst
151
+ ]);
152
+ }
153
+ };
154
+ }
155
+ function createPostgrestRateLimitStore(adapter) {
156
+ return {
157
+ async takeToken(host, cfg, cost) {
158
+ const result = await adapter.rawQuery("/rpc/hazo_collect_take_token_rpc", {
159
+ method: "POST",
160
+ headers: { "Content-Type": "application/json" },
161
+ body: JSON.stringify({
162
+ p_host: host,
163
+ p_requests_per_second: cfg.requestsPerSecond,
164
+ p_burst: cfg.burst,
165
+ p_cost: cost
166
+ })
167
+ });
168
+ const row = Array.isArray(result) ? result[0] : result;
169
+ return { allowed: Boolean(row.allowed), waitMs: Number(row.wait_ms) };
170
+ },
171
+ async applyBackoff(host, backoffUntilIso, cfg) {
172
+ await adapter.rawQuery("/rpc/hazo_collect_apply_backoff_rpc", {
173
+ method: "POST",
174
+ headers: { "Content-Type": "application/json" },
175
+ body: JSON.stringify({
176
+ p_host: host,
177
+ p_backoff_until: backoffUntilIso,
178
+ p_requests_per_second: cfg.requestsPerSecond,
179
+ p_burst: cfg.burst
180
+ })
181
+ });
182
+ }
183
+ };
184
+ }
185
+ var RATE_LIMIT_TABLE = "hazo_collect_rate_limits";
186
+ function createSqliteRateLimitStore(adapter, maxAttempts = 50) {
187
+ async function readRow(host) {
188
+ const rows = await adapter.rawQuery(`SELECT * FROM ${RATE_LIMIT_TABLE} WHERE host = ?`, {
189
+ params: [host]
98
190
  });
99
- const http = createHttpHelper(envelope.run_id);
100
- const ctx = {
101
- envelope,
102
- write: writeAdapter.write.bind(writeAdapter),
103
- http,
104
- log
105
- };
106
- try {
107
- const outcome = await def.run(ctx);
108
- return { ok: true, outcome };
109
- } catch (error) {
110
- return { ok: false, error };
191
+ return rows[0];
192
+ }
193
+ async function seedRow(host, cfg, tokens, nowIso) {
194
+ await adapter.rawQuery(
195
+ `INSERT OR IGNORE INTO ${RATE_LIMIT_TABLE}
196
+ (host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
197
+ VALUES (?, ?, ?, ?, ?, ?)`,
198
+ { params: [host, cfg.requestsPerSecond, cfg.burst, tokens, nowIso, nowIso] }
199
+ );
200
+ }
201
+ return {
202
+ async takeToken(host, cfg, cost) {
203
+ for (let attempt = 0; attempt < maxAttempts; attempt++) {
204
+ const existing = await readRow(host);
205
+ const nowMs = Date.now();
206
+ const nowIso = new Date(nowMs).toISOString();
207
+ if (!existing) {
208
+ await seedRow(host, cfg, cfg.burst, nowIso);
209
+ continue;
210
+ }
211
+ const computed = computeTokenBucket(
212
+ {
213
+ tokensAvailable: Number(existing.tokens_available),
214
+ lastRefillAtMs: Date.parse(existing.last_refill_at),
215
+ backoffUntilMs: existing.backoff_until ? Date.parse(existing.backoff_until) : null
216
+ },
217
+ cfg,
218
+ cost,
219
+ nowMs
220
+ );
221
+ const updated = await adapter.rawQuery(
222
+ `UPDATE ${RATE_LIMIT_TABLE}
223
+ SET tokens_available = ?, last_refill_at = ?, requests_per_second = ?, burst = ?, updated_at = ?
224
+ WHERE host = ?
225
+ AND last_refill_at = ?
226
+ AND IFNULL(backoff_until, '') = IFNULL(?, '')
227
+ RETURNING tokens_available`,
228
+ {
229
+ params: [
230
+ computed.tokensAvailable,
231
+ nowIso,
232
+ cfg.requestsPerSecond,
233
+ cfg.burst,
234
+ nowIso,
235
+ host,
236
+ existing.last_refill_at,
237
+ existing.backoff_until
238
+ ]
239
+ }
240
+ );
241
+ if (!updated.length) continue;
242
+ return { allowed: computed.allowed, waitMs: computed.waitMs };
243
+ }
244
+ throw new Error(`hazo_collect: exhausted retries taking a rate-limit token for host "${host}"`);
245
+ },
246
+ async applyBackoff(host, backoffUntilIso, cfg) {
247
+ const nowIso = (/* @__PURE__ */ new Date()).toISOString();
248
+ await seedRow(host, cfg, 0, nowIso);
249
+ await adapter.rawQuery(
250
+ `UPDATE ${RATE_LIMIT_TABLE}
251
+ SET tokens_available = 0,
252
+ backoff_until = CASE
253
+ WHEN backoff_until IS NULL OR backoff_until < ? THEN ?
254
+ ELSE backoff_until
255
+ END,
256
+ updated_at = ?
257
+ WHERE host = ?`,
258
+ { params: [backoffUntilIso, backoffUntilIso, nowIso, host] }
259
+ );
111
260
  }
261
+ };
262
+ }
263
+ async function createRateLimitStore(adapter) {
264
+ const dialect = await detectDialect(adapter);
265
+ switch (dialect) {
266
+ case "pg":
267
+ return createPgRateLimitStore(adapter);
268
+ case "postgrest":
269
+ return createPostgrestRateLimitStore(adapter);
270
+ case "sqlite":
271
+ return createSqliteRateLimitStore(adapter);
112
272
  }
113
- };
273
+ }
274
+ function hostnameOf(url) {
275
+ try {
276
+ const raw = url instanceof Request ? url.url : url;
277
+ return (typeof raw === "string" ? new URL(raw) : raw).hostname;
278
+ } catch {
279
+ return String(url);
280
+ }
281
+ }
282
+ function createRateLimitManager(opts) {
283
+ const hosts = opts.hosts ?? {};
284
+ const fetchFn = opts.fetchFn ?? fetchWithRequestId2;
285
+ const now = opts.now ?? (() => Date.now());
286
+ const sleepFn = opts.sleep ?? coreSleep;
287
+ const maxWaitMs = opts.maxWaitMs ?? 6e4;
288
+ let storePromise;
289
+ function getStore() {
290
+ if (opts.store) return Promise.resolve(opts.store);
291
+ if (!storePromise) storePromise = createRateLimitStore(opts.adapter);
292
+ return storePromise;
293
+ }
294
+ return {
295
+ createScopedFetch(waitTracker) {
296
+ return async (url, init) => {
297
+ const host = hostnameOf(url);
298
+ const cfg = hosts[host];
299
+ if (!cfg) return fetchFn(url, init);
300
+ const store = await getStore();
301
+ let waitedMs = 0;
302
+ while (true) {
303
+ const { allowed, waitMs } = await store.takeToken(host, cfg, 1);
304
+ if (allowed) break;
305
+ const sleepMs = Math.max(1, waitMs);
306
+ waitedMs += sleepMs;
307
+ if (waitedMs > maxWaitMs) break;
308
+ await sleepFn(sleepMs);
309
+ }
310
+ if (waitTracker) waitTracker.totalWaitMs += waitedMs;
311
+ const response = await fetchFn(url, init);
312
+ if (response.status === 429) {
313
+ const retryAfter = parseRetryAfterMs(response.headers?.get?.("Retry-After") ?? null, now());
314
+ const backoffUntilIso = new Date(now() + (retryAfter ?? DEFAULT_BACKOFF_MS)).toISOString();
315
+ await store.applyBackoff(host, backoffUntilIso, cfg);
316
+ }
317
+ return response;
318
+ };
319
+ }
320
+ };
321
+ }
322
+
323
+ // src/runtime/node.ts
324
+ function createNodeRuntime(options = {}) {
325
+ const baseFetch = options.fetch ?? fetchWithRequestId3;
326
+ const hosts = options.rateLimits ?? {};
327
+ const rateLimitingEnabled = Object.keys(hosts).length > 0;
328
+ const managers = /* @__PURE__ */ new WeakMap();
329
+ function managerFor(adapter) {
330
+ let manager = managers.get(adapter);
331
+ if (!manager) {
332
+ manager = createRateLimitManager({ adapter, hosts, fetchFn: baseFetch });
333
+ managers.set(adapter, manager);
334
+ }
335
+ return manager;
336
+ }
337
+ return {
338
+ async run(def, envelope, adapter) {
339
+ const writeAdapter = createWriteAdapter(adapter);
340
+ const log = createLogger("hazo_collect").child({
341
+ collector: def.manifest.name,
342
+ runId: envelope.run_id
343
+ });
344
+ const waitTracker = { totalWaitMs: 0 };
345
+ const scopedFetch = rateLimitingEnabled ? managerFor(adapter).createScopedFetch(waitTracker) : baseFetch;
346
+ const http = createHttpHelper(envelope.run_id, scopedFetch);
347
+ const ctx = {
348
+ envelope,
349
+ write: writeAdapter.write.bind(writeAdapter),
350
+ http,
351
+ log
352
+ };
353
+ try {
354
+ const outcome = await def.run(ctx);
355
+ return { ok: true, outcome, rateLimitWaitMs: waitTracker.totalWaitMs };
356
+ } catch (error) {
357
+ return { ok: false, error, rateLimitWaitMs: waitTracker.totalWaitMs };
358
+ }
359
+ }
360
+ };
361
+ }
362
+ var nodeRuntime = createNodeRuntime();
114
363
 
115
364
  // src/manager/index.ts
116
365
  async function buildDefaultSecretsProvider() {
@@ -193,7 +442,7 @@ async function upsertWatermark(adapter, plugin, watermark) {
193
442
  updated_at: nowISO()
194
443
  });
195
444
  }
196
- async function upsertHealth(adapter, plugin, status, now) {
445
+ async function upsertHealth(adapter, plugin, status, now, rateLimitWaitMs) {
197
446
  if (status === "success") {
198
447
  const qb = new QueryBuilder2().from("hazo_collect_plugin_health").onConflict(["plugin"]).doUpdate();
199
448
  await adapter.query(qb, "POST", {
@@ -202,7 +451,8 @@ async function upsertHealth(adapter, plugin, status, now) {
202
451
  last_run_at: now,
203
452
  last_status: "success",
204
453
  consecutive_failures: 0,
205
- state: "ok"
454
+ state: "ok",
455
+ last_rate_limit_wait_ms: rateLimitWaitMs
206
456
  });
207
457
  return;
208
458
  }
@@ -213,7 +463,8 @@ async function upsertHealth(adapter, plugin, status, now) {
213
463
  plugin,
214
464
  last_run_at: now,
215
465
  last_status: status,
216
- consecutive_failures: current + 1
466
+ consecutive_failures: current + 1,
467
+ last_rate_limit_wait_ms: rateLimitWaitMs
217
468
  };
218
469
  if (status === "failed") {
219
470
  upsertRow.state = "failed";
@@ -223,7 +474,7 @@ async function upsertHealth(adapter, plugin, status, now) {
223
474
  }
224
475
  function createManager(opts) {
225
476
  const { adapter, registry } = opts;
226
- const runtime = opts.runtime ?? nodeRuntime;
477
+ const runtime = opts.runtime ?? (opts.fetch || opts.rateLimits ? createNodeRuntime({ fetch: opts.fetch, rateLimits: opts.rateLimits }) : nodeRuntime);
227
478
  let _secretsProviderPromise;
228
479
  function getSecretsProvider() {
229
480
  if (opts.secrets) return Promise.resolve(opts.secrets);
@@ -243,6 +494,7 @@ function createManager(opts) {
243
494
  started_at
244
495
  });
245
496
  const dispatchResult = await runtime.run(entry.worker, envelope, adapter);
497
+ const rateLimitWaitMs = dispatchResult.rateLimitWaitMs ?? 0;
246
498
  let result;
247
499
  if (dispatchResult.ok) {
248
500
  const outcome = dispatchResult.outcome;
@@ -283,7 +535,7 @@ function createManager(opts) {
283
535
  if (result.status === "success" && result.watermark) {
284
536
  await upsertWatermark(adapter, envelope.plugin, result.watermark);
285
537
  }
286
- await upsertHealth(adapter, envelope.plugin, result.status, nowISO());
538
+ await upsertHealth(adapter, envelope.plugin, result.status, nowISO(), rateLimitWaitMs);
287
539
  return result;
288
540
  });
289
541
  }
@@ -470,10 +722,15 @@ async function persistRegistry(adapter, entries, opts) {
470
722
  });
471
723
  }
472
724
  export {
725
+ computeTokenBucket,
473
726
  createInMemoryRegistry,
474
727
  createManager,
728
+ createNodeRuntime,
729
+ createRateLimitManager,
730
+ createRateLimitStore,
475
731
  createWriteAdapter,
476
732
  discover,
477
733
  nodeRuntime,
734
+ parseRetryAfterMs,
478
735
  persistRegistry
479
736
  };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "hazo_collect",
3
- "version": "0.2.8",
3
+ "version": "0.3.0",
4
4
  "description": "Collector-manager engine for the Ocdata platform",
5
5
  "type": "module",
6
6
  "module": "./dist/index.js",
@@ -27,6 +27,9 @@
27
27
  },
28
28
  "files": [
29
29
  "dist",
30
+ "config/hazo_collect_config.ini.sample",
31
+ "db_setup_postgres.sql",
32
+ "db_setup_sqlite.sql",
30
33
  "README.md",
31
34
  "CHANGE_LOG.md",
32
35
  "SETUP_CHECKLIST.md"
@@ -43,13 +46,13 @@
43
46
  "ajv": "^8.17.1"
44
47
  },
45
48
  "peerDependencies": {
46
- "hazo_core": "^2.4.0",
47
- "hazo_connect": "^4.3.0",
48
- "hazo_secure": "^1.8.1",
49
+ "hazo_core": "^2.5.0",
50
+ "hazo_connect": "^4.3.5",
51
+ "hazo_secure": "^1.9.1",
49
52
  "react": "^18.0.0 || ^19.0.0",
50
53
  "react-dom": "^18.0.0 || ^19.0.0",
51
- "hazo_ui": "^6.12.0",
52
- "hazo_i18n": "^0.5.0"
54
+ "hazo_ui": "^6.21.0",
55
+ "hazo_i18n": "^0.17.0"
53
56
  },
54
57
  "peerDependenciesMeta": {
55
58
  "hazo_secure": {
@@ -78,9 +81,9 @@
78
81
  "@types/node": "^22.10.0",
79
82
  "@types/react": "^19.0.0",
80
83
  "@types/react-dom": "^19.0.0",
81
- "hazo_core": "^2.4.0",
82
- "hazo_connect": "^4.3.0",
83
- "hazo_secure": "^1.8.1"
84
+ "hazo_core": "^2.5.0",
85
+ "hazo_connect": "^4.3.5",
86
+ "hazo_secure": "^1.9.1"
84
87
  },
85
88
  "keywords": [],
86
89
  "author": "Pubs Abayasiri",