hazo_collect 0.2.7 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGE_LOG.md CHANGED
@@ -1,5 +1,98 @@
1
1
  # hazo_collect — Change Log
2
2
 
3
+ ## 0.3.0 — 2026-11-21 (faraday data-foundation ticket 05: shared per-host rate limit)
4
+
5
+ ### ⚠️ DDL required
6
+
7
+ New table `hazo_collect_rate_limits` (one token-bucket row per host) plus two
8
+ new Postgres RPC functions, `hazo_collect_take_token_rpc` and
9
+ `hazo_collect_apply_backoff_rpc`, and one new column on
10
+ `hazo_collect_plugin_health` (`last_rate_limit_wait_ms`, default 0).
11
+ Re-run `db_setup_postgres.sql` / `db_setup_sqlite.sql` (idempotent) or apply
12
+ `src/ddl/postgres.sql` / `src/ddl/sqlite.sql` directly. SQLite has no
13
+ RPC/plpgsql, so its atomic "take a token" step is an optimistic
14
+ compare-and-swap retry loop over the same table instead of a stored
15
+ function (see `src/server/rate-limit.ts`, `createSqliteRateLimitStore`) —
16
+ same technique hazo_jobs 0.25.0 used for its SQLite group-completion CAS.
17
+ PostgREST consumers must `GRANT EXECUTE` on both new functions to the role
18
+ PostgREST connects as — see README's "PostgREST setup" note.
19
+
20
+ ### Added
21
+
22
+ - **Shared, DB-backed per-host token-bucket rate limiting inside `ctx.http`.**
23
+ Consumers configure `requestsPerSecond` + `burst` per host via
24
+ `createManager({ rateLimits: { 'api.example.com': { requestsPerSecond: 5,
25
+ burst: 10 } } })`. A host absent from that map keeps today's unthrottled
26
+ behaviour (no limiting), so this is strictly additive/opt-in. The bucket
27
+ state lives in the database (`hazo_collect_rate_limits`) and is shared by
28
+ every worker/process collecting through it — not just this process's view
29
+ of the limit.
30
+ - **Postgres**: taking a token is one atomic statement,
31
+ `hazo_collect_take_token_rpc` — a single `UPDATE ... FROM (SELECT ...
32
+ FOR UPDATE) ... RETURNING`, so no lock is ever held across requests
33
+ (safe to call via PostgREST's `POST /rpc/hazo_collect_take_token_rpc`,
34
+ not just a raw SQL connection).
35
+ - **SQLite**: the same refill-on-read token-bucket math
36
+ (`computeTokenBucket` in `src/server/rate-limit.ts`) runs in an
37
+ optimistic CAS retry loop over the raw adapter, since SQLite has no
38
+ RPC/plpgsql equivalent.
39
+ - **429 + `Retry-After`** on a rate-limited host trigger backoff: the
40
+ shared bucket is drained and `backoff_until` is raised (monotonically)
41
+ via `hazo_collect_apply_backoff_rpc` / the SQLite equivalent, so every
42
+ caller sharing that host's bucket backs off — not just the process that
43
+ saw the 429.
44
+ - **Collector health** (`hazo_collect_plugin_health.last_rate_limit_wait_ms`)
45
+ records how long the most recent run waited on the limit.
46
+ - **`ManagerOptions.fetch`** — an injected fetch implementation used by
47
+ `ctx.http` in every run the manager dispatches. Lets consumers (and this
48
+ package's own tests) point collectors at a fake host while per-host rate
49
+ limiting still applies, without touching real network code.
50
+ `ManagerOptions.rateLimits` is the per-host config described above; both
51
+ are threaded through a new `createNodeRuntime({ fetch, rateLimits })`
52
+ factory (`nodeRuntime` stays the pre-0.3.0 default instance — no
53
+ behaviour change when neither option is passed).
54
+ - New `hazo_collect/server` exports: `createNodeRuntime`,
55
+ `createRateLimitManager`, `createRateLimitStore`, `computeTokenBucket`,
56
+ `parseRetryAfterMs`, and types `HostRateLimitConfig`, `RateLimitManager`,
57
+ `RateLimitStore`, `TakeTokenResult`, `FetchFn`, `WaitTracker`,
58
+ `NodeRuntimeOptions`.
59
+ - `config/hazo_collect_config.ini.sample` documents the per-host rate-limit
60
+ config shape (`[rate_limit:<host>]`) consumers should map into
61
+ `ManagerOptions.rateLimits` — this package has no ini-loading code of its
62
+ own, so the section is documentation, not a parsed config file.
63
+
64
+ ### Scope notes
65
+
66
+ - `RuntimeAdapter.run()`'s return type gained a required `rateLimitWaitMs`
67
+ field (present on both the `ok: true` and `ok: false` branches). This is
68
+ additive for any existing `RuntimeAdapter` implementation that already
69
+ destructures `{ ok, outcome }` / `{ ok, error }` — it simply won't read
70
+ the new field.
71
+
72
+ ## 0.2.8 — 2026-09-19
73
+
74
+ ### Added
75
+
76
+ Future-proofing i18n plumbing only — no behavior change. hazo_collect is a "zero-string" package
77
+ (headless collector-manager engine, no rendered UI; the string literals in the codebase are
78
+ internal error messages for plugin/collector authors, not end-user copy), so this wires the
79
+ canonical `hazo_core/i18n` bridge without adding any real strings yet:
80
+
81
+ - `src/locales/en.json` — empty flat catalog (`{}`).
82
+ - `src/lib/index.ts` now exports `t = createPackageT('hazo_collect', en)`, re-exported from the
83
+ main entry point. Loads the catalog via `createRequire` rather than
84
+ `import en from '../locales/en.json' with { type: 'json' }` — this package's `tsconfig.json` uses
85
+ `"module": "Node16"`, which throws TS2823 on import attributes syntax (a known incompatibility
86
+ found during the workspace's i18n adoption pass). `tsup.config.ts` gained a `lib/index` build
87
+ entry and copies `src/locales/en.json` to `dist/locales/en.json`. The `createRequire` lookup
88
+ tries a couple of relative depths (`../locales/en.json`, `./locales/en.json`,
89
+ `../../locales/en.json`) rather than one hardcoded path, since tsup's code-splitting extracts
90
+ this module into a shared chunk whose on-disk depth doesn't reliably match the source file's —
91
+ confirmed empirically against a real build. Falls back to `{}` rather than throwing.
92
+ - `hazo_i18n` added to `peerDependencies` (`^0.5.0`) and marked optional in
93
+ `peerDependenciesMeta` — `hazo_core` was already a required peer.
94
+ - Added an `hazo_collect.i18n` autotest scenario group (one trivial case) to the test-app.
95
+
3
96
  ## Unreleased — test tooling
4
97
 
5
98
  ### Fixed
package/README.md CHANGED
@@ -75,12 +75,17 @@ const result = await manager.runNow({ plugin: 'my_source' });
75
75
 
76
76
  | Export | Description |
77
77
  |---|---|
78
- | `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist |
79
- | `nodeRuntime` | Execution adapter for in-process Node.js collectors |
78
+ | `createManager(opts)` | Main orchestration pipeline — resolve → validate → run → persist. Accepts `fetch` + `rateLimits` (see "Per-host rate limiting" above) |
79
+ | `nodeRuntime` | Default execution adapter for in-process Node.js collectors (no fetch override, no rate limits) |
80
+ | `createNodeRuntime({ fetch, rateLimits })` | Build a `RuntimeAdapter` with an injected fetch and/or per-host rate limits |
80
81
  | `createInMemoryRegistry()` | In-memory registry backed by the SDK Map |
81
82
  | `persistRegistry(adapter, manifests)` | Upsert manifests into `hazo_collect_plugin_registry`; quarantines invalid entries |
82
83
  | `discover(opts)` | Scan directories for `manifest.json` files and return parsed entries |
83
84
  | `createWriteAdapter(adapter, runId)` | Create a `ctx.write` function bound to a specific run |
85
+ | `createRateLimitManager(opts)` | Build the `ctx.http` rate-limiting wrapper directly (advanced — `createManager`'s `rateLimits`/`fetch` cover the common case) |
86
+ | `createRateLimitStore(adapter)` | Pick the dialect-appropriate (`pg` / `postgrest` / `sqlite`) atomic token-bucket store for an adapter |
87
+ | `computeTokenBucket(state, cfg, cost, nowMs)` | Pure refill-on-read token-bucket math, shared by the SQLite CAS loop |
88
+ | `parseRetryAfterMs(header, nowMs)` | Parse a `Retry-After` header (seconds or HTTP-date) into a millisecond delay |
84
89
 
85
90
  ### `hazo_collect/sdk`
86
91
 
@@ -109,8 +114,70 @@ Run migrations from `hazo_collect/ddl/sqlite.sql` (SQLite) or `hazo_collect/ddl/
109
114
 
110
115
  - `hazo_collect_plugin_registry` — discovered manifest catalogue with validity + quarantine info
111
116
  - `hazo_collect_plugin_runs` — per-run scorecard (status, records, contract_version, watermark)
112
- - `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status)
117
+ - `hazo_collect_plugin_health` — rolling health state (consecutive failures, last status, last rate-limit wait)
113
118
  - `hazo_collect_landing` — idempotent landing zone for collector output rows
119
+ - `hazo_collect_rate_limits` — shared per-host token-bucket state (0.3.0+; see below)
120
+
121
+ ## Per-host rate limiting (`ctx.http`)
122
+
123
+ Every collector's `ctx.http` can enforce a shared, database-backed per-host
124
+ token bucket. Configure `requestsPerSecond` + `burst` per host on the
125
+ manager; a host with no entry keeps today's unthrottled behaviour:
126
+
127
+ ```ts
128
+ import { createManager } from 'hazo_collect/server';
129
+
130
+ const manager = createManager({
131
+ adapter,
132
+ registry,
133
+ rateLimits: {
134
+ 'api.example.com': { requestsPerSecond: 5, burst: 10 },
135
+ },
136
+ });
137
+ ```
138
+
139
+ Because the bucket lives in `hazo_collect_rate_limits` (not in process
140
+ memory), the limit is shared across every worker/process collecting through
141
+ the same database — not just this process's view of it. A `429` response
142
+ (honouring `Retry-After`, in seconds or as an HTTP-date) drains the shared
143
+ bucket and raises its backoff window, so every caller sharing that host
144
+ backs off, not just the one that saw the 429. `hazo_collect_plugin_health`
145
+ records how long the most recent run waited on the limit
146
+ (`last_rate_limit_wait_ms`).
147
+
148
+ ### Injected fetch
149
+
150
+ `createManager({ fetch })` lets you point `ctx.http` at a different fetch
151
+ implementation for every run the manager dispatches — the rate limiter still
152
+ applies on top of it. This is how both this package's own tests and a
153
+ consumer's tests can exercise `ctx.http` against a fake host without any
154
+ real network traffic:
155
+
156
+ ```ts
157
+ const manager = createManager({
158
+ adapter,
159
+ registry,
160
+ fetch: fakeFetch, // e.g. a jest.fn() resolving a canned Response
161
+ rateLimits: { 'fake-host.test': { requestsPerSecond: 10, burst: 10 } },
162
+ });
163
+ ```
164
+
165
+ `fetch` and `rateLimits` are ignored if you also pass your own `runtime`
166
+ (that runtime owns its own fetch wiring) — use `createNodeRuntime({ fetch,
167
+ rateLimits })` directly in that case.
168
+
169
+ ### PostgREST setup
170
+
171
+ If you run `createHazoConnect({ type: 'postgrest', ... })`, the rate limiter
172
+ calls the two new functions via `POST /rpc/<fn>`. Grant execute to the role
173
+ PostgREST connects as:
174
+
175
+ ```sql
176
+ GRANT EXECUTE ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) TO api_user;
177
+ GRANT EXECUTE ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) TO api_user;
178
+ ```
179
+
180
+ (substitute the role PostgREST actually connects as for `api_user`).
114
181
 
115
182
  ## Tailwind v4 (`@source` required)
116
183
 
@@ -0,0 +1,44 @@
1
+ ; hazo_collect configuration
2
+ ; Copy to hazo_collect_config.ini and fill in values for your environment.
3
+
4
+ ; [env]
5
+ ; Uncomment to hard-code the environment name (overrides HAZO_ENV / NODE_ENV).
6
+ ; env = development
7
+
8
+ ; [secrets]
9
+ ; Prefix for environment variables that hold plugin secrets.
10
+ ; A manifest declaring secrets = ["API_KEY"] will read env var ${env_prefix}_API_KEY.
11
+ ; env_prefix = HAZO_COLLECT_SECRET
12
+
13
+ ; [plugins]
14
+ ; Directory to scan for plugin manifests (future folder-scan discovery provider).
15
+ ; Each sub-directory should contain a manifest.json.
16
+ ; dir = ./plugins
17
+
18
+ ; [general]
19
+ ; Example setting
20
+ ; some_key = some_value
21
+
22
+ ; [rate_limit:<host>]
23
+ ; Per-host token-bucket rate limit enforced inside ctx.http (0.3.0+). One
24
+ ; section per host you want throttled — a host with no section here keeps
25
+ ; today's unthrottled behaviour. The bucket itself lives in the database
26
+ ; (hazo_collect_rate_limits), shared by every worker/process collecting
27
+ ; through it, so this config only needs to be loaded wherever createManager()
28
+ ; is called — it does not need to agree across processes.
29
+ ;
30
+ ; This file isn't parsed by hazo_collect itself (there's no ini-loading code
31
+ ; in this package yet) — map whatever ini/env/JSON config your app already
32
+ ; loads into the shape createManager() expects:
33
+ ;
34
+ ; createManager({
35
+ ; adapter,
36
+ ; registry,
37
+ ; rateLimits: {
38
+ ; 'api.example.com': { requestsPerSecond: 5, burst: 10 },
39
+ ; },
40
+ ; });
41
+ ;
42
+ ; [rate_limit:api.example.com]
43
+ ; requests_per_second = 5
44
+ ; burst = 10
@@ -0,0 +1,189 @@
1
+ -- hazo_collect — PostgreSQL DDL
2
+ -- Consumers MAY copy this verbatim or wrap with their own role grants.
3
+ -- Idempotent — safe to re-run.
4
+ --
5
+ -- Tables:
6
+ -- hazo_collect_plugin_runs — one row per plugin run (RunResult scorecard)
7
+ -- hazo_collect_plugin_health — latest health snapshot per plugin
8
+ -- hazo_collect_plugin_registry — discovered manifest catalogue
9
+ -- hazo_collect_watermarks — per-plugin incremental watermark
10
+ -- hazo_collect_landing — raw as-fetched payloads (idempotent on plugin+key)
11
+ -- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+)
12
+
13
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
14
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
15
+ plugin TEXT NOT NULL,
16
+ run_id TEXT NOT NULL UNIQUE,
17
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
18
+ started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
19
+ completed_at TIMESTAMPTZ,
20
+ status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
21
+ records_fetched INT NOT NULL DEFAULT 0,
22
+ records_written INT NOT NULL DEFAULT 0,
23
+ watermark TEXT,
24
+ errors JSONB NOT NULL DEFAULT '[]'::jsonb,
25
+ metrics JSONB,
26
+ correlation_id TEXT,
27
+ env TEXT
28
+ );
29
+
30
+ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
31
+ ON hazo_collect_plugin_runs (plugin, started_at DESC);
32
+
33
+ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_run_id_idx
34
+ ON hazo_collect_plugin_runs (run_id);
35
+
36
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
37
+ plugin TEXT PRIMARY KEY,
38
+ last_success_at TIMESTAMPTZ,
39
+ last_run_at TIMESTAMPTZ,
40
+ last_status TEXT,
41
+ state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
42
+ consecutive_failures INT NOT NULL DEFAULT 0,
43
+ last_rate_limit_wait_ms INT NOT NULL DEFAULT 0
44
+ );
45
+
46
+ -- 0.3.0: existing deployments — add the new column if upgrading in place.
47
+ ALTER TABLE hazo_collect_plugin_health
48
+ ADD COLUMN IF NOT EXISTS last_rate_limit_wait_ms INT NOT NULL DEFAULT 0;
49
+
50
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
51
+ name TEXT PRIMARY KEY,
52
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
53
+ version TEXT NOT NULL,
54
+ runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
55
+ manifest JSONB NOT NULL,
56
+ source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
57
+ discovered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
58
+ valid BOOLEAN NOT NULL DEFAULT TRUE,
59
+ quarantine_reason TEXT
60
+ );
61
+
62
+ CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
63
+ plugin TEXT PRIMARY KEY,
64
+ watermark TEXT NOT NULL,
65
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
66
+ );
67
+
68
+ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
69
+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
70
+ plugin TEXT NOT NULL,
71
+ run_id TEXT NOT NULL,
72
+ idempotency_key TEXT NOT NULL,
73
+ payload JSONB NOT NULL,
74
+ fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
75
+ window_since TEXT,
76
+ window_until TEXT,
77
+ CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
78
+ );
79
+
80
+ CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
81
+ ON hazo_collect_landing (plugin, fetched_at DESC);
82
+
83
+ -- ===========================================================================
84
+ -- Shared per-host rate limiting (0.3.0+) — see src/ddl/postgres.sql for the
85
+ -- full design notes. Mirrored here per repo convention.
86
+ -- ===========================================================================
87
+
88
+ CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
89
+ host TEXT PRIMARY KEY,
90
+ requests_per_second DOUBLE PRECISION NOT NULL,
91
+ burst INT NOT NULL,
92
+ tokens_available DOUBLE PRECISION NOT NULL,
93
+ last_refill_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
94
+ backoff_until TIMESTAMPTZ,
95
+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
96
+ );
97
+
98
+ DROP FUNCTION IF EXISTS hazo_collect_take_token_rpc(text, double precision, int, int);
99
+
100
+ CREATE OR REPLACE FUNCTION hazo_collect_take_token_rpc(
101
+ p_host text,
102
+ p_requests_per_second double precision,
103
+ p_burst int,
104
+ p_cost int DEFAULT 1
105
+ ) RETURNS TABLE(allowed boolean, wait_ms int)
106
+ LANGUAGE plpgsql
107
+ SECURITY DEFINER
108
+ AS $$
109
+ DECLARE
110
+ now_ts timestamptz := clock_timestamp();
111
+ BEGIN
112
+ INSERT INTO hazo_collect_rate_limits
113
+ (host, requests_per_second, burst, tokens_available, last_refill_at, updated_at)
114
+ VALUES (p_host, p_requests_per_second, p_burst, p_burst, now_ts, now_ts)
115
+ ON CONFLICT (host) DO NOTHING;
116
+
117
+ RETURN QUERY
118
+ UPDATE hazo_collect_rate_limits r
119
+ SET tokens_available = CASE
120
+ WHEN old.is_allowed THEN old.refilled - p_cost
121
+ ELSE old.refilled
122
+ END,
123
+ last_refill_at = now_ts,
124
+ requests_per_second = p_requests_per_second,
125
+ burst = p_burst,
126
+ updated_at = now_ts
127
+ FROM (
128
+ SELECT
129
+ host,
130
+ LEAST(
131
+ p_burst::double precision,
132
+ tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
133
+ ) AS refilled,
134
+ backoff_until,
135
+ (backoff_until IS NULL OR backoff_until <= now_ts)
136
+ AND LEAST(
137
+ p_burst::double precision,
138
+ tokens_available + EXTRACT(EPOCH FROM (now_ts - last_refill_at)) * p_requests_per_second
139
+ ) >= p_cost AS is_allowed
140
+ FROM hazo_collect_rate_limits
141
+ WHERE host = p_host
142
+ FOR UPDATE
143
+ ) AS old
144
+ WHERE r.host = old.host
145
+ RETURNING
146
+ old.is_allowed AS allowed,
147
+ CASE
148
+ WHEN old.is_allowed THEN 0
149
+ WHEN old.backoff_until IS NOT NULL AND old.backoff_until > now_ts
150
+ THEN CEIL(EXTRACT(EPOCH FROM (old.backoff_until - now_ts)) * 1000)::int
151
+ WHEN p_requests_per_second > 0
152
+ THEN CEIL((p_cost - old.refilled) / p_requests_per_second * 1000)::int
153
+ ELSE 2147483647
154
+ END AS wait_ms;
155
+ END;
156
+ $$;
157
+
158
+ COMMENT ON FUNCTION hazo_collect_take_token_rpc(text, double precision, int, int) IS
159
+ 'hazo_collect: atomically take a rate-limit token for a host (refill-on-read token bucket). '
160
+ 'Callable via PostgREST POST /rpc/hazo_collect_take_token_rpc. GRANT EXECUTE to the PostgREST role.';
161
+
162
+ DROP FUNCTION IF EXISTS hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int);
163
+
164
+ CREATE OR REPLACE FUNCTION hazo_collect_apply_backoff_rpc(
165
+ p_host text,
166
+ p_backoff_until timestamptz,
167
+ p_requests_per_second double precision,
168
+ p_burst int
169
+ ) RETURNS void
170
+ LANGUAGE plpgsql
171
+ SECURITY DEFINER
172
+ AS $$
173
+ BEGIN
174
+ INSERT INTO hazo_collect_rate_limits
175
+ (host, requests_per_second, burst, tokens_available, last_refill_at, backoff_until, updated_at)
176
+ VALUES (p_host, p_requests_per_second, p_burst, 0, clock_timestamp(), p_backoff_until, clock_timestamp())
177
+ ON CONFLICT (host) DO UPDATE
178
+ SET tokens_available = 0,
179
+ backoff_until = GREATEST(hazo_collect_rate_limits.backoff_until, EXCLUDED.backoff_until),
180
+ updated_at = clock_timestamp();
181
+ END;
182
+ $$;
183
+
184
+ COMMENT ON FUNCTION hazo_collect_apply_backoff_rpc(text, timestamptz, double precision, int) IS
185
+ 'hazo_collect: record a 429 + Retry-After for a host — drains the shared bucket and raises '
186
+ 'backoff_until so every caller sharing this host backs off. Callable via PostgREST POST '
187
+ '/rpc/hazo_collect_apply_backoff_rpc. GRANT EXECUTE to the PostgREST role.';
188
+
189
+ NOTIFY pgrst, 'reload schema';
@@ -0,0 +1,87 @@
1
+ -- hazo_collect — SQLite DDL
2
+ -- Idempotent — safe to re-run.
3
+ --
4
+ -- SQLite companion to postgres.sql.
5
+ -- Type mapping: UUID→TEXT, BOOLEAN→INTEGER(0/1), TIMESTAMPTZ→TEXT, JSONB→TEXT.
6
+ --
7
+ -- Tables:
8
+ -- hazo_collect_plugin_runs
9
+ -- hazo_collect_plugin_health
10
+ -- hazo_collect_plugin_registry
11
+ -- hazo_collect_watermarks
12
+ -- hazo_collect_landing
13
+ -- hazo_collect_rate_limits — shared per-host token-bucket state (0.3.0+);
14
+ -- see src/ddl/sqlite.sql for the CAS-loop design note.
15
+
16
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_runs (
17
+ id TEXT PRIMARY KEY,
18
+ plugin TEXT NOT NULL,
19
+ run_id TEXT NOT NULL UNIQUE,
20
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
21
+ started_at TEXT NOT NULL DEFAULT (datetime('now')),
22
+ completed_at TEXT,
23
+ status TEXT NOT NULL CHECK (status IN ('success', 'partial', 'failed', 'running')),
24
+ records_fetched INTEGER NOT NULL DEFAULT 0,
25
+ records_written INTEGER NOT NULL DEFAULT 0,
26
+ watermark TEXT,
27
+ errors TEXT NOT NULL DEFAULT '[]',
28
+ metrics TEXT,
29
+ correlation_id TEXT,
30
+ env TEXT
31
+ );
32
+
33
+ CREATE INDEX IF NOT EXISTS hazo_collect_plugin_runs_plugin_idx
34
+ ON hazo_collect_plugin_runs (plugin, started_at DESC);
35
+
36
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_health (
37
+ plugin TEXT PRIMARY KEY,
38
+ last_success_at TEXT,
39
+ last_run_at TEXT,
40
+ last_status TEXT,
41
+ state TEXT NOT NULL DEFAULT 'ok' CHECK (state IN ('ok', 'stale', 'failed')),
42
+ consecutive_failures INTEGER NOT NULL DEFAULT 0,
43
+ last_rate_limit_wait_ms INTEGER NOT NULL DEFAULT 0
44
+ );
45
+
46
+ CREATE TABLE IF NOT EXISTS hazo_collect_plugin_registry (
47
+ name TEXT PRIMARY KEY,
48
+ kind TEXT NOT NULL CHECK (kind IN ('source', 'sink')),
49
+ version TEXT NOT NULL,
50
+ runtime TEXT NOT NULL CHECK (runtime IN ('node', 'python')),
51
+ manifest TEXT NOT NULL,
52
+ source TEXT NOT NULL DEFAULT 'folder' CHECK (source IN ('folder', 'db')),
53
+ discovered_at TEXT NOT NULL DEFAULT (datetime('now')),
54
+ valid INTEGER NOT NULL DEFAULT 1,
55
+ quarantine_reason TEXT
56
+ );
57
+
58
+ CREATE TABLE IF NOT EXISTS hazo_collect_watermarks (
59
+ plugin TEXT PRIMARY KEY,
60
+ watermark TEXT NOT NULL,
61
+ updated_at TEXT NOT NULL DEFAULT (datetime('now'))
62
+ );
63
+
64
+ CREATE TABLE IF NOT EXISTS hazo_collect_landing (
65
+ id TEXT PRIMARY KEY,
66
+ plugin TEXT NOT NULL,
67
+ run_id TEXT NOT NULL,
68
+ idempotency_key TEXT NOT NULL,
69
+ payload TEXT NOT NULL,
70
+ fetched_at TEXT NOT NULL DEFAULT (datetime('now')),
71
+ window_since TEXT,
72
+ window_until TEXT,
73
+ CONSTRAINT hazo_collect_landing_plugin_key UNIQUE (plugin, idempotency_key)
74
+ );
75
+
76
+ CREATE INDEX IF NOT EXISTS hazo_collect_landing_plugin_idx
77
+ ON hazo_collect_landing (plugin, fetched_at DESC);
78
+
79
+ CREATE TABLE IF NOT EXISTS hazo_collect_rate_limits (
80
+ host TEXT PRIMARY KEY,
81
+ requests_per_second REAL NOT NULL,
82
+ burst INTEGER NOT NULL,
83
+ tokens_available REAL NOT NULL,
84
+ last_refill_at TEXT NOT NULL DEFAULT (datetime('now')),
85
+ backoff_until TEXT,
86
+ updated_at TEXT NOT NULL DEFAULT (datetime('now'))
87
+ );
@@ -0,0 +1,19 @@
1
+ // src/lib/index.ts
2
+ import { createRequire } from "module";
3
+ import { createPackageT } from "hazo_core/i18n";
4
+ function loadEnCatalog() {
5
+ const req = createRequire(import.meta.url);
6
+ for (const rel of ["../locales/en.json", "./locales/en.json", "../../locales/en.json"]) {
7
+ try {
8
+ return req(rel);
9
+ } catch {
10
+ }
11
+ }
12
+ return {};
13
+ }
14
+ var en = loadEnCatalog();
15
+ var t = createPackageT("hazo_collect", en);
16
+
17
+ export {
18
+ t
19
+ };