@flowsint/plugin-sdk 2.6.1 → 2.6.2

package/README.md

@@ -1,374 +1,206 @@
-# Flowsint enricher plugins (`@flowsint/plugin-sdk`)
+# @flowsint/plugin-sdk
 
-The SDK for Flowsint local enricher plugins: the plugin contract and one
-TypeScript interface per Flowsint entity type. Types only.
+Write Flowsint enricher plugins with typed entities. Import `Domain`, `Ip`,
+`Email` and every other Flowsint entity type, plus the plugin contract, so
+a wrong field name is a compile error instead of an empty node in the graph.
+Types only: nothing ships at runtime.
+
+## Install
 
 ```sh
-npm i -D @flowsint/plugin-sdk@<Flowsint version you target>
+npm i -D @flowsint/plugin-sdk
 ```
 
-The package version is the Flowsint version. A plugin built against
-`@flowsint/plugin-sdk@X.Y.Z` declares `"engines": { "flowsint": "^X.Y.Z" }`:
-removing or renaming a spec field only ships in a major release. This
-guarantee covers the published types only, not the app's runtime
-handling of what a plugin returns.
-
-A plugin is a directory dropped into Flowsint's plugins folder
-(`~/Library/Application Support/Flowsint/plugins/` on macOS,
-`%APPDATA%/Flowsint/plugins/` on Windows,
-`~/.config/Flowsint/plugins/` on Linux), containing:
-
-- `package.json` - see shape below.
-- `README.md` - **required**. Shown in Settings -> Plugins ("README"
-  button, rendered as markdown). Cover: what the dependency is, how to
-  install it, and your inputs/outputs contract. The section below
-  shows the layout Flowsint's own bundled plugins use.
-- Whatever `package.json`'s `"main"` points to. Your bundled entry
-  file, exporting `run` per the signature below.
-- `node_modules/`. Only if your bundler couldn't inline a dependency
-  (e.g. a native addon).
-
-Flowsint never runs `npm install` for you. Bundle your dependencies
-(esbuild, tsup, whatever) before distributing - your plugin must work
-by `require()` alone, on a machine with no package manager.
-
-## package.json
+## Quick start
+
+A plugin is a folder with a `package.json`, a `README.md` and one bundled
+CommonJS file. This one resolves a domain to its IP address.
+
+`package.json`:
 
 ```json
 {
-  "name": "my-plugin",
+  "name": "my-domain-to-ip",
   "version": "1.0.0",
   "main": "dist/index.js",
-  "author": "you",
   "engines": { "flowsint": "^2.6.1" },
   "flowsint": {
     "category": "Domain",
-    "description": "What this does, one sentence.",
+    "source": "dns",
     "inputs": { "type": "domain", "properties": [{ "name": "domain", "type": "string" }] },
-    "outputs": { "type": "ip", "properties": [{ "name": "address", "type": "string" }] },
-    "icon": "Globe",
-    "installUrl": "https://example.com/install-docs",
-    "params": [
-      {
-        "name": "LIMIT",
-        "type": "string",
-        "description": "Max results",
-        "default": "50",
-        "required": false
-      },
-      {
-        "name": "API_KEY",
-        "type": "vaultSecret",
-        "description": "Your API key",
-        "default": "",
-        "required": true
-      }
-    ]
+    "outputs": { "type": "ip", "properties": [{ "name": "address", "type": "string" }] }
   },
-  "dependencies": {}
+  "scripts": {
+    "build": "esbuild src/index.ts --bundle --platform=node --format=cjs --target=node20 --outfile=dist/index.js"
+  }
 }
 ```
 
-- `name` is your plugin's id - must be unique across every installed
-  plugin. Two plugins with the same `name`: the one Flowsint finds
-  first (alphabetical by directory name) loads, the other is skipped
-  with an error visible in Settings → Plugins.
-- `engines.flowsint` is a normal semver range, checked against the
-  running app version. Omit it to skip the check (not recommended).
-- `flowsint.category` and `flowsint.inputs` are required. Everything
-  else in the `flowsint` key is optional.
-- `flowsint.icon` is a [lucide-react](https://lucide.dev/icons/) icon
-  name.
-- `flowsint.installUrl` - link to your dependency's install docs, shown
-  as a clickable link in Settings → Plugins. Omit it if your plugin has
-  no external dependency (pure Node/OS APIs only). This is static
-  metadata, unrelated to `init()`'s `fix` message below - see "Checking
-  a dependency."
-- `flowsint.params` - optional. An array of
-  `{ name, type, description, default, required }` entries, shown
-  in the launch settings drawer before your plugin runs. A
-  `type: "vaultSecret"` entry is resolved against Flowsint's local
-  vault (the same store backing the chat feature's provider keys)
-  before your `run()` is called - you receive the decrypted secret
-  as a plain string in `params[name]` (or `null` if it's optional and
-  the user never set one), never a key id or reference.
-  `type: "select"` renders a dropdown of its
-  `options: [{ "label": "...", "value": "..." }]` (malformed entries
-  are dropped) and passes the picked `value`. `type: "date"` renders a
-  calendar and passes a `Date` at midnight UTC of the picked day
-  (`undefined` if left empty). `type: "number"` renders a numeric
-  input and passes a `number` (`undefined` if left empty).
-  `type: "boolean"` renders a switch and passes a `boolean` (`false`
-  if never touched). `type: "list"` renders a tags input and passes a
-  `string[]` (`[]` if empty). An invalid value fails the run before
-  `run()` is called. Any other `type` is passed through as the raw string the
-  user typed.
-
-## `main`'s export
-
-Two things your entry file can export - `run` is required, `init` is
-optional, nothing else is read. The contract lives in
-this package's `plugin.d.ts`, the one definition the app's plugin
-runner also imports. Shown here without
-its doc comments:
+`src/index.ts`:
 
 ```ts
-interface EnricherEntityInput {
-  type: string
-  properties: Record<string, unknown>
-}
-
-interface LocalEnricherNode {
-  id: string
-  nodeType: string
-  nodeProperties: Record<string, unknown> // typed per nodeType, see below
-  nodeLabel?: string // falls back to `nodeType`
-  nodeSize?: number // falls back to the app's default size
-  sources?: string[] // overrides the finding's `source`
-}
-
-interface LocalEnricherEdge {
-  id: string
-  source: string
-  target: string
-  label: string
-  sources?: string[]
-}
-
-interface LocalEnricherEvent {
-  timestamp: number
-  endTimestamp?: number | null
-  title: string
-  description?: string | null
-  type?: string
-  confidence?: number | null
-  nodeIds?: string[] // ids of nodes in this same chunk
-  tags?: string[]
-  properties?: Record<string, unknown>
-}
-
-interface LocalEnricherFinding {
-  nodes: LocalEnricherNode[]
-  edges: LocalEnricherEdge[]
-  events?: LocalEnricherEvent[]
+import { randomUUID } from 'node:crypto'
+import { lookup } from 'node:dns/promises'
+import type { Domain, Ip, LocalEnricherRun } from '@flowsint/plugin-sdk'
+
+export const run: LocalEnricherRun = async function* (entities, logger) {
+  for (const entity of entities) {
+    if (entity.type !== 'domain') continue
+    const domain: Domain = { domain: String(entity.properties.domain) }
+    const ip: Ip = { address: (await lookup(domain.domain)).address }
+    logger.info(`${domain.domain} resolves to ${ip.address}`)
+    const [domainId, ipId] = [randomUUID(), randomUUID()]
+    yield {
+      nodes: [
+        { id: domainId, nodeType: 'domain', nodeProperties: domain },
+        { id: ipId, nodeType: 'ip', nodeProperties: ip }
+      ],
+      edges: [{ id: randomUUID(), source: domainId, target: ipId, label: 'RESOLVES_TO' }]
+    }
+  }
 }
+```
 
-interface LocalEnricherLogger {
-  info(message: string): void
-  warn(message: string): void
-  error(message: string): void
-  success(message: string): void
-}
+Build it (`npm i -D @flowsint/plugin-sdk esbuild` first), then install it:
 
-// An async generator (`async function* run`): each yielded finding is a
-// chunk, merged into the graph as soon as it is yielded.
-function run(
-  entities: EnricherEntityInput[],
-  logger: LocalEnricherLogger,
-  params: Record<string, unknown>,
-  signal: AbortSignal
-): AsyncIterable<LocalEnricherFinding>
+```sh
+npm run build
 ```
 
-Yield one chunk per entity: the entity's node, what you found for it
-and the edges between them. **An edge's `source` and `target` (and an
-event's `nodeIds`) must be nodes of the same chunk.** When results are
-shared across entities, yield everything once at the end instead.
-Merging a chunk twice is a no-op.
+1. Copy the folder (`package.json`, `README.md`, `dist/`) into the plugins
+   folder: `~/Library/Application Support/Flowsint/plugins/` (macOS),
+   `%APPDATA%/Flowsint/plugins/` (Windows), `~/.config/Flowsint/plugins/`
+   (Linux).
+2. In Flowsint, open Settings -> Plugins and click "Reload plugins". Your
+   plugin shows `loaded`.
+3. Add a domain to a sketch and run your plugin on it. Your `logger` calls
+   appear in the sketch's logs.
 
-`signal` aborts after one hour: Flowsint stops reading your chunks
-and fails the run, keeping the chunks already merged. Pass it to
-`spawn(cmd, args, { signal })` and `fetch(url, { signal })`, and call
-`signal.throwIfAborted()` in a per-entity `catch` so an abort ends the
-run instead of being logged as one failed entity.
+## Entity types
 
-`entities` are not narrowed per type: an input can be a user-defined
-custom type. Check `entity.type` before reading `entity.properties`.
+| Export                                     | What it is                                                             |
+| ------------------------------------------ | ---------------------------------------------------------------------- |
+| `Domain`, `Ip`, `Email`, `Individual`, ... | One interface per Flowsint entity type: the shape of `nodeProperties`. |
+| `FlowsintTypeMap`                          | Type key (`"domain"`, `"ip"`, ...) to its interface.                   |
+| `FlowsintTypeKey`                          | Union of every type key.                                               |
 
-`params` mirrors your `flowsint.params` declaration by name - a
-plugin with no `params` field in `package.json` still receives an
-(empty) object, never `undefined`. See the `flowsint.params` bullet
-above for what a `vaultSecret` entry resolves to.
+A node is checked against its `nodeType`:
 
 ```ts
-type InitResult = { ok: true } | { ok: false; message: string; fix: string }
-
-// Optional - omit entirely if your plugin has no runtime precondition
-// to check. Called once at app boot and again whenever the user clicks
-// "Reload plugins" in Settings → Plugins - never on a timer, never
-// before a run. A missing/failing dependency keeps your plugin out of
-// the enricher picker until init() resolves { ok: true }.
-function init(): Promise<InitResult>
+// Compile error: Ip has no `ip` field, its primary field is `address`.
+const node: LocalEnricherNode = { id: 'i', nodeType: 'ip', nodeProperties: { ip: '8.8.8.8' } }
 ```
 
-`logger` calls show up live in the sketch's Logs panel; anything you
-`throw` surfaces as a failed run there too (chunks already yielded stay).
+## Plugin contract
 
-## Typing your plugin with `@flowsint/plugin-sdk`
+Your `main` file exports `run` (required) and `init` (optional). Nothing
+else is read.
 
-Import the contract instead of copying it. Types only: use
-`import type`, your bundler erases it and nothing ships at runtime.
+| Export | Signature                                                                           | Called                                                    |
+| ------ | ----------------------------------------------------------------------------------- | --------------------------------------------------------- |
+| `run`  | `async function* (entities, logger, params, signal)`, yields `LocalEnricherFinding` | Each time a user runs the plugin.                         |
+| `init` | `() => Promise<InitResult>`                                                         | At app start and on "Reload plugins". Never before a run. |
 
-```ts
-import type {
-  EnricherEntityInput,
-  LocalEnricherFinding,
-  LocalEnricherLogger,
-  LocalEnricherNode
-} from '@flowsint/plugin-sdk'
+### `run` arguments
 
-// Compile error: Ip has no `ip` field, its primary field is `address`.
-const node: LocalEnricherNode = { id, nodeType: 'ip', nodeProperties: { ip: '8.8.8.8' } }
-```
+| Argument   | Type                      | Content                                                                                                            |
+| ---------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `entities` | `EnricherEntityInput[]`   | `{ type, properties }` per selected entity. Not narrowed: check `type` first, an input can be a user-defined type. |
+| `logger`   | `LocalEnricherLogger`     | `info`, `warn`, `error`, `success`, shown live in the sketch's logs.                                               |
+| `params`   | `Record<string, unknown>` | Values of your `flowsint.params`, by name. `{}` when you declare none.                                             |
+| `signal`   | `AbortSignal`             | Aborts when the user stops the run or after 6 hours. Pass it to `fetch` and `spawn`.                               |
 
-`LocalEnricherNode` is discriminated by `nodeType`: `nodeProperties`
-is checked against that type's spec (the interfaces are generated from
-Flowsint's own type specs, so they never drift from them). Narrow on
-`nodeType` before reading properties
-(`nodes.filter((n) => n.nodeType === 'dnsrecord')`).
-
-`LocalEnricherRun` types `run` as a whole (`export const run:
-LocalEnricherRun = async function* (entities, logger, params, signal) {...}`).
-`PluginPackageJson` is the shape of your `package.json` and
-`LocalEnricherPlugin` the module the host loads from `main`.
-
-## Validating params before you spawn anything
-
-If your plugin shells out to a CLI (most do), the params you declared
-in `flowsint.params` will usually become argv values for that
-`spawn()` call. Calling `spawn(cmd, args)` with an argv array, never
-with `shell: true` and never by building a single command string
-yourself, already rules out classic shell-metacharacter injection
-(`;`, `|`, `` ` ``, `$()`) by construction: no shell ever parses the
-strings you pass, the OS execs your binary directly with each array
-entry as one literal argv token. Do not build commands by
-concatenating strings into `exec()`; use `spawn()` with an argv array
-every time.
-
-That guarantee alone is not a reason to pass a param straight through
-unchecked, though. A malformed value can still confuse your CLI's own
-argument parser (a stray value swallowed as the wrong flag's argument,
-a numeric flag rejecting a non-numeric string with a cryptic error) or
-trigger something your plugin didn't intend (a flag with a very
-different meaning than the one the param name suggests). Validate each
-param before using it, and fail loud - `throw` before calling `spawn`
-at all, not after - so the whole run aborts with a clear error instead
-of silently doing the wrong thing:
+### `LocalEnricherFinding` (one yielded chunk)
 
-```ts
-const INT_RE = /^\d+$/
-const NAME_LIST_RE = /^[A-Za-z0-9_-]+(,[A-Za-z0-9_-]+)*$/
-
-function requireInt(name: string, v: unknown): string | undefined {
-  if (typeof v !== 'string' || v.trim() === '') return undefined
-  const s = v.trim()
-  if (!INT_RE.test(s))
-    throw new Error(`Invalid ${name}: expected a non-negative integer, got "${s}"`)
-  return s
-}
+| Field    | Type                              | Rule                                                                                                                     |
+| -------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
+| `nodes`  | `LocalEnricherNode[]`             | `{ id, nodeType, nodeProperties, nodeLabel?, nodeSize?, sources? }`. Use a fresh `randomUUID()` per node.                |
+| `edges`  | `LocalEnricherEdge[]`             | `{ id, source, target, label, sources? }`. `source` and `target` are node ids **of the same chunk**.                     |
+| `events` | `LocalEnricherEvent[]` (optional) | Timeline entries: `{ timestamp, title, endTimestamp?, description?, type?, confidence?, nodeIds?, tags?, properties? }`. |
 
-function requireList(name: string, v: unknown, pattern: RegExp): string | undefined {
-  if (typeof v !== 'string' || v.trim() === '') return undefined
-  const s = v.trim()
-  if (!pattern.test(s))
-    throw new Error(`Invalid ${name}: "${s}" contains characters outside the allowed set`)
-  return s
-}
+Each chunk is merged into the graph as soon as it is yielded. Yield one
+chunk per entity. A `throw` fails the run; chunks already yielded stay.
 
-// usage: requireList('SOURCES', params.SOURCES, NAME_LIST_RE)
-```
+### `InitResult`
 
-A URL-shaped param (a proxy, a webhook) is simplest to validate with
-`new URL(value)` - it throws on anything that isn't a real URL, which
-is exactly the check you want, and it's already in Node's stdlib.
-
-Build the whole argv array **once, before your entity loop**, not
-inside the per-entity `try`/`catch` most plugins use around their
-`spawn()` call. If a param is invalid, that's a configuration
-mistake, not a search failure for one entity - it should abort the
-whole run with a thrown error, not get silently caught and logged as
-a per-entity warning while the rest of the batch runs on stale/wrong
-settings.
-
-Logging the exact command you're about to run (`logger.info(...)`,
-built from the same array you pass to `spawn`, never a re-derived
-string) costs one line and pays for itself the first time a param
-doesn't do what someone expected - the sketch's Logs panel then shows
-precisely what ran, not just "started" or "failed."
-
-Flowsint's own bundled plugins (seeded into your plugins folder by
-the app) all
-follow this pattern - `domain-to-dns-records`, `ip-to-ports`, and
-`username-to-social-accounts` in particular are worth reading if
-you want a fuller example with several param types (integers,
-comma-separated allowlisted lists, an exact-match set, a boolean
-flag) in one file.
-
-## Checking a dependency (optional `init`)
-
-If your plugin shells out to an external tool - a CLI, a binary that
-must already be on the user's `PATH` - check for it in `init()` instead
-of letting `run()` fail with a bare "spawn ENOENT". There's no shared
-helper to import (see "nothing else is read" above) - copy this
-pattern into your own `src/index.ts`:
+| Value                         | Effect                                                                                                                           |
+| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
+| `{ ok: true }`                | Plugin is available.                                                                                                             |
+| `{ ok: false, message, fix }` | Plugin is hidden from the enricher list. Settings -> Plugins shows `message` (what is wrong) and `fix` (plain text instruction). |
 
 ```ts
 import { spawn } from 'node:child_process'
+import type { InitResult } from '@flowsint/plugin-sdk'
 
-async function checkBinaryOnPath(bin: string, fix: string): Promise<InitResult> {
+export function init(): Promise<InitResult> {
   return new Promise((resolve) => {
-    const probe = spawn(process.platform === 'win32' ? 'where' : 'which', [bin])
-    probe.on('close', (code) =>
-      resolve(code === 0 ? { ok: true } : { ok: false, message: `${bin} not found on PATH`, fix })
-    )
-    probe.on('error', () => resolve({ ok: false, message: `${bin} not found on PATH`, fix }))
+    const fail = {
+      ok: false,
+      message: 'subfinder not found on PATH',
+      fix: 'brew install subfinder'
+    } as const
+    const probe = spawn(process.platform === 'win32' ? 'where' : 'which', ['subfinder'])
+    probe.on('close', (code) => resolve(code === 0 ? { ok: true } : fail))
+    probe.on('error', () => resolve(fail))
   })
 }
-
-export async function init(): Promise<InitResult> {
-  return checkBinaryOnPath(
-    'subfinder',
-    "install with `brew install subfinder` or see the plugin's README"
-  )
-}
 ```
 
-`message` is what's wrong, shown next to your plugin in Settings →
-Plugins. `fix` is free text rendered as-is, no link-parsing - a short
-human instruction, not necessarily a URL. The clickable link in
-Settings → Plugins comes from `flowsint.installUrl` in `package.json`
-instead (see above) - a separate, static piece of metadata that has
-nothing to do with `init()`.
-
-## Install / uninstall
-
-Develop wherever you like. The plugins directory above is an install
-target, not a workspace. Copy your plugin's folder into it, then
-either restart Flowsint or click "Reload plugins" in Settings ->
-Plugins. Both re-scan the plugins directory and re-run every plugin's
-`init()` check. To remove a plugin, delete its folder and do the same.
-There is no enable/disable toggle in v1. Settings -> Plugins is
-read-only, showing what's loaded, what's degraded and why, and what's
-failed to load entirely.
-
-Symlinking your dev folder into the plugins directory (instead of
-copying on every change) saves the copy step - rebuild, then just
-click "Reload plugins".
-
-## Testing your plugin
-
-No test harness ships in the app. "does it work" is checked by
-running it for real:
-
-1. Settings → Plugins - your plugin should show `loaded`. `degraded`
-   or `failed` shows the `init()` message/fix or load error right
-   there; fix that before anything else.
-2. Open a sketch, add an entity matching your `flowsint.inputs.type`.
-3. Pick your plugin from that entity's enricher list and run it.
-4. Bottom panel → "terminal" tab (`Cmd+Shift+T` to toggle) shows your
-   `logger.info/warn/error/success` calls live, plus anything your
-   `run`/`init` throws.
-5. Check the sketch's graph for the `nodes`/`edges` you returned.
-
-Changed the code? Rebuild, click "Reload plugins" in Settings →
-Plugins, re-run step 3 - no restart needed.
+## `package.json` manifest
+
+Typed by `PluginPackageJson`. Flowsint reads it without running your code.
+
+| Field                  | Required | Meaning                                                                         |
+| ---------------------- | -------- | ------------------------------------------------------------------------------- |
+| `name`                 | yes      | Plugin id, unique across installed plugins.                                     |
+| `main`                 | yes      | Bundled CommonJS entry.                                                         |
+| `engines.flowsint`     | no       | Flowsint versions supported, e.g. `^2.6.1`. Skipped when absent.                |
+| `flowsint.category`    | yes      | Group in the enricher list (`Domain`, `Ip`, ...).                               |
+| `flowsint.source`      | yes      | Provider slug stamped on every result (`dns`, `whois`): lowercase, digits, `_`. |
+| `flowsint.inputs`      | yes      | `{ type, properties }`: the entity type the plugin accepts.                     |
+| `flowsint.outputs`     | no       | `{ type, properties }`: the main entity type it produces.                       |
+| `flowsint.description` | no       | One sentence shown in the app.                                                  |
+| `flowsint.icon`        | no       | A [lucide](https://lucide.dev/icons/) icon name.                                |
+| `flowsint.installUrl`  | no       | Install docs of an external dependency, linked in Settings -> Plugins.          |
+| `flowsint.params`      | no       | Launch parameters, see below.                                                   |
+
+### `flowsint.params`
+
+Each entry is `{ name, type, description, default, required, options? }`
+(`EnricherParamSchemaItem`). The user fills them in before the run.
+
+| `type`          | Form control                              | Value in `params[name]`                              |
+| --------------- | ----------------------------------------- | ---------------------------------------------------- |
+| `string`, `url` | Text input                                | `string`                                             |
+| `number`        | Numeric input                             | `number`, `undefined` when empty                     |
+| `boolean`       | Switch                                    | `boolean`, `false` when untouched                    |
+| `date`          | Calendar                                  | `Date` at midnight UTC, `undefined` when empty       |
+| `list`          | Tags input                                | `string[]`, `[]` when empty                          |
+| `select`        | Dropdown of `options: [{ label, value }]` | The picked `value`                                   |
+| `vaultSecret`   | Key from the local vault                  | The decrypted secret, `null` when optional and unset |
+
+An invalid value fails the run before `run` is called.
+
+## Good practice
+
+- **Bundle everything.** Flowsint never runs `npm install`: your plugin must
+  work by `require()` alone. Ship `node_modules/` only for a native addon
+  the bundler cannot inline.
+- **Write a `README.md`.** It is required, and shown in Settings -> Plugins.
+  Say what the plugin needs and what it returns.
+- **Spawn with an argv array.** `spawn(cmd, args)`, never `shell: true` or a
+  command string: no shell, no injection.
+- **Validate params before the entity loop.** A bad param should fail the
+  whole run with a clear `throw`, not one entity at a time.
+- **Log what you run.** `logger.info` the exact argv: the logs then show
+  what happened, not just that it failed.
+- **Develop with a symlink.** Link your folder into the plugins folder, then
+  rebuild and click "Reload plugins". No restart needed.
+
+## Versioning
+
+The package version is the Flowsint version. Set `engines.flowsint` to `^`
+the version you installed. Removing or renaming an entity type or field,
+narrowing a field's type or making a field required only happens in a
+major release. This covers the types only, not how the app processes what
+a plugin returns.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@flowsint/plugin-sdk",
-  "version": "2.6.1",
+  "version": "2.6.2",
   "description": "SDK for Flowsint local enricher plugins: the plugin contract and the entity property types. Types only: consume with `import type`.",
   "license": "Apache-2.0",
   "author": "dextmorgn <contact@flowsint.io>",