@ever-works/firecrawl-plugin 1.0.0

package/README.md

@@ -0,0 +1,76 @@
+# @ever-works/firecrawl-plugin
+
+Firecrawl Plugin - Web search and content extraction using the Firecrawl API
+
+## Plugin metadata
+
+| Field        | Value                         |
+| ------------ | ----------------------------- |
+| ID           | `firecrawl`                   |
+| Category     | `search`                      |
+| Capabilities | `search`, `content-extractor` |
+| Author       | Ever Works Team               |
+| License      | AGPL-3.0                      |
+| Built-in     | yes                           |
+| Auto-enable  | no                            |
+
+## What does Firecrawl do?
+
+Firecrawl is a web scraping and search API that can search the web for relevant results and extract clean, well-formatted markdown content from any web page. It handles JavaScript rendering, anti-bot bypasses, and content cleaning automatically.
+
+## Why use it?
+
+- **Web search** — search the web and get structured results with content snippets
+- **Clean markdown output** — returns well-structured markdown from any web page
+- **JavaScript rendering** — handles dynamic/SPA pages that simple HTTP fetches miss
+- **Anti-bot handling** — bypasses common protections automatically
+- **Metadata extraction** — captures title, description, and other page metadata
+
+## How it works in Ever Works
+
+When enabled, Firecrawl can be used as both a search provider and content extractor during work generation. It searches the web for relevant information and extracts high-quality content from web pages, including JavaScript-heavy sites.
+
+## Getting started
+
+1. Create an account at [firecrawl.dev](https://firecrawl.dev)
+2. Copy your API key from the Firecrawl dashboard
+3. Enter the key in the **API Key** field below
+4. Enable this plugin to use it for search and content extraction
+
+## Settings
+
+- **API Key** (`apiKey`) — required, secret. Your Firecrawl API key (also configurable via the `PLUGIN_FIRECRAWL_API_KEY` environment variable).
+- Stored at user scope so each user can supply their own credentials.
+- Configuration mode is `hybrid` — admins can preset values and users may override them.
+
+## Troubleshooting
+
+| Symptom                                      | Likely cause                                                            | Fix                                                                                                                                                       |
+| -------------------------------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `401 Unauthorized` / `Authentication failed` | API key missing, revoked, or wrong key entered                          | Re-enter the **API Key** from the Firecrawl dashboard, or set `PLUGIN_FIRECRAWL_API_KEY` in the host environment for default fallback                     |
+| `429 Too Many Requests`                      | Free-tier or paid quota exhausted on Firecrawl                          | Throttle calls, wait for the quota reset, or upgrade the plan in the Firecrawl dashboard                                                                  |
+| Empty / sparse results                       | Query is too restrictive, time-range or domain filters are too narrow   | Broaden the query, relax `time_range` / `safesearch` / `include_domains` / `exclude_domains` filters, or raise `max_results`                              |
+| Plugin not used during work generation       | Another search plugin is set as the default for the `search` capability | In **Settings → Plugins**, set `firecrawl` as the default for `search`, or disable competing search plugins                                               |
+| `Failed to extract content` for a URL        | Page is gated by login, Cloudflare, or robots.txt; URL is malformed     | Verify the URL is publicly reachable; for protected pages enable a different `content-extractor` plugin (`scrapfly`, `notion-extractor`, `pdf-extractor`) |
+| `healthCheck` reports unhealthy              | API key invalid OR Firecrawl endpoint unreachable from the host         | Verify the key with a manual `curl` against the documented endpoint and confirm outbound HTTPS is allowed by the firewall                                 |
+
+## Local development
+
+This plugin ships built-in with the Ever Works platform. To work on it locally from the monorepo root:
+
+```bash
+pnpm install
+pnpm --filter @ever-works/firecrawl-plugin build
+pnpm --filter @ever-works/firecrawl-plugin test
+```
+
+## Documentation
+
+- [Ever Works documentation](https://docs.ever.works)
+- [Ever Works repository](https://github.com/ever-works/ever-works)
+- [Plugin system](../../plugin/README.md)
+- [Firecrawl homepage](https://firecrawl.dev)
+
+## License
+
+AGPL-3.0

package/package.json

@@ -0,0 +1,78 @@
+{
+  "name": "@ever-works/firecrawl-plugin",
+  "version": "1.0.0",
+  "description": "Firecrawl Plugin - Web search and content extraction using the Firecrawl API",
+  "author": {
+    "name": "Ever Co. LTD",
+    "email": "ever@ever.co",
+    "url": "https://ever.co"
+  },
+  "license": "AGPL-3.0",
+  "private": false,
+  "type": "module",
+  "main": "./dist/index.cjs",
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs"
+    }
+  },
+  "dependencies": {
+    "@mendable/firecrawl-js": "^4.12.1"
+  },
+  "peerDependencies": {
+    "@ever-works/plugin": "1.1.0"
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "tsup": "^8.4.0",
+    "typescript": "^5.7.3",
+    "vitest": "^4.1.0",
+    "@ever-works/plugin": "1.1.0"
+  },
+  "everworks": {
+    "plugin": {
+      "id": "firecrawl",
+      "name": "Firecrawl",
+      "version": "1.0.0",
+      "category": "search",
+      "capabilities": [
+        "search",
+        "content-extractor"
+      ],
+      "description": "Web search and content extraction using the Firecrawl API. Search the web and scrape pages for clean markdown content.",
+      "author": {
+        "name": "Ever Works Team"
+      },
+      "license": "AGPL-3.0",
+      "autoEnable": false,
+      "systemPlugin": false,
+      "builtIn": true,
+      "envVars": [
+        {
+          "name": "PLUGIN_FIRECRAWL_API_KEY",
+          "required": false,
+          "secret": true,
+          "description": "Firecrawl API key (optional - can be set via admin/user settings)"
+        }
+      ],
+      "distribution": "registry"
+    }
+  },
+  "publishConfig": {
+    "access": "restricted",
+    "registry": "https://registry.npmjs.org"
+  },
+  "scripts": {
+    "build": "tsc --noEmit && tsup",
+    "dev": "tsup --watch",
+    "type-check": "tsc --noEmit",
+    "clean": "rm -rf dist",
+    "test": "vitest run --passWithNoTests",
+    "test:watch": "vitest",
+    "test:coverage": "vitest run --coverage"
+  }
+}

package/src/__tests__/firecrawl.plugin.spec.ts

@@ -0,0 +1,291 @@
+import { describe, it, expect, vi, beforeEach } from 'vitest';
+import type { PluginContext, SearchOptions, ContentExtractionOptions } from '@ever-works/plugin';
+
+const { searchMock, scrapeMock, batchScrapeMock, FirecrawlAppCtorMock } = vi.hoisted(() => {
+	const search = vi.fn();
+	const scrape = vi.fn();
+	const batchScrape = vi.fn();
+	const ctor = vi.fn().mockImplementation(function () {
+		return { search, scrape, batchScrape };
+	});
+	return {
+		searchMock: search,
+		scrapeMock: scrape,
+		batchScrapeMock: batchScrape,
+		FirecrawlAppCtorMock: ctor
+	};
+});
+
+vi.mock('@mendable/firecrawl-js', () => ({
+	default: FirecrawlAppCtorMock
+}));
+
+const { FirecrawlPlugin } = await import('../firecrawl.plugin.js');
+
+const buildContext = (settings: Record<string, unknown> = {}): PluginContext =>
+	({
+		pluginId: 'firecrawl',
+		logger: {
+			log: vi.fn(),
+			debug: vi.fn(),
+			warn: vi.fn(),
+			error: vi.fn()
+		},
+		getSettings: vi.fn().mockResolvedValue(settings)
+	}) as unknown as PluginContext;
+
+describe('FirecrawlPlugin', () => {
+	let plugin: FirecrawlPlugin;
+
+	beforeEach(() => {
+		vi.clearAllMocks();
+		plugin = new FirecrawlPlugin();
+	});
+
+	describe('metadata', () => {
+		it('exposes stable identity fields', () => {
+			expect(plugin.id).toBe('firecrawl');
+			expect(plugin.name).toBe('Firecrawl');
+			expect(plugin.version).toBe('1.0.0');
+			expect(plugin.category).toBe('search');
+			expect(plugin.providerName).toBe('Firecrawl');
+		});
+
+		it('declares both search and content-extractor capabilities', () => {
+			expect(plugin.capabilities).toEqual(['search', 'content-extractor']);
+		});
+
+		it('uses hybrid configuration mode', () => {
+			expect(plugin.configurationMode).toBe('hybrid');
+		});
+	});
+
+	describe('settingsSchema', () => {
+		it('requires apiKey and marks it as user-scoped secret', () => {
+			expect(plugin.settingsSchema.required).toContain('apiKey');
+			const props = plugin.settingsSchema.properties as Record<string, Record<string, unknown>>;
+			expect(props.apiKey['x-secret']).toBe(true);
+			expect(props.apiKey['x-envVar']).toBe('PLUGIN_FIRECRAWL_API_KEY');
+		});
+	});
+
+	describe('search', () => {
+		const opts = (overrides: Partial<SearchOptions> = {}): SearchOptions => ({
+			query: 'hello',
+			settings: { apiKey: 'k' },
+			...overrides
+		});
+
+		it('throws when apiKey is missing', async () => {
+			await expect(plugin.search({ query: 'hello', settings: {} })).rejects.toThrow(/API key not configured/i);
+			expect(FirecrawlAppCtorMock).not.toHaveBeenCalled();
+		});
+
+		it('maps web results with description+title and derives source hostname', async () => {
+			searchMock.mockResolvedValueOnce({
+				web: [
+					{
+						title: 'Example',
+						url: 'https://www.example.com/page',
+						description: 'Example description'
+					}
+				]
+			});
+			const r = await plugin.search(opts({ limit: 5 }));
+
+			expect(FirecrawlAppCtorMock).toHaveBeenCalledWith({ apiKey: 'k' });
+			expect(searchMock).toHaveBeenCalledWith('hello', { limit: 5 });
+			expect(r.results).toHaveLength(1);
+			expect(r.results[0]).toMatchObject({
+				title: 'Example',
+				url: 'https://www.example.com/page',
+				snippet: 'Example description',
+				position: 1,
+				source: 'www.example.com'
+			});
+		});
+
+		it('falls back to markdown when description is absent', async () => {
+			searchMock.mockResolvedValueOnce({
+				web: [{ title: 't', url: 'https://example.com', markdown: '# header' }]
+			});
+			const r = await plugin.search(opts());
+			expect(r.results[0].snippet).toBe('# header');
+		});
+
+		it('handles missing fields gracefully', async () => {
+			searchMock.mockResolvedValueOnce({ web: [{}] });
+			const r = await plugin.search(opts());
+			expect(r.results[0]).toMatchObject({ title: '', url: '', snippet: '' });
+		});
+
+		it('returns an empty result list when web is missing', async () => {
+			searchMock.mockResolvedValueOnce({});
+			const r = await plugin.search(opts());
+			expect(r.results).toEqual([]);
+		});
+
+		it('logs and rethrows on SDK failure', async () => {
+			const ctx = buildContext();
+			await plugin.onLoad(ctx);
+			searchMock.mockRejectedValueOnce(new Error('boom'));
+			await expect(plugin.search(opts())).rejects.toThrow(/boom/);
+			expect(ctx.logger.error).toHaveBeenCalled();
+		});
+	});
+
+	describe('extract', () => {
+		const opts = (overrides: Partial<ContentExtractionOptions> = {}): ContentExtractionOptions => ({
+			url: 'https://example.com/post',
+			settings: { apiKey: 'k' },
+			...overrides
+		});
+
+		it('returns success with markdown, title, wordCount, readingTime', async () => {
+			scrapeMock.mockResolvedValueOnce({
+				markdown: 'one two three four five',
+				metadata: { title: 'Title', url: 'https://example.com/post' }
+			});
+			const r = await plugin.extract(opts());
+			expect(scrapeMock).toHaveBeenCalledWith('https://example.com/post', { formats: ['markdown'] });
+			expect(r.success).toBe(true);
+			expect(r.title).toBe('Title');
+			expect(r.markdown).toBe('one two three four five');
+			expect(r.wordCount).toBe(5);
+			expect(r.readingTime).toBe(1);
+		});
+
+		it('exposes finalUrl when metadata.url differs', async () => {
+			scrapeMock.mockResolvedValueOnce({
+				markdown: 'hi',
+				metadata: { title: 't', url: 'https://example.com/redirected' }
+			});
+			const r = await plugin.extract(opts());
+			expect(r.finalUrl).toBe('https://example.com/redirected');
+		});
+
+		it('returns failure when markdown is empty', async () => {
+			scrapeMock.mockResolvedValueOnce({ markdown: '', metadata: {} });
+			const r = await plugin.extract(opts());
+			expect(r.success).toBe(false);
+			expect(r.error).toMatch(/No content extracted/);
+		});
+
+		it('returns failure when SDK throws', async () => {
+			scrapeMock.mockRejectedValueOnce(new Error('extract failed'));
+			const r = await plugin.extract(opts());
+			expect(r.success).toBe(false);
+			expect(r.error).toMatch(/extract failed/);
+		});
+	});
+
+	describe('extractBatch', () => {
+		it('uses batch API when available', async () => {
+			batchScrapeMock.mockResolvedValueOnce({
+				data: [
+					{ markdown: 'one two', metadata: { title: 'A', url: 'https://a' } },
+					{ markdown: 'three four five', metadata: { title: 'B', url: 'https://b' } }
+				]
+			});
+			const r = await plugin.extractBatch(['https://a', 'https://b'], { settings: { apiKey: 'k' } });
+			expect(r).toHaveLength(2);
+			expect(r[0]).toMatchObject({ success: true, url: 'https://a', wordCount: 2 });
+			expect(r[1]).toMatchObject({ success: true, url: 'https://b', wordCount: 3 });
+		});
+
+		it('falls back to sequential extract when batch API throws', async () => {
+			batchScrapeMock.mockRejectedValueOnce(new Error('batch unavailable'));
+			scrapeMock
+				.mockResolvedValueOnce({ markdown: 'one two', metadata: { title: 'A', url: 'https://a' } })
+				.mockResolvedValueOnce({ markdown: 'three', metadata: { title: 'B', url: 'https://b' } });
+
+			const r = await plugin.extractBatch(['https://a', 'https://b'], { settings: { apiKey: 'k' } });
+			expect(r).toHaveLength(2);
+			expect(r[0]).toMatchObject({ success: true, wordCount: 2 });
+			expect(r[1]).toMatchObject({ success: true, wordCount: 1 });
+		});
+
+		it('reports per-item failure when batch returns empty markdown', async () => {
+			batchScrapeMock.mockResolvedValueOnce({
+				data: [{ markdown: '', metadata: { url: 'https://a' } }]
+			});
+			const r = await plugin.extractBatch(['https://a'], { settings: { apiKey: 'k' } });
+			expect(r[0].success).toBe(false);
+		});
+	});
+
+	describe('isAvailable', () => {
+		it('returns false without context', async () => {
+			expect(await plugin.isAvailable()).toBe(false);
+		});
+
+		it('reflects whether settings has an apiKey', async () => {
+			await plugin.onLoad(buildContext({ apiKey: 'k' }));
+			expect(await plugin.isAvailable()).toBe(true);
+
+			await plugin.onLoad(buildContext({}));
+			expect(await plugin.isAvailable()).toBe(false);
+		});
+	});
+
+	describe('validateConnection', () => {
+		it('fails fast without apiKey', async () => {
+			const r = await plugin.validateConnection({});
+			expect(r.success).toBe(false);
+			expect(r.message).toMatch(/not configured/i);
+		});
+
+		it('returns success when SDK search works', async () => {
+			searchMock.mockResolvedValueOnce({ web: [] });
+			const r = await plugin.validateConnection({ apiKey: 'k' });
+			expect(r.success).toBe(true);
+		});
+
+		it('returns failure when SDK throws', async () => {
+			searchMock.mockRejectedValueOnce(new Error('bad key'));
+			const r = await plugin.validateConnection({ apiKey: 'k' });
+			expect(r.success).toBe(false);
+		});
+	});
+
+	describe('canExtract / getSupportedFormats', () => {
+		it('accepts http and https URLs', async () => {
+			expect(await plugin.canExtract('https://example.com')).toBe(true);
+			expect(await plugin.canExtract('http://example.com')).toBe(true);
+		});
+
+		it('rejects non-http schemes and invalid URLs', async () => {
+			expect(await plugin.canExtract('ftp://example.com')).toBe(false);
+			expect(await plugin.canExtract('garbage')).toBe(false);
+		});
+
+		it('exposes only the markdown format', () => {
+			expect(plugin.getSupportedFormats()).toEqual(['markdown']);
+		});
+	});
+
+	describe('lifecycle', () => {
+		it('logs on load and clears context on unload', async () => {
+			const ctx = buildContext();
+			await plugin.onLoad(ctx);
+			expect(ctx.logger.log).toHaveBeenCalledWith('Firecrawl Plugin loaded');
+			await plugin.onUnload();
+			expect(await plugin.isAvailable()).toBe(false);
+		});
+	});
+
+	describe('healthCheck + manifest', () => {
+		it('reports healthy', async () => {
+			const h = await plugin.healthCheck();
+			expect(h.status).toBe('healthy');
+		});
+
+		it('returns a manifest aligned with plugin metadata', () => {
+			const m = plugin.getManifest();
+			expect(m.id).toBe('firecrawl');
+			expect(m.category).toBe('search');
+			expect(m.capabilities).toEqual(['search', 'content-extractor']);
+			expect(m.builtIn).toBe(true);
+		});
+	});
+});