@neottia/searchable-core 0.2.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dragos Cirjan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/NOTICE.md

@@ -0,0 +1,5 @@
+Neottia Searchable includes code and behavior adapted from @dragoscirjan/mcp-searchable.
+Copyright (c) 2025 Templ Project. Used under the MIT License.
+
+Mozilla Readability is distributed as a dependency under the Apache License 2.0.
+See https://github.com/mozilla/readability for its source and license.

package/README.md

@@ -0,0 +1,71 @@
+# @neottia/searchable-core
+
+Search, fetch, stash, grep, and ask services for repository-local web research. The package provides the shared five-tool registry and a concrete runtime used by the MCP, Pi, and OpenCode packages.
+
+## Install
+
+```sh
+pnpm add @neottia/searchable-core
+```
+
+Node.js 22.16 or newer is required. Canonical stash files and the SQLite FTS cache require a filesystem supported by `@neottia/repository-store`.
+
+## Configure
+
+Searchable contributes the canonical `modules.searchable` shard to `@neottia/config`. Hosts resolve the official registry once and pass the immutable shard to the runtime and tool context. `loadSearchableConfig()` remains available for standalone use and accepts the deprecated `skills.searchable` path.
+
+See the [unified configuration guide](../../docs/configuration.md) for shared files, profiles, precedence, secret handling, diagnostics, and migration. Configuration resolves in this order: defaults, global file, project file, selected global profile, selected project profile, environment bindings, and explicit runtime overrides. The YAML root must contain `version: 1`.
+
+Create `.neottia/config.yml` in the project working directory:
+
+```yaml
+version: 1
+modules:
+  searchable:
+    enabled: true
+    search:
+      provider: duckduckgo
+```
+
+DuckDuckGo requires no key. Google needs `google_api_key` and `google_cse_id`; Bing and Brave need their API keys. YAML credential values must use exact `${ENV_NAME}` references. The resolver expands only the winning reference and redacts all provider credentials from snapshots, provenance, and diagnostics. Service endpoints must use HTTP or HTTPS and cannot contain user information, a query string, or a fragment. See [`config.schema.json`](./config.schema.json) for the complete standalone shard.
+
+## Use the concrete runtime
+
+```ts
+import { createSearchableRuntime, findSearchableTool } from "@neottia/searchable-core";
+
+const runtime = createSearchableRuntime({ cwd: process.cwd() });
+const search = findSearchableTool("web_search");
+if (!search) throw new Error("web_search is not registered");
+
+const result = await search.run({ cwd: process.cwd(), services: runtime }, { query: "Neottia documentation" });
+console.log(result);
+await runtime.close();
+```
+
+The shared registry exposes `web_search`, `web_fetch`, `web_stash`, `web_grep`, and `web_ask`. Inputs and outputs pass the same Zod contracts for every delivery package.
+
+## Runtime behavior
+
+- Search supports DuckDuckGo, Google Custom Search, Bing, and Brave.
+- Fetch tries the configured `direct`, `jina`, and `wayback` strategies in order. Direct fetch blocks private and special-use destinations, validates redirects, and bounds decoded response bytes.
+- Stash writes canonical JSON records below `.neottia/searchable/pages/`. `.neottia/cache/searchable.sqlite` is a disposable FTS5 index with a separate limit of eight times the canonical quota plus 8 MiB.
+- Grep treats query text as bounded literal terms. It reloads canonical records before returning titles, URLs, and snippets.
+- Ask selects canonical stash context and calls the configured Ollama `/api/generate` endpoint. It parses newline-delimited JSON incrementally, bounds frames and the cumulative answer, and returns the source URLs.
+- `modules.searchable.enabled` must resolve to `true` before any service runs.
+
+Call `close()` when the host shuts down. Closing aborts new work and closes the injected transport.
+
+## Legacy migration
+
+`importLegacySearchableDatabase(store, {path: ".web_stash.db"})` previews by default. The source must be a direct child of the project root. Stop the legacy server and remove active WAL or SHM sidecars first. Set `preview: false` only after checking the report. The importer snapshots the database, checks integrity, reads only the `pages` table, waits for its worker to terminate, and leaves the source file unchanged.
+
+## Security boundary
+
+`web_fetch` accepts only HTTP and HTTPS URLs without user information. The default client rejects local, private, link-local, metadata, multicast, reserved, and mixed public/private DNS destinations. It pins the accepted address for the request and revalidates redirects. Jina and Wayback do not receive target URLs that contain query parameters.
+
+Ollama is the only trusted-local network exception. That exception is confined to the model client and does not change `web_fetch` policy. Treat fetched text and model answers as untrusted input.
+
+Errors use `{category, code, message, paths, details?}`. Diagnostics remove credential values and URL query strings. Provider response bodies and request objects are not included.
+
+Full user documentation is at [`docs/searchable/`](../../docs/searchable/index.md).

package/config.schema.json

@@ -0,0 +1,246 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "properties": {
+    "enabled": {
+      "default": false,
+      "type": "boolean"
+    },
+    "root": {
+      "default": ".neottia/searchable",
+      "type": "string",
+      "minLength": 1,
+      "maxLength": 1024,
+      "pattern": "^(?!\\.{1,2}(?:\\/|$))(?!.*(?:^|\\/)\\.{1,2}(?:\\/|$))(?!\\/)(?![A-Za-z]:)[^\\\\\\0\\r\\n]+$"
+    },
+    "search": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "provider": {
+          "default": "duckduckgo",
+          "type": "string",
+          "enum": ["duckduckgo", "google", "bing", "brave"]
+        },
+        "limit": {
+          "default": 5,
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 100
+        },
+        "timeout_ms": {
+          "default": 10000,
+          "type": "integer",
+          "exclusiveMinimum": 0,
+          "maximum": 300000
+        },
+        "credentials": {
+          "default": {},
+          "type": "object",
+          "properties": {
+            "google_api_key": {
+              "type": "string",
+              "pattern": "^\\$\\{[A-Za-z_][A-Za-z0-9_]*\\}$"
+            },
+            "google_cse_id": {
+              "type": "string",
+              "pattern": "^\\$\\{[A-Za-z_][A-Za-z0-9_]*\\}$"
+            },
+            "bing_api_key": {
+              "type": "string",
+              "pattern": "^\\$\\{[A-Za-z_][A-Za-z0-9_]*\\}$"
+            },
+            "brave_api_key": {
+              "type": "string",
+              "pattern": "^\\$\\{[A-Za-z_][A-Za-z0-9_]*\\}$"
+            }
+          },
+          "additionalProperties": false
+        },
+        "bing_api_endpoint": {
+          "default": "https://api.bing.microsoft.com/v7.0/search",
+          "type": "string",
+          "maxLength": 8192,
+          "pattern": "^[hH][tT][tT][pP][sS]?:\\/\\/(?![^/?#]*@)[^?#]+$",
+          "format": "uri"
+        }
+      },
+      "additionalProperties": false
+    },
+    "fetch": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "strategies": {
+          "default": ["direct", "jina", "wayback"],
+          "minItems": 1,
+          "maxItems": 3,
+          "type": "array",
+          "items": {
+            "type": "string",
+            "enum": ["direct", "jina", "wayback"]
+          },
+          "uniqueItems": true
+        },
+        "timeout_ms": {
+          "default": 10000,
+          "type": "integer",
+          "exclusiveMinimum": 0,
+          "maximum": 300000
+        },
+        "overall_timeout_ms": {
+          "default": 20000,
+          "type": "integer",
+          "exclusiveMinimum": 0,
+          "maximum": 600000
+        },
+        "max_response_bytes": {
+          "default": 10485760,
+          "type": "integer",
+          "exclusiveMinimum": 0,
+          "maximum": 10485760
+        }
+      },
+      "additionalProperties": false
+    },
+    "grep": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "limit": {
+          "default": 5,
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 100
+        },
+        "snippet_bytes": {
+          "default": 512,
+          "type": "integer",
+          "minimum": 64,
+          "maximum": 16384
+        }
+      },
+      "additionalProperties": false
+    },
+    "ask": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "limit": {
+          "default": 3,
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 100
+        },
+        "context_bytes": {
+          "default": 131072,
+          "type": "integer",
+          "minimum": 256,
+          "maximum": 4194304
+        }
+      },
+      "additionalProperties": false
+    },
+    "ollama": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "endpoint": {
+          "default": "http://localhost:11434",
+          "type": "string",
+          "maxLength": 8192,
+          "pattern": "^[hH][tT][tT][pP][sS]?:\\/\\/(?![^/?#]*@)[^?#]+$",
+          "format": "uri"
+        },
+        "model": {
+          "default": "llama3",
+          "type": "string",
+          "minLength": 1,
+          "pattern": "\\S"
+        },
+        "timeout_ms": {
+          "default": 60000,
+          "type": "integer",
+          "exclusiveMinimum": 0,
+          "maximum": 600000
+        }
+      },
+      "additionalProperties": false
+    },
+    "cache": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "max_age_ms": {
+          "default": 300000,
+          "type": "integer",
+          "minimum": 0,
+          "maximum": 9007199254740991
+        },
+        "stale_policy": {
+          "default": "prompt",
+          "type": "string",
+          "enum": ["prompt", "rebuild", "fail"]
+        }
+      },
+      "additionalProperties": false
+    },
+    "security": {
+      "default": {},
+      "type": "object",
+      "properties": {
+        "limits": {
+          "default": {},
+          "type": "object",
+          "properties": {
+            "max_query_bytes": {
+              "default": 16384,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 16384
+            },
+            "max_url_bytes": {
+              "default": 8192,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 8192
+            },
+            "max_title_bytes": {
+              "default": 4096,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 4096
+            },
+            "max_content_bytes": {
+              "default": 10485760,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 10485760
+            },
+            "max_results": {
+              "default": 100,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 100
+            },
+            "max_result_bytes": {
+              "default": 4194304,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 4194304
+            },
+            "max_storage_bytes": {
+              "default": 268435456,
+              "type": "integer",
+              "exclusiveMinimum": 0,
+              "maximum": 268435456
+            }
+          },
+          "additionalProperties": false
+        }
+      },
+      "additionalProperties": false
+    }
+  },
+  "additionalProperties": false
+}