@crowi/plugin-search-opensearch 0.1.0-alpha.0

package/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 Sotaro KARASAWA <sotaro.k@gmail.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

package/README.md

@@ -0,0 +1,165 @@
+# @crowi/plugin-search-opensearch
+
+OpenSearch search driver for Crowi 2.0. Indexes pages on create /
+update / delete, serves the wiki search box, and rebuilds the whole
+index from scratch on demand. Targets a `<indexName>-current` alias so
+a rebuild can swap the underlying index atomically.
+
+The plugin is a sibling of [`@crowi/plugin-search-elasticsearch`](../plugin-search-elasticsearch):
+the wire-level document shape, alias name, mapping JSONs and query DSL
+are identical, so a cluster migration between the two backends is a
+re-point + rebuild rather than a mapping rewrite.
+
+## Install
+
+```bash
+crowi-admin plugin add @crowi/plugin-search-opensearch
+```
+
+(or, in dev: `pnpm --filter @crowi/api add -D @crowi/plugin-search-opensearch`)
+
+## Configure
+
+### 1. Activate the driver in `crowi.config.json`
+
+```jsonc
+{
+  "plugins": ["@crowi/plugin-search-opensearch"],
+  "search": { "driver": "opensearch" }
+}
+```
+
+A server restart is required when `search.driver` changes — Crowi
+reads this file once at boot.
+
+### 2. Fill in connection settings in the admin UI
+
+Open `/admin/plugins` and edit `@crowi/plugin-search-opensearch`:
+
+- **`url`** — `https://[user:pass@]host[:port][/indexName]`. The URL
+  embeds the cluster password, so it is encrypted at rest with
+  `CROWI_ENCRYPTION_KEY`. Only Basic Auth via the URL is supported
+  (AWS SigV4 / IAM auth is intentionally out of scope; a managed
+  OpenSearch deployment using fine-grained access control still works
+  with a Basic Auth user).
+- **`indexName`** — base index name (default `crowi`). The driver
+  reads / writes the `<indexName>-current` alias.
+- **`requestTimeout`** — per-request timeout in ms (default `5000`).
+- **`analyzer`** — `default` / `kuromoji` / `sudachi` (see below).
+
+The admin UI is the single source of truth for these settings — there
+is no env-variable fallback.
+
+### 3. Build the initial index
+
+```bash
+crowi-admin search rebuild
+```
+
+This creates a fresh `<indexName>-<timestamp>-<rand>` index, indexes
+all pages in 2000-document bulk batches with pre-fetched bookmark
+counts, and atomically swaps `<indexName>-current` to the new index.
+
+## Hot-reload (no restart needed)
+
+This plugin implements `reconfigure`, so **saving connection settings
+in the admin UI applies without a server restart**. When you save:
+
+- the `url` / `indexName` / `requestTimeout` / `analyzer` changes are
+  picked up by the live driver,
+- a fresh OpenSearch client is built and the previous one is closed
+  in the background (its HTTP keep-alive pool drains),
+- the admin UI shows a "saved — applied immediately" toast.
+
+Mechanics: the driver holds a module-scope state ref; each operation
+(`query` / `index` / `remove` / `rebuild`) snapshots the state once at
+the top of the call, so a save that lands mid-request cannot retarget
+an inflight operation onto a different cluster. The next request sees
+the new settings.
+
+### Caveats
+
+- **Analyzer changes need a manual rebuild.** Switching `analyzer`
+  (`default` / `kuromoji` / `sudachi`) updates the setting immediately,
+  but the **existing index keeps its old analyzer** — analyzers are
+  fixed at index-creation time. Run `crowi-admin search rebuild` to
+  create a new index with the new analyzer and swap the alias to it.
+- **Empty `url` → configured `url` is restart-only.** If `url` was
+  empty at boot, the driver is not registered and there is nothing for
+  `reconfigure` to mutate; configure a `url` and restart once. After
+  that, all further changes hot-reload. Clearing a configured `url`
+  (configured → empty) *is* handled live — search requests then fail
+  with a clear `Search not configured` error until a `url` is set again.
+- A rebuild that is already running when you reconfigure runs to
+  completion against the cluster / index name it started with.
+
+## Analyzer flavours
+
+| Analyzer | Cluster requirement |
+|---|---|
+| `default` | No extra OpenSearch plugin. |
+| `kuromoji` | `analysis-kuromoji` (Apache 2.0, from `opensearch-project/analysis-kuromoji`). Unlike Elasticsearch, this is a **separate distribution** from OpenSearch core — install it on every cluster node with `bin/opensearch-plugin install analysis-kuromoji` and restart. |
+| `sudachi` | Third-party `analysis-sudachi` (OpenSearch-compatible fork from WorksApplications) + a dictionary. Operators must bundle these into a custom image. Picking this without the plugin makes `rebuild()` fail. |
+
+For a wiki with mostly Japanese content, `kuromoji` is the typical
+choice. The mapping JSON (`src/mappings/kuromoji.json`) names the
+analyzer as `kuromoji`, which is the identifier both the Elastic and
+OpenSearch distributions install under, so no per-engine tweak is
+needed in the mapping itself.
+
+## Trying it locally
+
+The default `docker compose` stack ships an **Elasticsearch** service
+for the sibling plugin, not OpenSearch. To exercise this plugin
+locally, add an override file:
+
+```yaml
+# compose.override.yml
+services:
+  opensearch:
+    image: opensearchproject/opensearch:2.18.0
+    environment:
+      - discovery.type=single-node
+      - plugins.security.disabled=true
+      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=Crowi-Dev-Passw0rd!
+    ports:
+      - "9201:9200"
+```
+
+Then:
+
+1. `docker compose up -d opensearch` — brings up OpenSearch on `:9201`
+   (`:9200` is taken by the dev Elasticsearch service if you left it
+   running).
+2. In `/admin/plugins`, set `@crowi/plugin-search-opensearch`'s `url`
+   to `http://opensearch:9201/crowi` (or `http://localhost:9201/crowi`
+   from outside the compose network) and save; restart once if the
+   driver was previously unconfigured.
+3. From `/admin/search`, run a rebuild to populate the index.
+4. Change `requestTimeout` (or point `indexName` at a freshly rebuilt
+   index) and save **without restarting**. The next search query uses
+   the new settings — confirmed by the api log line
+   `reconfigured opensearch search driver (...)`.
+
+## Why a separate plugin from `@crowi/plugin-search-elasticsearch`?
+
+Both backends speak the same query DSL today, but the SDKs differ
+(`@opensearch-project/opensearch` vs `@elastic/elasticsearch`) and the
+backends are intentionally allowed to diverge — OpenSearch's neural /
+k-NN extensions, the Elastic-licensed features in newer ES versions,
+and the analyzer-plugin distribution stories are all separate concerns.
+Keeping the drivers in distinct npm packages means each can pin its own
+SDK / mapping fork without forcing the other to follow.
+
+The parse-query / query-builder / mapping JSON files are 1:1 copies of
+the ES plugin's today, but **on purpose** — we will revisit a shared
+core when a third driver arrives.
+
+## See also
+
+- [`@crowi/plugin-search-elasticsearch`](../plugin-search-elasticsearch) —
+  the sibling Elasticsearch driver.
+- RFC-0001 §"Search" for the search-driver plugin architecture.
+- [`@crowi/plugin-storage-aws-s3`](../plugin-storage-aws-s3) — the
+  reference implementation of the same state-ref + snapshot hot-reload
+  pattern.

package/dist/index.d.mts

@@ -0,0 +1,268 @@
+import { z } from 'zod/v3';
+import { SearchDriver, PluginLogger, SearchQueryViewer, SearchQueryGrants, CrowiPlugin } from '@crowi/plugin-api';
+import { Client } from '@opensearch-project/opensearch';
+
+/**
+ * OpenSearch driver implementing the `SearchDriver` contract. Owns
+ * the Client, the `${indexName}-current` alias (legacy ops compat),
+ * single-doc index / remove, query against the alias, and rebuild-
+ * from-scratch in 2k-doc bulk batches with bookmark counts pre-fetched
+ * in one aggregate. Document field shape (path / body / username /
+ * grant / granted_users / *_count / *_at) matches the ES plugin's
+ * shape so a cluster migration is a re-point + rebuild rather than a
+ * mapping rewrite.
+ *
+ * SDK note: `@opensearch-project/opensearch` 3.x returns
+ * `{ body, statusCode, ... }` wrappers around every API response (the
+ * shape inherited from the old `elasticsearch-js` 7.x line). The
+ * Elasticsearch 9 client we use for `@crowi/plugin-search-elasticsearch`
+ * collapsed those wrappers — so every call site here unwraps `body`
+ * explicitly. Bulk requests likewise take `{ body: operations }`, not
+ * the ES 9 `{ operations }` keyword.
+ */
+
+type Analyzer = 'default' | 'kuromoji' | 'sudachi';
+interface OpenSearchDriverConfig {
+    url: string;
+    indexName: string;
+    requestTimeout: number;
+    analyzer: Analyzer;
+}
+interface OpenSearchDriverDeps {
+    log?: PluginLogger;
+    /**
+     * Iterate every page in the Mongo Page collection in cursor-style.
+     * Plugin can't import the Page model directly, so the manager wires
+     * this in from `ctx.model('Page')`. Each yielded doc is the lean
+     * shape produced by `Page.getStreamOfFindAll({ publicOnly: false })`.
+     */
+    iteratePages?: (handler: (page: PageStreamDoc) => Promise<void>) => Promise<void>;
+    /** Total page count, used for progress reporting. */
+    countAllPages?: () => Promise<number>;
+    /**
+     * Bulk-fetch bookmark counts for every page in one Mongo aggregate.
+     * Avoids the per-doc N+1 lookup the legacy rebuild used. Returns a
+     * `Map<pageId, count>`; pages without bookmarks may be absent
+     * (caller defaults to 0).
+     */
+    getBookmarkCountsBulk?: () => Promise<Map<string, number>>;
+    /** Total user count, used to scale the bookmark-count factor. */
+    countUsers?: () => Promise<number>;
+}
+/** The lean Page document shape we expect from the rebuild stream. */
+interface PageStreamDoc {
+    _id: {
+        toString: () => string;
+    } | string;
+    path: string;
+    redirectTo: string | null;
+    status: string;
+    grant: number;
+    grantedUsers?: Array<{
+        toString: () => string;
+    } | string>;
+    creator?: {
+        username?: string;
+    };
+    revision?: {
+        body?: string;
+    };
+    liker?: unknown[];
+    commentCount?: number;
+    bookmarkCount?: number;
+    createdAt?: Date;
+    updatedAt?: Date;
+}
+interface OpenSearchDriver extends SearchDriver {
+    /** Currently-targeted alias name (`<indexName>-current`). Exposed for tests / admin UI. */
+    readonly aliasName: string;
+    /** OpenSearch node URI parsed out of `config.url`. */
+    readonly node: string;
+    /** Base index name (without timestamp / `-current` suffix). */
+    readonly baseIndexName: string;
+    /** Test-only handle to the underlying client. */
+    readonly client: Client;
+}
+/**
+ * Mutable driver state. `createOpenSearchDriver` receives a ref to
+ * this; each driver method snapshots the fields it needs *once at the
+ * top* of the call, so a concurrent `reconfigure` cannot swap the
+ * client / index name mid-operation. `reconfigure` mutates the fields
+ * in place via {@link applyConfigInPlace}; the next call sees the new
+ * values. An empty `url` leaves `client` as `null` — the methods then
+ * throw a `Search not configured` error rather than touching a stale
+ * client.
+ */
+interface OSDriverState {
+    /** `null` when `url` is empty (driver configured-but-disabled). */
+    client: Client | null;
+    /** OpenSearch node URI parsed out of `config.url`; empty string when `url` is empty. */
+    node: string;
+    /** Base index name (without timestamp / `-current` suffix). */
+    baseIndexName: string;
+    /** Runtime alias the driver reads / writes (`<baseIndexName>-current`). */
+    aliasName: string;
+    analyzer: Analyzer;
+    requestTimeout: number;
+}
+/**
+ * Build a fresh {@link OSDriverState} from a config. An empty `url`
+ * yields a disabled state (`client: null`) instead of throwing — the
+ * driver stays registered but every method rejects with a
+ * `Search not configured` error.
+ */
+declare function applyConfig(config: OpenSearchDriverConfig): OSDriverState;
+/**
+ * Mutate `target` in place to reflect `config`. Used by `reconfigure`:
+ * the old client reference is returned so the caller can `close()` it
+ * (fire-and-forget) once the swap is done — inflight operations have
+ * already snapshotted the old client and will run to completion.
+ */
+declare function applyConfigInPlace(target: OSDriverState, config: OpenSearchDriverConfig): {
+    oldClient: Client | null;
+};
+/**
+ * Build the search driver around an {@link OSDriverState} ref. Methods
+ * snapshot `state` *once at the top* — a `reconfigure` running
+ * concurrently with an inflight call cannot swap the client mid-call;
+ * the next call sees the new client / index name.
+ */
+declare function createOpenSearchDriver(state: OSDriverState, deps?: OpenSearchDriverDeps): OpenSearchDriver;
+
+/**
+ * Search-string parser for the OpenSearch driver.
+ *
+ * Splits a free-form query into positive / negative keywords and
+ * phrases. Lifted from the legacy `packages/api/src/service/query.ts`
+ * with no behaviour changes — preserved here as a plugin-private
+ * helper because the parser is currently OpenSearch / Elasticsearch
+ * specific (the +/- and `"phrase"` syntax maps directly to
+ * `multi_match` queries). When a future driver wants the same shape,
+ * factor it back into `@crowi/plugin-api`.
+ */
+type PositiveAndNegative<T> = {
+    positive: T;
+    negative: T;
+};
+type ParsedSearchQuery = {
+    keywords: PositiveAndNegative<string[]>;
+    phrases: PositiveAndNegative<string[]>;
+};
+declare const parseQuery: (query: string) => ParsedSearchQuery;
+
+/**
+ * Build an OpenSearch search request body from the SearchQuery shape
+ * exposed by `@crowi/plugin-api`. The driver passes a parsed
+ * keyword/phrase tree plus the viewer + grants from the original
+ * SearchQuery; this module composes them into a single bool query.
+ *
+ * The wire shape is identical to Elasticsearch's query DSL (OpenSearch
+ * forked from ES 7.10.2 and has kept the search API surface compatible),
+ * so the builder is a 1:1 copy of `@crowi/plugin-search-elasticsearch`'s
+ * builder for now. Kept private here rather than shared because the two
+ * drivers may diverge in the future (e.g. OpenSearch's neural / k-NN
+ * extensions) and hard-coupling them would block that.
+ *
+ * Design notes:
+ *   - All filters are composed at the top-level `bool`. We never nest
+ *     a second `bool` for the same operator type (must / filter /
+ *     should / must_not), so the generated body is small and easy to
+ *     diff in tests.
+ *   - The grant filter mirrors the legacy ES Searcher precisely:
+ *     a non-public page (RESTRICTED / SPECIFIED / OWNER) is hidden
+ *     unless its `username` field matches the viewer's username.
+ *     For SPECIFIED / OWNER / RESTRICTED pages, we additionally allow
+ *     the page through if `granted_users` contains the viewer id —
+ *     the legacy query only checked `username`, but the new
+ *     SearchableDoc lets us index `granted_users` precisely so we
+ *     can express "shared with me" as well.
+ *   - Type filter (portal / public / user) reproduces the legacy
+ *     `path.raw` regex / prefix queries.
+ */
+
+type FunctionScoreParams = {
+    fieldValueFactor: {
+        field: string;
+        factor?: number;
+        modifier?: 'log' | 'log1p' | 'log2p' | 'ln' | 'ln1p' | 'ln2p' | 'square' | 'sqrt' | 'reciprocal' | 'none';
+        missing: number;
+    };
+    boostMode: 'multiply' | 'replace' | 'sum' | 'avg' | 'max' | 'min';
+};
+interface BuildSearchBodyParams {
+    parsed: ParsedSearchQuery;
+    pathPrefix?: string;
+    viewer?: SearchQueryViewer;
+    grants?: SearchQueryGrants;
+    functionScore?: FunctionScoreParams;
+    from: number;
+    size: number;
+}
+/**
+ * Build the OpenSearch search request body. Returns an object suitable
+ * for `client.search({ index, body })`.
+ */
+declare function buildSearchBody(params: BuildSearchBodyParams): {
+    from: number;
+    size: number;
+    sort: Array<Record<string, unknown>>;
+    highlight: Record<string, unknown>;
+    query: Record<string, unknown>;
+    _source: string[];
+};
+
+/**
+ * @crowi/plugin-search-opensearch — search driver registering
+ * `'opensearch'` against the SearchRegistry.
+ *
+ * Activation: add this plugin to the runner's `crowi.config.json`
+ * `plugins` array and set `search.driver: 'opensearch'`. Configure
+ * via the Mongo Config namespace `plugin:@crowi/plugin-search-opensearch:*`
+ * — operators set the connection URL exclusively from the admin UI.
+ */
+
+declare const OpenSearchConfigSchema: z.ZodObject<{
+    /**
+     * `https://[user:pass@]host[:port][/indexName]`. Empty string keeps
+     * the driver registered but disabled — `query()` will throw a
+     * helpful error and `index()` becomes a no-op.
+     *
+     * Marked `@sensitive` because the URL embeds the cluster password
+     * (Bonsai-style `https://USER:PASS@HOST/INDEX`); we don't want
+     * Mongo to keep it in plaintext.
+     */
+    url: z.ZodDefault<z.ZodString>;
+    /**
+     * Base index name. Used as the `indexName` if not provided in the
+     * URL path. The runtime alias `${indexName}-current` is what the
+     * driver actually targets for read / write.
+     */
+    indexName: z.ZodDefault<z.ZodString>;
+    requestTimeout: z.ZodDefault<z.ZodNumber>;
+    /**
+     * Mapping flavour. Cluster requirements:
+     *   - `default`: no extra OpenSearch plugin.
+     *   - `kuromoji`: `analysis-kuromoji` plugin (Apache 2.0, a
+     *     separate distribution from OpenSearch core — install via
+     *     `bin/opensearch-plugin install analysis-kuromoji`).
+     *   - `sudachi`: `analysis-sudachi` (OpenSearch-compatible fork
+     *     from WorksApplications) + dictionary; operators must build
+     *     a derived image. Picking this without the plugin makes
+     *     `rebuild()` fail.
+     */
+    analyzer: z.ZodDefault<z.ZodEnum<["default", "kuromoji", "sudachi"]>>;
+}, "strict", z.ZodTypeAny, {
+    url: string;
+    indexName: string;
+    requestTimeout: number;
+    analyzer: "default" | "kuromoji" | "sudachi";
+}, {
+    url?: string | undefined;
+    indexName?: string | undefined;
+    requestTimeout?: number | undefined;
+    analyzer?: "default" | "kuromoji" | "sudachi" | undefined;
+}>;
+type OpenSearchConfig = z.infer<typeof OpenSearchConfigSchema>;
+declare const plugin: CrowiPlugin;
+
+export { type Analyzer, type OSDriverState, type OpenSearchConfig, OpenSearchConfigSchema, type OpenSearchDriver, type OpenSearchDriverConfig, type OpenSearchDriverDeps, type PageStreamDoc, applyConfig, applyConfigInPlace, buildSearchBody, createOpenSearchDriver, plugin as default, parseQuery };