@havesomecode/kibana-mcp-server 1.1.2 → 1.2.0

package/README.md

@@ -5,6 +5,8 @@ Read-only MCP server for agent-driven log investigation against Kibana-backed se
 The server is intentionally small:
 
 - `setup` saves one or more machine-level Kibana profiles for later threads
+- `bootstrap` verifies only the Kibana connection, creates an empty managed catalog, saves credentials, and registers Codex without prompts or config-file edits
+- `configure_index` validates and persists exactly one index or pattern after the user explicitly names it
 - `configure` still exists for advanced MCP clients that want to drive credentials and source catalogs at runtime
 - `describe_fields` exposes effective field capabilities for one configured source
 - `discover` lists configured logical sources and field hints
@@ -17,7 +19,7 @@ It is designed as a general-purpose log investigation MCP for environments where
 
 - Node.js 22+
 - A Kibana-compatible read endpoint reachable with basic auth
-- A source catalog JSON file describing the logical log sources to expose, or the bundled `config/sources.example.json`
+- One or more Elasticsearch index patterns. `bootstrap` generates the source catalog; hand-authored catalogs remain supported for advanced setups.
 
 ## Project Status
 
@@ -25,10 +27,11 @@ This repo is live for external adoption and AI-agent use.
 
 - Homepage: `https://havesomecode.github.io/kibana-mcp-server/`
 - Published package: `npx -y @havesomecode/kibana-mcp-server`
+- Agent skill: `npx skills add Havesomecode/kibana-mcp-server --skill kibana-log-investigation`
 - Repo-local Codex plugin install remains a first-class path for cloned checkouts
 
-- Guaranteed: repo-local Codex plugin install (see `INSTALL.md`)
-- Guaranteed: guided machine setup through `npm run setup`
+- Guaranteed: prompt-free package bootstrap and Codex registration (see `INSTALL.md`)
+- Supported: guided machine setup through `npm run setup` for hand-authored catalogs
 - Available: published package install via `npx -y @havesomecode/kibana-mcp-server`
 - Active: semantic-release automation for tags, npm publish, and GitHub Releases
 - Support posture and compatibility details live in `docs/project/support-policy.md`
@@ -45,17 +48,44 @@ This repo is live for external adoption and AI-agent use.
 
 ## Installation
 
+### Prompt-free connection bootstrap
+
+Pre-provision the connection values in the environment:
+
 ```bash
-npm install
-npm run build
-npm run setup
+export KIBANA_BASE_URL='https://kibana.example.com'
+export KIBANA_USERNAME='elastic'
+export KIBANA_PASSWORD='use-a-secret-provider'
+
+npx -y @havesomecode/kibana-mcp-server bootstrap \
+  --client codex
 ```
 
-## Use In Codex
+This command performs no prompts and does not edit Codex configuration files directly. It verifies only the Kibana connection, writes an empty managed source catalog on first install, stores credentials securely, and registers the exact running package transport. It does not list, scan, infer, or configure indexes.
+
+After bootstrap, call `discover`. When no source is configured, the agent must ask the user which exact Kibana index or index pattern to use. Only after the user answers may the agent call `configure_index` with that value. That tool validates only the approved index and persists it without accepting credentials.
+
+Bootstrap reruns preserve sources that were configured through this explicit flow. It refuses to replace a hand-authored or modified catalog unless `--replace` is supplied explicitly; `--replace` creates an empty managed catalog.
+
+Never pass passwords as command-line arguments. Use `KIBANA_PASSWORD`, `--password-env NAME`, or `--password-stdin`.
+
+### Install the agent skill
+
+The repository contains a standard Agent Skill discoverable by the `skills` CLI and skills.sh:
+
+```bash
+npx skills add Havesomecode/kibana-mcp-server \
+  --skill kibana-log-investigation \
+  --agent codex \
+  --global \
+  --yes
+```
 
-This repo includes a repo-scoped Codex plugin so a cloned checkout can install the MCP without any extra packaging or hosted infrastructure.
+The skill teaches compatible agents to run the same verified bootstrap instead of editing MCP config files. Its source is `skills/kibana-log-investigation/SKILL.md`.
 
-If you do not need the repo-local plugin workflow, the published package is already available through `npx -y @havesomecode/kibana-mcp-server`.
+## Use In Codex
+
+The prompt-free package bootstrap above is the primary Codex path. The repo-scoped plugin remains available for contributors working from a cloned checkout.
 
 ### Quick path
 
@@ -68,19 +98,14 @@ If you do not need the repo-local plugin workflow, the published package is alre
 ```bash
 npm install
 npm run build
-npm run setup
 ```
 
 4. Open the cloned repo in Codex.
 5. Open the plugin directory in Codex and install `Kibana Log Investigation` from the repo marketplace.
    - if the current model cannot complete that install itself, do the Codex UI click manually and let the agent continue with configuration afterward
 6. Restart Codex if the new MCP server does not appear immediately.
-7. Run guided setup once:
-   - `npm run setup`
-   - provide the environment name, Kibana base URL, username, password, and a source catalog file to import
-   - use the bundled `config/sources.example.json` unless you already have a better catalog JSON
-8. Let later threads reuse the saved default profile automatically.
-   - if you add more than one environment during setup, select non-default ones with `KIBANA_PROFILE=<PROFILE_NAME>`
+7. Run `node dist/src/index.js bootstrap --client none` to verify the connection and save an empty profile. The repo plugin already supplies the client registration.
+8. In a fresh thread, call `discover`, ask the user for the exact index or pattern, then call `configure_index` only after they answer.
 
 Repo-scoped plugin files:
 
@@ -93,12 +118,21 @@ Repo-scoped plugin files:
 If you are handing only the repo link to another Codex agent, this usually works:
 
 ```text
-Clone this repo, ensure Node.js 22+ is installed at user level for the current OS, run npm install, npm run build, and npm run setup, install the repo plugin named "Kibana Log Investigation", import a source catalog during setup, and verify discover plus one query work in Codex without needing a later manual configure step.
+Install the `kibana-log-investigation` skill from Havesomecode/kibana-mcp-server, then run its connection-only bootstrap with the provisioned KIBANA_BASE_URL, KIBANA_USERNAME, and KIBANA_PASSWORD values. Do not edit Codex config files, import the example catalog, or inspect indexes. After install, ask me which exact index or pattern to configure before calling configure_index.
 ```
 
 ## Configuration
 
-The preferred path is guided machine setup:
+The preferred path is deterministic connection-only bootstrap:
+
+- `npx -y @havesomecode/kibana-mcp-server bootstrap --client codex`
+- verifies `/api/status` without touching an index endpoint
+- creates an empty managed source catalog on first install
+- rejects conflicting MCP registrations instead of overwriting them
+- saves non-secret profile metadata and credentials for later threads
+- exposes `configure_index` for one user-approved index or pattern at a time
+
+Guided setup remains available for hand-authored source catalogs:
 
 - `npm run setup` or `node dist/src/index.js setup`
 - saves non-secret profile metadata in the user’s machine-level app config directory
@@ -107,6 +141,7 @@ The preferred path is guided machine setup:
   - Windows Credential Manager
   - Linux Secret Service when available
 - imports the selected source catalog into machine-local state so later threads do not depend on the repo checkout
+- requires an explicit catalog path; type `example` to opt into the bundled example deliberately
 
 The server also supports two advanced compatibility paths:
 
@@ -265,6 +300,21 @@ npm run dev
 
 ## Tool Shapes
 
+### `configure_index`
+
+Use this tool only after the user explicitly names the exact index or index pattern to configure. It accepts no credentials and never lists or scans for candidate indexes.
+
+Input:
+
+```json
+{
+  "index": "app-logs-*",
+  "source_name": "Application logs"
+}
+```
+
+The tool inspects fields and runs one bounded validation search only for that index. It then persists the generated source in the managed catalog. Optional `source_id`, `time_field`, and `replace` fields support explicit collision or schema handling.
+
 ### `configure`
 
 Input:

package/dist/src/bootstrap.js

@@ -0,0 +1,426 @@
+import { createHash, randomUUID } from "node:crypto";
+import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { CodexClientRegistrar } from "./client_registration.js";
+import { sourceCatalogSchema, sourceDefinitionSchema } from "./config.js";
+import { KibanaClient } from "./kibana_client.js";
+import { resolveProfilePaths } from "./profile_paths.js";
+import { ProfileStore, deriveProfileId } from "./profile_store.js";
+import { SecretStoreError, createSecretStore } from "./secret_store.js";
+import { acquireStateLock } from "./state_lock.js";
+const DEFAULT_TIMEOUT_MS = 10000;
+const DEFAULT_MCP_NAME = "kibana-log-investigation";
+const KNOWN_ENDPOINT_PATHS = [
+    "/internal/search/es",
+    "/api/data_views/fields_for_wildcard",
+    "/api/index_patterns/_fields_for_wildcard",
+];
+export function buildGeneratedSource(rawIndexes, sourceName, sourceId) {
+    const indexes = normalizeIndexes(rawIndexes);
+    const resolvedSourceId = normalizeSourceId(sourceId ?? sourceName ?? indexes[0] ?? "logs");
+    const resolvedSourceName = sourceName?.trim() || resolvedSourceId;
+    const indexConfig = indexes.length === 1 ? indexes[0] : indexes;
+    return {
+        id: resolvedSourceId,
+        name: resolvedSourceName,
+        description: `Generated from Elasticsearch index patterns: ${indexes.join(", ")}.`,
+        tags: ["generated"],
+        timeField: "@timestamp",
+        backend: {
+            kind: "kibana_internal_search_es",
+            path: "/internal/search/es",
+            index: indexConfig,
+        },
+        schema: {
+            kind: "kibana_data_views_fields",
+            index: indexConfig,
+        },
+        fieldHints: [],
+        defaultTextFields: ["message"],
+        evidenceFields: [],
+    };
+}
+export function chooseGeneratedSourceFields(fields, explicitTimeField) {
+    if (fields.length === 0) {
+        throw new Error("The index pattern resolved to no fields. Check the index and Kibana permissions.");
+    }
+    const byName = new Map(fields.map((field) => [field.name, field]));
+    const requestedTimeField = explicitTimeField?.trim();
+    if (requestedTimeField && !byName.has(requestedTimeField)) {
+        throw new Error(`Requested time field '${requestedTimeField}' was not found in the index pattern.`);
+    }
+    const timeField = requestedTimeField ||
+        ["@timestamp", "timestamp", "event.created", "event.ingested", "date"].find((name) => {
+            const field = byName.get(name);
+            return field?.type === "date" || field?.type === "date_nanos";
+        }) ||
+        fields
+            .filter((field) => field.type === "date" || field.type === "date_nanos")
+            .map((field) => field.name)
+            .sort((left, right) => left.localeCompare(right))[0];
+    if (!timeField) {
+        throw new Error("Could not determine a time field from the discovered schema. Retry configure_index with time_field set explicitly.");
+    }
+    const preferredTextFields = ["message", "event.original", "log.original"];
+    const defaultTextFields = preferredTextFields.filter((name) => byName.has(name));
+    if (defaultTextFields.length === 0) {
+        const fallbackTextField = fields
+            .filter((field) => field.searchable !== false && (field.type === "text" || field.type === "match_only_text"))
+            .map((field) => field.name)
+            .sort((left, right) => left.localeCompare(right))[0];
+        if (fallbackTextField) {
+            defaultTextFields.push(fallbackTextField);
+        }
+    }
+    const evidenceFields = [
+        "trace.id",
+        "traceId",
+        "trace_id",
+        "correlation_id",
+        "request.id",
+        "transaction.id",
+        "service.name",
+        "host.name",
+    ].filter((name) => byName.has(name));
+    return { timeField, defaultTextFields, evidenceFields };
+}
+export async function runBootstrap(rawOptions, dependencies = {}) {
+    const paths = dependencies.paths ?? resolveProfilePaths();
+    const releaseLock = await acquireStateLock(paths.stateRoot);
+    try {
+        return await runBootstrapLocked(rawOptions, { ...dependencies, paths });
+    }
+    finally {
+        await releaseLock();
+    }
+}
+async function runBootstrapLocked(rawOptions, dependencies) {
+    const normalizedOptions = normalizeOptions(rawOptions);
+    const options = {
+        ...normalizedOptions,
+        packageSpecifier: normalizedOptions.packageSpecifier ||
+            (normalizedOptions.client === "codex" ? await resolveCurrentPackageSpecifier() : ""),
+    };
+    const paths = dependencies.paths ?? resolveProfilePaths();
+    const profileStore = dependencies.profileStore ?? new ProfileStore(paths);
+    const secretStore = dependencies.secretStore ?? createSecretStore();
+    const verifier = dependencies.verifier ?? new NetworkBootstrapVerifier();
+    const stateBefore = await profileStore.load();
+    const existingProfile = stateBefore.profiles.find((profile) => profile.name.toLowerCase() === options.profileName.toLowerCase());
+    const profileId = existingProfile?.id ??
+        deriveProfileId(options.profileName, stateBefore.profiles.map((profile) => profile.id));
+    const sourceCatalogPath = join(paths.sourceCatalogsDir, `${profileId}.json`);
+    const profileCatalog = existingProfile
+        ? await readOptionalFile(existingProfile.sourceCatalogPath)
+        : undefined;
+    const targetCatalog = await readOptionalFile(sourceCatalogPath);
+    const protectedCatalog = [profileCatalog, targetCatalog].find((content) => content !== undefined && !isGeneratedCatalog(content));
+    if (!options.replaceExisting && protectedCatalog !== undefined) {
+        throw new Error(`Profile '${options.profileName}' would replace an existing hand-authored, modified, or orphaned source catalog. Re-run with --replace to replace it explicitly.`);
+    }
+    const connection = {
+        baseUrl: options.baseUrl,
+        username: options.username,
+        password: options.password,
+        timeoutMs: options.timeoutMs,
+    };
+    const registrar = resolveRegistrar(options, dependencies.clientRegistrar);
+    if (registrar) {
+        await registrar.preflight();
+    }
+    await verifier.verify(connection);
+    const existingGeneratedCatalog = [targetCatalog, profileCatalog]
+        .filter((content) => content !== undefined)
+        .find((content) => isGeneratedCatalog(content));
+    const catalog = !options.replaceExisting && existingGeneratedCatalog
+        ? sourceCatalogSchema.parse(JSON.parse(existingGeneratedCatalog))
+        : createGeneratedCatalog([]);
+    const previousCatalog = targetCatalog;
+    const previousSecret = await loadPreviousSecret(secretStore, profileId);
+    let localStateChanged = false;
+    let secretPossiblyChanged = false;
+    try {
+        await writeJsonAtomically(sourceCatalogPath, catalog);
+        localStateChanged = true;
+        secretPossiblyChanged = true;
+        await secretStore.save(profileId, {
+            username: options.username,
+            password: options.password,
+        });
+        await profileStore.upsertProfile({
+            id: profileId,
+            name: options.profileName,
+            baseUrl: options.baseUrl,
+            timeoutMs: options.timeoutMs,
+            sourceCatalogPath,
+        }, { makeDefault: options.makeDefault });
+        if (registrar) {
+            await registrar.register();
+        }
+        return {
+            profileName: options.profileName,
+            profileId,
+            sourceCount: catalog.sources.length,
+            sourceCatalogPath,
+            client: options.client,
+            registered: options.client === "codex",
+            verified: true,
+        };
+    }
+    catch (error) {
+        const rollbackErrors = [];
+        if (registrar) {
+            await rollbackClientRegistration(registrar).catch((rollbackError) => {
+                rollbackErrors.push(asErrorMessage(rollbackError));
+            });
+        }
+        if (localStateChanged) {
+            await profileStore.replaceState(stateBefore).catch((rollbackError) => {
+                rollbackErrors.push(asErrorMessage(rollbackError));
+            });
+            await restoreFile(sourceCatalogPath, previousCatalog).catch((rollbackError) => {
+                rollbackErrors.push(asErrorMessage(rollbackError));
+            });
+            if (secretPossiblyChanged) {
+                await restoreSecret(secretStore, profileId, previousSecret).catch((rollbackError) => {
+                    rollbackErrors.push(asErrorMessage(rollbackError));
+                });
+            }
+        }
+        if (rollbackErrors.length > 0) {
+            throw new Error(`${asErrorMessage(error)} Rollback also failed: ${rollbackErrors.join("; ")}`, { cause: error });
+        }
+        throw error;
+    }
+}
+class NetworkBootstrapVerifier {
+    async verify(connection) {
+        await new KibanaClient(connection).verifyConnection();
+    }
+}
+function resolveRegistrar(options, injectedRegistrar) {
+    if (options.client === "none") {
+        return undefined;
+    }
+    return (injectedRegistrar ??
+        new CodexClientRegistrar({
+            mcpName: options.mcpName,
+            profileName: options.profileName,
+            packageSpecifier: options.packageSpecifier,
+        }));
+}
+function normalizeOptions(options) {
+    const profileName = options.profileName.trim();
+    if (!profileName || !/^[A-Za-z0-9][A-Za-z0-9 _-]*$/.test(profileName)) {
+        throw new Error("Profile name must start with a letter or number and contain only letters, numbers, spaces, dashes, or underscores.");
+    }
+    const baseUrl = normalizeBaseUrl(options.baseUrl);
+    const username = options.username.trim();
+    const password = options.password;
+    if (!username)
+        throw new Error("Kibana username is required.");
+    if (!password)
+        throw new Error("Kibana password is required.");
+    if (options.client !== "codex" && options.client !== "none") {
+        throw new Error("Client must be either 'codex' or 'none'.");
+    }
+    const timeoutMs = options.timeoutMs ?? DEFAULT_TIMEOUT_MS;
+    if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || timeoutMs > 120000) {
+        throw new Error("Timeout must be a positive number no greater than 120000 milliseconds.");
+    }
+    return {
+        ...options,
+        profileName,
+        baseUrl,
+        username,
+        password,
+        timeoutMs,
+        mcpName: options.mcpName?.trim() || DEFAULT_MCP_NAME,
+        packageSpecifier: options.packageSpecifier?.trim() || "",
+        replaceExisting: options.replaceExisting ?? false,
+    };
+}
+export async function resolveCurrentPackageSpecifier() {
+    const moduleDirectory = dirname(fileURLToPath(import.meta.url));
+    const candidates = [
+        join(moduleDirectory, "..", "package.json"),
+        join(moduleDirectory, "..", "..", "package.json"),
+    ];
+    for (const candidate of candidates) {
+        const content = await readOptionalFile(candidate);
+        if (!content)
+            continue;
+        try {
+            const parsed = JSON.parse(content);
+            if (typeof parsed.name === "string" && typeof parsed.version === "string") {
+                return `${parsed.name}@${parsed.version}`;
+            }
+        }
+        catch {
+            // Try the next package.json candidate.
+        }
+    }
+    throw new Error("Could not resolve the running package version for deterministic registration.");
+}
+export function createGeneratedCatalog(sourceOrSources) {
+    const sources = Array.isArray(sourceOrSources) ? sourceOrSources : [sourceOrSources];
+    const parsedSources = sources.map((source) => sourceDefinitionSchema.parse(source));
+    return sourceCatalogSchema.parse({
+        generatedBy: {
+            tool: "@havesomecode/kibana-mcp-server",
+            formatVersion: 1,
+            sourceHash: hashSources(parsedSources),
+        },
+        sources: parsedSources,
+    });
+}
+export function parseGeneratedCatalog(content) {
+    try {
+        const raw = JSON.parse(content);
+        const parsed = sourceCatalogSchema.parse(raw);
+        if (!isPlainRecord(raw) || !hasExactKeys(raw, ["generatedBy", "sources"]))
+            return undefined;
+        if (!isPlainRecord(raw.generatedBy))
+            return undefined;
+        if (!hasExactKeys(raw.generatedBy, ["formatVersion", "sourceHash", "tool"])) {
+            return undefined;
+        }
+        if (!Array.isArray(raw.sources))
+            return undefined;
+        if (!parsed.generatedBy || parsed.generatedBy.sourceHash !== hashSources(raw.sources)) {
+            return undefined;
+        }
+        return parsed;
+    }
+    catch {
+        return undefined;
+    }
+}
+function isGeneratedCatalog(content) {
+    return parseGeneratedCatalog(content) !== undefined;
+}
+function hashSources(sources) {
+    return createHash("sha256").update(JSON.stringify(sources)).digest("hex");
+}
+function isPlainRecord(value) {
+    return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+function hasExactKeys(record, expectedKeys) {
+    const keys = Object.keys(record).sort();
+    return JSON.stringify(keys) === JSON.stringify([...expectedKeys].sort());
+}
+async function loadPreviousSecret(secretStore, profileId) {
+    try {
+        return await secretStore.load(profileId);
+    }
+    catch (error) {
+        if (error instanceof SecretStoreError && error.code === "NOT_FOUND") {
+            return undefined;
+        }
+        throw error;
+    }
+}
+function normalizeIndexes(rawIndexes) {
+    const indexes = [...new Set(rawIndexes.map((index) => index.trim()).filter(Boolean))];
+    if (indexes.length === 0) {
+        throw new Error("At least one Elasticsearch index pattern is required.");
+    }
+    return indexes;
+}
+function normalizeSourceId(value) {
+    const normalized = value
+        .trim()
+        .toLowerCase()
+        .replace(/\*+$/g, "")
+        .replace(/[^a-z0-9]+/g, "-")
+        .replace(/^-+|-+$/g, "");
+    return normalized || "logs";
+}
+function normalizeBaseUrl(rawValue) {
+    let parsed;
+    try {
+        parsed = new URL(rawValue.trim());
+    }
+    catch {
+        throw new Error("Kibana base URL must be a valid http or https URL.");
+    }
+    if (parsed.protocol !== "http:" && parsed.protocol !== "https:") {
+        throw new Error("Kibana base URL must use http or https.");
+    }
+    if (parsed.username || parsed.password) {
+        throw new Error("Kibana base URL must not embed credentials.");
+    }
+    if (parsed.search || parsed.hash) {
+        throw new Error("Kibana base URL must not include a query string or fragment.");
+    }
+    if (KNOWN_ENDPOINT_PATHS.some((path) => parsed.pathname.includes(path))) {
+        throw new Error("Kibana base URL must not include endpoint paths such as /internal/search/es.");
+    }
+    return parsed.toString().replace(/\/+$/, "");
+}
+async function readOptionalFile(path) {
+    try {
+        return await readFile(path, "utf8");
+    }
+    catch (error) {
+        if (isMissingFileError(error))
+            return undefined;
+        throw error;
+    }
+}
+async function writeJsonAtomically(path, value) {
+    await writeTextAtomically(path, `${JSON.stringify(value, null, 2)}\n`);
+}
+async function writeTextAtomically(path, content) {
+    const temporaryPath = `${path}.${process.pid}.${randomUUID()}.tmp`;
+    await mkdir(dirname(path), { recursive: true });
+    try {
+        await writeFile(temporaryPath, content, {
+            encoding: "utf8",
+            mode: 0o600,
+        });
+        await rename(temporaryPath, path);
+    }
+    finally {
+        await rm(temporaryPath, { force: true }).catch(() => { });
+    }
+}
+async function restoreFile(path, previousContent) {
+    if (previousContent === undefined) {
+        await rm(path, { force: true });
+        return;
+    }
+    await writeTextAtomically(path, previousContent);
+}
+async function restoreSecret(secretStore, profileId, previousSecret) {
+    if (previousSecret) {
+        await secretStore.save(profileId, previousSecret);
+        return;
+    }
+    await secretStore.delete(profileId);
+}
+function isMissingFileError(error) {
+    return Boolean(error && typeof error === "object" && "code" in error && error.code === "ENOENT");
+}
+function formatIndex(index) {
+    if (!index)
+        return "unknown";
+    return Array.isArray(index) ? index.join(",") : index;
+}
+function asErrorMessage(error) {
+    return error instanceof Error ? error.message : String(error);
+}
+async function rollbackClientRegistration(registrar) {
+    let lastError;
+    for (let attempt = 0; attempt < 2; attempt += 1) {
+        try {
+            await registrar.rollback();
+            return;
+        }
+        catch (error) {
+            lastError = error;
+        }
+    }
+    throw lastError;
+}