mikroscope 0.0.1 → 1.0.0

package/README.md

@@ -1,8 +1,8 @@
 # MikroScope
 
-**Log sidecar using NDJSON: ingest, query, retention, and alerts.**
+**Ultralight log sidecar for Node.js that turns NDJSON into instant queries and actionable webhook alerts.**
 
-MikroScope runs next to your service, writes/reads `.ndjson` logs, indexes them into SQLite, and exposes an HTTP API for search and aggregation.
+MikroScope runs next to your service, writes/reads `.ndjson` logs, indexes them into SQLite, and exposes an HTTP API for search and aggregation with millisecond latency.
 
 ## What You Get
 
@@ -21,30 +21,28 @@ MikroScope runs next to your service, writes/reads `.ndjson` logs, indexes them
 
 ## Install
 
-| Requirement                   | Notes                                        |
-|-------------------------------|----------------------------------------------|
-| Node.js `>= 24`               | Required to run MikroScope                   |
-| npm                           | Required for npm install / npx flows         |
-| `curl` or `wget`              | Used by installer to download release assets |
-| `tar` (or `unzip` for `.zip`) | Needed to extract release archive            |
-
-| Method                    | Recommended for                           | Command                            |
-|---------------------------|-------------------------------------------|------------------------------------|
-| npm global install        | Node/npm environments with persistent CLI | `npm install -g mikroscope`        |
-| npx                       | One-off execution without global install  | `npx mikroscope --help`            |
-| One-line installer        | VM/server installs without npm dependency | See command below                  |
-| Non-interactive installer | CI/provisioning via release assets        | See command below                  |
-| Manual release archive    | Pinned/manual installs                    | See "Manual Release Install" below |
+Requirements:
+
+- Node.js `>= 24` (required to run MikroScope).
+- `npm` (required for npm install / npx flows).
+- `curl` or `wget` (used by installer to download release assets).
+- `tar` (or `unzip` for `.zip`) to extract release archives.
+
+Install methods:
+
+- `npm install -g mikroscope` for persistent CLI usage in Node/npm environments.
+- `npx mikroscope --help` for one-off execution without global install.
+- One-line installer (below) for VM/server installs without npm dependency.
+- Non-interactive installer (below) for CI/provisioning.
+- Manual release archive install (see "Manual Release Install" below) for pinned/manual installs.
 
 Installer behavior:
 
-| Step                    | Result                                                     |
-|-------------------------|------------------------------------------------------------|
-| Download latest release | Fetches latest tagged archive from GitHub Releases         |
-| Verify checksum         | Uses `SHA256SUMS.txt` when available                       |
-| Expand archive          | Installs binaries/docs under your chosen install directory |
-| Prompt for config       | Writes host/port/path/auth settings to a local env file    |
-| Create wrapper          | Adds a `mikroscope` command in your chosen bin directory   |
+- Download latest release from GitHub Releases.
+- Verify checksum using `SHA256SUMS.txt` when available.
+- Expand archive into your chosen install directory.
+- Prompt for host/port/path/auth config values.
+- Create a `mikroscope` wrapper command in your chosen bin directory.
 
 npm install examples:
 
@@ -77,7 +75,7 @@ If `mikroscope` is not found after install, add your chosen bin directory to `PA
 ### Manual Release Install
 
 ```bash
-VERSION=0.0.1
+VERSION=1.0.0
 curl -LO "https://github.com/mikaelvesavuori/mikroscope/releases/download/v${VERSION}/mikroscope-${VERSION}.tar.gz"
 curl -LO "https://github.com/mikaelvesavuori/mikroscope/releases/download/v${VERSION}/SHA256SUMS.txt"
 shasum -a 256 -c SHA256SUMS.txt
@@ -127,6 +125,252 @@ curl -sS "http://127.0.0.1:4310/api/logs?field=producerId&value=backend-api&limi
 
 If `/docs` is blank (for example blocked CDN scripts), use `/openapi.json` directly.
 
+## Configuration Inputs
+
+`mikroscope serve`, `mikroscope index`, `mikroscope query`, and `mikroscope aggregate` resolve configuration with layered precedence:
+
+1. Built-in defaults
+1. JSON config file (`--config` or `MIKROSCOPE_CONFIG_PATH`; default: `./mikroscope.config.json` if present)
+1. Environment variables (`MIKROSCOPE_*`)
+1. Direct CLI flags (highest precedence)
+
+### mikroscope.config.json
+
+MikroScope will automatically read `./mikroscope.config.json` when it exists in your current working directory.
+
+Quick setup:
+
+1. Copy the example template from this repo:
+
+```bash
+cp ./examples/mikroscope.config.json ./mikroscope.config.json
+```
+
+1. Set your real secrets/tokens in `./mikroscope.config.json`.
+1. Start MikroScope:
+
+```bash
+mikroscope serve
+```
+
+If your config is not in the current directory, set an explicit path:
+
+```bash
+mikroscope serve --config /absolute/or/relative/path/mikroscope.config.json
+```
+
+Config file keys use camelCase (for example `ingestQueueFlushMs`, `alertErrorThreshold`) and match the server option names.
+
+Template (`./examples/mikroscope.config.json`):
+
+```json
+{
+  "dbPath": "./data/mikroscope.db",
+  "logsPath": "./logs",
+  "host": "127.0.0.1",
+  "port": 4310,
+  "protocol": "http",
+  "apiToken": "api-token-123",
+  "ingestProducers": "backend-token=backend-api,frontend-token=frontend-web",
+  "ingestAsyncQueue": true,
+  "alertWebhookUrl": "https://example.com/webhook"
+}
+```
+
+You can still override config-file values via env vars or CLI flags at runtime.
+
+Use it explicitly:
+
+```bash
+mikroscope serve --config ./mikroscope.config.json
+```
+
+Env-only example:
+
+```bash
+MIKROSCOPE_DB_PATH=./data/mikroscope.db \
+MIKROSCOPE_LOGS_PATH=./logs \
+MIKROSCOPE_PORT=4310 \
+MIKROSCOPE_API_TOKEN=api-token-123 \
+mikroscope serve
+```
+
+Programmatic usage with direct params:
+
+```ts
+import { resolveMikroScopeServerOptions } from "./src/application/config/resolveMikroScopeServerOptions.js";
+import { startMikroScopeServer } from "./src/server.js";
+
+const options = resolveMikroScopeServerOptions({
+  configFilePath: "./mikroscope.config.json",
+  overrides: {
+    port: 4320
+  }
+});
+
+await startMikroScopeServer(options);
+```
+
+## Generate Mock Data
+
+If you want realistic logs to inspect quickly, generate synthetic NDJSON files from a source checkout of this repository.
+
+```bash
+# Standard dataset (writes to ./logs)
+npm run mock-data
+
+# Smaller/faster dataset
+npm run mock-data:quick
+```
+
+What this does:
+
+- Writes normal logs to `./logs/generated/*.ndjson`.
+- Writes audit logs to `./logs/audit/generated/*.ndjson`.
+- Uses deterministic random data (seeded) so runs are reproducible.
+
+Useful tuning options:
+
+- `MOCK_LOG_DAYS` (default: `21`)
+- `MOCK_LOGS_PER_DAY` (default: `1200`)
+- `MOCK_AUDIT_LOGS_PER_DAY` (default: `150`)
+- `MOCK_LOG_TENANTS` (default: `320`)
+- `MOCK_LOG_SEED` (default: `2602`)
+- `MOCK_LOG_OUT_DIR` (default: `./logs`)
+
+Example custom run:
+
+```bash
+MOCK_LOG_DAYS=14 \
+MOCK_LOGS_PER_DAY=800 \
+MOCK_AUDIT_LOGS_PER_DAY=100 \
+MOCK_LOG_TENANTS=120 \
+npm run mock-data
+```
+
+Then index and run:
+
+```bash
+npm run index
+npm start
+```
+
+## Optional UI: MikroScope Console
+
+MikroScope works with the optional [MikroScope Console](https://github.com/mikaelvesavuori/mikroscope-console), a static web UI for exploring logs and correlations.
+
+Basic setup:
+
+1. Start MikroScope:
+
+```bash
+mikroscope serve --host 127.0.0.1 --port 4310 --logs ./logs --db ./data/mikroscope.db --cors-allow-origin http://127.0.0.1:4320
+```
+
+1. Install Console (from the console repo README):
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/mikaelvesavuori/mikroscope-console/main/install.sh -o install.sh && sh install.sh && rm install.sh
+```
+
+1. Set Console API target in `public/config.json`:
+
+```json
+{
+  "apiOrigin": "http://127.0.0.1:4310"
+}
+```
+
+1. Serve Console static files and open it:
+
+```bash
+npx http-server public -p 4320 -c-1
+```
+
+MikroScope endpoints used by Console:
+
+- `GET /health`
+- `GET /api/logs`
+- `GET /api/logs/aggregate`
+- `GET /api/alerts/config`
+- `PUT /api/alerts/config`
+- `POST /api/alerts/test-webhook`
+
+## Manual Webhook Alert Demo
+
+If you want to manually experience alerting (not test runner output), use the helper script:
+
+```bash
+# Start demo in error-threshold mode
+npm run demo:alerts -- up error
+
+# Trigger ERROR logs (should fire webhook)
+npm run demo:alerts -- trigger-error
+
+# Check alerting state from /health
+npm run demo:alerts -- status
+
+# Inspect recent webhook + server logs
+npm run demo:alerts -- logs
+
+# Stop demo
+npm run demo:alerts -- down
+```
+
+No-logs mode:
+
+```bash
+npm run demo:alerts -- up nologs
+```
+
+Then wait ~60-70 seconds without sending logs and check `npm run demo:alerts -- logs` for a `rule: "no_logs"` webhook.
+
+## Remote Alert Configuration
+
+Alert config can be managed over API and is persisted to disk so it survives restarts/reboots.
+
+Default config file path:
+
+- `<db-directory>/mikroscope.alert-config.json`
+
+Override path:
+
+- CLI: `--alert-config-path /path/to/mikroscope.alert-config.json`
+- Env: `MIKROSCOPE_ALERT_CONFIG_PATH=/path/to/mikroscope.alert-config.json`
+
+Examples:
+
+```bash
+# Read current policy
+curl -sS "http://127.0.0.1:4310/api/alerts/config" \
+  -H "Authorization: Bearer api-token-123"
+
+# Update and persist policy
+curl -sS "http://127.0.0.1:4310/api/alerts/config" \
+  -H "Authorization: Bearer api-token-123" \
+  -H "Content-Type: application/json" \
+  -X PUT \
+  --data '{
+    "enabled": true,
+    "webhookUrl": "https://example.com/webhook",
+    "intervalMs": 30000,
+    "windowMinutes": 5,
+    "errorThreshold": 20,
+    "noLogsThresholdMinutes": 0,
+    "cooldownMs": 300000,
+    "webhookTimeoutMs": 5000,
+    "webhookRetryAttempts": 3,
+    "webhookBackoffMs": 250
+  }'
+
+# Send a manual test webhook event
+curl -sS "http://127.0.0.1:4310/api/alerts/test-webhook" \
+  -H "Authorization: Bearer api-token-123" \
+  -H "Content-Type: application/json" \
+  -X POST \
+  --data '{}'
+```
+
 ## CLI Commands
 
 | Command                                                                      | Use case                          |
@@ -143,6 +387,9 @@ If `/docs` is blank (for example blocked CDN scripts), use `/openapi.json` direc
 | `POST /api/ingest`        | `Bearer <ingest-token>` mapped by `--ingest-producers`, or Basic auth if configured | Always server-assigned. Incoming `producerId` in payload is ignored/overridden. |
 | `GET /api/logs`           | `Bearer <api-token>` and/or Basic auth (if enabled)                                 | N/A                                                                             |
 | `GET /api/logs/aggregate` | `Bearer <api-token>` and/or Basic auth (if enabled)                                 | N/A                                                                             |
+| `GET /api/alerts/config`  | `Bearer <api-token>` and/or Basic auth (if enabled)                                 | N/A                                                                             |
+| `PUT /api/alerts/config`  | `Bearer <api-token>` and/or Basic auth (if enabled)                                 | N/A                                                                             |
+| `POST /api/alerts/test-webhook` | `Bearer <api-token>` and/or Basic auth (if enabled)                          | N/A                                                                             |
 | `POST /api/reindex`       | `Bearer <api-token>` and/or Basic auth (if enabled)                                 | N/A                                                                             |
 
 Notes:
@@ -171,6 +418,9 @@ Notes:
 |--------|-----------------------|-------------------------------------------|------------------------------------------|
 | `GET`  | `/health`             | None                                      | Runtime health and policy/status details |
 | `POST` | `/api/ingest`         | Ingest bearer token mapping or Basic auth | Accept and persist inbound logs          |
+| `GET`  | `/api/alerts/config`  | API bearer token and/or Basic auth        | Read active alert policy + config path   |
+| `PUT`  | `/api/alerts/config`  | API bearer token and/or Basic auth        | Update and persist alert policy          |
+| `POST` | `/api/alerts/test-webhook` | API bearer token and/or Basic auth   | Send manual webhook test event           |
 | `GET`  | `/api/logs`           | API bearer token and/or Basic auth        | Paginated log query                      |
 | `GET`  | `/api/logs/aggregate` | API bearer token and/or Basic auth        | Bucketed counts                          |
 | `POST` | `/api/reindex`        | API bearer token and/or Basic auth        | Full DB reset + reindex from logs        |
@@ -207,8 +457,10 @@ Query parameters for `/api/logs/aggregate`:
 |-----------------------|------------------------|------------------------------------------------|
 | `--logs`              | `./logs`               | NDJSON root directory                          |
 | `--db`                | `./data/mikroscope.db` | SQLite database file                           |
+| `--config`            | `./mikroscope.config.json` | JSON config file path (if present)         |
 | `--host`              | `127.0.0.1`            | Bind host                                      |
 | `--port`              | `4310`                 | Bind port                                      |
+| `--protocol`          | `http`                 | Listener protocol (`http` or `https`)          |
 | `--https`             | `false`                | Enable HTTPS listener                          |
 | `--tls-cert`          | none                   | TLS certificate path (required with `--https`) |
 | `--tls-key`           | none                   | TLS key path (required with `--https`)         |
@@ -253,6 +505,7 @@ Query parameters for `/api/logs/aggregate`:
 | `--alert-webhook-timeout-ms`        | `5000`   | Webhook timeout per request                     |
 | `--alert-webhook-retry-attempts`    | `3`      | Webhook retry attempts per alert                |
 | `--alert-webhook-backoff-ms`        | `250`    | Base backoff between webhook retries            |
+| `--alert-config-path`               | db dir   | Path to persisted alert config JSON             |
 
 ## Example Integrations
 
@@ -333,5 +586,5 @@ Use this only if you want to develop MikroScope itself.
 ```bash
 npm install
 npm run index
-npm run start
+npm start
 ```