@tikoci/rosetta 0.3.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (10) hide show
  1. package/README.md +135 -197
  2. package/package.json +1 -1
  3. package/src/db.ts +31 -1
  4. package/src/extract-test-results.ts +359 -0
  5. package/src/mcp-http.test.ts +443 -0
  6. package/src/mcp.ts +92 -17
  7. package/src/query.test.ts +47 -3
  8. package/src/query.ts +38 -3
  9. package/src/release.test.ts +106 -0
  10. package/src/setup.ts +22 -0
package/README.md CHANGED
@@ -1,66 +1,111 @@
1
1
  # rosetta
2
2
 
3
- MCP server for searching [MikroTik RouterOS documentation](https://help.mikrotik.com/docs/spaces/ROS/overview). Gives your AI assistant searchable access to 317 documentation pages, 4,860 property definitions, 40,000-entry command tree, and 144 hardware product specs — with direct links to help.mikrotik.com.
3
+ MCP server that gives AI assistants searchable access to the complete [MikroTik RouterOS documentation](https://help.mikrotik.com/docs/spaces/ROS/overview) — 317 pages, 4,860 properties, 40,000-entry command tree, hardware specs for 144 products, and direct links to help.mikrotik.com.
4
4
 
5
- Tested with **Claude Desktop**, **Claude Code**, **VS Code Copilot** (including Copilot CLI), **Cursor**, and **OpenAI Codex** on macOS, Linux, and Windows.
5
+ If you need MikroTik docs, you likely have a MikroTik. Install rosetta once as a container on your router using [RouterOS /app](#install-on-mikrotik-app), and any AI assistant on the network can use it. Or [run it locally](#install-locally-with-bun) on your workstation.
6
6
 
7
- ## What is SQL-as-RAG?
7
+ ### SQL-as-RAG
8
8
 
9
- Most retrieval-augmented generation (RAG) systems use vector embeddings to search documentation. This project takes a different approach: **SQLite [FTS5](https://www.sqlite.org/fts5.html) full-text search as the retrieval layer** — what we call SQL-as-RAG.
9
+ Instead of vector embeddings, rosetta uses **SQLite [FTS5](https://www.sqlite.org/fts5.html) full-text search** as the retrieval layer — SQL-as-RAG. For structured technical docs, [BM25 ranking](https://www.sqlite.org/fts5.html#the_bm25_function) with [porter stemming](https://www.sqlite.org/fts5.html#porter_tokenizer) beats vector similarity: terms like `dhcp-snooping` and `/ip/firewall/filter` are exact tokens, not fuzzy embeddings. No API keys, no vector database — just a single SQLite file that searches in milliseconds.
10
10
 
11
- For structured technical documentation like RouterOS, full-text search with [BM25 ranking](https://www.sqlite.org/fts5.html#the_bm25_function) beats vector similarity. Technical terms like "dhcp-snooping" or "/ip/firewall/filter" are exact tokens — [porter stemming](https://www.sqlite.org/fts5.html#porter_tokenizer) and proximity matching handle the rest. No embedding pipeline, no vector database, no API keys. Just a single SQLite file that searches in milliseconds.
12
-
13
- The data flows: **HTML docs → SQLite extraction → FTS5 indexes → MCP tools → your AI assistant.** The database is built once from MikroTik's official Confluence documentation export, then the MCP server exposes 11 search tools over stdio or HTTP transport.
14
-
15
- ## What's Inside
11
+ ### What's Inside
16
12
 
17
13
  - **317 documentation pages** from MikroTik's official help site (~515K words)
18
14
  - **4,860 property definitions** with types, defaults, and descriptions
19
15
  - **5,114 commands** in the RouterOS command hierarchy (551 directories, 34K arguments)
20
16
  - **1,034 callout blocks** — warnings, notes, and tips with important caveats
21
17
  - **144 hardware products** — CPU, RAM, storage, ports, PoE, wireless, license level, pricing
18
+ - **2,874 performance benchmarks** — ethernet and IPSec throughput test results for 125 devices (64/512/1518-byte packets, multiple routing/bridging modes), plus block diagrams for 110
22
19
  - **46 RouterOS versions tracked** (7.9 through 7.23beta2) for command history
23
20
  - Direct links to help.mikrotik.com for every page and section
24
21
 
25
- ## Quick Start
22
+ ---
26
23
 
27
- ## MCP Discovery Status
24
+ ## Install on MikroTik (/app)
28
25
 
29
- - GitHub MCP Registry listing: planned
30
- - Official MCP Registry publication: metadata is now prepared in `server.json`
26
+ RouterOS 7.22+ includes the [/app](https://help.mikrotik.com/docs/spaces/ROS/pages/328068) feature for running containers directly on the router. This is the simplest way to deploy rosetta — install once, and any AI assistant on your network can connect to the MCP endpoint URL shown in the router UI.
31
27
 
32
- Local install remains the primary path today (`bunx @tikoci/rosetta`).
28
+ **Requirements:** RouterOS 7.22+, x86 or ARM64 architecture (CCR, RB5009, hAP ax series, CHR, etc.), container package installed, device-mode enabled.
33
29
 
34
- When ready to publish to the official registry:
30
+ ### 1. Enable containers (two reboots required)
35
31
 
36
- ```sh
37
- brew install mcp-publisher
38
- mcp-publisher validate server.json
39
- mcp-publisher login github
40
- mcp-publisher publish server.json
32
+ If you haven't already enabled the container package and device-mode:
33
+
34
+ ```routeros
35
+ # Install the container package (router reboots automatically)
36
+ /system/package/update/check-for-updates duration=10s
37
+ /system/package/enable container
38
+ # Apply changes restarts the router
41
39
  ```
42
40
 
43
- After publication, the server should be discoverable via:
41
+ After reboot:
44
42
 
45
- ```sh
46
- curl "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.tikoci/rosetta"
43
+ ```routeros
44
+ # Enable container device-mode (requires physical power cycle or button press — follow the on-screen prompt)
45
+ /system/device-mode/update mode=advanced container=yes
47
46
  ```
48
47
 
49
- ### Option A: Install with Bun (recommended)
48
+ See MikroTik's [Container documentation](https://help.mikrotik.com/docs/spaces/ROS/pages/Container) for full prerequisites and troubleshooting.
49
+
50
+ ### 2. Add the rosetta app
51
+
52
+ ```routeros
53
+ /app/add use-https=yes disabled=no yaml="name: rosetta
54
+ descr: \"RouterOS Docs for AI assistants - use URL as MCP server\"
55
+ page: https://tikoci.github.io/p/rosetta
56
+ category: development
57
+ icon: https://tikoci.github.io/p/rosetta.svg
58
+ default-credentials: \"none - just use 'ui-url' as the MCP server in your AI assistant\"
59
+ url-path: /mcp
60
+ auto-update: true
61
+ services:
62
+ rosetta:
63
+ image: ghcr.io/tikoci/rosetta:latest
64
+ container_name: mcp-server
65
+ ports:
66
+ - 9803:8080/tcp:web
67
+ "
68
+ ```
50
69
 
51
- Zero install, zero config, no binary signing issues. Requires [Bun](https://bun.sh/) — no Gatekeeper or SmartScreen warnings since there's no compiled binary to sign.
70
+ That's it. RouterOS downloads the container image, configures networking and firewall redirects, and starts the MCP server. The `auto-update: true` setting pulls the latest image on each boot.
52
71
 
53
- **1. Install Bun** (if you don't have it):
72
+ ### 3. Get the MCP endpoint URL
54
73
 
55
- ```sh
56
- # macOS / Linux
57
- curl -fsSL https://bun.sh/install | bash
74
+ The URL to use with your AI assistant is shown as **UI URL** in WebFig (App → rosetta), or from the CLI:
58
75
 
59
- # Windows
60
- powershell -c "irm bun.sh/install.ps1 | iex"
76
+ ```routeros
77
+ :put [/app/get rosetta ui-url]
61
78
  ```
62
79
 
63
- **2. Configure your MCP client** with `bunx @tikoci/rosetta` as the command. No setup step needed — the database downloads automatically on first launch (~50 MB compressed).
80
+ This URL includes the `/mcp` path and is ready to paste into any MCP client that supports HTTP transport. With `use-https=yes`, the URL uses HTTPS with a MikroTik-managed `*.routingthecloud.net` certificate.
81
+
82
+ ### 4. Configure your AI assistant
83
+
84
+ Point any HTTP-capable MCP client at the URL from the previous step:
85
+
86
+ ```json
87
+ { "url": "https://app-rosetta.XXX.routingthecloud.net/mcp" }
88
+ ```
89
+
90
+ > **CHR note:** Cloud Hosted Router in free or trial mode does not include the `/ip/cloud` service needed for HTTPS certificates. Set `use-https=no` on the /app — the URL will use HTTP instead. The UI URL always reflects the correct protocol.
91
+
92
+ > **HTTP option:** On any platform, you may choose `use-https=no` if you prefer HTTP or are on an isolated network.
93
+
94
+ ---
95
+
96
+ ## Install Locally (with Bun)
97
+
98
+ Run rosetta on your workstation using [Bun](https://bun.sh/). The MCP server runs over stdio — no network configuration needed. The database downloads automatically on first launch (~50 MB compressed).
99
+
100
+ ### Quick setup
101
+
102
+ ```sh
103
+ bunx @tikoci/rosetta --setup
104
+ ```
105
+
106
+ This downloads the database and prints config snippets for all supported MCP clients. Copy-paste the config for your client and you're done.
107
+
108
+ ### Or configure manually
64
109
 
65
110
  <details>
66
111
  <summary><b>VS Code Copilot</b></summary>
@@ -98,8 +143,6 @@ Edit your Claude Desktop config file:
98
143
  - **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
99
144
  - **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
100
145
 
101
- Add (or merge into existing config):
102
-
103
146
  ```json
104
147
  {
105
148
  "mcpServers": {
@@ -111,22 +154,9 @@ Add (or merge into existing config):
111
154
  }
112
155
  ```
113
156
 
114
- > **PATH note:** Claude Desktop on macOS doesn't always inherit your shell PATH. If `bunx` isn't found, use the full path — typically `~/.bun/bin/bunx`. Run `which bunx` to find it, or use `bunx @tikoci/rosetta --setup` which prints the full-path config for you.
115
-
116
- Then **restart Claude Desktop**.
117
-
118
- </details>
119
-
120
- <details>
121
- <summary><b>GitHub Copilot CLI</b></summary>
122
-
123
- Inside a `copilot` session, type `/mcp add` to open the interactive form:
157
+ > **PATH note:** Claude Desktop on macOS doesn't always inherit your shell PATH. If `bunx` isn't found, use the full path (typically `~/.bun/bin/bunx`). Run `bunx @tikoci/rosetta --setup` to print the full-path config.
124
158
 
125
- - **Server Name:** `routeros-rosetta`
126
- - **Server Type:** 2 (STDIO)
127
- - **Command:** `bunx @tikoci/rosetta`
128
-
129
- Press <kbd>Tab</kbd> to navigate fields, <kbd>Ctrl+S</kbd> to save.
159
+ Restart Claude Desktop after editing.
130
160
 
131
161
  </details>
132
162
 
@@ -155,21 +185,38 @@ Open **Settings → MCP** and add a new server:
155
185
  codex mcp add rosetta -- bunx @tikoci/rosetta
156
186
  ```
157
187
 
158
- > **Note:** ChatGPT Apps require a remote HTTPS MCP endpoint and cannot use local stdio servers like this one. Codex (CLI and desktop app) supports stdio and works with `bunx`.
188
+ > **Note:** ChatGPT Apps require a remote HTTPS MCP endpoint. Use the [MikroTik /app install](#install-on-mikrotik-app) or another container platform for a hosted endpoint, or Codex CLI for local stdio.
159
189
 
160
190
  </details>
161
191
 
162
- **That's it.** First launch takes a moment to download the database; subsequent starts are instant. The database is stored in `~/.rosetta/ros-help.db`.
192
+ <details>
193
+ <summary><b>GitHub Copilot CLI</b></summary>
194
+
195
+ Inside a `copilot` session, type `/mcp add`:
196
+
197
+ - **Server Name:** `routeros-rosetta`
198
+ - **Server Type:** 2 (STDIO)
199
+ - **Command:** `bunx @tikoci/rosetta`
200
+
201
+ </details>
202
+
203
+ **Install Bun** (if you don't have it):
204
+
205
+ ```sh
206
+ # macOS / Linux
207
+ curl -fsSL https://bun.sh/install | bash
208
+
209
+ # Windows
210
+ powershell -c "irm bun.sh/install.ps1 | iex"
211
+ ```
163
212
 
164
- > **Verify it works:** Run `bunx @tikoci/rosetta --setup` to see the database status and print config for all MCP clients.
165
- >
166
- > **Auto-update:** `bunx` checks the npm registry each session and uses the latest published version automatically. No manual update needed. (Note: the `~/.rosetta/ros-help.db` database persists across updates — it's re-downloaded only when missing or when you run `--setup --force`.)
213
+ > **Auto-update:** `bunx` checks the npm registry each session and uses the latest published version automatically. The database in `~/.rosetta/ros-help.db` persists across updates.
167
214
 
168
- ### Option B: Pre-built binary (no runtime needed)
215
+ ---
169
216
 
170
- Download a compiled binary from [Releases](https://github.com/tikoci/rosetta/releases) — no Bun, Node.js, or other tools required.
217
+ ## Install from Binary
171
218
 
172
- **1. Download** the ZIP for your platform from the [latest release](https://github.com/tikoci/rosetta/releases/latest):
219
+ Download a compiled binary from [Releases](https://github.com/tikoci/rosetta/releases) — no Bun, Node.js, or other runtime needed.
173
220
 
174
221
  | Platform | File |
175
222
  |----------|------|
@@ -178,23 +225,16 @@ Download a compiled binary from [Releases](https://github.com/tikoci/rosetta/rel
178
225
  | Windows | `rosetta-windows-x64.zip` |
179
226
  | Linux | `rosetta-linux-x64.zip` |
180
227
 
181
- Extract the ZIP to a permanent location (e.g., `~/rosetta` or `C:\rosetta`).
182
-
183
- **2. Run setup** to download the database and see MCP client config:
184
-
185
228
  ```sh
186
- ./rosetta --setup
229
+ ./rosetta --setup # downloads DB + prints MCP client config
187
230
  ```
188
231
 
189
- On Windows: `.\rosetta.exe --setup`
190
-
191
- > **macOS Gatekeeper:** If macOS blocks the binary: `xattr -d com.apple.quarantine ./rosetta` or go to **System Settings → Privacy & Security → Allow Anyway**.
192
- >
232
+ > **macOS Gatekeeper:** `xattr -d com.apple.quarantine ./rosetta` or System Settings → Privacy & Security → Allow Anyway.
193
233
  > **Windows SmartScreen:** Click **More info → Run anyway**.
194
234
 
195
- **3. Configure your MCP client** using the config printed by `--setup`. It uses the full path to the binary — paste it into your MCP client's config as shown.
235
+ ---
196
236
 
197
- ### Try It
237
+ ## Try It
198
238
 
199
239
  Ask your AI assistant questions like:
200
240
 
@@ -204,6 +244,7 @@ Ask your AI assistant questions like:
204
244
  - *"What are the firewall filter default chains?"*
205
245
  - *"Show me warnings about hardware offloading"*
206
246
  - *"Which MikroTik routers have L3HW offload, and more than 8 ports of 48V PoE? Include cost."*
247
+ - *"Compare the RB5009 and CCR2004 IPSec throughput at 1518-byte packets."*
207
248
 
208
249
  ## MCP Tools
209
250
 
@@ -219,7 +260,7 @@ The server provides 11 tools, designed to work together:
219
260
  | `routeros_search_callouts` | Search warnings, notes, and tips across all pages |
220
261
  | `routeros_search_changelogs` | Search parsed changelog entries — filter by version range, category, breaking changes |
221
262
  | `routeros_command_version_check` | Check which RouterOS versions include a command |
222
- | `routeros_device_lookup` | Hardware specs for 144 MikroTik products — filter by architecture, RAM, storage, PoE, wireless, LTE |
263
+ | `routeros_device_lookup` | Hardware specs for 144 MikroTik products — filter by architecture, RAM, storage, PoE, wireless, LTE. Includes ethernet/IPSec benchmarks and block diagrams for most devices |
223
264
  | `routeros_stats` | Database health: page/property/command counts, coverage stats |
224
265
  | `routeros_current_versions` | Fetch current RouterOS versions from MikroTik (live) |
225
266
 
@@ -232,15 +273,15 @@ The AI assistant typically starts with `routeros_search`, then drills into speci
232
273
  | **First launch is slow** | One-time database download (~50 MB). Subsequent starts are instant. |
233
274
  | **`npx @tikoci/rosetta` fails** | This package requires Bun, not Node.js. Use `bunx` instead of `npx`. |
234
275
  | **`npm install -g` then `rosetta` fails** | Global npm install works if Bun is on PATH — it delegates to `bun` at runtime. But prefer `bunx` — it's simpler and auto-updates. |
235
- | **ChatGPT Apps can't connect with `bunx @tikoci/rosetta`** | Expected: ChatGPT Apps supports remote HTTPS MCP endpoints, not local stdio command launch. Use OpenAI Codex for local stdio, or deploy/tunnel a remote MCP URL for ChatGPT. |
276
+ | **ChatGPT Apps can't connect** | ChatGPT Apps require a remote HTTPS MCP endpoint. Use the [MikroTik /app install](#install-on-mikrotik-app) for a hosted endpoint, or Codex CLI for local stdio. |
236
277
  | **Claude Desktop can't find `bunx`** | Claude Desktop on macOS may not inherit shell PATH. Use the full path to bunx (run `which bunx` to find it, typically `~/.bun/bin/bunx`). `bunx @tikoci/rosetta --setup` prints the full-path config. |
237
278
  | **macOS Gatekeeper blocks binary** | Use `bunx` install (no Gatekeeper issues), or: `xattr -d com.apple.quarantine ./rosetta` |
238
279
  | **Windows SmartScreen warning** | Use `bunx` install (no SmartScreen issues), or click **More info → Run anyway** |
239
- | **How to update** | `bunx` always uses the latest published version. For binaries, re-download from [Releases](https://github.com/tikoci/rosetta/releases/latest). |
280
+ | **How to update** | `bunx` always uses the latest published version. For binaries, re-download from [Releases](https://github.com/tikoci/rosetta/releases/latest). MikroTik /app with `auto-update: true` pulls the latest image on each boot. |
240
281
 
241
282
  ## HTTP Transport
242
283
 
243
- Most MCP clients use stdio (the default). Some — like the OpenAI platform and remote/LAN setups — require an HTTP endpoint instead. Rosetta supports the [MCP Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports#streamable-http) via the `--http` flag:
284
+ The [MikroTik /app install](#install-on-mikrotik-app) is the easiest way to get an HTTP endpoint. For other setups, rosetta supports the [MCP Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-03-26/basic/transports#streamable-http) via `--http`:
244
285
 
245
286
  ```sh
246
287
  rosetta --http # http://localhost:8080/mcp
@@ -254,155 +295,47 @@ Then point your MCP client at the URL:
254
295
  { "url": "http://localhost:8080/mcp" }
255
296
  ```
256
297
 
257
- **Key facts:**
258
-
259
- - **Read-only** — the server queries a local SQLite database. It does not store data, accept uploads, or modify anything.
260
- - **No authentication** — designed for local/trusted-network use. For public exposure, put it behind a reverse proxy (nginx, caddy) with TLS and auth.
261
- - **TLS built-in** — for direct HTTPS without a proxy: `--tls-cert cert.pem --tls-key key.pem` (or `TLS_CERT_PATH` + `TLS_KEY_PATH` env vars)
262
- - **Defaults to localhost** — binding to all interfaces (`--host 0.0.0.0`) requires an explicit flag and logs a warning.
263
- - **Origin validation** — rejects cross-origin requests to prevent DNS rebinding attacks.
264
- - **Stdio remains default** — `--http` is opt-in. Existing stdio configs are unaffected.
265
-
266
- The `PORT`, `HOST`, `TLS_CERT_PATH`, and `TLS_KEY_PATH` environment variables are supported (lower precedence than CLI flags).
298
+ - **Read-only** — queries a local SQLite database, stores nothing.
299
+ - **No authentication** — designed for local/trusted-network use. Use a reverse proxy for public exposure.
300
+ - **TLS built-in** — `--tls-cert cert.pem --tls-key key.pem` for direct HTTPS without a proxy.
301
+ - **Defaults to localhost** — LAN binding (`--host 0.0.0.0`) requires an explicit flag.
267
302
 
268
303
  ## Container Images
269
304
 
270
- Release CI publishes multi-arch OCI images to:
271
-
272
- - Docker Hub: `ammo74/rosetta`
273
- - GHCR: `ghcr.io/tikoci/rosetta`
274
-
275
- Tags per release:
276
-
277
- - `${version}` (example: `v0.2.1`)
278
- - `latest`
279
- - `sha-<12-char-commit>`
305
+ Multi-arch OCI images (linux/amd64 + linux/arm64) are published with each release:
280
306
 
281
- Container defaults:
282
-
283
- - Starts in HTTP mode (`--http`) on `0.0.0.0`
284
- - Uses `PORT` if set, otherwise 8080
285
- - Uses HTTPS only when both `TLS_CERT_PATH` and `TLS_KEY_PATH` are set
286
-
287
- Examples:
307
+ - `ghcr.io/tikoci/rosetta` (GitHub Container Registry)
308
+ - `ammo74/rosetta` (Docker Hub)
288
309
 
289
310
  ```sh
290
311
  docker run --rm -p 8080:8080 ghcr.io/tikoci/rosetta:latest
291
312
  ```
292
313
 
293
- ```sh
294
- docker run --rm -p 8443:8443 \
295
- -e PORT=8443 \
296
- -e TLS_CERT_PATH=/certs/cert.pem \
297
- -e TLS_KEY_PATH=/certs/key.pem \
298
- -v "$PWD/certs:/certs:ro" \
299
- ghcr.io/tikoci/rosetta:latest
300
- ```
301
-
302
- ## Building from Source
303
-
304
- For contributors or when you have access to the MikroTik HTML documentation export.
305
-
306
- ### Prerequisites
307
-
308
- - [Bun](https://bun.sh/) v1.1+
309
- - RouterOS HTML documentation export (Confluence space export)
310
- - Internet access to [tikoci/restraml GitHub Pages](https://tikoci.github.io/restraml/) for command-tree extraction
311
-
312
- `make extract` and `make extract-full` fetch `inspect.json` from restraml GitHub Pages by default. You can still pass a local source explicitly:
313
-
314
- ```sh
315
- bun run src/extract-commands.ts /path/to/restraml/docs/7.22.1/extra/inspect.json
316
- bun run src/extract-all-versions.ts /path/to/restraml/docs
317
- ```
318
-
319
- ### Build
320
-
321
- ```sh
322
- git clone https://github.com/tikoci/rosetta.git
323
- cd rosetta
324
- bun install
325
- ```
326
-
327
- Place the Confluence HTML export in `box/documents-export-<date>/ROS/` and symlink `box/latest` to it:
328
-
329
- ```sh
330
- ln -s documents-export-<date> box/latest
331
- ```
332
-
333
- Then:
334
-
335
- ```sh
336
- make extract # HTML → properties → commands (single version) → link
337
- # or
338
- make extract-full # Same but with all 46 RouterOS versions
339
- ```
340
-
341
- ### Development
342
-
343
- ```sh
344
- bun test # Run tests (in-memory SQLite, no DB needed)
345
- bun run typecheck # Type check
346
- make lint # Biome linter
347
- bun run src/mcp.ts # Start MCP server in dev mode
348
- ```
349
-
350
- The repo includes `.vscode/mcp.json` — opening the folder in VS Code automatically configures Copilot to use the dev server.
351
-
352
- ### Creating a Release
353
-
354
- The Makefile handles the full release flow — preflight checks, cross-compile, git tag, push, and GitHub Release upload:
355
-
356
- ```sh
357
- make release VERSION=v0.1.0
358
- ```
359
-
360
- This cross-compiles to macOS (arm64 + x64), Windows (x64), and Linux (x64), creates ZIP archives, compresses the database, tags the commit, and creates a GitHub Release with all artifacts.
361
-
362
- The release workflow also publishes OCI images to Docker Hub (`ammo74/rosetta`) and GHCR (`ghcr.io/tikoci/rosetta`) using crane (no Docker daemon required in CI).
363
-
364
- ## Project Structure
365
-
366
- ```text
367
- src/
368
- ├── mcp.ts # MCP server (11 tools, stdio + HTTP) + CLI dispatch
369
- ├── setup.ts # --setup: DB download + MCP client config
370
- ├── query.ts # NL → FTS5 query planner, BM25 ranking
371
- ├── db.ts # SQLite schema, WAL mode, FTS5 triggers
372
- ├── extract-html.ts # Confluence HTML → pages + callouts
373
- ├── extract-properties.ts # Property table extraction
374
- ├── extract-commands.ts # inspect.json → commands (version-aware)
375
- ├── extract-all-versions.ts # Batch extract all 46 versions
376
- ├── extract-devices.ts # Product matrix CSV → devices table
377
- ├── link-commands.ts # Command ↔ page mapping
378
- └── query.test.ts # Tests (in-memory SQLite fixtures)
379
-
380
- scripts/
381
- └── build-release.ts # Cross-compile + package releases
382
- └── container-entrypoint.sh # OCI image runtime entrypoint (HTTP default)
383
- ```
314
+ These are the same images used by the [MikroTik /app install](#install-on-mikrotik-app). Tags: `latest`, version (e.g., `v0.2.1`), and `sha-<commit>`.
384
315
 
385
316
  ## Data Sources
386
317
 
387
- The database combines three sources of MikroTik data into a single SQLite file with full-text search ([FTS5](https://www.sqlite.org/fts5.html) with [porter stemming](https://www.sqlite.org/fts5.html#porter_tokenizer) and [BM25 ranking](https://www.sqlite.org/fts5.html#the_bm25_function)):
318
+ The database combines multiple MikroTik data sources into a single SQLite file with [FTS5](https://www.sqlite.org/fts5.html) full-text search, [porter stemming](https://www.sqlite.org/fts5.html#porter_tokenizer), and [BM25 ranking](https://www.sqlite.org/fts5.html#the_bm25_function):
388
319
 
389
- - **HTML Documentation** — Confluence space export from help.mikrotik.com (March 2026). The 317 pages are broken out several ways: by section heading, callout boxes (warnings, tips, notes), and property tables (attribute name/type/default/description). This is the richest source — ~515K words of official documentation with direct links back to help.mikrotik.com.
320
+ - **HTML Documentation** — Confluence space export from help.mikrotik.com (March 2026). 317 pages broken into sections, callouts, and property tables (~515K words) with links back to help.mikrotik.com.
390
321
 
391
- - **Command Tree** — `inspect.json` files from [tikoci/restraml](https://github.com/tikoci/restraml), which runs `/console/inspect` requests (`child`, `syntax`, `completion`) against RouterOS CHR under QEMU via GitHub Actions for every version since 7.9. This gives the MCP server structured knowledge of whether a command or argument exists in a particular version (46 versions tracked: 7.9–7.23beta2). The blind spot: these come from CHR with all extra-packages on x86, so commands from packages not available on CHR (like `zerotier` and Wi-Fi driver packages) are missing — the HTML docs cover those.
322
+ - **Command Tree** — `inspect.json` from [tikoci/restraml](https://github.com/tikoci/restraml), generated by running `/console/inspect` against RouterOS CHR under QEMU for every version since 7.9 (46 versions tracked: 7.9–7.23beta2).
392
323
 
393
- - **Product Matrix** — CSV export from mikrotik.com/products/matrix (144 products, 34 columns). Hardware specs, license levels, and pricing — lets the AI answer questions like *"Which routers have L3HW offload and 8+ ports of 48V PoE?"* The CSV requires manual download (the old POST API was removed when the site was redesigned in late 2025).
324
+ - **Product Matrix** — CSV export from mikrotik.com/products/matrix (144 products, 34 columns). Hardware specs, license levels, and pricing.
325
+
326
+ - **Device Benchmarks** — Ethernet bridging/routing and IPSec throughput test results scraped from individual product pages on mikrotik.com (2,874 measurements across 125 devices; 64/512/1518-byte packets, multiple configurations). Also captures block diagram image URLs for 110 devices.
394
327
 
395
328
  Documentation covers RouterOS **v7 only** and aligns with the long-term release (~7.22) at export time. v6 had different syntax and major subsystems — answers for v6 are unreliable.
396
329
 
397
330
  ## Database (Standalone)
398
331
 
399
- The SQLite database is distributed separately from the MCP server code via GitHub Releases:
332
+ The SQLite database is downloadable on its own from [GitHub Releases](https://github.com/tikoci/rosetta/releases):
400
333
 
401
334
  ```text
402
335
  https://github.com/tikoci/rosetta/releases/latest/download/ros-help.db.gz
403
336
  ```
404
337
 
405
- The MCP server downloads this automatically on first run (or via `--setup`), but the database is usable on its own with any SQLite client:
338
+ Use it with any SQLite client:
406
339
 
407
340
  ```sh
408
341
  sqlite3 ros-help.db "SELECT title, url FROM pages_fts WHERE pages_fts MATCH 'DHCP lease' ORDER BY rank LIMIT 5;"
@@ -420,8 +353,13 @@ sqlite3 ros-help.db "SELECT title, url FROM pages_fts WHERE pages_fts MATCH 'DHC
420
353
  | `command_versions` | 1.67M | Junction table: which command paths exist in which RouterOS versions (7.9–7.23beta2) |
421
354
  | `ros_versions` | 46 | Tracked RouterOS versions with channel (stable/development) |
422
355
  | `devices` | 144 | MikroTik hardware — CPU, RAM, storage, ports, PoE, wireless, license level, MSRP |
356
+ | `device_test_results` | 2,874 | Ethernet and IPSec throughput benchmarks for 125 devices — packet sizes, modes, Mbps/Kpps |
357
+
358
+ Each content table has a corresponding FTS5 index (e.g., `pages_fts`, `properties_fts`, `devices_fts`).
359
+
360
+ ## Contributing
423
361
 
424
- Each content table has a corresponding [FTS5](https://www.sqlite.org/fts5.html) index (e.g., `pages_fts`, `properties_fts`, `devices_fts`) using the [porter](https://www.sqlite.org/fts5.html#porter_tokenizer) stemming tokenizer for natural language search with [BM25 ranking](https://www.sqlite.org/fts5.html#the_bm25_function).
362
+ See [CONTRIBUTING.md](CONTRIBUTING.md) for building from source, running tests, development setup, and the release process.
425
363
 
426
364
  ## License
427
365
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@tikoci/rosetta",
3
- "version": "0.3.0",
3
+ "version": "0.4.0",
4
4
  "description": "RouterOS documentation as SQLite FTS5 — RAG search + command glossary via MCP",
5
5
  "type": "module",
6
6
  "license": "MIT",
package/src/db.ts CHANGED
@@ -250,9 +250,20 @@ export function initDb() {
250
250
  sim_slots INTEGER,
251
251
  memory_cards TEXT,
252
252
  usb_type TEXT,
253
- msrp_usd REAL
253
+ msrp_usd REAL,
254
+ product_url TEXT,
255
+ block_diagram_url TEXT
254
256
  );`);
255
257
 
258
+ // Migration: add product_url and block_diagram_url columns if missing
259
+ const devCols = db.prepare("PRAGMA table_info(devices)").all() as Array<{ name: string }>;
260
+ if (!devCols.some((c) => c.name === "product_url")) {
261
+ db.run("ALTER TABLE devices ADD COLUMN product_url TEXT;");
262
+ }
263
+ if (!devCols.some((c) => c.name === "block_diagram_url")) {
264
+ db.run("ALTER TABLE devices ADD COLUMN block_diagram_url TEXT;");
265
+ }
266
+
256
267
  db.run(`CREATE VIRTUAL TABLE IF NOT EXISTS devices_fts USING fts5(
257
268
  product_name, product_code, architecture, cpu,
258
269
  content=devices,
@@ -275,6 +286,23 @@ export function initDb() {
275
286
  VALUES (new.id, new.product_name, new.product_code, new.architecture, new.cpu);
276
287
  END;`);
277
288
 
289
+ // -- Device test results (from mikrotik.com product pages) --
290
+
291
+ db.run(`CREATE TABLE IF NOT EXISTS device_test_results (
292
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
293
+ device_id INTEGER NOT NULL REFERENCES devices(id),
294
+ test_type TEXT NOT NULL,
295
+ mode TEXT NOT NULL,
296
+ configuration TEXT NOT NULL,
297
+ packet_size INTEGER NOT NULL,
298
+ throughput_kpps REAL,
299
+ throughput_mbps REAL,
300
+ UNIQUE(device_id, test_type, mode, configuration, packet_size)
301
+ );`);
302
+
303
+ db.run(`CREATE INDEX IF NOT EXISTS idx_device_tests_device ON device_test_results(device_id);`);
304
+ db.run(`CREATE INDEX IF NOT EXISTS idx_device_tests_type ON device_test_results(test_type);`);
305
+
278
306
  // -- Changelogs (parsed per-entry from MikroTik download server) --
279
307
 
280
308
  db.run(`CREATE TABLE IF NOT EXISTS changelogs (
@@ -330,6 +358,8 @@ export function getDbStats() {
330
358
  commands: count("SELECT COUNT(*) AS c FROM commands"),
331
359
  commands_linked: count("SELECT COUNT(*) AS c FROM commands WHERE page_id IS NOT NULL"),
332
360
  devices: count("SELECT COUNT(*) AS c FROM devices"),
361
+ device_test_results: count("SELECT COUNT(*) AS c FROM device_test_results"),
362
+ devices_with_tests: count("SELECT COUNT(DISTINCT device_id) AS c FROM device_test_results"),
333
363
  changelogs: count("SELECT COUNT(*) AS c FROM changelogs"),
334
364
  changelog_versions: count("SELECT COUNT(DISTINCT version) AS c FROM changelogs"),
335
365
  ros_versions: count("SELECT COUNT(*) AS c FROM ros_versions"),