tradeblocks-mcp 3.11.0 → 4.1.0

package/README.md

@@ -14,6 +14,102 @@ Model Context Protocol (MCP) server for options trading analysis. Works with Cla
 
 ## Installation
 
+### Prerequisites (before using npm or npx)
+
+Open a terminal (PowerShell on Windows) and run `node --version`, `npm --version`,
+and `npx --version`. TradeBlocks' packaged server requires Node **18 or newer**;
+Node **24 LTS** is recommended (and used for development). Your chosen client
+may require a newer Node version, so Node 24 is the easiest shared choice.
+
+If any command is missing, install Node **with npm**, then close and reopen the terminal:
+
+| Platform | Install Node/npm                                                                                                                                                                           |
+| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| macOS    | Use the macOS installer from [nodejs.org](https://nodejs.org/en/download), or `brew install node` if you already use Homebrew.                                                             |
+| Windows  | Use the Windows installer from [nodejs.org](https://nodejs.org/en/download), or `winget install OpenJS.NodeJS.LTS`. Reopen PowerShell afterward.                                           |
+| Linux    | Follow the Linux instructions at [nodejs.org](https://nodejs.org/en/download) for your distribution or version manager; choose Node 24 with npm rather than an older distribution package. |
+
+Then repeat the three version checks. Install/open your chosen client separately:
+[Claude Desktop](https://claude.ai/download),
+[Claude Code](https://code.claude.com/docs/en/setup),
+[Codex CLI](https://developers.openai.com/codex/cli/) (`npm install -g @openai/codex`),
+or [Gemini CLI](https://geminicli.com/docs/get-started/installation/)
+(`npm install -g @google/gemini-cli`). Setup does not install any of these.
+Desktop's configuration directory must already exist; on Linux follow your Desktop
+distribution's instructions. Local setup does not need OO, a provider key, or a skills plugin.
+
+### Guided local setup
+
+```bash
+# Interactive: select a client and folder, review changes, explicitly approve
+npx tradeblocks-mcp setup
+
+# Agent/non-interactive: preview only (non-zero consent_required result)
+npx tradeblocks-mcp setup --client codex --folder "/path/to/backtests" --json
+
+# Apply the reviewed preview and verify
+npx tradeblocks-mcp setup --client codex --folder "/path/to/backtests" --yes --json
+```
+
+Client choices are `claude-desktop`, `claude-code`, `codex`, and `gemini`.
+Interactive setup shows all four clients numbered with detected/not-detected
+markers and accepts a number or client name; it never selects one automatically.
+Without `--json`, the preview and result are readable prose, including the
+quoted launch command, verification, client action and next steps.
+Folder paths are resolved to absolute paths, including paths with spaces. Setup
+previews the exact file, entry and official user-scope registration command.
+It can create a missing data folder only under the same consent. Without a TTY,
+setup never prompts and requires `--client`, `--folder`, and `--yes` to write.
+`--json` emits exactly one final result object on stdout; a redacted preview goes
+to stderr before any write, even with `--yes`. Inspect `plannedChange` in a
+preview run before approving it with `--yes`. Exit 0 means both registration
+(or an equivalent existing entry) **and** MCP verification succeeded.
+
+A differing `tradeblocks` entry needs separate replacement permission:
+interactive setup asks separately, or agents add **both** `--replace` and
+`--yes`. Replacement preserves existing `env` keys **and their values** and removes
+other TradeBlocks-specific launch settings. Unknown entry fields (including URLs)
+are conflicts, not equivalent stdio registrations. Gemini replacement refuses
+environment values containing `=` before writing: its CLI truncates those values;
+update the launch manually in its settings file while retaining the environment.
+Previews show differing field names and environment key names, never existing
+values or client diagnostics. Desktop updates preserve unrelated settings/servers,
+use a same-directory temporary file and atomic rename, and back up existing files.
+For all CLI clients, setup backs up an existing config before registration and
+restores it if remove/add fails or read-back differs from the preview. The vendor
+CLI may itself normalize or remove unrelated settings on successful registration
+(Claude Code can drop unknown top-level keys); setup does not promise to undo a
+successful vendor rewrite. The backup retains the original bytes. Backups may
+contain secrets; protect them like the original and remove them when no longer needed.
+
+Setup refuses unreadable or malformed configuration without rewriting it. This
+includes Gemini settings with comments: repair/export strict JSON yourself
+before using guided setup; manual configuration remains available below.
+CLI registration uses `claude mcp add -s user`, `codex mcp add`, or
+`gemini mcp add -s user`, not project-scope edits. Desktop uses the per-OS paths
+below. The registered command uses the absolute `npx` found on your PATH.
+For Desktop, setup also supplies Node/npm's directory in the server's `PATH`
+when no existing server `PATH` is present, because a GUI may not inherit an
+nvm shell's PATH. Moving/upgrading that Node installation can require setup
+again; an existing server `PATH` is preserved, not overridden.
+
+Verification starts the **read-back configured server subprocess**, performs MCP
+`initialize` and `tools/list` with a 120-second timeout, and reports server
+name/version and tool count. It does **not** prove that the AI client itself
+connected. Restart/reopen Desktop or start a new CLI-client session afterward.
+If verification fails, registration may already have succeeded: inspect the
+result's `appliedChange`, fix the indicated prerequisites, and rerun.
+The bounded two-minute window allows for a first npx launch downloading the
+package and native DuckDB binaries. Setup explains that possibility before
+verification starts.
+The server's first npx launch may need network access to download TradeBlocks;
+setup does not install clients, Node/npm, or change system settings.
+
+For **Claude Code only**, the separate
+[tradeblocks-skills plugin](https://github.com/tradeblocks-org/tradeblocks-skills)
+is optional and is not installed by setup. Manual CSV import and core analysis
+remain available without the plugin, OO, or market-data credentials.
+
 ### Option 1: npx (All Platforms)
 
 Run directly without installation:
@@ -37,7 +133,7 @@ npm install
 npm run build -w packages/mcp-server
 
 # Run the server
-node packages/mcp-server/server/index.js ~/Trading/backtests
+node packages/mcp-server/server/cli.js ~/Trading/backtests
 ```
 
 ## Quick Start
@@ -251,23 +347,17 @@ Or set `TRADEBLOCKS_NO_AUTH=true` in `.env`.
 
 ## Agent Skills
 
-For guided conversational workflows, install the bundled agent skills:
+For guided conversational workflows, install the agent skills from their standalone plugin,
+[tradeblocks-skills](https://github.com/tradeblocks-org/tradeblocks-skills). In Claude Code:
 
 ```bash
-# Install skills to Claude Code
-tradeblocks-mcp install-skills
-
-# Install to other platforms
-tradeblocks-mcp install-skills --platform codex
-tradeblocks-mcp install-skills --platform gemini
-
-# Check installation status
-tradeblocks-mcp check-skills
-
-# Remove skills
-tradeblocks-mcp uninstall-skills
+/plugin marketplace add tradeblocks-org/tradeblocks-skills
+/plugin install tradeblocks@tradeblocks-skills
 ```
 
+The former `install-skills`, `check-skills`, and `uninstall-skills` commands now only print these
+instructions.
+
 Skills provide structured prompts for tasks like:
 
 - Strategy health checks
@@ -275,7 +365,68 @@ Skills provide structured prompts for tasks like:
 - Portfolio addition recommendations
 - Correlation analysis
 
-See [Agent Skills README](../agent-skills/README.md) for details.
+See the [tradeblocks-skills README](https://github.com/tradeblocks-org/tradeblocks-skills) for details.
+
+## Prompts
+
+The same MCP server exposes five prompts in stdio and HTTP. In Claude Code, when
+the server is named `tradeblocks`, pick them from the `/` menu, where they appear
+as `/tradeblocks:bring-in-oo-backtest (MCP)`, `/tradeblocks:is-this-optimum-real (MCP)`,
+`/tradeblocks:stress-oo-portfolio (MCP)`, `/tradeblocks:live-vs-oo (MCP)` and
+`/tradeblocks:allocate-oo-portfolio (MCP)`, or type
+`/mcp__tradeblocks__bring-in-oo-backtest` (likewise for the others).
+Other MCP clients can list and get these prompts through their prompt interface.
+The server name is chosen by the user; these are not Option Omega tool prefixes.
+
+Each prompt takes optional arguments, typed after the command in Claude Code in
+this order, for example
+`/mcp__tradeblocks__bring-in-oo-backtest <savedBacktestId> my-strategy`:
+`bring-in-oo-backtest` and `stress-oo-portfolio` take `ooId` (an OO saved
+backtest or portfolio ID, or a run's `runId`) and `block`; `is-this-optimum-real`
+takes `optimizationId`; `live-vs-oo` takes `block`; `allocate-oo-portfolio`
+takes `ooId` (saved portfolio ID or run ID) and `block`. Without arguments the
+prompts work from the conversation.
+
+| Prompt                  | Purpose                                                                                                               |
+| ----------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| `bring-in-oo-backtest`  | Capture a backtest's trade log as a block, or import an OO-exported trade CSV.                                        |
+| `is-this-optimum-real`  | Compare the optimizer's best cell with a stable-region candidate using scratch runs and TradeBlocks robustness tests. |
+| `stress-oo-portfolio`   | Stress a portfolio's economic trades in an existing or imported block.                                                |
+| `live-vs-oo`            | Compare live trades in a reporting log with an OO reference backtest's block.                                         |
+| `allocate-oo-portfolio` | Check TradeBlocks allocation leads in OO shared-funds runs; never alter the saved source.                             |
+
+TradeBlocks never calls Option Omega. With the `tradeblocks-skills` Claude Code
+plugin installed, the import prompt uses `/tradeblocks:oo-capture`, and the
+optimum prompt directs to `/tradeblocks:is-this-optimum-real`. That skill
+captures each scratch run separately and combines verified trades under
+distinct strategies in one trade-only comparison block for a paired
+best-minus-centre test. This test covers jointly traded days, not the
+optimizer's selection from the grid. Without the plugin,
+`bring-in-oo-backtest` asks for OO's trade-log and daily-log CSVs and imports
+both in one `import_csv` call with `dailyLogPath`, so the block carries OO's
+marked daily curve; it says so when the daily log is missing. The optimum
+prompt without the plugin has each run's trade-log CSV exported from OO and
+imported into separate blocks: it runs tests against zero per block and states
+that no paired difference ran. The CSV must
+be readable by the TradeBlocks server; for Docker/HTTP, put it inside the
+mounted server data directory. If unavailable, stop instead of rebuilding
+CSV from model responses.
+
+`allocate-oo-portfolio` and the read-only `stress-oo-portfolio` use
+`/tradeblocks:oo-capture` for a saved portfolio only when the installed skill
+describes saved-portfolio capture (older plugin versions do not); otherwise
+they use an existing block, or import OO's portfolio trade-log CSV
+with `import_csv` and optionally its whole-book daily log with `dailyLogPath`.
+TradeBlocks' correlation, marginal, tail, what-if and health results are
+trade-derived counterfactual proposals, never OO marked equity or OO headline
+figures. The whole-book curve has no per-member marks. A candidate is OO-tested
+only after `get_portfolio_status` reports complete and `get_portfolio_results`
+is read; saving requires an explicit request and a new portfolio. OO's headline
+and marked-equity figures remain OO's; TradeBlocks' realized trade statistics
+do not replace marked-account drawdown.
+
+`live-vs-oo` reads the reporting log from the OO reference block's own folder
+(see below); put the CSV there instead of importing it as a separate block.
 
 ## Block Directory Structure
 
@@ -297,20 +448,25 @@ backtests/
 **tradelog.csv** - Trade records with these key columns:
 
 - Date Opened, Time Opened, Date Closed, Time Closed
-- P/L (gross profit/loss)
-- Strategy, Legs (or Symbol)
-- No. of Contracts, Premium (optional)
+- P/L (Option Omega exports already include fees; `import_csv` defaults to
+  `plBasis: "net_includes_fees"`. Use `"gross_before_fees"` only if fees still need deducting)
+- Strategy (optional; the block ID is used when missing), Legs (or Symbol)
+- No. of Contracts, Premium (optional; dollars per contract: `250` and `250.00` both mean $250)
+- P/L % (optional; when numeric, report tools use Option Omega's value as the trade's `plPct`;
+  otherwise `plPct` is P/L ÷ |Premium × No. of Contracts| × 100)
 
 **dailylog.csv** - Daily portfolio values:
 
 - Date
-- Net Liquidity (or Portfolio Value, Equity)
+- Net Liquidity (or Portfolio Value, Value, Equity)
 - P/L, Drawdown % (optional)
 
 **Flexible Detection**: Files don't need standard names. The server detects CSV types by examining column headers (ISS-006).
 
 ## Available Tools
 
+These tables show common tools; [MCP Tools Reference](../../docs/mcp-tools.md) lists every registered tool.
+
 ### Core Tools
 
 | Tool                      | Description                                           |
@@ -321,6 +477,12 @@ backtests/
 | `get_strategy_comparison` | Compare strategies within a block                     |
 | `compare_blocks`          | Compare statistics across multiple blocks             |
 
+Trade-calendar dates in tool output are calendar days in `YYYY-MM-DD` form and name
+the same day on every server timezone. That covers a trade's opened or closed day, a
+daily-log day, and any range or window built from them, such as `dateRange`,
+walk-forward windows, `peakExposure` dates, and `run_sql` DATE columns. Real instants,
+such as `calculatedAt`, sync times, and market-feed timestamps, remain ISO timestamps.
+
 ### Analysis Tools
 
 | Tool                     | Description                                              |
@@ -348,21 +510,35 @@ backtests/
 
 ### Market Data Tools
 
-| Tool                         | Description                                                                        |
-| ---------------------------- | ---------------------------------------------------------------------------------- |
-| `import_market_csv`          | Import market data CSV with column mapping                                         |
-| `import_from_database`       | Import from external DuckDB databases                                              |
-| `import_flat_file`           | Import a local Parquet or CSV flat file for a ticker/timespan                      |
-| `fetch_bars`                 | Fetch daily or intraday OHLCV bars from configured provider                        |
-| `fetch_quotes`               | Fetch option minute quotes from configured provider                                |
-| `fetch_chain`                | Fetch option chain snapshot for an underlying on a given date                      |
-| `compute_vix_context`        | Compute cross-ticker VIX regime fields for a date range                            |
-| `refresh_market_data`        | Composite daily refresh: fetch bars, auto-fire VIX context, return coverage report |
-| `enrich_market_data`         | Compute ~40 derived indicators from raw OHLCV                                      |
-| `enrich_trades`              | Enrich trades with market context (lookahead-free)                                 |
-| `analyze_regime_performance` | Analyze P&L by market regime                                                       |
-| `suggest_filters`            | Suggest trade filters based on market conditions                                   |
-| `calculate_orb`              | Opening range breakout analysis from intraday bars                                 |
+| Tool                         | Description                                                                         |
+| ---------------------------- | ----------------------------------------------------------------------------------- |
+| `import_market_csv`          | Import market data CSV with column mapping                                          |
+| `import_from_database`       | Import from external DuckDB databases                                               |
+| `import_flat_file`           | Import a local Parquet or CSV flat file for a ticker/timespan                       |
+| `fetch_bars`                 | Fetch daily or intraday OHLCV bars from configured provider                         |
+| `fetch_quotes`               | Fetch option minute quotes from configured provider                                 |
+| `fetch_chain`                | Fetch option chain snapshot for an underlying on a given date                       |
+| `compute_vix_context`        | Compute cross-ticker VIX regime fields for a date range                             |
+| `refresh_market_data`        | Composite daily refresh: fetch bars, auto-fire VIX context, return coverage report  |
+| `enrich_market_data`         | Compute ~40 derived indicators from raw OHLCV                                       |
+| `enrich_trades`              | Enrich trades with causal market/dataset values, provenance and custom fields       |
+| `analyze_regime_performance` | Analyze P&L by presets or any registered field (numeric bins or categorical values) |
+| `suggest_filters`            | Default market filters or optional fields with generic threshold/exclusion sweeps   |
+| `calculate_orb`              | Opening range breakout analysis from intraday bars                                  |
+| `import_daily_dataset`       | Import or replace a user Static Dataset with per-column timing                      |
+| `list_daily_datasets`        | List user datasets, fields, timing and coverage                                     |
+| `delete_daily_dataset`       | Remove a dataset and its registered fields                                          |
+| `list_analysis_fields`       | Discover built-in and dataset fields at runtime                                     |
+
+Since 4.1.0, user daily datasets use `<dataset>.<column>` field names in profile filters,
+predictive reports, regime analysis and threshold sweeps. `import_daily_dataset` takes
+`name`, exactly one of inline `content` or `csvPath` under the server's data root, and optional
+`ticker`, `columnTiming` and `knownBy` (`HH:mm` ET). Re-import replaces the named dataset
+atomically, without changing market data or requiring provider credentials. Date-only close
+values are available the following calendar day at 00:00 ET; date-only open values require
+`knownBy`; timestamped rows are as-of. Sparse columns carry forward their latest available
+observation without a freshness cutoff. See the [dataset tools reference](../../docs/mcp-tools.md#market-analysis)
+and [4.1.0 behavior changes](../../releases/v4.1.0.md#behavior-changes).
 
 ### Strategy Profile Tools
 
@@ -376,11 +552,41 @@ backtests/
 | `validate_entry_filters`  | Test each entry filter's contribution to edge                |
 | `portfolio_structure_map` | Regime x structure coverage matrix across strategies         |
 
+`validate_entry_filters` names unknown market fields with reasons in
+`unresolvable_filters`, warnings, and its summary; they are excluded from ablation
+so known filters remain analyzable. If all market filters are unknown, it returns
+`error: "no_resolvable_filters"`. `analyze_structure_fit` reports the same exclusions
+and keeps its remaining dimensions. Known fields with missing daily values retain
+their existing missing-data treatment.
+
 ### Import Tools
 
-| Tool         | Description                                                                     |
-| ------------ | ------------------------------------------------------------------------------- |
-| `import_csv` | Import a CSV file as a new block _(CLI only - not available in Claude Desktop)_ |
+| Tool         | Description                                              |
+| ------------ | -------------------------------------------------------- |
+| `import_csv` | Import a CSV file as a block from a server-readable path |
+
+`import_csv` accepts `csvPath`, `blockName`, optional `csvType` (default `tradelog`),
+optional `dailyLogPath` for a paired daily log, and `plBasis` (default
+`net_includes_fees` for Option Omega exports; `gross_before_fees` when fee columns
+still need deducting). Both paths must be readable by the server: local for stdio,
+or inside its mounted data directory for Docker/HTTP. Paths can be absolute, use
+`~`, or use filename-only search via `searchPaths` (default: Downloads, Desktop, Documents).
+The pair creates one block with a stamped `tradelog.csv` and verbatim `dailylog.csv`.
+A bad daily log rejects the whole import without creating a block. The usual
+result fields describe the primary CSV; its `dateRange` holds the first and last
+calendar dates (`YYYY-MM-DD`).
+Paired results also include `dailyLog.recordCount` and `dailyLog.dateRange`
+in the same calendar-date form. Unfiltered `get_statistics` uses daily-log drawdown;
+strategy-filtered statistics use the trade log instead.
+
+Before writing a block, `import_csv` refuses an invalid opened/closed trade or reporting
+calendar date, missing or unparseable P/L, or missing/unparseable daily-log value. For
+gross-before-fees trade imports, both commission values are also required. Errors
+identify the CSV file line including the header; a rejected pair leaves no block.
+Daily-log aliases load as net liquidity in imported and folder-discovered blocks;
+folder loading drops a row with a missing/unparseable value rather than using zero.
+`recordCount` counts loaded rows, and a missing Strategy in a trade log uses the
+block ID in the receipt and in subsequent tools.
 
 ## Development
 
@@ -428,6 +634,6 @@ See the [Market Data Guide](../../docs/market-data.md) for import examples, tick
 
 - [Usage Guide](../../docs/usage.md) - Detailed usage examples and workflows
 - [Web Platforms Guide](../../docs/web-platforms.md) - Connect to ChatGPT, Google AI Studio, Julius
-- [Agent Skills](../agent-skills/README.md) - Conversational workflows for guided analysis
+- [Agent Skills](https://github.com/tradeblocks-org/tradeblocks-skills) - Conversational workflows for guided analysis
 - [Market Data Guide](../../docs/market-data.md) - Import workflow, Massive API, and column mapping reference
 - [Main Application](../../README.md) - Web-based UI for TradeBlocks