@langwatch/mcp-server 0.0.5 → 0.1.0

package/.env.example

@@ -0,0 +1,2 @@
+LANGWATCH_API_KEY=
+ANTHROPIC_API_KEY=

package/.eslintrc.cjs

@@ -1,4 +1,3 @@
-/** @type {import("eslint").Linter.Config} */
 const config = {
   parser: "@typescript-eslint/parser",
   parserOptions: {

package/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+## [0.1.0](https://github.com/langwatch/langwatch/compare/mcp-server@v0.0.5...mcp-server@v0.1.0) (2025-09-19)
+
+
+### Features
+
+* added auto setup functionality for langwatch mcp ([#617](https://github.com/langwatch/langwatch/issues/617)) ([8c95b07](https://github.com/langwatch/langwatch/commit/8c95b07598a74285940b0c9267368543a9ced5e0))
+* ci/cd steps for all packages and deployables, including improvements to caching and bundle sizes ([#351](https://github.com/langwatch/langwatch/issues/351)) ([e67a169](https://github.com/langwatch/langwatch/commit/e67a1694fec2f96479266454403928e9dc68a20f))
+
+
+### Bug Fixes
+
+* add missing dotenv dependency for running tests ([fb706ce](https://github.com/langwatch/langwatch/commit/fb706ceef9a298d070b264ad8b6da7c2df5e2a5d))
+* judge agent for mcp-server test ([cd8e378](https://github.com/langwatch/langwatch/commit/cd8e3783ec02f02174ecb5fd86fa86c3f11e1734))
+* mcp-server ci ([0ab6e51](https://github.com/langwatch/langwatch/commit/0ab6e513129d9b1fbdb7a696ce1d99ed6093dea3))
+* run claude-code on the CI ([d760307](https://github.com/langwatch/langwatch/commit/d760307807c72a2a0e995a4f0a42845c2cc5114a))
+
+
+### Documentation
+
+* add detailed markdown documentation for LangWatch eval notebook ([#618](https://github.com/langwatch/langwatch/issues/618)) ([525b62a](https://github.com/langwatch/langwatch/commit/525b62ad6ea01f122297b1a3fd1eb7e842479f19))
+* added mcp-server contributing guide ([19d1431](https://github.com/langwatch/langwatch/commit/19d14313824663842e5bba3a98986b9b80382300))
+* improve notebook descriptions ([fa1f267](https://github.com/langwatch/langwatch/commit/fa1f26705bfff3143dbd6d16edfdae86bd5ce6bd))
+
+
+### Code Refactoring
+
+* split tool call fix helper ([c95028f](https://github.com/langwatch/langwatch/commit/c95028fba882357b33ca975e9d08ceabfe5cfc1c))

package/CONTRIBUTING.md

@@ -0,0 +1,96 @@
+# Contributing to LangWatch MCP Server
+
+Thank you for your interest in contributing to the LangWatch MCP Server! This guide will help you get set up for development and understand our testing approach.
+
+## Development Setup
+
+### Prerequisites
+
+- Node.js and pnpm
+- Python with uv package manager
+- Git
+
+### Getting Started
+
+1. **Clone the repository and navigate to the MCP server directory:**
+   ```bash
+   git clone https://github.com/langwatch/langwatch.git
+   cd langwatch/mcp-server
+   ```
+
+2. **Install dependencies and build the MCP server:**
+   ```bash
+   pnpm install
+   pnpm run build
+   ```
+
+3. **Configure environment variables:**
+   ```bash
+   cp .env.example .env
+   ```
+
+   Fill in the following required variables in your `.env` file:
+   - `LANGWATCH_API_KEY` - Your LangWatch project API key
+   - `ANTHROPIC_API_KEY` - Your Anthropic API key for Claude Code integration
+
+4. **Install Python dependencies (for evaluation notebooks):**
+   ```bash
+   uv sync
+   ```
+
+## Testing Approach
+
+This project follows the **[Agent Testing Pyramid](https://scenario.langwatch.ai/best-practices/the-agent-testing-pyramid/)** methodology, which provides a structured approach to testing AI agents across three layers:
+
+### 1. Unit Tests (Foundation)
+Traditional software tests for deterministic components like API connections, data pipelines, and error handling.
+
+### 2. Evals & Optimization (Middle Layer)
+Component-level evaluation and optimization of probabilistic AI components, including prompt effectiveness and retrieval accuracy.
+
+### 3. Simulations (Peak)
+End-to-end testing that validates the complete agent behavior in realistic scenarios.
+
+## Running Tests
+
+### Quick Evaluations (Jupyter Notebook)
+
+For rapid iteration and component testing:
+
+```bash
+# Open the evaluation notebook in VS Code/Cursor
+code tests/evaluations.ipynb
+```
+
+The notebook contains lightweight tests that directly test the MCP server with a "mocked" coding agent on single files. These are useful for:
+- Quick validation of MCP tool functionality
+- Testing individual instrumentation patterns
+- Rapid prototyping of new features
+
+### End-to-End Simulations
+
+For comprehensive system validation:
+
+```bash
+pnpm test
+```
+
+This runs full simulation tests using the Scenario framework, which:
+- Launches actual Claude Code sessions
+- Uses the MCP server in a real development environment
+- Tests complete workflows on entire codebases
+- Validates that the agent can successfully instrument various AI frameworks (OpenAI, LangChain, DSPy, etc.)
+
+When tests run successfully, you'll see:
+- LangWatch Scenario interface opening
+- Terminal output showing Claude Code using MCP tools
+- Validation of code instrumentation at the end of each scenario
+
+## Questions?
+
+If you encounter any issues or have questions about the setup, please:
+- Check existing GitHub issues
+- Create a new issue with detailed reproduction steps
+- Join our Discord community for real-time support
+
+Happy contributing! 🚀

package/README.md

@@ -1,8 +1,8 @@
 # LangWatch 🏰 MCP Server
 
-The LangWatch MCP Server is a tool designed to aid finding, searching, and looking up LLM traces from the LangWatch platform via the [Model Context Protocol](https://modelcontextprotocol.io/introduction).
+The LangWatch MCP Server is a tool designed to automatically instrument your AI code with LangWatch monitoring via the [Model Context Protocol](https://modelcontextprotocol.io/introduction).
 
-This server facilitates LLM development by allowing the agent to search for traces, understand all the steps in between a problematic output and try to fix the issue.
+This server facilitates LLM development by helping AI coding assistants automatically add LangWatch instrumentation to your codebase, then use those traces to analyze and debug the very AI agents they're building.
 
 ## Setup in your Codebase
 
@@ -15,8 +15,9 @@ Check out [LangWatch integration guide](https://docs.langwatch.ai/integration/ov
 3. Set the "name" as "LangWatch"
 4. Set the "type" to `command`
 5. Set the "command" to `npx -y @langwatch/mcp-server --apiKey=sk-lw-...`
+
 - `--apiKey`: Your LangWatch API key. This is mandatory and must be provided.
-- `--endpoint`: *Optional* The endpoint for the LangWatch API. Defaults to `https://app.langwatch.ai` if not specified.
+- `--endpoint`: _Optional_ The endpoint for the LangWatch API. Defaults to `https://app.langwatch.ai` if not specified.
 
 > [!TIP]
 > To aid in securing your keys, the MCP will first look at the global system environment variables `LANGWATCH_API_KEY` and `LANGWATCH_ENDPOINT` to check if they have values as well as looking at arguments passed into the server on start.
@@ -31,6 +32,12 @@ Check out [LangWatch integration guide](https://docs.langwatch.ai/integration/ov
 
 The MCP Server provides the following tools:
 
+### `fetch_langwatch_docs`
+
+- **Description:** Fetches the LangWatch docs for understanding how to implement LangWatch in your codebase.
+- **Parameters:**
+  - `url`: (Optional) The full url of the specific doc page. If not provided, the docs index will be fetched.
+
 ### `get_latest_traces`
 
 - **Description:** Retrieves the latest LLM traces.
@@ -49,12 +56,13 @@ The MCP Server provides the following tools:
 To use these tools within Cursor, follow these steps:
 
 1. **Open the Cursor Chat view:**
-    - `Cmd + I`
+
+   - `Cmd + I`
 
 2. **Ensure the MCP server is running:**
 
 3. **Interact with your Agent:**
-    - Ask a question like the following to test the tools are accessible: *Note: When the tool is detected, you'll need to run `Run tool` in the chat view for it to be called.
+   - Ask a question like the following to test the tools are accessible: \*Note: When the tool is detected, you'll need to run `Run tool` in the chat view for it to be called.
 
 > "I just ran into an issue while debugging, can you check the latest traces and fix it?"
 
@@ -64,7 +72,6 @@ To use these tools within Cursor, follow these steps:
 <img alt="LangWatch Logo" src="../assets/mcp-server/cursor-example.light.webp" width="900">
 </picture>
 
-
 ## 🛟 Support
 
 If you have questions or need help, join our community: