langsmith-cli 0.4.0__tar.gz → 0.4.2__tar.gz

{langsmith_cli-0.4.0 → langsmith_cli-0.4.2}/.claude-plugin/marketplace.json

@@ -6,14 +6,14 @@
   },
   "metadata": {
     "description": "LangSmith CLI plugin marketplace",
-    "version": "0.4.0"
+    "version": "0.4.2"
   },
   "plugins": [
     {
       "name": "langsmith-cli",
       "source": "./",
       "description": "A context-efficient interface for LangSmith observability and evaluations.",
-      "version": "0.4.0",
+      "version": "0.4.2",
       "author": {
         "name": "Gigaverse",
         "email": "aviadr1@gmail.com"

{langsmith_cli-0.4.0 → langsmith_cli-0.4.2}/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "langsmith-cli",
-  "version": "0.4.0",
+  "version": "0.4.2",
   "description": "A context-efficient interface for LangSmith observability and evaluations.",
   "author": {
     "name": "Aviad Rozenhek",

{langsmith_cli-0.4.0 → langsmith_cli-0.4.2}/CLAUDE.md

@@ -889,6 +889,68 @@ def test_mycommand_list(runner):
 
 **Python Version:** >=3.12
 
+## Releasing
+
+### ALWAYS Use the Release Script
+
+**Never manually bump versions or create tags/releases.** Use `scripts/release.sh` which handles everything:
+
+```bash
+# Patch bump (0.4.0 → 0.4.1) — default
+./scripts/release.sh
+
+# Minor bump (0.4.0 → 0.5.0)
+./scripts/release.sh minor
+
+# Major bump (0.4.0 → 1.0.0)
+./scripts/release.sh major
+
+# Explicit version
+./scripts/release.sh 0.5.0
+
+# Skip tests for docs-only releases
+./scripts/release.sh --skip-tests
+
+# Auto-confirm (no prompts)
+./scripts/release.sh -y
+```
+
+### What the Release Script Does
+
+1. **Bumps version in all 4 files simultaneously:**
+   - `pyproject.toml`
+   - `.claude-plugin/plugin.json`
+   - `.claude-plugin/marketplace.json` (both `metadata.version` and `plugins[0].version`)
+   - `uv.lock`
+2. **Runs quality checks:** ruff lint, ruff format, pyright
+3. **Runs tests** (unless `--skip-tests`)
+4. **Creates git commit and annotated tag**
+5. **Pushes to remote** (triggers CI → PyPI publish → GitHub release)
+
+### Version Files — Never Edit Manually
+
+These files contain version strings that **must stay in sync**:
+- `pyproject.toml` — PyPI package version
+- `.claude-plugin/plugin.json` — Plugin version for Claude Code
+- `.claude-plugin/marketplace.json` — Marketplace listing version (2 locations)
+- `uv.lock` — Auto-updated by the script
+
+Editing any of these manually creates version drift. The release script is the **single source of truth** for version bumps.
+
+### When to Release
+
+- **Code changes:** Always use `./scripts/release.sh` (runs tests by default)
+- **Docs/skill-only changes:** Use `./scripts/release.sh --skip-tests` (still bumps version)
+- **After updating SKILL.md:** A release is needed for plugin users to get the updated skill
+
+### CI/CD Pipeline (Automated)
+
+On tag push (`v*`), GitHub Actions automatically:
+1. Runs full test suite
+2. Builds wheel and sdist
+3. Publishes to PyPI (Trusted Publishing, no tokens)
+4. Creates GitHub release with artifacts
+
 ## Git Workflow
 
 Per docs/dev/SESSION_DIRECTIVES.md:

{langsmith_cli-0.4.0 → langsmith_cli-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: langsmith-cli
-Version: 0.4.0
+Version: 0.4.2
 Summary: Context-efficient CLI for LangSmith. Built for humans and agents.
 Project-URL: Homepage, https://github.com/aviadr1/langsmith-cli
 Project-URL: Repository, https://github.com/aviadr1/langsmith-cli

langsmith_cli-0.4.2/docs/devto-article.md

@@ -0,0 +1,214 @@
+---
+title: "I Replaced My LangSmith MCP Server with a CLI That Only Loads When You Need It"
+published: false
+description: "How langsmith-cli gives you 100% MCP parity, 96% less context per query, and features the MCP server doesn't have — all in a single pip install."
+tags: langsmith, llmops, cli, claude
+cover_image:
+---
+
+If you're using LangSmith with Claude Code (or any AI coding agent), you're probably running the official MCP server. It works. But every session, it injects **~5,000 tokens** of tool schemas into your context window — whether you touch LangSmith or not.
+
+I built [langsmith-cli](https://github.com/gigaverse-app/langsmith-cli) to fix that. It's a standalone CLI *and* a Claude Code plugin that replaces the always-on MCP server with an **on-demand skill** that only loads when your agent actually needs to talk to LangSmith.
+
+And it does more than the MCP server does.
+
+## The Problem with MCP Servers
+
+MCP servers are always-on. The moment your agent session starts, every tool definition gets loaded into context. For LangSmith's MCP server, that's 66 parameters across multiple tools — around 5,000 tokens of JSON schema sitting in your context window whether you ever query a trace or not.
+
+For agents that need to do many things — write code, run tests, debug, *and occasionally* check LangSmith — this is wasteful. Context is your agent's working memory. Every token of schema is a token not available for reasoning.
+
+## The Fix: On-Demand Skills Instead of Always-On Schemas
+
+`langsmith-cli` takes a different approach. Instead of an MCP server that injects schemas at session start, it's a CLI tool with a skill file that **only loads when the agent actually invokes it**:
+
+```bash
+# Install the CLI
+uv tool install langsmith-cli
+
+# Add as Claude Code plugin
+claude plugin marketplace add gigaverse-app/langsmith-cli
+claude plugin install langsmith-cli@langsmith-cli
+```
+
+Sessions that never touch LangSmith pay **zero context tokens**. When the agent *does* need observability data, it invokes the skill and gets a comprehensive reference for the full CLI — every command, every flag, with usage patterns and examples. Then it runs shell commands:
+
+```bash
+# Get the latest failed run with only the fields you need
+langsmith-cli --json runs get-latest --project my-app \
+  --failed --fields id,name,error
+```
+
+No always-on server. No startup schema tax. The skill loads on-demand, and `--fields` keeps the *response* data lean too.
+
+## 96% Token Reduction with `--fields`
+
+This is the feature that matters most for agents. A typical LangSmith run object is **20KB** — easily 1,000+ tokens. With `--fields`, you get only what you asked for:
+
+```bash
+# Full run object: ~1000 tokens
+langsmith-cli --json runs get abc-123
+
+# Just what you need: ~40 tokens
+langsmith-cli --json runs get abc-123 --fields name,status,error
+```
+
+`--fields` works on every list and get command: runs, projects, datasets, examples, prompts. Your agent stays lean.
+
+## Built for Two Audiences
+
+Most developer tools pick one audience. `langsmith-cli` serves both:
+
+**For humans** — rich terminal tables with color-coded statuses, smart column truncation, syntax highlighting:
+
+```bash
+langsmith-cli runs list --project my-app --status error --last 24h
+```
+
+```
+┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┓
+┃ Name         ┃ Status     ┃ Tokens ┃ Latency  ┃ Error       ┃
+┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━┩
+│ extractor    │ error      │ 2,340  │ 3.2s     │ Rate limit  │
+│ classifier   │ error      │ 1,102  │ 12.4s    │ Timeout     │
+└──────────────┴────────────┴────────┴──────────┴─────────────┘
+```
+
+**For agents** — add `--json` as the first flag and everything switches: strict JSON to stdout, diagnostics to stderr, zero formatting noise:
+
+```bash
+langsmith-cli --json runs list --project my-app --status error --limit 5
+```
+
+One flag. Two completely different UX modes.
+
+## Features the MCP Server Doesn't Have
+
+`langsmith-cli` has 100% parity with the official MCP server (all 66 parameters mapped). But it also has features the MCP server can't offer:
+
+### Live Monitoring with `runs watch`
+
+A real-time streaming dashboard in your terminal:
+
+```bash
+langsmith-cli runs watch --project my-app
+```
+
+### One-Command Debugging with `runs get-latest`
+
+No more `list | jq | get` pipelines:
+
+```bash
+# Before: three commands piped together
+langsmith-cli --json runs list --project X --limit 1 \
+  | jq -r '.[0].id' \
+  | xargs langsmith-cli --json runs get
+
+# After: one command
+langsmith-cli --json runs get-latest --project X --fields inputs,outputs,error
+```
+
+### Stratified Sampling with `runs sample`
+
+Build statistically sound eval datasets:
+
+```bash
+langsmith-cli runs sample \
+  --stratify-by tag:length,tag:content_type \
+  --dimension-values "short|long,news|gaming" \
+  --samples-per-combination 5 \
+  --output eval_samples.jsonl
+```
+
+### Aggregate Analytics with `runs analyze`
+
+Group-by metrics without leaving the terminal:
+
+```bash
+langsmith-cli --json runs analyze \
+  --group-by tag:model \
+  --metrics count,error_rate,p50_latency,avg_cost
+```
+
+### Schema Discovery with `runs fields` / `runs describe`
+
+Don't know what fields your runs have? Discover them:
+
+```bash
+langsmith-cli --json runs fields --include inputs,outputs
+# Returns field paths, types, presence rates, even language distribution
+```
+
+### Tag & Metadata Discovery
+
+```bash
+langsmith-cli runs tags --project my-app
+langsmith-cli runs metadata-keys --project my-app
+```
+
+### Bulk Export with Pattern Filenames
+
+```bash
+langsmith-cli runs export ./traces \
+  --project my-app --roots --limit 1000 \
+  --filename-pattern "{name}-{run_id}"
+```
+
+### Production Run to Eval Example in One Command
+
+```bash
+langsmith-cli --json examples from-run <run-id> --dataset my-eval-set
+```
+
+## Smart Filtering That Translates to FQL
+
+Nobody wants to write raw Filter Query Language. The CLI translates human-friendly flags automatically:
+
+```bash
+# These flags...
+langsmith-cli runs list --tag summarizer --failed --last 24h --slow
+
+# ...become this FQL:
+# and(has(tags, "summarizer"), eq(error, true),
+#     gt(start_time, "2026-03-03T..."), gt(latency, "5s"))
+```
+
+Time presets like `--recent` (last hour), `--today`, `--last 7d`, and `--since 2026-01-01` all work. Content search with `--grep` supports regex and field-specific matching. Everything composes.
+
+## What's New in v0.4.0
+
+The v0.4.0 release focused on type safety and code quality:
+
+- **Zero pyright errors** — every function has proper type annotations. `client: langsmith.Client`, not `client: Any`. Return types are real SDK Pydantic models, not `object`.
+- **`datasets delete`** command with confirmation prompts and JSON mode support
+- **Improved error handling** across prompts and runs commands using specific SDK exception types (`LangSmithNotFoundError`, `LangSmithConflictError`) instead of broad `except Exception`
+- **702 unit tests** passing with real Pydantic model instances (no MagicMock for test data)
+
+## Getting Started
+
+```bash
+# Install
+uv tool install langsmith-cli
+# or: pip install langsmith-cli
+
+# Authenticate
+export LANGSMITH_API_KEY="lsv2_..."
+# or: langsmith-cli auth login
+
+# Start exploring
+langsmith-cli runs list --project my-app --last 24h
+langsmith-cli --json runs get-latest --failed --fields name,error
+```
+
+If you're using Claude Code, add the plugin for the best agent experience:
+
+```bash
+claude plugin marketplace add gigaverse-app/langsmith-cli
+claude plugin install langsmith-cli@langsmith-cli
+```
+
+---
+
+The code is MIT licensed and on GitHub: [gigaverse-app/langsmith-cli](https://github.com/gigaverse-app/langsmith-cli)
+
+If you're building with LangSmith and tired of context-heavy MCP servers, give it a try. Happy to hear feedback in the issues.

{langsmith_cli-0.4.0 → langsmith_cli-0.4.2}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "langsmith-cli"
 # IMPORTANT: When bumping this version, also update .claude-plugin/plugin.json
-version = "0.4.0"
+version = "0.4.2"
 description = "Context-efficient CLI for LangSmith. Built for humans and agents."
 readme = "README.md"
 requires-python = ">=3.12"

{langsmith_cli-0.4.0 → langsmith_cli-0.4.2}/skills/langsmith/SKILL.md

@@ -99,10 +99,21 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
 
 ## API Reference
 
+### Authentication
+- `langsmith-cli auth login`: Configure API key (saves to global config).
+  - `--local`: Save to `.env` in current directory instead.
+
 ### Projects
 - `langsmith-cli --json projects list [OPTIONS]`: List all projects.
+  - `--limit <n>`: Max results (default: 100, use 0 for no limit)
+  - `--name <text>`: Filter by exact name
+  - `--name-pattern <pattern>`: Wildcard filter (e.g., `'*prod*'`)
+  - `--name-regex <regex>`: Regex filter
+  - `--has-runs`: Show only projects with runs
+  - `--sort-by <field>`: Sort by field (name, run_count). Prefix `-` for descending
   - `--fields <comma-separated>`: Select specific fields (e.g., `id,name`)
   - `--output <file>`: Write to file instead of stdout
+  - See [Projects Reference](references/projects.md) for full options and output fields.
 - `langsmith-cli --json projects get <name-or-id>`: Get project details (UUID auto-detected).
   - `--include-stats/--no-stats`: Include/exclude run statistics (default: include)
   - `--fields <comma-separated>`: Select fields
@@ -114,9 +125,24 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
 - `langsmith-cli --json runs list [OPTIONS]`: List recent runs.
   - `--project <name>`: Filter by project name (default: "default").
   - `--project-id <uuid>`: Filter by project UUID (bypasses name resolution, faster).
+  - **Multi-project:** `--project-name <text>`, `--project-name-exact <text>`, `--project-name-pattern <pattern>`, `--project-name-regex <regex>`
   - `--limit <n>`: Max results (default 10, keep it small).
   - `--status <success|error>`: Filter by status.
+  - **Convenience shortcuts:** `--failed`, `--succeeded`, `--slow` (>5s), `--recent` (last hour), `--today`
   - `--filter <string>`: Advanced FQL query string (see FQL examples below).
+  - `--roots`: Show only root traces (recommended for cleaner output).
+  - `--trace-id <uuid>`: Get all runs in a specific trace tree.
+  - `--run-type <type>`: Filter by type (llm, chain, tool, retriever, etc).
+  - `--tag <tag>`: Filter by tag (repeatable for AND logic).
+  - `--name-pattern <pattern>`: Wildcard filter on run names (client-side).
+  - `--name-regex <regex>`: Regex filter on run names (client-side).
+  - `--model <name>`: Filter by model name (e.g., `gpt-4`, `claude-3`).
+  - `--since <time>`: Runs since time (ISO, `3d`, or `3 days ago`).
+  - `--last <duration>`: Runs from last duration (e.g., `24h`, `7d`).
+  - `--min-latency <dur>` / `--max-latency <dur>`: Latency range (e.g., `2s`, `500ms`).
+  - `--trace-filter <fql>` / `--tree-filter <fql>`: Filter on root trace / any run in tree.
+  - `--sort-by <field>`: Sort by field (name, status, latency, start_time). Prefix `-` for descending.
+  - `--format <table|json|csv|yaml>`: Output format.
   - **Content Search Options:**
     - `--query <text>`: Server-side full-text search (fast, but only first ~250 chars indexed).
     - `--grep <pattern>`: Client-side content search (unlimited content, supports regex).
@@ -126,7 +152,7 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
   - `--fields <comma-separated>`: Reduce output size (e.g., `id,name,status,error`).
   - `--output <file>`: Write to file (JSONL format) instead of stdout.
   - `--no-truncate`: Show full content in table columns (only affects table output, not JSON).
-  - `--roots`: Show only root traces (recommended for cleaner output).
+  - See [Runs Reference](references/runs.md) for full field list and examples.
 - `langsmith-cli --json runs get <id> [OPTIONS]`: Get details of a single run.
   - `--fields <comma-separated>`: Only return specific fields (e.g., `inputs,outputs,error`).
 - `langsmith-cli --json runs get-latest [OPTIONS]`: Get the most recent run matching filters.
@@ -139,6 +165,21 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
   - Example: `langsmith-cli --json runs get-latest --project-name-pattern "prd/*" --succeeded --roots`
   - **Before (complex):** `langsmith-cli --json runs list --project X --limit 1 --roots | jq -r '.[0].id' | xargs langsmith-cli --json runs get --fields inputs,outputs`
   - **After (simple):** `langsmith-cli --json runs get-latest --project X --roots --fields inputs,outputs`
+- `langsmith-cli --json runs search <query> [OPTIONS]`: Full-text search across runs.
+  - `--project <name>`: Project name (default: "default").
+  - Multi-project: `--project-name-pattern`, `--project-name-regex`, etc.
+  - `--limit <n>`: Max results (default: 10).
+  - `--roots`: Show only root traces.
+  - `--in <all|inputs|outputs|error>`: Where to search (default: all).
+  - `--input-contains <text>`: Filter by content in inputs.
+  - `--output-contains <text>`: Filter by content in outputs.
+  - `--since <time>` / `--last <duration>`: Time filters.
+  - `--format <table|json|csv|yaml>`: Output format.
+  - Example: `langsmith-cli --json runs search "timeout" --in error --project myapp`
+- `langsmith-cli runs watch [OPTIONS]`: Live monitoring dashboard (interactive, no `--json`).
+  - `--project <name>`: Project to monitor (default: "default").
+  - Multi-project: `--project-name-pattern`, `--project-name-regex`, etc.
+  - `--interval <seconds>`: Refresh interval (default: 2).
 - `langsmith-cli runs view-file <pattern> [OPTIONS]`: View runs from JSONL files with table display.
   - **Use this to read files created by `--output`** - don't use the Read tool on JSONL files (they can be 30K+ tokens).
   - `<pattern>`: File path or glob pattern (e.g., `samples.jsonl`, `data/*.jsonl`).
@@ -215,16 +256,29 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
   - `--output <file>`: Write to file instead of stdout
 - `langsmith-cli --json datasets get <id> [--fields id,name,description]`: Get dataset details.
 - `langsmith-cli --json datasets create <name>`: Create a dataset.
+  - `--description <text>`: Dataset description.
+  - `--type [kv|llm|chat]`: Dataset type (default: kv).
 - `langsmith-cli --json datasets delete <name-or-id> --confirm`: Delete a dataset.
 - `langsmith-cli --json datasets push <file.jsonl> --dataset <name>`: Upload examples from JSONL.
+- See [Datasets Reference](references/datasets.md) for full options and output fields.
 - `langsmith-cli --json examples list --dataset <name> [OPTIONS]`: List examples in a dataset.
+  - `--limit <n>` / `--offset <n>`: Pagination.
+  - `--splits <comma-separated>`: Filter by splits (e.g., `train,test`).
+  - `--as-of <tag-or-timestamp>`: Version snapshot.
+  - `--filter <fql>`: Advanced FQL query.
+  - `--metadata <json>`: Filter by metadata.
   - `--fields <comma-separated>`: Select fields (e.g., `id,inputs,outputs`)
   - `--output <file>`: Write to file instead of stdout
 - `langsmith-cli --json examples get <id> [--fields id,inputs,outputs]`: Get example details.
 - `langsmith-cli --json examples create --dataset <name> --inputs <json> --outputs <json>`: Add an example.
+  - `--metadata <json>`: Custom metadata.
+  - `--split <name>`: Split name (e.g., `train`, `test`).
 - `langsmith-cli --json examples update <id> --inputs <json> --outputs <json>`: Update an example.
+  - `--metadata <json>`: New metadata.
+  - `--split <name>`: New split name.
 - `langsmith-cli --json examples delete <id> [<id>...] --confirm`: Delete examples (supports bulk).
 - `langsmith-cli --json examples from-run <run-id> --dataset <name>`: Create example from a run.
+- See [Examples Reference](references/examples.md) for full options and output fields.
 
 ### Prompts
 - `langsmith-cli --json prompts list [OPTIONS]`: List prompt repositories.
@@ -235,9 +289,20 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
   - `--include-model`: Include model configuration
   - `--fields <comma-separated>`: Select fields
 - `langsmith-cli --json prompts push <name> <file_path>`: Push a local file as a prompt.
+  - `--description <text>`: Prompt description.
+  - `--tags <comma-separated>`: Tags.
+  - `--is-public <bool>`: Make public.
 - `langsmith-cli --json prompts create <name> [--description <text>]`: Create a new prompt.
+  - `--tags <comma-separated>`: Tags.
+  - `--is-public <bool>`: Make public.
 - `langsmith-cli --json prompts delete <name> --confirm`: Delete a prompt.
 - `langsmith-cli --json prompts commits <name> [--limit N]`: List prompt versions.
+  - `--offset <n>`: Skip N commits.
+  - `--include-model`: Include model configuration.
+  - `--fields <comma-separated>`: Select fields.
+  - `--count`: Return only the count of commits.
+  - `--output <file>`: Write to file.
+- See [Prompts Reference](references/prompts.md) for full options and output fields.
 
 ### Self (Installation Management)
 - `langsmith-cli self detect`: Show installation details (version, install method, paths).
@@ -295,7 +360,7 @@ langsmith-cli --json projects list | jq -r '.[].name' | grep -E "(prd|stg)/"
 langsmith-cli --json projects list --name-regex "^(prd|stg)/" --fields name
 ```
 
-### Pattern 2: Get Latest Run Without Nested Commands
+### Pattern 3: Get Latest Run Without Nested Commands
 ```bash
 # ❌ BAD (requires jq + nested command)
 langsmith-cli --json runs get $(
@@ -307,7 +372,7 @@ langsmith-cli --json runs get $(
 langsmith-cli --json runs get-latest --project X --roots --fields inputs,outputs
 ```
 
-### Pattern 3: Get Latest Error from Production
+### Pattern 4: Get Latest Error from Production
 ```bash
 # ❌ BAD (complex piping)
 for project in $(langsmith-cli --json projects list | jq -r '.[].name' | grep "prd/"); do
@@ -318,7 +383,7 @@ done | jq -s '.[0]'
 langsmith-cli --json runs get-latest --project-name-pattern "prd/*" --failed --fields id,name,error
 ```
 
-### Pattern 4: Filter Projects by Pattern
+### Pattern 5: Filter Projects by Pattern
 ```bash
 # Filter by substring
 langsmith-cli --json projects list --name "production" --fields name
@@ -330,7 +395,7 @@ langsmith-cli --json projects list --name-pattern "*prod*" --fields name
 langsmith-cli --json projects list --name-regex "^(prd|stg)/.*" --fields name
 ```
 
-### Pattern 5: Get Latest Successful Run from Multiple Projects
+### Pattern 6: Get Latest Successful Run from Multiple Projects
 ```bash
 # Searches across all matching projects
 langsmith-cli --json runs get-latest \