langsmith-cli 0.3.4__tar.gz → 0.4.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (97) hide show
  1. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.claude-plugin/marketplace.json +2 -2
  2. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.claude-plugin/plugin.json +1 -1
  3. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/CLAUDE.md +141 -16
  4. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/PKG-INFO +1 -1
  5. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/pyproject.toml +1 -1
  6. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/SKILL.md +27 -0
  7. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/references/datasets.md +42 -18
  8. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/references/examples.md +82 -2
  9. langsmith_cli-0.4.0/skills/langsmith/references/projects.md +141 -0
  10. langsmith_cli-0.4.0/skills/langsmith/references/prompts.md +250 -0
  11. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/references/runs.md +67 -4
  12. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/commands/datasets.py +62 -1
  13. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/commands/examples.py +125 -6
  14. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/commands/projects.py +141 -6
  15. langsmith_cli-0.4.0/src/langsmith_cli/commands/prompts.py +378 -0
  16. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/commands/runs.py +222 -0
  17. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/utils.py +10 -10
  18. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/conftest.py +76 -1
  19. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_datasets.py +95 -0
  20. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_examples.py +283 -1
  21. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_projects.py +326 -0
  22. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_prompts.py +439 -1
  23. langsmith_cli-0.4.0/tests/test_runs_export.py +399 -0
  24. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_smoke.py +40 -29
  25. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/uv.lock +1 -1
  26. langsmith_cli-0.3.4/skills/langsmith/references/projects.md +0 -59
  27. langsmith_cli-0.3.4/skills/langsmith/references/prompts.md +0 -130
  28. langsmith_cli-0.3.4/src/langsmith_cli/commands/prompts.py +0 -172
  29. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.env.example +0 -0
  30. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.github/dependabot.yml +0 -0
  31. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.github/workflows/ci.yml +0 -0
  32. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.github/workflows/dependency-review.yml +0 -0
  33. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.github/workflows/publish.yml +0 -0
  34. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.gitignore +0 -0
  35. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.pre-commit-config.yaml +0 -0
  36. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/.python-version +0 -0
  37. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/AGENTS.md +0 -0
  38. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/LICENSE +0 -0
  39. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/README.md +0 -0
  40. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/RELEASING.md +0 -0
  41. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/COMMANDS_DESIGN.md +0 -0
  42. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/PIPES_TO_CLI_REFERENCE.md +0 -0
  43. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/PRD.md +0 -0
  44. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/QOL_FEATURES.md +0 -0
  45. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/QOL_IMPROVEMENTS.md +0 -0
  46. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/TLDR.md +0 -0
  47. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/CI_BEST_PRACTICES.md +0 -0
  48. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/CODECOV_SETUP.md +0 -0
  49. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/IMPLEMENTATION_PLAN.md +0 -0
  50. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/LANGSMITH_TEAM_QUESTIONS.md +0 -0
  51. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/MCP_PARITY.md +0 -0
  52. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/PUBLISHING.md +0 -0
  53. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/PYPI_SETUP_SUMMARY.md +0 -0
  54. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/SESSION_DIRECTIVES.md +0 -0
  55. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/TESTING_PERFORMANCE.md +0 -0
  56. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/TESTING_STRATEGY.md +0 -0
  57. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/docs/dev/TYPE_SAFETY_GUIDE.md +0 -0
  58. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/main.py +0 -0
  59. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/scripts/install.ps1 +0 -0
  60. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/scripts/install.py +0 -0
  61. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/scripts/install.sh +0 -0
  62. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/scripts/release.sh +0 -0
  63. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/scripts/test_installer.sh +0 -0
  64. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/scripts/uninstall.py +0 -0
  65. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/docs/examples.md +0 -0
  66. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/docs/reference.md +0 -0
  67. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/references/fql.md +0 -0
  68. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/references/installation.md +0 -0
  69. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/skills/langsmith/references/troubleshooting.md +0 -0
  70. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/__init__.py +0 -0
  71. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/cli_logging.py +0 -0
  72. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/commands/auth.py +0 -0
  73. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/commands/self_cmd.py +0 -0
  74. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/config.py +0 -0
  75. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/field_analysis.py +0 -0
  76. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/filters.py +0 -0
  77. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/src/langsmith_cli/main.py +0 -0
  78. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_auth.py +0 -0
  79. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_config.py +0 -0
  80. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_e2e.py +0 -0
  81. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_fetch_helpers.py +0 -0
  82. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_field_analysis.py +0 -0
  83. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_filters.py +0 -0
  84. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_logging.py +0 -0
  85. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_main.py +0 -0
  86. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_output_flag.py +0 -0
  87. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_analyze.py +0 -0
  88. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_discovery.py +0 -0
  89. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_fields.py +0 -0
  90. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_get.py +0 -0
  91. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_list.py +0 -0
  92. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_roots.py +0 -0
  93. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_sample.py +0 -0
  94. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_search.py +0 -0
  95. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_runs_view.py +0 -0
  96. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_self.py +0 -0
  97. {langsmith_cli-0.3.4 → langsmith_cli-0.4.0}/tests/test_utils.py +0 -0
@@ -6,14 +6,14 @@
6
6
  },
7
7
  "metadata": {
8
8
  "description": "LangSmith CLI plugin marketplace",
9
- "version": "0.2.9"
9
+ "version": "0.4.0"
10
10
  },
11
11
  "plugins": [
12
12
  {
13
13
  "name": "langsmith-cli",
14
14
  "source": "./",
15
15
  "description": "A context-efficient interface for LangSmith observability and evaluations.",
16
- "version": "0.2.9",
16
+ "version": "0.4.0",
17
17
  "author": {
18
18
  "name": "Gigaverse",
19
19
  "email": "aviadr1@gmail.com"
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "langsmith-cli",
3
- "version": "0.3.4",
3
+ "version": "0.4.0",
4
4
  "description": "A context-efficient interface for LangSmith observability and evaluations.",
5
5
  "author": {
6
6
  "name": "Aviad Rozenhek",
@@ -107,7 +107,10 @@ main.py (entry point)
107
107
  │ └── login
108
108
  ├── projects (group)
109
109
  │ ├── list
110
- └── create
110
+ ├── get
111
+ │ ├── create
112
+ │ ├── update
113
+ │ └── delete
111
114
  ├── runs (group)
112
115
  │ ├── list
113
116
  │ ├── get
@@ -122,20 +125,29 @@ main.py (entry point)
122
125
  │ ├── metadata-keys
123
126
  │ ├── fields
124
127
  │ ├── describe
125
- └── view-file
128
+ ├── view-file
129
+ │ └── export
126
130
  ├── datasets (group)
127
131
  │ ├── list
128
132
  │ ├── get
129
133
  │ ├── create
134
+ │ ├── delete
130
135
  │ └── push
131
136
  ├── examples (group)
132
137
  │ ├── list
133
138
  │ ├── get
134
- └── create
139
+ ├── create
140
+ │ ├── update
141
+ │ ├── delete
142
+ │ └── from-run
135
143
  ├── prompts (group)
136
144
  │ ├── list
137
145
  │ ├── get
138
- └── push
146
+ ├── pull
147
+ │ ├── push
148
+ │ ├── create
149
+ │ ├── delete
150
+ │ └── commits
139
151
  └── self (group)
140
152
  ├── detect
141
153
  └── update
@@ -143,18 +155,27 @@ main.py (entry point)
143
155
 
144
156
  ### Key Design Patterns
145
157
 
146
- **1. Lazy Loading Performance Pattern**
158
+ **1. Direct Imports for Strong Types**
147
159
  ```python
148
- # BAD - Top-level import (loads SDK immediately)
149
- from langsmith import Client
160
+ # GOOD - Direct imports of langsmith types for strong typing
161
+ import langsmith
162
+ from langsmith.schemas import Dataset, TracerSessionResult, Run
150
163
 
151
- # GOOD - Import inside command function
152
- @runs.command("list")
153
- def list_runs(...):
154
- import langsmith # Only loads when command executes
155
- client = langsmith.Client()
164
+ def resolve_dataset(client: langsmith.Client, name: str) -> Dataset:
165
+ ...
166
+
167
+ # BAD - using `object` or `Any` as type (loses all type safety)
168
+ def resolve_dataset(client: object, name: str) -> object: ...
169
+
170
+ # ❌ BAD - TYPE_CHECKING / __future__ annotations workaround (unnecessary complexity)
171
+ from __future__ import annotations
172
+ from typing import TYPE_CHECKING
173
+ if TYPE_CHECKING:
174
+ import langsmith # Don't do this
156
175
  ```
157
176
 
177
+ **Note on CLI startup performance:** `utils.py` already imports `langsmith` at module level (needed for shared helpers), and every command module imports `utils.py`. Since the cost is already paid at startup, import langsmith types directly everywhere — no lazy loading workarounds needed for type annotations. If a module does NOT import utils.py and doesn't need langsmith at import time, keep heavy imports inside command functions.
178
+
158
179
  **2. Dual Output Pattern (JSON vs Rich Tables)**
159
180
  ```python
160
181
  # All commands check ctx.obj.get("json") flag
@@ -530,6 +551,107 @@ def create_dataset(name="test", example_count=10) -> Dataset:
530
551
  - E2E tests validate full SDK interaction (require API key)
531
552
  - Aim for 100% coverage of new code
532
553
 
554
+ ### Handling Mixed stdout/stderr in Tests (CliRunner)
555
+
556
+ Click's `CliRunner` mixes stdout and stderr. When commands use `logger.info()` before JSON output, the `--json` test output may contain non-JSON lines. Use the shared `parse_json_output()` helper from conftest:
557
+
558
+ ```python
559
+ from conftest import parse_json_output
560
+
561
+ def test_json_output(runner, tmp_path):
562
+ result = runner.invoke(cli, ["--json", "runs", "export", str(tmp_path), ...])
563
+ data = parse_json_output(result.output) # Finds last JSON line
564
+ assert data["status"] == "success"
565
+ ```
566
+
567
+ **Anti-pattern:** Don't use `json.loads(result.output)` directly when commands emit progress messages — it will fail with `JSONDecodeError`.
568
+
569
+ ### SDK Exception Types in Tests
570
+
571
+ Always use real SDK exception types in test side_effects, never bare `Exception`:
572
+
573
+ ```python
574
+ # ❌ BAD - bare Exception won't be caught by specific handlers
575
+ mock_client.delete_example.side_effect = Exception("Not found")
576
+
577
+ # ✅ GOOD - matches the actual exception hierarchy
578
+ from langsmith.utils import LangSmithNotFoundError
579
+ mock_client.delete_example.side_effect = LangSmithNotFoundError("Not found")
580
+ ```
581
+
582
+ ## Reusable Patterns Across Commands
583
+
584
+ ### Name-or-ID Resolution (`resolve_project()`, `resolve_dataset()`)
585
+
586
+ When a command accepts a name or ID argument, use the existing resolve helpers:
587
+ - `projects.py: resolve_project(client, name_or_id, include_stats=True)`
588
+ - `datasets.py: resolve_dataset(client, name_or_id)`
589
+
590
+ These helpers follow a consistent pattern:
591
+ 1. Auto-detect UUIDs via `_looks_like_uuid()` to save an API call
592
+ 2. Fall back from name to ID lookup
593
+ 3. Catch `(LangSmithNotFoundError, LangSmithError, ValueError)` for friendly errors
594
+ 4. Raise `click.ClickException` on not-found
595
+
596
+ ```python
597
+ from langsmith_cli.commands.projects import resolve_project
598
+
599
+ project = resolve_project(client, name_or_id, include_stats=True)
600
+ ```
601
+
602
+ ### Shared Filter Building (utils.py: `build_runs_list_filter()`)
603
+
604
+ Never manually reconstruct status/tag/time filters. Use the canonical helper:
605
+
606
+ ```python
607
+ combined_filter, error_filter = build_runs_list_filter(
608
+ filter_=filter_, status=status, tag=tag, since=since, last=last
609
+ )
610
+ ```
611
+
612
+ ### Shared Multi-Project Fetching (utils.py: `fetch_from_projects()`)
613
+
614
+ Never write manual project iteration loops. Use `fetch_from_projects()`:
615
+
616
+ ```python
617
+ def _fetch_runs(c, proj, **kw):
618
+ if proj is not None:
619
+ return c.list_runs(project_name=proj, **kw)
620
+ return c.list_runs(**kw)
621
+
622
+ result = fetch_from_projects(client, projects, _fetch_runs, project_query=pq, ...)
623
+ raise_if_all_failed_with_suggestions(result, client, pq, logger, "runs")
624
+ ```
625
+
626
+ ### Destructive Operations Pattern
627
+
628
+ For delete/destructive commands, follow this pattern:
629
+ 1. Require `--confirm` flag (skip with `click.confirm(abort=True)`)
630
+ 2. Resolve entity first (verify it exists), then perform the operation
631
+ 3. Use specific SDK exceptions (`LangSmithNotFoundError`), never broad `except Exception`
632
+ 4. In JSON mode, return `{"status": "success", ...}` on success
633
+
634
+ ### Filename Sanitization
635
+
636
+ When writing user-controlled filenames, sanitize with regex:
637
+ ```python
638
+ import re
639
+ safe = re.sub(r'[<>:"/\\|?*\x00-\x1f]', "_", name)
640
+ safe = safe.strip(". ")
641
+ ```
642
+
643
+ ### Common Anti-Patterns to Avoid
644
+
645
+ 1. **`hasattr()` on Pydantic models** → Use `p.field is not None` instead
646
+ 2. **Broad `except Exception`** in loops → Catch specific SDK exceptions
647
+ 3. **Manual filter construction** → Use `build_runs_list_filter()`
648
+ 4. **Manual project iteration** → Use `fetch_from_projects()`
649
+ 5. **Duplicated utility logic** → Extract to helpers (e.g., `normalize_split()`)
650
+ 6. **`--flag default=True is_flag=True`** → No-op; use `--flag/--no-flag` pair
651
+ 7. **ThreadPoolExecutor for local file I/O** → Sequential is faster (no GIL benefit)
652
+ 8. **`TYPE_CHECKING` / `__future__` annotations workaround** → Import types directly; langsmith is already loaded at startup
653
+ 9. **`client: object` or `client: Any`** → Use `client: langsmith.Client` for strong types
654
+
533
655
  ## Engineering Standards (from docs/AGENTS.md)
534
656
 
535
657
  ### 1. Type Safety (Zero Tolerance for Weak Types)
@@ -542,10 +664,13 @@ def create_dataset(name="test", example_count=10) -> Dataset:
542
664
  - ✅ Never create duplicate response models - reuse SDK models with field selection
543
665
  - **Current Status**: Fully implemented - all commands use SDK Pydantic models with type-safe attribute access
544
666
 
545
- ### 2. Performance (100ms Rule)
546
- - ❌ NO top-level imports of heavy libraries (langsmith, rich, pandas)
547
- - Libraries imported inside command functions (lazy loading)
548
- - **Current Status**: Properly implemented
667
+ ### 2. Performance (Keep CLI Fast)
668
+ - ❌ Don't add new top-level imports of heavy libraries to modules that don't already need them
669
+ - Don't introduce new heavy dependencies without measuring CLI startup impact
670
+ - `langsmith` and `rich` are already loaded at startup (via utils.py) — use them directly for strong types
671
+ - ✅ If adding a new module that doesn't need langsmith at import time, keep heavy imports inside functions
672
+ - ✅ Always prefer strong types over lazy loading workarounds — don't sacrifice type safety for startup time
673
+ - **Current Status**: langsmith/rich load at startup (~300ms). Acceptable. Don't make it worse.
549
674
 
550
675
  ### 3. Architecture (Pure Logic vs View)
551
676
  - Logic Layer: Returns Pydantic models/typed objects, never prints
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: langsmith-cli
3
- Version: 0.3.4
3
+ Version: 0.4.0
4
4
  Summary: Context-efficient CLI for LangSmith. Built for humans and agents.
5
5
  Project-URL: Homepage, https://github.com/aviadr1/langsmith-cli
6
6
  Project-URL: Repository, https://github.com/aviadr1/langsmith-cli
@@ -1,7 +1,7 @@
1
1
  [project]
2
2
  name = "langsmith-cli"
3
3
  # IMPORTANT: When bumping this version, also update .claude-plugin/plugin.json
4
- version = "0.3.4"
4
+ version = "0.4.0"
5
5
  description = "Context-efficient CLI for LangSmith. Built for humans and agents."
6
6
  readme = "README.md"
7
7
  requires-python = ">=3.12"
@@ -103,7 +103,12 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
103
103
  - `langsmith-cli --json projects list [OPTIONS]`: List all projects.
104
104
  - `--fields <comma-separated>`: Select specific fields (e.g., `id,name`)
105
105
  - `--output <file>`: Write to file instead of stdout
106
+ - `langsmith-cli --json projects get <name-or-id>`: Get project details (UUID auto-detected).
107
+ - `--include-stats/--no-stats`: Include/exclude run statistics (default: include)
108
+ - `--fields <comma-separated>`: Select fields
106
109
  - `langsmith-cli --json projects create <name>`: Create a new project.
110
+ - `langsmith-cli --json projects update <name-or-id> --name <new> --description <desc>`: Update project.
111
+ - `langsmith-cli --json projects delete <name-or-id> --confirm`: Delete a project.
107
112
 
108
113
  ### Runs (Traces)
109
114
  - `langsmith-cli --json runs list [OPTIONS]`: List recent runs.
@@ -192,6 +197,17 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
192
197
  - Returns: `{"fields": [{"path": "inputs.query", "type": "string", "present_pct": 98.0, "length": {"min": 5, "max": 500, "avg": 89}, "languages": {"en": 80.0}}, ...], "total_runs": 100}`
193
198
  - Example: `langsmith-cli --json runs describe --include inputs,outputs`
194
199
  - Example: `langsmith-cli --json runs describe --project my-project --no-language`
200
+ - `langsmith-cli runs export <directory> [OPTIONS]`: Export runs as individual JSON files.
201
+ - `--project <name>`: Project name (required)
202
+ - `--limit <n>`: Max runs to export (default: 50)
203
+ - `--status <success|error>`: Filter by status
204
+ - `--roots`: Export only root traces
205
+ - `--tag <tag>`: Filter by tag (repeatable)
206
+ - `--last <duration>`: Time window (e.g., `24h`, `7d`)
207
+ - `--fields <comma-separated>`: Reduce exported file size
208
+ - `--filename-pattern <pattern>`: Custom filenames (placeholders: `{run_id}`, `{name}`, `{index}`, `{trace_id}`)
209
+ - Example: `langsmith-cli runs export ./traces --project my-project --roots --limit 100`
210
+ - Example: `langsmith-cli --json runs export ./errors --project my-project --status error --last 24h --fields name,inputs,outputs,error`
195
211
 
196
212
  ### Datasets & Examples
197
213
  - `langsmith-cli --json datasets list [OPTIONS]`: List datasets.
@@ -199,18 +215,29 @@ langsmith-cli --json runs list --project my-project --limit 5 2>&1
199
215
  - `--output <file>`: Write to file instead of stdout
200
216
  - `langsmith-cli --json datasets get <id> [--fields id,name,description]`: Get dataset details.
201
217
  - `langsmith-cli --json datasets create <name>`: Create a dataset.
218
+ - `langsmith-cli --json datasets delete <name-or-id> --confirm`: Delete a dataset.
219
+ - `langsmith-cli --json datasets push <file.jsonl> --dataset <name>`: Upload examples from JSONL.
202
220
  - `langsmith-cli --json examples list --dataset <name> [OPTIONS]`: List examples in a dataset.
203
221
  - `--fields <comma-separated>`: Select fields (e.g., `id,inputs,outputs`)
204
222
  - `--output <file>`: Write to file instead of stdout
205
223
  - `langsmith-cli --json examples get <id> [--fields id,inputs,outputs]`: Get example details.
206
224
  - `langsmith-cli --json examples create --dataset <name> --inputs <json> --outputs <json>`: Add an example.
225
+ - `langsmith-cli --json examples update <id> --inputs <json> --outputs <json>`: Update an example.
226
+ - `langsmith-cli --json examples delete <id> [<id>...] --confirm`: Delete examples (supports bulk).
227
+ - `langsmith-cli --json examples from-run <run-id> --dataset <name>`: Create example from a run.
207
228
 
208
229
  ### Prompts
209
230
  - `langsmith-cli --json prompts list [OPTIONS]`: List prompt repositories.
210
231
  - `--fields <comma-separated>`: Select fields (e.g., `repo_handle,description`)
211
232
  - `--output <file>`: Write to file instead of stdout
212
233
  - `langsmith-cli --json prompts get <name> [--commit <hash>]`: Fetch a prompt template.
234
+ - `langsmith-cli --json prompts pull <name> [--commit <hash>]`: Pull full prompt content (manifest, examples).
235
+ - `--include-model`: Include model configuration
236
+ - `--fields <comma-separated>`: Select fields
213
237
  - `langsmith-cli --json prompts push <name> <file_path>`: Push a local file as a prompt.
238
+ - `langsmith-cli --json prompts create <name> [--description <text>]`: Create a new prompt.
239
+ - `langsmith-cli --json prompts delete <name> --confirm`: Delete a prompt.
240
+ - `langsmith-cli --json prompts commits <name> [--limit N]`: List prompt versions.
214
241
 
215
242
  ### Self (Installation Management)
216
243
  - `langsmith-cli self detect`: Show installation details (version, install method, paths).
@@ -9,12 +9,16 @@ langsmith-cli --json datasets list [OPTIONS]
9
9
  ```
10
10
 
11
11
  **Options:**
12
- - `--limit INTEGER` - Maximum results (default: 10)
12
+ - `--limit INTEGER` - Maximum results (default: 20)
13
13
  - `--name TEXT` - Filter by exact dataset name
14
- - `--name-contains TEXT` - Filter by name substring (case-insensitive)
14
+ - `--name-contains TEXT` - Filter by name substring
15
15
  - `--dataset-ids TEXT` - Comma-separated list of dataset UUIDs
16
16
  - `--data-type TEXT` - Filter by type: `kv`, `llm`, or `chat`
17
- - `--metadata JSON` - Filter by metadata (JSON object)
17
+ - `--metadata TEXT` - Filter by metadata (JSON string)
18
+ - `--exclude TEXT` - Exclude items containing substring (repeatable)
19
+ - `--fields TEXT` - Comma-separated field names to include
20
+ - `--count` - Output only the count of results
21
+ - `--output TEXT` - Write output to file (JSONL format)
18
22
 
19
23
  **Output Fields:**
20
24
  - `id` (UUID) - Dataset identifier
@@ -43,17 +47,21 @@ langsmith-cli --json datasets list --data-type llm
43
47
  Get dataset details.
44
48
 
45
49
  ```bash
46
- langsmith-cli --json datasets get <dataset-id-or-name>
50
+ langsmith-cli --json datasets get <dataset-id> [OPTIONS]
47
51
  ```
48
52
 
49
53
  **Arguments:**
50
- - `dataset-id-or-name` (required) - Dataset UUID or exact name
54
+ - `dataset-id` (required) - Dataset UUID
55
+
56
+ **Options:**
57
+ - `--fields TEXT` - Comma-separated field names to include
58
+ - `--output TEXT` - Write output to file (JSON format)
51
59
 
52
60
  **Output:** Complete dataset object with all metadata
53
61
 
54
62
  **Example:**
55
63
  ```bash
56
- langsmith-cli --json datasets get "my-test-dataset"
64
+ langsmith-cli --json datasets get "ae99b6fa-a6db-4f1c-8868-bc6764f4c29e"
57
65
  ```
58
66
 
59
67
  ### `datasets create`
@@ -69,8 +77,7 @@ langsmith-cli --json datasets create <name> [OPTIONS]
69
77
 
70
78
  **Options:**
71
79
  - `--description TEXT` - Dataset description
72
- - `--type TEXT` - Dataset type: `kv` (default), `llm`, or `chat`
73
- - `--metadata JSON` - Custom metadata (JSON object)
80
+ - `--type [kv|llm|chat]` - Dataset type (default: kv)
74
81
 
75
82
  **Output:** Created dataset object
76
83
 
@@ -78,8 +85,28 @@ langsmith-cli --json datasets create <name> [OPTIONS]
78
85
  ```bash
79
86
  langsmith-cli --json datasets create "qa-pairs" \
80
87
  --description "Question answering test set" \
81
- --type kv \
82
- --metadata '{"source": "production", "version": "1.0"}'
88
+ --type kv
89
+ ```
90
+
91
+ ### `datasets delete`
92
+
93
+ Delete a dataset by name or ID.
94
+
95
+ ```bash
96
+ langsmith-cli --json datasets delete <name-or-id> --confirm
97
+ ```
98
+
99
+ **Arguments:**
100
+ - `name-or-id` (required) - Dataset name or UUID (auto-detected)
101
+
102
+ **Options:**
103
+ - `--confirm` - Skip confirmation prompt (required for non-interactive use)
104
+
105
+ **Output:** `{"status": "success", "name": "<dataset-name>"}`
106
+
107
+ **Example:**
108
+ ```bash
109
+ langsmith-cli --json datasets delete "old-test-data" --confirm
83
110
  ```
84
111
 
85
112
  ### `datasets push`
@@ -87,27 +114,24 @@ langsmith-cli --json datasets create "qa-pairs" \
87
114
  Bulk upload examples from JSONL file.
88
115
 
89
116
  ```bash
90
- langsmith-cli --json datasets push <dataset-name> <file.jsonl> [OPTIONS]
117
+ langsmith-cli --json datasets push <file.jsonl> [OPTIONS]
91
118
  ```
92
119
 
93
120
  **Arguments:**
94
- - `dataset-name` (required) - Target dataset name (creates if doesn't exist)
95
121
  - `file.jsonl` (required) - Path to JSONL file
96
122
 
123
+ **Options:**
124
+ - `--dataset TEXT` - Target dataset name (creates if doesn't exist; defaults to filename)
125
+
97
126
  **JSONL Format:**
98
127
  ```jsonl
99
128
  {"inputs": {"query": "What is AI?"}, "outputs": {"answer": "Artificial Intelligence..."}}
100
129
  {"inputs": {"query": "Define ML"}, "outputs": {"answer": "Machine Learning..."}}
101
130
  ```
102
131
 
103
- **Options:**
104
- - `--description TEXT` - Description for new dataset
105
- - `--type TEXT` - Type for new dataset: `kv`, `llm`, or `chat`
106
-
107
132
  **Output:** Upload summary with count of examples added
108
133
 
109
134
  **Example:**
110
135
  ```bash
111
- langsmith-cli --json datasets push "my-dataset" examples.jsonl
136
+ langsmith-cli --json datasets push examples.jsonl --dataset "my-dataset"
112
137
  ```
113
-
@@ -10,7 +10,7 @@ langsmith-cli --json examples list [OPTIONS]
10
10
 
11
11
  **Options:**
12
12
  - `--dataset TEXT` (required) - Dataset name or UUID
13
- - `--limit INTEGER` - Maximum results (default: 10, max: 100)
13
+ - `--limit INTEGER` - Maximum results (default: 20)
14
14
  - `--offset INTEGER` - Skip N examples (default: 0)
15
15
  - `--example-ids TEXT` - Comma-separated list of example UUIDs
16
16
  - `--filter TEXT` - Advanced FQL query
@@ -19,6 +19,10 @@ langsmith-cli --json examples list [OPTIONS]
19
19
  - `--as-of TEXT` - Version tag or ISO timestamp
20
20
  - `--inline-s3-urls BOOLEAN` - Inline S3 URLs: `true` or `false`
21
21
  - `--include-attachments BOOLEAN` - Include attachments: `true` or `false`
22
+ - `--exclude TEXT` - Exclude items containing substring (repeatable)
23
+ - `--fields TEXT` - Comma-separated field names to include
24
+ - `--count` - Output only the count of results
25
+ - `--output TEXT` - Write output to file (JSONL format)
22
26
 
23
27
  **Output Fields:**
24
28
  - `id` (UUID) - Example identifier
@@ -61,6 +65,8 @@ langsmith-cli --json examples get <example-id> [OPTIONS]
61
65
 
62
66
  **Options:**
63
67
  - `--as-of TEXT` - Version tag or ISO timestamp
68
+ - `--fields TEXT` - Comma-separated field names to include
69
+ - `--output TEXT` - Write output to file (JSON format)
64
70
 
65
71
  **Output:** Complete example object
66
72
 
@@ -83,7 +89,6 @@ langsmith-cli --json examples create [OPTIONS]
83
89
  - `--outputs JSON` - Expected output data as JSON object
84
90
  - `--metadata JSON` - Custom metadata as JSON object
85
91
  - `--split TEXT` - Split name (e.g., "train", "test", "validation")
86
- - `--source-run-id UUID` - Source run UUID if example from trace
87
92
 
88
93
  **Output:** Created example object
89
94
 
@@ -104,3 +109,78 @@ langsmith-cli --json examples create \
104
109
  --split train
105
110
  ```
106
111
 
112
+ ### `examples update`
113
+
114
+ Update an existing example's inputs, outputs, metadata, or split.
115
+
116
+ ```bash
117
+ langsmith-cli --json examples update <example-id> [OPTIONS]
118
+ ```
119
+
120
+ **Arguments:**
121
+ - `example-id` (required) - Example UUID
122
+
123
+ **Options:**
124
+ - `--inputs JSON` - New input data as JSON
125
+ - `--outputs JSON` - New output data as JSON
126
+ - `--metadata JSON` - New metadata as JSON
127
+ - `--split TEXT` - New split name
128
+
129
+ At least one option is required.
130
+
131
+ **Output:** Updated example data
132
+
133
+ **Example:**
134
+ ```bash
135
+ langsmith-cli --json examples update <uuid> \
136
+ --outputs '{"answer": "Updated answer"}' \
137
+ --metadata '{"reviewed": true}'
138
+ ```
139
+
140
+ ### `examples delete`
141
+
142
+ Delete one or more examples by ID. Supports bulk deletion with partial failure reporting.
143
+
144
+ ```bash
145
+ langsmith-cli --json examples delete <example-id> [<example-id>...] [OPTIONS]
146
+ ```
147
+
148
+ **Arguments:**
149
+ - `example-ids` (required) - One or more example UUIDs
150
+
151
+ **Options:**
152
+ - `--confirm` - Skip confirmation prompt
153
+
154
+ **Output:** `{"status": "success", "deleted": [...], "errors": [...]}`
155
+
156
+ **Examples:**
157
+ ```bash
158
+ # Delete single example
159
+ langsmith-cli --json examples delete <uuid> --confirm
160
+
161
+ # Bulk delete
162
+ langsmith-cli --json examples delete <uuid1> <uuid2> <uuid3> --confirm
163
+ ```
164
+
165
+ ### `examples from-run`
166
+
167
+ Create an example from a run's inputs and outputs.
168
+
169
+ ```bash
170
+ langsmith-cli --json examples from-run <run-id> --dataset <name>
171
+ ```
172
+
173
+ **Arguments:**
174
+ - `run-id` (required) - Run UUID to create example from
175
+
176
+ **Options:**
177
+ - `--dataset TEXT` (required) - Dataset name to add the example to
178
+
179
+ **Output:** Created example object
180
+
181
+ **Example:**
182
+ ```bash
183
+ # Turn a good run into a training example
184
+ langsmith-cli --json examples from-run <run-uuid> --dataset "training-data"
185
+ ```
186
+
@@ -0,0 +1,141 @@
1
+ ## Projects
2
+
3
+ ### `projects list`
4
+
5
+ List all LangSmith projects (sessions).
6
+
7
+ ```bash
8
+ langsmith-cli --json projects list [OPTIONS]
9
+ ```
10
+
11
+ **Options:**
12
+ - `--limit INTEGER` - Maximum number of projects to return (default: 100, use 0 for no limit)
13
+ - `--name TEXT` - Filter by exact project name
14
+ - `--name-pattern TEXT` - Filter by name with wildcards (e.g. `'*prod*'`)
15
+ - `--name-regex TEXT` - Filter by name with regex
16
+ - `--reference-dataset-name TEXT` - Filter projects by reference dataset name
17
+ - `--has-runs` - Show only projects with runs (run_count > 0)
18
+ - `--sort-by TEXT` - Sort by field (name, run_count). Prefix with `-` for descending
19
+ - `--exclude TEXT` - Exclude items containing substring (repeatable)
20
+ - `--fields TEXT` - Comma-separated field names to include
21
+ - `--count` - Output only the count of results
22
+ - `--output TEXT` - Write output to file (JSONL format)
23
+ - `--format [table|json|csv|yaml]` - Output format
24
+
25
+ **Output Fields:**
26
+ - `id` (UUID) - Project identifier
27
+ - `name` (string) - Project name
28
+ - `description` (string|null) - Project description
29
+ - `created_at` (datetime) - Creation timestamp
30
+ - `run_count` (integer) - Number of runs in project
31
+ - `latency_p50` (float|null) - Median latency in seconds
32
+ - `latency_p99` (float|null) - 99th percentile latency
33
+ - `first_start_time` (datetime|null) - First run start time
34
+ - `last_start_time` (datetime|null) - Most recent run start time
35
+ - `feedback_stats` (object|null) - Feedback statistics
36
+ - `total_tokens` (integer|null) - Total tokens used
37
+ - `prompt_tokens` (integer|null) - Prompt tokens
38
+ - `completion_tokens` (integer|null) - Completion tokens
39
+ - `total_cost` (float|null) - Total cost in USD
40
+
41
+ **Example:**
42
+ ```bash
43
+ langsmith-cli --json projects list --limit 5
44
+ ```
45
+
46
+ ### `projects create`
47
+
48
+ Create a new project.
49
+
50
+ ```bash
51
+ langsmith-cli --json projects create <name> [OPTIONS]
52
+ ```
53
+
54
+ **Arguments:**
55
+ - `name` (required) - Project name
56
+
57
+ **Options:**
58
+ - `--description TEXT` - Project description
59
+
60
+ **Output:** Created project object
61
+
62
+ **Example:**
63
+ ```bash
64
+ langsmith-cli --json projects create "my-experiment" --description "Testing new prompt"
65
+ ```
66
+
67
+ ### `projects get`
68
+
69
+ Get details of a single project by name or ID.
70
+
71
+ ```bash
72
+ langsmith-cli --json projects get <name-or-id> [OPTIONS]
73
+ ```
74
+
75
+ **Arguments:**
76
+ - `name-or-id` (required) - Project name or UUID (UUIDs auto-detected)
77
+
78
+ **Options:**
79
+ - `--include-stats/--no-stats` - Include/exclude run statistics (default: include)
80
+ - `--fields TEXT` - Comma-separated fields to return
81
+ - `--output FILE` - Write output to file
82
+
83
+ **Output:** Full project object (same fields as `projects list`)
84
+
85
+ **Examples:**
86
+ ```bash
87
+ # Get by name
88
+ langsmith-cli --json projects get "my-project"
89
+
90
+ # Get by UUID (auto-detected, saves an API call)
91
+ langsmith-cli --json projects get "f47ac10b-58cc-4372-a567-0e02b2c3d479"
92
+
93
+ # Get with field pruning
94
+ langsmith-cli --json projects get "my-project" --fields name,run_count,error_rate
95
+ ```
96
+
97
+ ### `projects update`
98
+
99
+ Update a project's name or description.
100
+
101
+ ```bash
102
+ langsmith-cli --json projects update <name-or-id> [OPTIONS]
103
+ ```
104
+
105
+ **Arguments:**
106
+ - `name-or-id` (required) - Project name or UUID
107
+
108
+ **Options:**
109
+ - `--name TEXT` - New project name
110
+ - `--description TEXT` - New project description
111
+
112
+ At least one of `--name` or `--description` is required.
113
+
114
+ **Output:** Updated project object
115
+
116
+ **Example:**
117
+ ```bash
118
+ langsmith-cli --json projects update "old-name" --name "new-name" --description "Updated"
119
+ ```
120
+
121
+ ### `projects delete`
122
+
123
+ Delete a project.
124
+
125
+ ```bash
126
+ langsmith-cli --json projects delete <name-or-id> [OPTIONS]
127
+ ```
128
+
129
+ **Arguments:**
130
+ - `name-or-id` (required) - Project name or UUID
131
+
132
+ **Options:**
133
+ - `--confirm` - Skip confirmation prompt
134
+
135
+ **Output:** Success status
136
+
137
+ **Example:**
138
+ ```bash
139
+ langsmith-cli --json projects delete "test-project" --confirm
140
+ ```
141
+