evalmarket-cli 0.1.0__tar.gz → 0.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (27) hide show
  1. evalmarket_cli-0.1.1/PKG-INFO +19 -0
  2. evalmarket_cli-0.1.1/README.md +12 -0
  3. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/_version.py +1 -1
  4. evalmarket_cli-0.1.0/PKG-INFO +0 -69
  5. evalmarket_cli-0.1.0/README.md +0 -62
  6. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/.gitignore +0 -0
  7. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/pyproject.toml +0 -0
  8. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/__init__.py +0 -0
  9. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/__main__.py +0 -0
  10. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/__init__.py +0 -0
  11. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/client.py +0 -0
  12. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/experiments.py +0 -0
  13. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/importer.py +0 -0
  14. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/models.py +0 -0
  15. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/resources.py +0 -0
  16. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/braintrust/validation.py +0 -0
  17. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/cli.py +0 -0
  18. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/entrypoint.py +0 -0
  19. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/json_data.py +0 -0
  20. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/__init__.py +0 -0
  21. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/importer.py +0 -0
  22. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/models.py +0 -0
  23. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/playground.py +0 -0
  24. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/prompts.py +0 -0
  25. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/resources.py +0 -0
  26. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/src/evalmarket/langsmith/validation.py +0 -0
  27. {evalmarket_cli-0.1.0 → evalmarket_cli-0.1.1}/tests/test_installed.py +0 -0
@@ -0,0 +1,19 @@
1
+ Metadata-Version: 2.5
2
+ Name: evalmarket-cli
3
+ Version: 0.1.1
4
+ Summary: Import purchased EvalMarket evaluations into your evaluation platform
5
+ Requires-Python: >=3.10
6
+ Description-Content-Type: text/markdown
7
+
8
+ # EvalMarket CLI
9
+
10
+ Import an extracted EvalMarket evaluation into your own Braintrust project or LangSmith workspace. The CLI uses the platform files supplied with the evaluation and verifies the imported resources.
11
+
12
+ Install or upgrade in a Python 3.10+ virtual environment:
13
+
14
+ ```sh
15
+ python -m pip install --upgrade evalmarket-cli
16
+ evalmarket --version
17
+ ```
18
+
19
+ Run `evalmarket import braintrust` or `evalmarket import langchain` from your extracted evaluation folder. For platform-specific prerequisites, see the [Braintrust guide](https://docs.evalmarket.ai/buyers/platform-integrations/braintrust/) or the [LangSmith guide](https://docs.evalmarket.ai/buyers/platform-integrations/langsmith/).
@@ -0,0 +1,12 @@
1
+ # EvalMarket CLI
2
+
3
+ Import an extracted EvalMarket evaluation into your own Braintrust project or LangSmith workspace. The CLI uses the platform files supplied with the evaluation and verifies the imported resources.
4
+
5
+ Install or upgrade in a Python 3.10+ virtual environment:
6
+
7
+ ```sh
8
+ python -m pip install --upgrade evalmarket-cli
9
+ evalmarket --version
10
+ ```
11
+
12
+ Run `evalmarket import braintrust` or `evalmarket import langchain` from your extracted evaluation folder. For platform-specific prerequisites, see the [Braintrust guide](https://docs.evalmarket.ai/buyers/platform-integrations/braintrust/) or the [LangSmith guide](https://docs.evalmarket.ai/buyers/platform-integrations/langsmith/).
@@ -1,3 +1,3 @@
1
1
  """Version used by the CLI and distribution metadata."""
2
2
 
3
- __version__ = "0.1.0"
3
+ __version__ = "0.1.1"
@@ -1,69 +0,0 @@
1
- Metadata-Version: 2.5
2
- Name: evalmarket-cli
3
- Version: 0.1.0
4
- Summary: Import purchased EvalMarket evaluations into your evaluation platform
5
- Requires-Python: >=3.10
6
- Description-Content-Type: text/markdown
7
-
8
- # EvalMarket Python importer
9
-
10
- Customers install the versioned Python wheel from the marketplace's [CLI download page](https://www.evalmarket.ai/import-guide/cli). This page and the wheel ship together with the website. Python 3.10+ is required. The distribution has no Python runtime dependencies. LangSmith imports also need the official LangSmith CLI on PATH and its authentication/profile setup. Braintrust uses a local API key.
11
-
12
- Download the wheel into a folder outside the extracted evaluation. In an activated Python virtual environment, run these commands from the wheel's folder:
13
-
14
- ```sh
15
- python -I -m pip install --no-index --no-deps --upgrade ./evalmarket_cli-0.1.0-py3-none-any.whl
16
- evalmarket --version
17
- evalmarket import /path/to/braintrust
18
- evalmarket import /path/to/langchain --profile my-profile
19
- ```
20
-
21
- For upgrades, download the newer wheel and repeat the install with its exact filename in the same environment. Existing extracted evaluation folders can be reused. The package has not been published to a registry.
22
-
23
- For development in the API environment, run `uv sync --locked` in `apps/api`,
24
- then `uv run evalmarket import --help`.
25
- The module entry point is `python -I -m evalmarket` in an installed environment.
26
-
27
- The CLI preserves existing model selection, confirmation, readback verification,
28
- partial-failure handling, and explicit experiment/Playground opt-in. It imports
29
- an extracted platform folder. It does not download purchases or unpack ZIPs.
30
-
31
- ## Source layout
32
-
33
- ```text
34
- src/evalmarket/
35
- cli.py Argument parsing and error presentation
36
- entrypoint.py Isolated process startup
37
- json_data.py Shared file-size limit, strict JSON parsing and readback comparisons
38
- braintrust/
39
- validation.py Pure native package validation shared with the API
40
- client.py HTTP, pagination, experiment-start acknowledgement
41
- resources.py Resource identifiers and model labels
42
- models.py Account model discovery and selection
43
- importer.py Import orchestration and readback
44
- experiments.py Explicit optional experiment start
45
- langsmith/
46
- validation.py Pure prompt-manifest validation shared with the API
47
- resources.py Resource identifiers
48
- models.py Saved judge-model selection
49
- importer.py Native CLI import orchestration and authentication
50
- prompts.py Private answering prompt creation and readback
51
- playground.py Explicit optional browser handoff
52
- ```
53
-
54
- Use explicit module imports and keep pure validators independent of platform
55
- clients. A new platform should have its own modules and explicit CLI routing;
56
- there is no plugin framework or stable Python SDK contract in this release.
57
- The version in `_version.py` drives distribution metadata and `--version`.
58
-
59
- ## Build and verify
60
-
61
- From the repository root, run:
62
-
63
- ```sh
64
- uv build --no-sources packages/evalmarket --out-dir dist
65
- ```
66
-
67
- Install the resulting wheel in a clean environment. Run `packages/evalmarket/tests/test_installed.py` with its Python interpreter. CI tests the installed CLI on Windows and Linux with Python 3.10 and 3.13. API tests exercise the provider workflows, ZIP delivery, and shared validators.
68
-
69
- For a new customer release, bump the version, build the wheel and source distribution, and copy the reviewed wheel to `apps/app/public/downloads/` under its original filename. Update the download page and verify the public wheel before API cutover. Obtain release authorization before publication or deployment.
@@ -1,62 +0,0 @@
1
- # EvalMarket Python importer
2
-
3
- Customers install the versioned Python wheel from the marketplace's [CLI download page](https://www.evalmarket.ai/import-guide/cli). This page and the wheel ship together with the website. Python 3.10+ is required. The distribution has no Python runtime dependencies. LangSmith imports also need the official LangSmith CLI on PATH and its authentication/profile setup. Braintrust uses a local API key.
4
-
5
- Download the wheel into a folder outside the extracted evaluation. In an activated Python virtual environment, run these commands from the wheel's folder:
6
-
7
- ```sh
8
- python -I -m pip install --no-index --no-deps --upgrade ./evalmarket_cli-0.1.0-py3-none-any.whl
9
- evalmarket --version
10
- evalmarket import /path/to/braintrust
11
- evalmarket import /path/to/langchain --profile my-profile
12
- ```
13
-
14
- For upgrades, download the newer wheel and repeat the install with its exact filename in the same environment. Existing extracted evaluation folders can be reused. The package has not been published to a registry.
15
-
16
- For development in the API environment, run `uv sync --locked` in `apps/api`,
17
- then `uv run evalmarket import --help`.
18
- The module entry point is `python -I -m evalmarket` in an installed environment.
19
-
20
- The CLI preserves existing model selection, confirmation, readback verification,
21
- partial-failure handling, and explicit experiment/Playground opt-in. It imports
22
- an extracted platform folder. It does not download purchases or unpack ZIPs.
23
-
24
- ## Source layout
25
-
26
- ```text
27
- src/evalmarket/
28
- cli.py Argument parsing and error presentation
29
- entrypoint.py Isolated process startup
30
- json_data.py Shared file-size limit, strict JSON parsing and readback comparisons
31
- braintrust/
32
- validation.py Pure native package validation shared with the API
33
- client.py HTTP, pagination, experiment-start acknowledgement
34
- resources.py Resource identifiers and model labels
35
- models.py Account model discovery and selection
36
- importer.py Import orchestration and readback
37
- experiments.py Explicit optional experiment start
38
- langsmith/
39
- validation.py Pure prompt-manifest validation shared with the API
40
- resources.py Resource identifiers
41
- models.py Saved judge-model selection
42
- importer.py Native CLI import orchestration and authentication
43
- prompts.py Private answering prompt creation and readback
44
- playground.py Explicit optional browser handoff
45
- ```
46
-
47
- Use explicit module imports and keep pure validators independent of platform
48
- clients. A new platform should have its own modules and explicit CLI routing;
49
- there is no plugin framework or stable Python SDK contract in this release.
50
- The version in `_version.py` drives distribution metadata and `--version`.
51
-
52
- ## Build and verify
53
-
54
- From the repository root, run:
55
-
56
- ```sh
57
- uv build --no-sources packages/evalmarket --out-dir dist
58
- ```
59
-
60
- Install the resulting wheel in a clean environment. Run `packages/evalmarket/tests/test_installed.py` with its Python interpreter. CI tests the installed CLI on Windows and Linux with Python 3.10 and 3.13. API tests exercise the provider workflows, ZIP delivery, and shared validators.
61
-
62
- For a new customer release, bump the version, build the wheel and source distribution, and copy the reviewed wheel to `apps/app/public/downloads/` under its original filename. Update the download page and verify the public wheel before API cutover. Obtain release authorization before publication or deployment.