unstract-cli 0.1.0rc1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- unstract_cli-0.1.0rc1/.claude/skills/bump-client-pins/SKILL.md +108 -0
- unstract_cli-0.1.0rc1/.github/workflows/ci.yml +74 -0
- unstract_cli-0.1.0rc1/.github/workflows/release.yml +226 -0
- unstract_cli-0.1.0rc1/.gitignore +10 -0
- unstract_cli-0.1.0rc1/PKG-INFO +197 -0
- unstract_cli-0.1.0rc1/README.md +181 -0
- unstract_cli-0.1.0rc1/install.sh +42 -0
- unstract_cli-0.1.0rc1/pyproject.toml +60 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/__init__.py +3 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/__main__.py +100 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/app.py +321 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/__init__.py +0 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/clone_cmd.py +297 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/common.py +114 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/config_cmd.py +534 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/docstudio_cmd.py +298 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/platform_cmd.py +659 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/commands/whisper_cmd.py +419 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/config.py +745 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/__init__.py +0 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/clients.py +417 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/discover.py +189 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/errors.py +443 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/output.py +431 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/overlay.py +50 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/params.py +518 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/platform.py +102 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/core/poll.py +432 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/overlay.toml +7 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/specs/README.md +28 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/specs/docstudio.json +1077 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/specs/llmwhisperer.json +2709 -0
- unstract_cli-0.1.0rc1/src/unstract_cli/specs/provenance.json +16 -0
- unstract_cli-0.1.0rc1/tests/__init__.py +0 -0
- unstract_cli-0.1.0rc1/tests/conftest.py +81 -0
- unstract_cli-0.1.0rc1/tests/derived_flags.json +500 -0
- unstract_cli-0.1.0rc1/tests/test_cli.py +401 -0
- unstract_cli-0.1.0rc1/tests/test_clients.py +227 -0
- unstract_cli-0.1.0rc1/tests/test_commands.py +3657 -0
- unstract_cli-0.1.0rc1/tests/test_config.py +629 -0
- unstract_cli-0.1.0rc1/tests/test_contract.py +154 -0
- unstract_cli-0.1.0rc1/tests/test_discover.py +553 -0
- unstract_cli-0.1.0rc1/tests/test_errors.py +245 -0
- unstract_cli-0.1.0rc1/tests/test_output.py +229 -0
- unstract_cli-0.1.0rc1/tests/test_params.py +313 -0
- unstract_cli-0.1.0rc1/tests/test_platform.py +143 -0
- unstract_cli-0.1.0rc1/tests/test_poll.py +649 -0
- unstract_cli-0.1.0rc1/tests/test_specs.py +44 -0
- unstract_cli-0.1.0rc1/tests/test_workflows.py +168 -0
- unstract_cli-0.1.0rc1/uv.lock +433 -0
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: bump-client-pins
|
|
3
|
+
description: Bump the exact `unstract-client` / `llmwhisperer-client` pins, re-sync the vendored specs to match, and cut a CLI release. Use whenever a new version of either client is released, when `tests/test_specs.py` or `tests/test_contract.py` fails, when the CLI is missing a flag for an endpoint the API already has, or when someone asks to "bump the client", "update the pins", "refresh the specs", or "release the CLI". Reach for this even when the request sounds like a plain dependency bump — the pins, the vendored specs and `provenance.json` have to move together or the CLI derives flags the pinned client cannot carry.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Bumping the client pins
|
|
7
|
+
|
|
8
|
+
The CLI derives its flags and help text from two published clients and from
|
|
9
|
+
vendored copies of the specs those clients were generated from. That makes a pin
|
|
10
|
+
bump three coupled edits, not one: the pin, the spec, and the provenance record.
|
|
11
|
+
Move one without the others and the tests say so — which is the point of them.
|
|
12
|
+
|
|
13
|
+
## The pieces
|
|
14
|
+
|
|
15
|
+
| Thing | Where |
|
|
16
|
+
|---|---|
|
|
17
|
+
| Exact pins | `pyproject.toml`, `[project].dependencies` |
|
|
18
|
+
| Vendored specs | `src/unstract_cli/specs/{docstudio,llmwhisperer}.json` |
|
|
19
|
+
| Provenance | `src/unstract_cli/specs/provenance.json` (client pin, upstream repo, commit, sha256) |
|
|
20
|
+
| Coherence tests | `tests/test_specs.py`, `tests/test_contract.py`, `tests/derived_flags.json` |
|
|
21
|
+
| Release | `.github/workflows/release.yml`, `workflow_dispatch` |
|
|
22
|
+
|
|
23
|
+
`src/unstract_cli/specs/README.md` explains the vendoring rule in place; read it
|
|
24
|
+
if any of the below is unclear.
|
|
25
|
+
|
|
26
|
+
## The sequence
|
|
27
|
+
|
|
28
|
+
1. **Move the pins** in `pyproject.toml` to the released versions you are
|
|
29
|
+
upgrading to. They are exact (`==`) on purpose: the CLI's published surface is
|
|
30
|
+
derived from these clients, so a client that moves reshapes the CLI.
|
|
31
|
+
|
|
32
|
+
2. **Relock:** `uv lock` then `uv sync --extra dev --python 3.12`. CI installs
|
|
33
|
+
from `uv.lock`, not from a fresh resolve, so a lockfile left behind means the
|
|
34
|
+
gate tests a dependency set nobody ships.
|
|
35
|
+
|
|
36
|
+
3. **Re-sync each vendored spec from the commit the pinned client was generated
|
|
37
|
+
from.** The chain is: the client repo's release tag → its `tools/gen_sdk.sh`,
|
|
38
|
+
which records the upstream service repo, path and revision the spec was copied
|
|
39
|
+
from → the spec file committed in that client at that tag. Copy that file here
|
|
40
|
+
byte-for-byte. Copying from anywhere else — upstream `main`, a newer service
|
|
41
|
+
commit — is what `tests/test_contract.py` guards: a spec parameter the pinned
|
|
42
|
+
client has no argument for cannot become a flag.
|
|
43
|
+
|
|
44
|
+
4. **Update `provenance.json`** for each spec you moved. Set `client` to the
|
|
45
|
+
exact pin you wrote in `pyproject.toml` (`unstract-client==X.Y.Z`); the
|
|
46
|
+
tests compare the two, so a pin moved without its spec fails here. Check
|
|
47
|
+
`repo`, `path` and `commit` against what that client's `tools/gen_sdk.sh`
|
|
48
|
+
records, because an upstream that moved its spec file leaves `repo` and
|
|
49
|
+
`path` stale and the tests cannot see it: they check the pin, the `sha256`
|
|
50
|
+
and the entry names, nothing about where the file came from. The `sha256`
|
|
51
|
+
is of the file you just wrote (`sha256sum src/unstract_cli/specs/<file>`).
|
|
52
|
+
This record is what lets the next person tell a current copy from a stale
|
|
53
|
+
one.
|
|
54
|
+
|
|
55
|
+
5. **Run the tests:** `uv run pytest -q`.
|
|
56
|
+
|
|
57
|
+
- `test_specs.py` fails if a vendored file stops matching its recorded
|
|
58
|
+
sha256, if its `client` no longer equals the pin in `pyproject.toml`, or if
|
|
59
|
+
a spec has no provenance entry. It is the cheap check that steps 1, 3 and
|
|
60
|
+
4 actually agree.
|
|
61
|
+
- `test_contract.py` fails if a spec parameter the pinned client cannot accept
|
|
62
|
+
would have become a flag, and separately if the derived flags stop matching
|
|
63
|
+
`tests/derived_flags.json`.
|
|
64
|
+
|
|
65
|
+
6. **If `derived_flags.json` fails, read the difference before refreshing it.**
|
|
66
|
+
The failure names the flags that moved. A flag missing from the new set is a
|
|
67
|
+
flag the CLI has stopped offering; a narrowed choice or changed type is a value
|
|
68
|
+
the CLI used to take and now rejects. Once you have decided the change is
|
|
69
|
+
intended, refresh it deliberately:
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
UNSTRACT_CLI_REFRESH_FLAG_SNAPSHOT=1 uv run pytest -q tests/test_contract.py
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
and commit the snapshot in the same PR, so the diff shows what the CLI's
|
|
76
|
+
surface gained or lost.
|
|
77
|
+
|
|
78
|
+
7. **Lint:** `uv run ruff check . && uv run ruff format --check .` — the release
|
|
79
|
+
run repeats exactly this, so a failure here is a failure there.
|
|
80
|
+
|
|
81
|
+
## Versioning and release
|
|
82
|
+
|
|
83
|
+
Choose the bump by what changed for CLI users: **minor** for new or changed
|
|
84
|
+
flags and commands, **patch** for fixes that leave the surface identical.
|
|
85
|
+
|
|
86
|
+
Do not bump `__version__` in `src/unstract_cli/__init__.py` in your PR. The
|
|
87
|
+
in-repo value names the last released version; `release.yml` reads it, applies
|
|
88
|
+
the bump chosen at dispatch and commits the result itself.
|
|
89
|
+
|
|
90
|
+
Release by dispatching **Release Tag and Publish Package** on `main`:
|
|
91
|
+
|
|
92
|
+
- `version_bump: none` publishes the version already in the repo — what the
|
|
93
|
+
first release of a version needs.
|
|
94
|
+
- `pre_release: true` publishes `<next-version>rcN` and deliberately leaves the
|
|
95
|
+
committed version alone, counting N up from the rc tags already published for
|
|
96
|
+
that target. To promote to stable, dispatch again with it off **and the same
|
|
97
|
+
`version_bump`**: the workflow recomputes the target from that input every
|
|
98
|
+
time, so a different bump publishes a different version than the one the rc
|
|
99
|
+
tested.
|
|
100
|
+
- It publishes to PyPI **before** it tags and releases, because publishing is the
|
|
101
|
+
only step that cannot be undone: a failure before it leaves nothing to
|
|
102
|
+
unpublish, and one after it is retried by hand against a live artifact.
|
|
103
|
+
|
|
104
|
+
## Upstream
|
|
105
|
+
|
|
106
|
+
If a client pin is missing an endpoint the service already offers, the fix is in
|
|
107
|
+
that client, not here — see the `spec-upgrade` skill in `unstract-python-client`
|
|
108
|
+
and `llm-whisperer-python-client`. Bump the pin here once it is released.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
pull_request:
|
|
5
|
+
push:
|
|
6
|
+
branches: [main]
|
|
7
|
+
|
|
8
|
+
jobs:
|
|
9
|
+
# Offline by design: no network, no credentials, sub-second. Live round trips
|
|
10
|
+
# are a manual pre-release step, not a per-PR gate.
|
|
11
|
+
test:
|
|
12
|
+
runs-on: ubuntu-latest
|
|
13
|
+
steps:
|
|
14
|
+
- uses: actions/checkout@v4
|
|
15
|
+
- uses: astral-sh/setup-uv@v6
|
|
16
|
+
with:
|
|
17
|
+
version: "0.6.14"
|
|
18
|
+
enable-cache: true
|
|
19
|
+
# Synced from uv.lock rather than resolved fresh, so the gate tests the
|
|
20
|
+
# dependency set an install actually gets.
|
|
21
|
+
- run: uv sync --extra dev --python 3.12
|
|
22
|
+
- run: uv run ruff check .
|
|
23
|
+
- run: uv run ruff format --check .
|
|
24
|
+
- run: uv run pytest -q
|
|
25
|
+
# `uv run` resolves from uv.lock, whose click sits in the middle of the
|
|
26
|
+
# range the pin allows. Both ends are what an install in the wild gets --
|
|
27
|
+
# the floor exactly, not the newest release that satisfies it -- and
|
|
28
|
+
# discovery is a published contract read out of Click's own objects, so
|
|
29
|
+
# both ends are run and their answers compared.
|
|
30
|
+
- run: uv pip install 'click==8.1.0'
|
|
31
|
+
- run: .venv/bin/python -m pytest -q
|
|
32
|
+
- run: .venv/bin/python -m unstract_cli -o json --discover full > floor.json
|
|
33
|
+
- run: uv pip install -U 'click>=8.1,<9'
|
|
34
|
+
- run: .venv/bin/python -m pytest -q
|
|
35
|
+
- run: .venv/bin/python -m unstract_cli -o json --discover full > latest.json
|
|
36
|
+
- run: diff floor.json latest.json
|
|
37
|
+
|
|
38
|
+
# The wheel rather than the checkout: packaging and entry points are declared
|
|
39
|
+
# in `pyproject.toml` and are not exercised by running the tests in place.
|
|
40
|
+
package:
|
|
41
|
+
runs-on: ubuntu-latest
|
|
42
|
+
env:
|
|
43
|
+
VIRTUAL_ENV: /tmp/smoke
|
|
44
|
+
steps:
|
|
45
|
+
- uses: actions/checkout@v4
|
|
46
|
+
- uses: astral-sh/setup-uv@v6
|
|
47
|
+
with:
|
|
48
|
+
version: "0.6.14"
|
|
49
|
+
enable-cache: true
|
|
50
|
+
- run: uv build --out-dir dist
|
|
51
|
+
- run: uv venv --python 3.12 /tmp/smoke
|
|
52
|
+
- run: uv pip install dist/*.whl
|
|
53
|
+
- name: Both entry points answer
|
|
54
|
+
run: |
|
|
55
|
+
/tmp/smoke/bin/unstract --version
|
|
56
|
+
/tmp/smoke/bin/unstract-cli --version
|
|
57
|
+
/tmp/smoke/bin/unstract --help > /dev/null
|
|
58
|
+
- name: The published surfaces render from an installed wheel
|
|
59
|
+
run: |
|
|
60
|
+
/tmp/smoke/bin/unstract -o json --discover full > discover.json
|
|
61
|
+
python -c "import json; d = json.load(open('discover.json')); assert d['ok'] and d['data']['commands']"
|
|
62
|
+
# A usage error still renders the envelope and still exits 2.
|
|
63
|
+
set +e
|
|
64
|
+
/tmp/smoke/bin/unstract -o json whisper status > usage.json
|
|
65
|
+
code=$?
|
|
66
|
+
set -e
|
|
67
|
+
test "$code" = "2"
|
|
68
|
+
python -c "import json; d = json.load(open('usage.json')); assert d['ok'] is False and d['error']['exit_code'] == 2"
|
|
69
|
+
- name: A fresh install can write its own config
|
|
70
|
+
env:
|
|
71
|
+
UNSTRACT_CONFIG: /tmp/smoke-config.toml
|
|
72
|
+
run: |
|
|
73
|
+
/tmp/smoke/bin/unstract config init < /dev/null
|
|
74
|
+
test -s /tmp/smoke-config.toml
|
|
@@ -0,0 +1,226 @@
|
|
|
1
|
+
name: Release Tag and Publish Package
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
workflow_dispatch:
|
|
5
|
+
inputs:
|
|
6
|
+
version_bump:
|
|
7
|
+
description: "Version bump type."
|
|
8
|
+
required: true
|
|
9
|
+
default: "patch"
|
|
10
|
+
type: choice
|
|
11
|
+
options:
|
|
12
|
+
- patch
|
|
13
|
+
- minor
|
|
14
|
+
- major
|
|
15
|
+
pre_release:
|
|
16
|
+
description: "Publish a release candidate (`<version>rcN`) instead of the version itself. Dispatch again with the same bump and this off to promote it to stable."
|
|
17
|
+
required: false
|
|
18
|
+
default: false
|
|
19
|
+
type: boolean
|
|
20
|
+
release_notes:
|
|
21
|
+
description: "Release notes (optional)"
|
|
22
|
+
required: false
|
|
23
|
+
type: string
|
|
24
|
+
|
|
25
|
+
concurrency:
|
|
26
|
+
group: release
|
|
27
|
+
cancel-in-progress: false
|
|
28
|
+
|
|
29
|
+
jobs:
|
|
30
|
+
release-and-publish:
|
|
31
|
+
runs-on: ubuntu-latest
|
|
32
|
+
# Dispatchable from any branch by anyone who can dispatch it, and a
|
|
33
|
+
# candidate reaches PyPI without a commit to review: the published artefact
|
|
34
|
+
# must come from the reviewed branch.
|
|
35
|
+
if: github.ref == 'refs/heads/main'
|
|
36
|
+
permissions:
|
|
37
|
+
contents: write
|
|
38
|
+
# Publishing is by PyPI Trusted Publisher, so there is no API token.
|
|
39
|
+
id-token: write
|
|
40
|
+
steps:
|
|
41
|
+
# Checked before the checkout, so a missing one fails on the reason
|
|
42
|
+
# rather than on a token that renders empty three steps later.
|
|
43
|
+
- name: Check the release credentials are visible to this repository
|
|
44
|
+
env:
|
|
45
|
+
APP_CLIENT_ID: ${{ vars.PUSH_TO_MAIN_APP_CLIENT_ID }}
|
|
46
|
+
APP_PRIVATE_KEY: ${{ secrets.PUSH_TO_MAIN_APP_PRIVATE_KEY }}
|
|
47
|
+
run: |
|
|
48
|
+
missing=""
|
|
49
|
+
[ -n "$APP_CLIENT_ID" ] || missing="$missing vars.PUSH_TO_MAIN_APP_CLIENT_ID"
|
|
50
|
+
[ -n "$APP_PRIVATE_KEY" ] || missing="$missing secrets.PUSH_TO_MAIN_APP_PRIVATE_KEY"
|
|
51
|
+
if [ -n "$missing" ]; then
|
|
52
|
+
echo "Missing:$missing" >&2
|
|
53
|
+
echo "An organisation-level variable or secret also has to list this repository." >&2
|
|
54
|
+
exit 1
|
|
55
|
+
fi
|
|
56
|
+
|
|
57
|
+
- name: Generate GitHub App Token
|
|
58
|
+
id: generate-token
|
|
59
|
+
uses: actions/create-github-app-token@v3
|
|
60
|
+
with:
|
|
61
|
+
client-id: ${{ vars.PUSH_TO_MAIN_APP_CLIENT_ID }}
|
|
62
|
+
private-key: ${{ secrets.PUSH_TO_MAIN_APP_PRIVATE_KEY }}
|
|
63
|
+
owner: Zipstack
|
|
64
|
+
repositories: |
|
|
65
|
+
unstract-cli
|
|
66
|
+
|
|
67
|
+
- uses: actions/checkout@v4
|
|
68
|
+
with:
|
|
69
|
+
token: ${{ steps.generate-token.outputs.token }}
|
|
70
|
+
fetch-depth: 0
|
|
71
|
+
|
|
72
|
+
- name: Configure Git
|
|
73
|
+
run: |
|
|
74
|
+
git config --global user.name "github-actions[bot]"
|
|
75
|
+
git config --global user.email "github-actions[bot]@users.noreply.github.com"
|
|
76
|
+
|
|
77
|
+
- uses: actions/setup-python@v5
|
|
78
|
+
with:
|
|
79
|
+
python-version: "3.12"
|
|
80
|
+
|
|
81
|
+
- uses: astral-sh/setup-uv@v6
|
|
82
|
+
with:
|
|
83
|
+
version: "0.6.14"
|
|
84
|
+
enable-cache: true
|
|
85
|
+
|
|
86
|
+
# The same install as ci.yml, so what the release run lints and tests is
|
|
87
|
+
# what the PR gate lints and tests.
|
|
88
|
+
- run: uv sync --extra dev --python 3.12
|
|
89
|
+
|
|
90
|
+
# Staged locally only: nothing is committed, tagged or released until the
|
|
91
|
+
# checks and the build have passed, so a failure leaves main untouched.
|
|
92
|
+
- name: Compute new version
|
|
93
|
+
id: version
|
|
94
|
+
run: |
|
|
95
|
+
VERSION_FILE=src/unstract_cli/__init__.py
|
|
96
|
+
CURRENT_VERSION=$(sed -nE 's/^__version__ = "(.*)"/\1/p' "$VERSION_FILE")
|
|
97
|
+
echo "Current version: $CURRENT_VERSION"
|
|
98
|
+
|
|
99
|
+
IFS='.' read -r MAJOR MINOR PATCH <<< "$CURRENT_VERSION"
|
|
100
|
+
case "${{ github.event.inputs.version_bump }}" in
|
|
101
|
+
major) MAJOR=$((MAJOR + 1)); MINOR=0; PATCH=0 ;;
|
|
102
|
+
minor) MINOR=$((MINOR + 1)); PATCH=0 ;;
|
|
103
|
+
patch) PATCH=$((PATCH + 1)) ;;
|
|
104
|
+
esac
|
|
105
|
+
NEXT_VERSION="$MAJOR.$MINOR.$PATCH"
|
|
106
|
+
|
|
107
|
+
# A pre-release is a candidate for NEXT_VERSION, not a version of its
|
|
108
|
+
# own, so it never moves the committed one: the file keeps naming the
|
|
109
|
+
# last stable release, and repeat dispatches count up from the rc tags
|
|
110
|
+
# already published for that target.
|
|
111
|
+
if [ "${{ github.event.inputs.pre_release }}" = "true" ]; then
|
|
112
|
+
HIGHEST_RC=$(git tag -l "v${NEXT_VERSION}rc*" \
|
|
113
|
+
| sed -nE "s/^v${NEXT_VERSION}rc([0-9]+)$/\1/p" | sort -n | tail -1)
|
|
114
|
+
NEW_VERSION="${NEXT_VERSION}rc$(( ${HIGHEST_RC:-0} + 1 ))"
|
|
115
|
+
else
|
|
116
|
+
NEW_VERSION="$NEXT_VERSION"
|
|
117
|
+
fi
|
|
118
|
+
|
|
119
|
+
echo "New version: $NEW_VERSION"
|
|
120
|
+
echo "version=$NEW_VERSION" >> "$GITHUB_OUTPUT"
|
|
121
|
+
|
|
122
|
+
sed -i "s/^__version__ = \".*\"/__version__ = \"$NEW_VERSION\"/" "$VERSION_FILE"
|
|
123
|
+
|
|
124
|
+
if git rev-parse -q --verify "refs/tags/v$NEW_VERSION" >/dev/null; then
|
|
125
|
+
echo "Tag v$NEW_VERSION already exists. Exiting..."
|
|
126
|
+
exit 1
|
|
127
|
+
fi
|
|
128
|
+
|
|
129
|
+
- name: Verify version update
|
|
130
|
+
run: |
|
|
131
|
+
BUILT_VERSION=$(uv run python -c "import unstract_cli; print(unstract_cli.__version__)")
|
|
132
|
+
echo "Package version: $BUILT_VERSION"
|
|
133
|
+
echo "Target version: ${{ steps.version.outputs.version }}"
|
|
134
|
+
if [ "$BUILT_VERSION" != "${{ steps.version.outputs.version }}" ]; then
|
|
135
|
+
echo "Version mismatch! Exiting..."
|
|
136
|
+
exit 1
|
|
137
|
+
fi
|
|
138
|
+
|
|
139
|
+
- name: Run linting
|
|
140
|
+
run: |
|
|
141
|
+
uv run ruff check .
|
|
142
|
+
uv run ruff format --check .
|
|
143
|
+
|
|
144
|
+
- name: Run tests
|
|
145
|
+
run: uv run pytest -q
|
|
146
|
+
|
|
147
|
+
- name: Build package
|
|
148
|
+
run: uv build
|
|
149
|
+
|
|
150
|
+
# Publishing is the one step that cannot be undone, so it runs last, and
|
|
151
|
+
# the release is created as a draft so a publish that fails leaves
|
|
152
|
+
# nothing public. Until then the bump commit exists only in the tag, so
|
|
153
|
+
# a failed publish leaves main untouched.
|
|
154
|
+
- name: Commit version bump and create draft release
|
|
155
|
+
env:
|
|
156
|
+
RELEASE_NOTES: ${{ github.event.inputs.release_notes }}
|
|
157
|
+
GITHUB_TOKEN: ${{ steps.generate-token.outputs.token }}
|
|
158
|
+
run: |
|
|
159
|
+
NEW_VERSION="${{ steps.version.outputs.version }}"
|
|
160
|
+
|
|
161
|
+
# A pre-release leaves the committed version alone.
|
|
162
|
+
if [ "${{ github.event.inputs.pre_release }}" = "true" ]; then
|
|
163
|
+
git checkout -- src/unstract_cli/__init__.py
|
|
164
|
+
elif ! git diff --quiet; then
|
|
165
|
+
git add src/unstract_cli/__init__.py
|
|
166
|
+
git commit -m "chore: bump version to $NEW_VERSION [skip ci]"
|
|
167
|
+
fi
|
|
168
|
+
|
|
169
|
+
git tag "v$NEW_VERSION"
|
|
170
|
+
git push origin "v$NEW_VERSION"
|
|
171
|
+
|
|
172
|
+
if [ -z "$RELEASE_NOTES" ]; then
|
|
173
|
+
gh release create "v$NEW_VERSION" \
|
|
174
|
+
--title "Release v$NEW_VERSION" \
|
|
175
|
+
--generate-notes \
|
|
176
|
+
--draft \
|
|
177
|
+
${{ github.event.inputs.pre_release == 'true' && '--prerelease' || '' }}
|
|
178
|
+
else
|
|
179
|
+
gh release create "v$NEW_VERSION" \
|
|
180
|
+
--title "Release v$NEW_VERSION" \
|
|
181
|
+
--notes "$RELEASE_NOTES" \
|
|
182
|
+
--generate-notes \
|
|
183
|
+
--draft \
|
|
184
|
+
${{ github.event.inputs.pre_release == 'true' && '--prerelease' || '' }}
|
|
185
|
+
fi
|
|
186
|
+
|
|
187
|
+
echo "Created draft release v$NEW_VERSION"
|
|
188
|
+
|
|
189
|
+
- name: Publish to PyPI
|
|
190
|
+
run: uv publish
|
|
191
|
+
|
|
192
|
+
# Everything the run created before the publish, so a rerun after the
|
|
193
|
+
# cause is fixed starts clean rather than on the tag guard. Each removal
|
|
194
|
+
# tolerates its target being absent: the failure may have come partway
|
|
195
|
+
# through creating them.
|
|
196
|
+
- name: Remove the tag and draft release a failed publish leaves behind
|
|
197
|
+
if: failure() && steps.version.outcome == 'success'
|
|
198
|
+
env:
|
|
199
|
+
GITHUB_TOKEN: ${{ steps.generate-token.outputs.token }}
|
|
200
|
+
run: |
|
|
201
|
+
NEW_VERSION="${{ steps.version.outputs.version }}"
|
|
202
|
+
# The wheel and the sdist upload separately, so a failed publish can
|
|
203
|
+
# still have put the version on PyPI; then the tag has to stay.
|
|
204
|
+
if curl -fsS -o /dev/null "https://pypi.org/pypi/unstract-cli/$NEW_VERSION/json"; then
|
|
205
|
+
echo "PyPI serves $NEW_VERSION; keeping the tag and draft release." >&2
|
|
206
|
+
exit 0
|
|
207
|
+
fi
|
|
208
|
+
gh release delete "v$NEW_VERSION" --yes || true
|
|
209
|
+
git push origin ":refs/tags/v$NEW_VERSION" || true
|
|
210
|
+
|
|
211
|
+
# If this fails the published version is on PyPI and in the tag but not
|
|
212
|
+
# on main; `git push origin v<version>:main` finishes the job by hand.
|
|
213
|
+
- name: Push version bump
|
|
214
|
+
if: github.event.inputs.pre_release != 'true'
|
|
215
|
+
run: git push origin main
|
|
216
|
+
|
|
217
|
+
- name: Publish release
|
|
218
|
+
env:
|
|
219
|
+
GITHUB_TOKEN: ${{ steps.generate-token.outputs.token }}
|
|
220
|
+
run: gh release edit "v${{ steps.version.outputs.version }}" --draft=false
|
|
221
|
+
|
|
222
|
+
- name: Success message
|
|
223
|
+
run: |
|
|
224
|
+
echo "Published ${{ steps.version.outputs.version }} to PyPI with uv publish using Trusted Publishers"
|
|
225
|
+
echo "Release: https://github.com/${{ github.repository }}/releases/tag/v${{ steps.version.outputs.version }}"
|
|
226
|
+
echo "PyPI: https://pypi.org/project/unstract-cli/${{ steps.version.outputs.version }}/"
|
|
@@ -0,0 +1,197 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: unstract-cli
|
|
3
|
+
Version: 0.1.0rc1
|
|
4
|
+
Summary: Unified, LLM-friendly CLI for the Unstract suite of products
|
|
5
|
+
Requires-Python: >=3.12
|
|
6
|
+
Requires-Dist: click<9,>=8.1
|
|
7
|
+
Requires-Dist: llmwhisperer-client==2.9.0
|
|
8
|
+
Requires-Dist: requests>=2.32.3
|
|
9
|
+
Requires-Dist: tomli-w>=1.0
|
|
10
|
+
Requires-Dist: unstract-client==1.7.0
|
|
11
|
+
Provides-Extra: dev
|
|
12
|
+
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
13
|
+
Requires-Dist: pyyaml>=6.0; extra == 'dev'
|
|
14
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
|
|
17
|
+
# unstract-cli
|
|
18
|
+
|
|
19
|
+
`unstract` — one CLI for the Unstract suite: extract a document with
|
|
20
|
+
LLMWhisperer, run it through a Document Studio API deployment, get structured
|
|
21
|
+
JSON back. It also clones one organization's resources into another.
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
curl -LsSf https://raw.githubusercontent.com/Zipstack/unstract-cli/main/install.sh | sh
|
|
25
|
+
unstract auth login # asks for your keys, checks them, stores them
|
|
26
|
+
unstract docstudio deployment ls # what can I run?
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
For an agent or CI, no prompts and no file — the environment is the profile:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
export UNSTRACT_ORG_ID=... UNSTRACT_DEPLOYMENT_KEY=... LLMWHISPERER_API_KEY=...
|
|
33
|
+
unstract -o json whisper extract ./doc.pdf
|
|
34
|
+
unstract -o json docstudio deployment run invoice-parser ./doc.pdf
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
The installer fetches `uv` if it is missing and installs the CLI with it; `uv`
|
|
38
|
+
brings its own Python, so nothing on the machine has to match. Already have
|
|
39
|
+
`uv`? `uv tool install git+https://github.com/Zipstack/unstract-cli` is the same
|
|
40
|
+
thing. Set `UNSTRACT_CLI_SOURCE` to install a branch or a local checkout
|
|
41
|
+
instead.
|
|
42
|
+
|
|
43
|
+
Or run it without installing: `uvx --from git+https://github.com/Zipstack/unstract-cli unstract --discover groups`.
|
|
44
|
+
|
|
45
|
+
## Output
|
|
46
|
+
|
|
47
|
+
`unstract` prints a table by default — in a terminal and in a pipe alike, so
|
|
48
|
+
what you see while trying something is what a script sees running it.
|
|
49
|
+
|
|
50
|
+
**Parsing anything? Pass `-o json`.** stdout then carries exactly one envelope,
|
|
51
|
+
on success and on failure alike:
|
|
52
|
+
|
|
53
|
+
```json
|
|
54
|
+
{"ok": true, "data": {...}, "error": null, "meta": {"contract_version": 1}}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`-o json` output depends on nothing but the command and its arguments — not the
|
|
58
|
+
terminal, not the config, not the environment. `-o raw` prints one field
|
|
59
|
+
unwrapped, for piping a document's text somewhere else. Diagnostics, warnings
|
|
60
|
+
and progress always go to stderr.
|
|
61
|
+
|
|
62
|
+
Consuming the JSON: ignore fields you do not recognise, and refuse a
|
|
63
|
+
`meta.contract_version` above the one you were written against. `unstract
|
|
64
|
+
--discover full` publishes the whole contract alongside every command and flag.
|
|
65
|
+
|
|
66
|
+
If a coding agent is driving (detected from the environment it sets), the
|
|
67
|
+
*default* becomes json. `--agent yes|no` forces that either way, and an explicit
|
|
68
|
+
`-o` always wins over both.
|
|
69
|
+
|
|
70
|
+
Failures exit non-zero with a stable code. The codes are this CLI's own
|
|
71
|
+
convention, not a service's — they are the `ExitCode` enum in
|
|
72
|
+
`core/errors.py`, and `--discover full` publishes the table so a caller does not
|
|
73
|
+
have to copy it:
|
|
74
|
+
|
|
75
|
+
| Code | Meaning |
|
|
76
|
+
|------|---------|
|
|
77
|
+
| 0 | success |
|
|
78
|
+
| 1 | generic failure |
|
|
79
|
+
| 2 | usage error |
|
|
80
|
+
| 3 | authentication failed |
|
|
81
|
+
| 4 | not found |
|
|
82
|
+
| 5 | validation failed — also a completed run in which a document failed; the full result, successful documents included, is in `error.details` |
|
|
83
|
+
| 6 | rate limited |
|
|
84
|
+
| 7 | timed out (the job handle is in the error payload — resume, do not resubmit) |
|
|
85
|
+
| 8 | server error |
|
|
86
|
+
| 9 | result already consumed (one-shot read; use `--save` next time) |
|
|
87
|
+
| 10 | the result was read but could not be saved — it is in `error.details` |
|
|
88
|
+
| 130 | interrupted (128 + SIGINT) — the user stopped it, not a failure |
|
|
89
|
+
|
|
90
|
+
## Credentials
|
|
91
|
+
|
|
92
|
+
Three keys, each for one job:
|
|
93
|
+
|
|
94
|
+
- **LLMWhisperer key** — extracts text (`whisper …`). Minted in the LLMWhisperer
|
|
95
|
+
console.
|
|
96
|
+
- **Deployment key** — runs deployments (`deployment run`, `deployment status`).
|
|
97
|
+
Shown on the API deployment's own page in the Unstract UI; one an
|
|
98
|
+
organisation admin mints under **Settings → Global API Deployment Keys**
|
|
99
|
+
covers every deployment in the organisation.
|
|
100
|
+
- **Platform key** — identifies the organisation and lists what is in it
|
|
101
|
+
(`auth whoami`, `deployment ls`). Minted by an organisation admin under
|
|
102
|
+
**Settings → Platform API Keys**.
|
|
103
|
+
|
|
104
|
+
`auth login` takes whichever of the three you have, checks the two it can
|
|
105
|
+
(`whoami` for the platform key, the usage endpoint for the LLMWhisperer key; a
|
|
106
|
+
deployment key has nothing side-effect-free to call and is stored as given) and
|
|
107
|
+
writes them to one profile. Run it again to rotate a key. A login that stores a
|
|
108
|
+
different host drops the profile's other keys rather than leave them beside a
|
|
109
|
+
server that never accepted them: it asks first, or without a terminal fails
|
|
110
|
+
until `--force`. Without a terminal pass them as flags — `--platform-key`, `--deployment-key`, `--llmwhisperer-key`,
|
|
111
|
+
any one of them `-` to read from stdin.
|
|
112
|
+
|
|
113
|
+
## Configuration
|
|
114
|
+
|
|
115
|
+
`~/.unstract/config.toml`, or a project-local `.unstract.toml` found by upward
|
|
116
|
+
search, or `$UNSTRACT_CONFIG`, or `--config`. Every setting resolves
|
|
117
|
+
**flag > env > profile > built-in default**, and the CLI is fully usable with no
|
|
118
|
+
config file at all. The flag tier is the connection options on each product
|
|
119
|
+
group — `--base-url`, `--api-key`, `--org-id` and `--platform-key` on
|
|
120
|
+
`docstudio`, `--base-url`/`--api-key` on `whisper`, `--base-url`/`--platform-key`
|
|
121
|
+
on `auth` — which override the profile for that one invocation without writing
|
|
122
|
+
anything.
|
|
123
|
+
|
|
124
|
+
```toml
|
|
125
|
+
default_profile = "cloud-us"
|
|
126
|
+
|
|
127
|
+
[profiles.cloud-us.llmwhisperer]
|
|
128
|
+
base_url = "https://llmwhisperer-api.us-central.unstract.com/api/v2"
|
|
129
|
+
api_key = "env:LLMWHISPERER_API_KEY"
|
|
130
|
+
|
|
131
|
+
[profiles.cloud-us.docstudio]
|
|
132
|
+
base_url = "https://us-central.unstract.com"
|
|
133
|
+
org_id = "org_ABC123"
|
|
134
|
+
api_key = "env:UNSTRACT_DEPLOYMENT_KEY"
|
|
135
|
+
platform_key = "env:UNSTRACT_PLATFORM_KEY"
|
|
136
|
+
|
|
137
|
+
# Only for a deployment whose key differs from the one above.
|
|
138
|
+
[profiles.cloud-us.deployments."invoice-parser"]
|
|
139
|
+
api_key = "env:INVOICE_PARSER_KEY"
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
`deployment run` and `deployment status` take the API name as `deployment ls`
|
|
143
|
+
prints it. `ls` itself authenticates with the platform key and refuses
|
|
144
|
+
`--api-key`, which on `docstudio` means a deployment key. The key for a run resolves `--api-key` > `$UNSTRACT_DEPLOYMENT_KEY` >
|
|
145
|
+
the deployment's own entry > the profile's `api_key`, so most profiles need no
|
|
146
|
+
`deployments` section at all; `config set docstudio api_key <key> --deployment
|
|
147
|
+
<api_name>` writes one. `org_id` lives on the `docstudio` block — `auth login`
|
|
148
|
+
and `auth whoami` write the one the platform key resolves there. `config init`
|
|
149
|
+
writes this shape minus `platform_key` and the `deployments` entry — both are
|
|
150
|
+
the exception, not the starting point — plus a `cloud-eu` profile and an
|
|
151
|
+
`onprem-example` shape to copy for a self-hosted install; only the *active*
|
|
152
|
+
profile is ever resolved.
|
|
153
|
+
|
|
154
|
+
Either form works for a credential. `auth login` writes keys literally, having
|
|
155
|
+
checked them at the moment it writes. `env:VAR_NAME` indirection — what `config
|
|
156
|
+
init` writes and what the example above uses — keeps the secret out of the file,
|
|
157
|
+
so it stays safe to copy or commit; that is the form for a shared machine or a
|
|
158
|
+
CI checkout. Either way the file is created `0600`, and `config doctor` warns
|
|
159
|
+
when its mode is wider than that.
|
|
160
|
+
|
|
161
|
+
`unstract config doctor` reports where each setting resolved from — including
|
|
162
|
+
whether an `env:` reference is actually set in the current process — without
|
|
163
|
+
echoing any value. `--probe` also checks the two keys that can be
|
|
164
|
+
checked — the platform key and the LLMWhisperer key, the same two `auth login`
|
|
165
|
+
checks — and, with a platform key, warns about a `deployments` entry the
|
|
166
|
+
organisation no longer has. It exits non-zero when one of its own checks failed, so a setup script can
|
|
167
|
+
branch on it.
|
|
168
|
+
|
|
169
|
+
A project-local `.unstract.toml` **found by upward search** may not supply a
|
|
170
|
+
key or `base_url`. Those are ignored, with a warning; everything else in it —
|
|
171
|
+
profile selection, `org_id` — applies as usual. A checkout you did not write is
|
|
172
|
+
not trusted to name the host your key is sent to. Name the file explicitly
|
|
173
|
+
(`--config` or `$UNSTRACT_CONFIG`) and it is honoured in full.
|
|
174
|
+
|
|
175
|
+
What that protects is the key and the host, not the routing: `org_id` and
|
|
176
|
+
profile selection stay repo-controllable by design, so a project file can still
|
|
177
|
+
decide *which* organisation a command runs against on a host you trust. Read
|
|
178
|
+
one before you run inside a checkout you did not write.
|
|
179
|
+
|
|
180
|
+
`clone` is the exception, and it is an operator command: a human moving one
|
|
181
|
+
organisation's resources into another, holding two admin Platform keys. It is
|
|
182
|
+
not part of the document-processing path the rest of this CLI wraps, so an agent
|
|
183
|
+
serving a user request should not reach for it unasked. It talks to two
|
|
184
|
+
deployments at once, which no single profile describes, so it takes both
|
|
185
|
+
endpoints as flags and both keys from `UNSTRACT_SRC_PLATFORM_KEY` /
|
|
186
|
+
`UNSTRACT_TGT_PLATFORM_KEY` — two keys for two organisations, so it reads
|
|
187
|
+
neither the profile's `platform_key` nor `$UNSTRACT_PLATFORM_KEY`. It exits 0
|
|
188
|
+
when nothing failed, which is not the same as everything having moved: oversize
|
|
189
|
+
and unsupported documents are skipped by design, and `data.skipped` counts them.
|
|
190
|
+
|
|
191
|
+
## Development
|
|
192
|
+
|
|
193
|
+
```bash
|
|
194
|
+
uv venv && uv pip install -e '.[dev]'
|
|
195
|
+
uv run pytest # offline; no network, no credentials
|
|
196
|
+
uv run ruff check .
|
|
197
|
+
```
|