sublinear 0.0.0.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- sublinear-0.0.0.1/PKG-INFO +134 -0
- sublinear-0.0.0.1/README.md +119 -0
- sublinear-0.0.0.1/pyproject.toml +35 -0
- sublinear-0.0.0.1/setup.cfg +4 -0
- sublinear-0.0.0.1/src/sublinear/__init__.py +11 -0
- sublinear-0.0.0.1/src/sublinear/core.py +402 -0
- sublinear-0.0.0.1/src/sublinear.egg-info/PKG-INFO +134 -0
- sublinear-0.0.0.1/src/sublinear.egg-info/SOURCES.txt +9 -0
- sublinear-0.0.0.1/src/sublinear.egg-info/dependency_links.txt +1 -0
- sublinear-0.0.0.1/src/sublinear.egg-info/requires.txt +3 -0
- sublinear-0.0.0.1/src/sublinear.egg-info/top_level.txt +1 -0
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sublinear
|
|
3
|
+
Version: 0.0.0.1
|
|
4
|
+
Summary: Python SDK for Sublinear foundation reward models for robotics.
|
|
5
|
+
Author: Sublinear
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://sublinear.dev
|
|
8
|
+
Project-URL: Documentation, https://sublinear.dev/docs
|
|
9
|
+
Keywords: robotics,reinforcement-learning,reward-models,video-language-models,embodied-ai
|
|
10
|
+
Requires-Python: >=3.10
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
Requires-Dist: numpy>=1.24
|
|
13
|
+
Requires-Dist: opencv-python-headless>=4.8
|
|
14
|
+
Requires-Dist: requests>=2.31
|
|
15
|
+
|
|
16
|
+
# Sublinear Python SDK
|
|
17
|
+
|
|
18
|
+
The official Python client for submitting robot videos and trajectories to Sublinear foundation reward models.
|
|
19
|
+
|
|
20
|
+
## Installation
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
pip install sublinear
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The PyPI distribution is named `sublinear`, while the Python import package is named `sublinear`:
|
|
27
|
+
|
|
28
|
+
```python
|
|
29
|
+
from sublinear import Sublinear
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Quick start
|
|
33
|
+
|
|
34
|
+
Create an API key from the Sublinear dashboard, then submit a local MP4 video:
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
import os
|
|
38
|
+
from sublinear import Sublinear
|
|
39
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"])
|
|
40
|
+
response = client.responses.create(model="sole-r1", task_description="Fold the clothing.", video_path="episode.mp4")
|
|
41
|
+
print(response.rewards)
|
|
42
|
+
print(response.success)
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
You can also provide RGB video frames directly as a list or NumPy array with shape `(frames, height, width, 3)`:
|
|
46
|
+
|
|
47
|
+
```python
|
|
48
|
+
import os
|
|
49
|
+
import numpy as np
|
|
50
|
+
from sublinear import Sublinear
|
|
51
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"])
|
|
52
|
+
video_frames = np.load("video_frames.npy")
|
|
53
|
+
response = client.responses.create(model="robometer", task_description="Fold the clothing.", video_frames=video_frames)
|
|
54
|
+
print(response.rewards)
|
|
55
|
+
print(response.success)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Multiple videos
|
|
59
|
+
|
|
60
|
+
Pass `video_paths` to process multiple videos. The SDK submits jobs concurrently, up to `max_parallel_jobs` at a time.
|
|
61
|
+
|
|
62
|
+
```python
|
|
63
|
+
import os
|
|
64
|
+
from sublinear import Sublinear
|
|
65
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"], max_parallel_jobs=8)
|
|
66
|
+
response = client.responses.create(model="sole-r1", task_description="Pick up the cube.", video_paths=["episode_1.mp4", "episode_2.mp4"])
|
|
67
|
+
print(response.rewards)
|
|
68
|
+
print(response.success)
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Available models
|
|
72
|
+
|
|
73
|
+
- `sole-r1` — reasoning-based robot trajectory evaluation
|
|
74
|
+
- `robometer` — dense task-progress reward estimation
|
|
75
|
+
- `topreward` — visual reward estimation across robot tasks
|
|
76
|
+
- `roboreward` — general-purpose robot reward prediction
|
|
77
|
+
|
|
78
|
+
## Documentation
|
|
79
|
+
|
|
80
|
+
Full documentation and additional input formats are available at [sublinear.dev/docs](https://sublinear.dev/docs).
|
|
81
|
+
|
|
82
|
+
## Package layout
|
|
83
|
+
|
|
84
|
+
This project uses a `src` layout:
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
sublinear/
|
|
88
|
+
├── pyproject.toml
|
|
89
|
+
├── README.md
|
|
90
|
+
└── src/
|
|
91
|
+
└── sublinear/
|
|
92
|
+
├── __init__.py
|
|
93
|
+
└── core.py
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Your `src/sublinear/__init__.py` should expose the public client:
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
from .core import Sublinear
|
|
100
|
+
__all__ = ["Sublinear"]
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Development
|
|
104
|
+
|
|
105
|
+
Install the package locally in editable mode:
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
python -m pip install -e .
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Build the source distribution and wheel:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
python -m pip install --upgrade build
|
|
115
|
+
python -m build
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Before publishing, verify the package metadata and archives:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
python -m pip install --upgrade twine
|
|
122
|
+
python -m twine check dist/*
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Publish to PyPI:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
python -m twine upload dist/*
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## License
|
|
132
|
+
|
|
133
|
+
MIT
|
|
134
|
+
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# Sublinear Python SDK
|
|
2
|
+
|
|
3
|
+
The official Python client for submitting robot videos and trajectories to Sublinear foundation reward models.
|
|
4
|
+
|
|
5
|
+
## Installation
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
pip install sublinear
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
The PyPI distribution is named `sublinear`, while the Python import package is named `sublinear`:
|
|
12
|
+
|
|
13
|
+
```python
|
|
14
|
+
from sublinear import Sublinear
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
Create an API key from the Sublinear dashboard, then submit a local MP4 video:
|
|
20
|
+
|
|
21
|
+
```python
|
|
22
|
+
import os
|
|
23
|
+
from sublinear import Sublinear
|
|
24
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"])
|
|
25
|
+
response = client.responses.create(model="sole-r1", task_description="Fold the clothing.", video_path="episode.mp4")
|
|
26
|
+
print(response.rewards)
|
|
27
|
+
print(response.success)
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
You can also provide RGB video frames directly as a list or NumPy array with shape `(frames, height, width, 3)`:
|
|
31
|
+
|
|
32
|
+
```python
|
|
33
|
+
import os
|
|
34
|
+
import numpy as np
|
|
35
|
+
from sublinear import Sublinear
|
|
36
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"])
|
|
37
|
+
video_frames = np.load("video_frames.npy")
|
|
38
|
+
response = client.responses.create(model="robometer", task_description="Fold the clothing.", video_frames=video_frames)
|
|
39
|
+
print(response.rewards)
|
|
40
|
+
print(response.success)
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Multiple videos
|
|
44
|
+
|
|
45
|
+
Pass `video_paths` to process multiple videos. The SDK submits jobs concurrently, up to `max_parallel_jobs` at a time.
|
|
46
|
+
|
|
47
|
+
```python
|
|
48
|
+
import os
|
|
49
|
+
from sublinear import Sublinear
|
|
50
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"], max_parallel_jobs=8)
|
|
51
|
+
response = client.responses.create(model="sole-r1", task_description="Pick up the cube.", video_paths=["episode_1.mp4", "episode_2.mp4"])
|
|
52
|
+
print(response.rewards)
|
|
53
|
+
print(response.success)
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Available models
|
|
57
|
+
|
|
58
|
+
- `sole-r1` — reasoning-based robot trajectory evaluation
|
|
59
|
+
- `robometer` — dense task-progress reward estimation
|
|
60
|
+
- `topreward` — visual reward estimation across robot tasks
|
|
61
|
+
- `roboreward` — general-purpose robot reward prediction
|
|
62
|
+
|
|
63
|
+
## Documentation
|
|
64
|
+
|
|
65
|
+
Full documentation and additional input formats are available at [sublinear.dev/docs](https://sublinear.dev/docs).
|
|
66
|
+
|
|
67
|
+
## Package layout
|
|
68
|
+
|
|
69
|
+
This project uses a `src` layout:
|
|
70
|
+
|
|
71
|
+
```text
|
|
72
|
+
sublinear/
|
|
73
|
+
├── pyproject.toml
|
|
74
|
+
├── README.md
|
|
75
|
+
└── src/
|
|
76
|
+
└── sublinear/
|
|
77
|
+
├── __init__.py
|
|
78
|
+
└── core.py
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Your `src/sublinear/__init__.py` should expose the public client:
|
|
82
|
+
|
|
83
|
+
```python
|
|
84
|
+
from .core import Sublinear
|
|
85
|
+
__all__ = ["Sublinear"]
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Development
|
|
89
|
+
|
|
90
|
+
Install the package locally in editable mode:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
python -m pip install -e .
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Build the source distribution and wheel:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
python -m pip install --upgrade build
|
|
100
|
+
python -m build
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Before publishing, verify the package metadata and archives:
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
python -m pip install --upgrade twine
|
|
107
|
+
python -m twine check dist/*
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
Publish to PyPI:
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
python -m twine upload dist/*
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## License
|
|
117
|
+
|
|
118
|
+
MIT
|
|
119
|
+
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=77", "wheel"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "sublinear"
|
|
7
|
+
version = "0.0.0.1"
|
|
8
|
+
description = "Python SDK for Sublinear foundation reward models for robotics."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.10"
|
|
11
|
+
license = "MIT"
|
|
12
|
+
authors = [
|
|
13
|
+
{ name = "Sublinear" },
|
|
14
|
+
]
|
|
15
|
+
keywords = [
|
|
16
|
+
"robotics",
|
|
17
|
+
"reinforcement-learning",
|
|
18
|
+
"reward-models",
|
|
19
|
+
"video-language-models",
|
|
20
|
+
"embodied-ai",
|
|
21
|
+
]
|
|
22
|
+
dependencies = [
|
|
23
|
+
"numpy>=1.24",
|
|
24
|
+
"opencv-python-headless>=4.8",
|
|
25
|
+
"requests>=2.31",
|
|
26
|
+
]
|
|
27
|
+
|
|
28
|
+
[project.urls]
|
|
29
|
+
Homepage = "https://sublinear.dev"
|
|
30
|
+
Documentation = "https://sublinear.dev/docs"
|
|
31
|
+
|
|
32
|
+
[tool.setuptools.packages.find]
|
|
33
|
+
where = ["src"]
|
|
34
|
+
include = ["sublinear*"]
|
|
35
|
+
|
|
@@ -0,0 +1,402 @@
|
|
|
1
|
+
import mimetypes
|
|
2
|
+
import time
|
|
3
|
+
from pathlib import Path
|
|
4
|
+
from types import SimpleNamespace
|
|
5
|
+
from typing import Any, BinaryIO
|
|
6
|
+
import os
|
|
7
|
+
|
|
8
|
+
import requests
|
|
9
|
+
import numpy as np
|
|
10
|
+
|
|
11
|
+
import tempfile
|
|
12
|
+
from concurrent.futures import ThreadPoolExecutor
|
|
13
|
+
|
|
14
|
+
import cv2
|
|
15
|
+
from itertools import repeat
|
|
16
|
+
|
|
17
|
+
import threading
|
|
18
|
+
from collections.abc import Callable
|
|
19
|
+
|
|
20
|
+
SUPABASE_FUNCTIONS_URL = (
|
|
21
|
+
"https://qttoxuyjvehacmihvkzu.supabase.co/functions/v1"
|
|
22
|
+
)
|
|
23
|
+
|
|
24
|
+
VideoData = bytes | bytearray | memoryview | BinaryIO
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
def load_video_frames(video_path):
|
|
29
|
+
import cv2
|
|
30
|
+
frames = []
|
|
31
|
+
cap = cv2.VideoCapture(video_path)
|
|
32
|
+
if not cap.isOpened():
|
|
33
|
+
raise ValueError("Error opening video file")
|
|
34
|
+
while True:
|
|
35
|
+
ret, frame = cap.read()
|
|
36
|
+
if not ret:
|
|
37
|
+
break
|
|
38
|
+
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
|
39
|
+
frames.append(frame)
|
|
40
|
+
cap.release()
|
|
41
|
+
return frames
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
def get_video_dimensions(
|
|
45
|
+
video_path: str | Path,
|
|
46
|
+
) -> tuple[int, int]:
|
|
47
|
+
capture = cv2.VideoCapture(str(video_path))
|
|
48
|
+
try:
|
|
49
|
+
if not capture.isOpened():
|
|
50
|
+
raise ValueError(f"Unable to inspect video: {video_path}")
|
|
51
|
+
width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
|
|
52
|
+
height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
|
|
53
|
+
finally:
|
|
54
|
+
capture.release()
|
|
55
|
+
if width < 1 or height < 1:
|
|
56
|
+
raise ValueError(f"Invalid video dimensions: {video_path}")
|
|
57
|
+
return width, height
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
|
|
62
|
+
def get_final_video_frames(
|
|
63
|
+
model: str,
|
|
64
|
+
video_paths: list = None,
|
|
65
|
+
video_path: str = None,
|
|
66
|
+
###### optional, in case you want to give two videos with different views instead of one video that has a single view or has combined side-by-side views
|
|
67
|
+
video_paths_external_view: list = None,
|
|
68
|
+
video_paths_wrist_view: list = None,
|
|
69
|
+
video_path_external_view: str = None,
|
|
70
|
+
video_path_wrist_view: str = None,
|
|
71
|
+
######
|
|
72
|
+
###### optional, in case you want to give frames instead of video paths
|
|
73
|
+
video_frames: list | None = None,
|
|
74
|
+
video_frames_external_view: list | None = None,
|
|
75
|
+
video_frames_wrist_view: list | None = None,
|
|
76
|
+
###### optional
|
|
77
|
+
view_type_per_video: list = None,
|
|
78
|
+
view_type: str = None,
|
|
79
|
+
verbose: bool = True,
|
|
80
|
+
):
|
|
81
|
+
final_video_paths=None
|
|
82
|
+
final_video_paths_external_view=None
|
|
83
|
+
final_video_paths_wrist_view=None
|
|
84
|
+
videos=None
|
|
85
|
+
single_video = False
|
|
86
|
+
if video_frames is not None:
|
|
87
|
+
# if video_frames[0] is a list of frames, then we have multiple videos worth of frames. if video_frames is a single list of frames, then we have one video worth of frames.
|
|
88
|
+
if isinstance(video_frames[0], list) or (isinstance(video_frames[0], np.ndarray) and video_frames[0].ndim == 4):
|
|
89
|
+
single_video = False
|
|
90
|
+
else:
|
|
91
|
+
single_video = True
|
|
92
|
+
#
|
|
93
|
+
if view_type_per_video is None and view_type is not None:
|
|
94
|
+
if single_video:
|
|
95
|
+
view_type_per_video = [view_type]
|
|
96
|
+
else:
|
|
97
|
+
view_type_per_video = [view_type] * len(video_frames)
|
|
98
|
+
#
|
|
99
|
+
elif video_frames_external_view is not None or video_frames_wrist_view is not None:
|
|
100
|
+
if video_frames_external_view is not None:
|
|
101
|
+
if isinstance(video_frames_external_view[0], list) or (isinstance(video_frames_external_view[0], np.ndarray) and video_frames_external_view[0].ndim == 4):
|
|
102
|
+
single_video = False
|
|
103
|
+
else:
|
|
104
|
+
single_video = True
|
|
105
|
+
if video_frames_wrist_view is not None:
|
|
106
|
+
if isinstance(video_frames_wrist_view[0], list) or (isinstance(video_frames_wrist_view[0], np.ndarray) and video_frames_wrist_view[0].ndim == 4):
|
|
107
|
+
single_video = False
|
|
108
|
+
else:
|
|
109
|
+
single_video = True
|
|
110
|
+
#
|
|
111
|
+
if view_type_per_video is None and view_type is not None:
|
|
112
|
+
if single_video:
|
|
113
|
+
view_type_per_video = [view_type]
|
|
114
|
+
else:
|
|
115
|
+
view_type_per_video = [view_type] * max([len(video_frames_external_view or []), len(video_frames_wrist_view or [])])
|
|
116
|
+
else:
|
|
117
|
+
if video_paths is not None:
|
|
118
|
+
final_video_paths = video_paths
|
|
119
|
+
elif video_paths is None and video_path is not None:
|
|
120
|
+
final_video_paths = [video_path]
|
|
121
|
+
single_video = True
|
|
122
|
+
#
|
|
123
|
+
if view_type_per_video is None and view_type is not None:
|
|
124
|
+
view_type_per_video = [view_type] * len(final_video_paths)
|
|
125
|
+
#
|
|
126
|
+
else:
|
|
127
|
+
if video_paths_external_view is None and video_path_external_view is not None:
|
|
128
|
+
final_video_paths_external_view = [video_path_external_view]
|
|
129
|
+
single_video = True
|
|
130
|
+
if video_paths_wrist_view is None and video_path_wrist_view is not None:
|
|
131
|
+
final_video_paths_wrist_view = [video_path_wrist_view]
|
|
132
|
+
single_video = True
|
|
133
|
+
#
|
|
134
|
+
if view_type_per_video is None and view_type is not None:
|
|
135
|
+
view_type_per_video = [view_type] * max([len(final_video_paths or []), len(final_video_paths_external_view or []), len(final_video_paths_wrist_view or [])])
|
|
136
|
+
#
|
|
137
|
+
############ EXTRACT VIDEO FRAMES FOR ALL VIDEOS AS A LIST OF LISTS
|
|
138
|
+
#
|
|
139
|
+
if video_frames is not None:
|
|
140
|
+
if single_video:
|
|
141
|
+
videos = [video_frames]
|
|
142
|
+
else:
|
|
143
|
+
videos = video_frames
|
|
144
|
+
elif video_frames_external_view is not None or video_frames_wrist_view is not None:
|
|
145
|
+
if video_frames_external_view is not None and video_frames_wrist_view is not None:
|
|
146
|
+
if single_video:
|
|
147
|
+
video_frames_external_view = [video_frames_external_view]
|
|
148
|
+
video_frames_wrist_view = [video_frames_wrist_view]
|
|
149
|
+
#
|
|
150
|
+
if len(video_frames_external_view) != len(video_frames_wrist_view):
|
|
151
|
+
raise ValueError(f"Number of videos in video_frames_external_view {len(video_frames_external_view)} does not match number of videos in video_frames_wrist_view {len(video_frames_wrist_view)}")
|
|
152
|
+
#
|
|
153
|
+
if model in ['sole-r1']:
|
|
154
|
+
videos = []
|
|
155
|
+
for video_idx in range(len(video_frames_external_view)):
|
|
156
|
+
frames_external = video_frames_external_view[video_idx]
|
|
157
|
+
frames_wrist = video_frames_wrist_view[video_idx]
|
|
158
|
+
if not len(frames_external) == len(frames_wrist):
|
|
159
|
+
raise ValueError(f"Number of frames in external view video {len(frames_external)} does not match number of frames in wrist view video {len(frames_wrist)} for video_idx {video_idx}")
|
|
160
|
+
frames_combined = [np.concatenate((frames_external[i], frames_wrist[i]), axis=1) for i in range(len(frames_external))]
|
|
161
|
+
videos.append(frames_combined)
|
|
162
|
+
view_type_per_video = ['external and wrist'] * len(videos)
|
|
163
|
+
else:
|
|
164
|
+
if video_frames_external_view is not None and video_frames_wrist_view is not None:
|
|
165
|
+
raise ValueError(f"Both video_frames_external_view and video_frames_wrist_view are provided, but model '{model}' does not support both views. Please provide only one view.")
|
|
166
|
+
else:
|
|
167
|
+
if video_frames_external_view is not None and video_frames_wrist_view is None:
|
|
168
|
+
if single_video:
|
|
169
|
+
video_frames_external_view = [video_frames_external_view]
|
|
170
|
+
videos = video_frames_external_view
|
|
171
|
+
view_type_per_video = ['external'] * len(video_frames_external_view)
|
|
172
|
+
if verbose:
|
|
173
|
+
print(f"Using videos from video_frames_external_view with view type 'external'")
|
|
174
|
+
elif video_frames_external_view is None and video_frames_wrist_view is not None:
|
|
175
|
+
if single_video:
|
|
176
|
+
video_frames_wrist_view = [video_frames_wrist_view]
|
|
177
|
+
videos = video_frames_wrist_view
|
|
178
|
+
view_type_per_video = ['wrist'] * len(video_frames_wrist_view)
|
|
179
|
+
if verbose:
|
|
180
|
+
print(f"Using videos from video_frames_wrist_view with view type 'wrist'")
|
|
181
|
+
else:
|
|
182
|
+
raise ValueError("video_frames cannot be None if video_paths is None")
|
|
183
|
+
else:
|
|
184
|
+
if final_video_paths is None and final_video_paths_external_view is not None and final_video_paths_wrist_view is not None:
|
|
185
|
+
# concatenate external and wrist view videos side by side and use that as input to the model
|
|
186
|
+
# if False:
|
|
187
|
+
# videos = []
|
|
188
|
+
# for video_idx in range(len(video_paths_external_view)):
|
|
189
|
+
# frames_external = load_video_frames(video_paths_external_view[video_idx])
|
|
190
|
+
# frames_wrist = load_video_frames(video_paths_wrist_view[video_idx])
|
|
191
|
+
# if not len(frames_external) == len(frames_wrist):
|
|
192
|
+
# raise ValueError(f"Number of frames in external view video {len(frames_external)} does not match number of frames in wrist view video {len(frames_wrist)} for video_idx {video_idx}")
|
|
193
|
+
# frames_combined = [np.concatenate((frames_external[i], frames_wrist[i]), axis=1) for i in range(len(frames_external))]
|
|
194
|
+
# videos.append(frames_combined)
|
|
195
|
+
if model in ['sole-r1']:
|
|
196
|
+
view_type_per_video = ['external and wrist'] * len(final_video_paths_external_view)
|
|
197
|
+
else:
|
|
198
|
+
raise ValueError(f"Both video_paths_external_view and video_paths_wrist_view are provided, but model '{model}' does not support both views. Please provide only one view.")
|
|
199
|
+
else:
|
|
200
|
+
if final_video_paths is None and final_video_paths_external_view is not None and final_video_paths_wrist_view is None:
|
|
201
|
+
final_video_paths = final_video_paths_external_view
|
|
202
|
+
final_video_paths_external_view = None
|
|
203
|
+
view_type_per_video = ['external'] * len(final_video_paths)
|
|
204
|
+
if verbose:
|
|
205
|
+
print(f"Using videos from video_paths_external_view with view type 'external'")
|
|
206
|
+
elif final_video_paths is None and final_video_paths_external_view is None and final_video_paths_wrist_view is not None:
|
|
207
|
+
final_video_paths = final_video_paths_wrist_view
|
|
208
|
+
final_video_paths_wrist_view = None
|
|
209
|
+
view_type_per_video = ['wrist'] * len(final_video_paths)
|
|
210
|
+
if verbose:
|
|
211
|
+
print(f"Using videos from video_paths_wrist_view with view type 'wrist'")
|
|
212
|
+
elif final_video_paths is None:
|
|
213
|
+
raise ValueError("video_paths cannot be None if video_paths_external_view and video_paths_wrist_view are both not provided")
|
|
214
|
+
#
|
|
215
|
+
# videos = []
|
|
216
|
+
# for video_path in video_paths:
|
|
217
|
+
# frames = load_video_frames(video_path)
|
|
218
|
+
# videos.append(frames)
|
|
219
|
+
#
|
|
220
|
+
if not videos and not final_video_paths and not final_video_paths_external_view and not final_video_paths_wrist_view:
|
|
221
|
+
raise ValueError("No videos were provided.")
|
|
222
|
+
elif not videos is None:
|
|
223
|
+
video_count = len(videos)
|
|
224
|
+
elif not final_video_paths is None:
|
|
225
|
+
video_count = len(final_video_paths)
|
|
226
|
+
elif not final_video_paths_external_view is None:
|
|
227
|
+
video_count = len(final_video_paths_external_view)
|
|
228
|
+
elif not final_video_paths_wrist_view is None:
|
|
229
|
+
video_count = len(final_video_paths_wrist_view)
|
|
230
|
+
#
|
|
231
|
+
if view_type_per_video is None and not videos is None:
|
|
232
|
+
view_type_per_video = ['external'] * len(videos)
|
|
233
|
+
elif view_type_per_video is None and final_video_paths is not None:
|
|
234
|
+
view_type_per_video = ['external'] * len(final_video_paths)
|
|
235
|
+
elif view_type_per_video is None and final_video_paths_external_view is not None:
|
|
236
|
+
view_type_per_video = ['external'] * len(final_video_paths_external_view)
|
|
237
|
+
elif view_type_per_video is None and final_video_paths_wrist_view is not None:
|
|
238
|
+
view_type_per_video = ['wrist'] * len(final_video_paths_wrist_view)
|
|
239
|
+
#
|
|
240
|
+
elif len(view_type_per_video) != video_count:
|
|
241
|
+
if len(view_type_per_video) < video_count and len(set(view_type_per_video)) == 1:
|
|
242
|
+
if verbose:
|
|
243
|
+
print(f"Extending view_type_per_video (length {len(view_type_per_video)}) to match number of videos (length {video_count})")
|
|
244
|
+
view_type_per_video = view_type_per_video * video_count
|
|
245
|
+
else:
|
|
246
|
+
raise ValueError(f"Length of view_type_per_video ({len(view_type_per_video)}) does not match number of videos ({video_count})")
|
|
247
|
+
#
|
|
248
|
+
return videos, final_video_paths, final_video_paths_external_view, final_video_paths_wrist_view, view_type_per_video, single_video, video_count
|
|
249
|
+
|
|
250
|
+
def video_plot(*args, **kwargs):
|
|
251
|
+
print('Thank you for trying out Sublinear! We are still in development and will be launching the API soon.')
|
|
252
|
+
|
|
253
|
+
class Sublinear:
|
|
254
|
+
def __init__(
|
|
255
|
+
self,
|
|
256
|
+
api_key: str,
|
|
257
|
+
base_url: str = SUPABASE_FUNCTIONS_URL,
|
|
258
|
+
max_parallel_jobs: int = 8,
|
|
259
|
+
):
|
|
260
|
+
if not api_key:
|
|
261
|
+
raise ValueError("api_key cannot be empty")
|
|
262
|
+
if max_parallel_jobs < 1:
|
|
263
|
+
raise ValueError("max_parallel_jobs must be at least 1")
|
|
264
|
+
self.base_url = base_url.rstrip("/")
|
|
265
|
+
self.max_parallel_jobs = max_parallel_jobs
|
|
266
|
+
self.session = requests.Session()
|
|
267
|
+
self.session.headers.update({
|
|
268
|
+
"Authorization": f"Bearer {api_key}",
|
|
269
|
+
})
|
|
270
|
+
self.responses = self
|
|
271
|
+
#
|
|
272
|
+
def cancel_job(self, job_id: str) -> None:
|
|
273
|
+
response = self.session.post(
|
|
274
|
+
f"{self.base_url}/create-inference-job",
|
|
275
|
+
json={
|
|
276
|
+
"action": "cancel",
|
|
277
|
+
"job_id": job_id,
|
|
278
|
+
},
|
|
279
|
+
timeout=10,
|
|
280
|
+
)
|
|
281
|
+
# Cancellation losing a race with completion is not retriable.
|
|
282
|
+
if response.status_code != 409:
|
|
283
|
+
response.raise_for_status()
|
|
284
|
+
#
|
|
285
|
+
def create_job(
|
|
286
|
+
self,
|
|
287
|
+
input_json: dict[str, Any],
|
|
288
|
+
*,
|
|
289
|
+
video: VideoData,
|
|
290
|
+
filename: str,
|
|
291
|
+
model: str,
|
|
292
|
+
on_job_created: Callable[[str], None] | None = None,
|
|
293
|
+
) -> dict[str, Any]:
|
|
294
|
+
downsample_to = input_json.get("downsample_to",10)
|
|
295
|
+
if (
|
|
296
|
+
isinstance(downsample_to, bool)
|
|
297
|
+
or not isinstance(downsample_to, int)
|
|
298
|
+
or downsample_to < 1
|
|
299
|
+
or downsample_to > 100
|
|
300
|
+
):
|
|
301
|
+
raise ValueError("downsample_to must be an integer between 1 and 100")
|
|
302
|
+
content_type = (
|
|
303
|
+
mimetypes.guess_type(filename)[0]
|
|
304
|
+
or "application/octet-stream"
|
|
305
|
+
)
|
|
306
|
+
response = self.session.post(
|
|
307
|
+
f"{self.base_url}/create-inference-job",
|
|
308
|
+
json={
|
|
309
|
+
"action": "create",
|
|
310
|
+
"filename": filename,
|
|
311
|
+
"input_json": input_json,
|
|
312
|
+
"content_type": content_type,
|
|
313
|
+
"model": model,
|
|
314
|
+
},
|
|
315
|
+
timeout=30,
|
|
316
|
+
)
|
|
317
|
+
response.raise_for_status()
|
|
318
|
+
job = response.json()
|
|
319
|
+
if on_job_created is not None:
|
|
320
|
+
on_job_created(job["job_id"])
|
|
321
|
+
# Do not use self.session for the signed upload URL.
|
|
322
|
+
# The storage host should not receive the API key.
|
|
323
|
+
upload_response = requests.put(
|
|
324
|
+
job["upload_url"],
|
|
325
|
+
data=video,
|
|
326
|
+
headers={
|
|
327
|
+
"Content-Type": content_type,
|
|
328
|
+
},
|
|
329
|
+
timeout=(30, 1800),
|
|
330
|
+
)
|
|
331
|
+
upload_response.raise_for_status()
|
|
332
|
+
start_response = self.session.post(
|
|
333
|
+
f"{self.base_url}/create-inference-job",
|
|
334
|
+
json={
|
|
335
|
+
"action": "start",
|
|
336
|
+
"job_id": job["job_id"],
|
|
337
|
+
},
|
|
338
|
+
timeout=60,
|
|
339
|
+
)
|
|
340
|
+
start_response.raise_for_status()
|
|
341
|
+
return start_response.json()
|
|
342
|
+
#
|
|
343
|
+
def get_job(
|
|
344
|
+
self,
|
|
345
|
+
job_id: str,
|
|
346
|
+
) -> dict[str, Any]:
|
|
347
|
+
response = self.session.get(
|
|
348
|
+
f"{self.base_url}/get-inference-job",
|
|
349
|
+
params={
|
|
350
|
+
"job_id": job_id,
|
|
351
|
+
},
|
|
352
|
+
timeout=30,
|
|
353
|
+
)
|
|
354
|
+
response.raise_for_status()
|
|
355
|
+
return response.json()
|
|
356
|
+
#
|
|
357
|
+
def wait_for_job(
|
|
358
|
+
self,
|
|
359
|
+
job_id: str,
|
|
360
|
+
poll_interval: float = 2.0,
|
|
361
|
+
timeout: float = 6000,
|
|
362
|
+
) -> dict[str, Any]:
|
|
363
|
+
deadline = time.monotonic() + timeout
|
|
364
|
+
while True:
|
|
365
|
+
job = self.get_job(job_id)
|
|
366
|
+
status = job["status"]
|
|
367
|
+
if status == "completed":
|
|
368
|
+
return job
|
|
369
|
+
if status == "failed":
|
|
370
|
+
raise RuntimeError(
|
|
371
|
+
job.get("error")
|
|
372
|
+
or "Inference job failed."
|
|
373
|
+
)
|
|
374
|
+
if status == "canceled":
|
|
375
|
+
raise RuntimeError("Inference job was canceled.")
|
|
376
|
+
if time.monotonic() >= deadline:
|
|
377
|
+
raise TimeoutError(
|
|
378
|
+
f"Job did not finish within {timeout} seconds."
|
|
379
|
+
)
|
|
380
|
+
time.sleep(poll_interval)
|
|
381
|
+
#
|
|
382
|
+
def create(
|
|
383
|
+
self,
|
|
384
|
+
*,
|
|
385
|
+
model: str,
|
|
386
|
+
video_paths: list | None = None,
|
|
387
|
+
video_path: str | None = None,
|
|
388
|
+
video_paths_external_view: list | None = None,
|
|
389
|
+
video_paths_wrist_view: list | None = None,
|
|
390
|
+
video_path_external_view: str | None = None,
|
|
391
|
+
video_path_wrist_view: str | None = None,
|
|
392
|
+
video_frames: list | np.ndarray | None = None,
|
|
393
|
+
video_frames_external_view: list | np.ndarray | None = None,
|
|
394
|
+
video_frames_wrist_view: list | np.ndarray | None = None,
|
|
395
|
+
view_type_per_video: list | None = None,
|
|
396
|
+
view_type: str | None = None,
|
|
397
|
+
verbose: bool = True,
|
|
398
|
+
**input_json: Any,
|
|
399
|
+
) -> SimpleNamespace | list[SimpleNamespace]:
|
|
400
|
+
print('Thank you for trying out Sublinear! We are still in development and will be launching the API soon.')
|
|
401
|
+
return
|
|
402
|
+
|
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: sublinear
|
|
3
|
+
Version: 0.0.0.1
|
|
4
|
+
Summary: Python SDK for Sublinear foundation reward models for robotics.
|
|
5
|
+
Author: Sublinear
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://sublinear.dev
|
|
8
|
+
Project-URL: Documentation, https://sublinear.dev/docs
|
|
9
|
+
Keywords: robotics,reinforcement-learning,reward-models,video-language-models,embodied-ai
|
|
10
|
+
Requires-Python: >=3.10
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
Requires-Dist: numpy>=1.24
|
|
13
|
+
Requires-Dist: opencv-python-headless>=4.8
|
|
14
|
+
Requires-Dist: requests>=2.31
|
|
15
|
+
|
|
16
|
+
# Sublinear Python SDK
|
|
17
|
+
|
|
18
|
+
The official Python client for submitting robot videos and trajectories to Sublinear foundation reward models.
|
|
19
|
+
|
|
20
|
+
## Installation
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
pip install sublinear
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
The PyPI distribution is named `sublinear`, while the Python import package is named `sublinear`:
|
|
27
|
+
|
|
28
|
+
```python
|
|
29
|
+
from sublinear import Sublinear
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Quick start
|
|
33
|
+
|
|
34
|
+
Create an API key from the Sublinear dashboard, then submit a local MP4 video:
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
import os
|
|
38
|
+
from sublinear import Sublinear
|
|
39
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"])
|
|
40
|
+
response = client.responses.create(model="sole-r1", task_description="Fold the clothing.", video_path="episode.mp4")
|
|
41
|
+
print(response.rewards)
|
|
42
|
+
print(response.success)
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
You can also provide RGB video frames directly as a list or NumPy array with shape `(frames, height, width, 3)`:
|
|
46
|
+
|
|
47
|
+
```python
|
|
48
|
+
import os
|
|
49
|
+
import numpy as np
|
|
50
|
+
from sublinear import Sublinear
|
|
51
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"])
|
|
52
|
+
video_frames = np.load("video_frames.npy")
|
|
53
|
+
response = client.responses.create(model="robometer", task_description="Fold the clothing.", video_frames=video_frames)
|
|
54
|
+
print(response.rewards)
|
|
55
|
+
print(response.success)
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Multiple videos
|
|
59
|
+
|
|
60
|
+
Pass `video_paths` to process multiple videos. The SDK submits jobs concurrently, up to `max_parallel_jobs` at a time.
|
|
61
|
+
|
|
62
|
+
```python
|
|
63
|
+
import os
|
|
64
|
+
from sublinear import Sublinear
|
|
65
|
+
client = Sublinear(api_key=os.environ["SUBLINEAR_API_KEY"], max_parallel_jobs=8)
|
|
66
|
+
response = client.responses.create(model="sole-r1", task_description="Pick up the cube.", video_paths=["episode_1.mp4", "episode_2.mp4"])
|
|
67
|
+
print(response.rewards)
|
|
68
|
+
print(response.success)
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Available models
|
|
72
|
+
|
|
73
|
+
- `sole-r1` — reasoning-based robot trajectory evaluation
|
|
74
|
+
- `robometer` — dense task-progress reward estimation
|
|
75
|
+
- `topreward` — visual reward estimation across robot tasks
|
|
76
|
+
- `roboreward` — general-purpose robot reward prediction
|
|
77
|
+
|
|
78
|
+
## Documentation
|
|
79
|
+
|
|
80
|
+
Full documentation and additional input formats are available at [sublinear.dev/docs](https://sublinear.dev/docs).
|
|
81
|
+
|
|
82
|
+
## Package layout
|
|
83
|
+
|
|
84
|
+
This project uses a `src` layout:
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
sublinear/
|
|
88
|
+
├── pyproject.toml
|
|
89
|
+
├── README.md
|
|
90
|
+
└── src/
|
|
91
|
+
└── sublinear/
|
|
92
|
+
├── __init__.py
|
|
93
|
+
└── core.py
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
Your `src/sublinear/__init__.py` should expose the public client:
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
from .core import Sublinear
|
|
100
|
+
__all__ = ["Sublinear"]
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Development
|
|
104
|
+
|
|
105
|
+
Install the package locally in editable mode:
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
python -m pip install -e .
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Build the source distribution and wheel:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
python -m pip install --upgrade build
|
|
115
|
+
python -m build
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Before publishing, verify the package metadata and archives:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
python -m pip install --upgrade twine
|
|
122
|
+
python -m twine check dist/*
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Publish to PyPI:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
python -m twine upload dist/*
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## License
|
|
132
|
+
|
|
133
|
+
MIT
|
|
134
|
+
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
README.md
|
|
2
|
+
pyproject.toml
|
|
3
|
+
src/sublinear/__init__.py
|
|
4
|
+
src/sublinear/core.py
|
|
5
|
+
src/sublinear.egg-info/PKG-INFO
|
|
6
|
+
src/sublinear.egg-info/SOURCES.txt
|
|
7
|
+
src/sublinear.egg-info/dependency_links.txt
|
|
8
|
+
src/sublinear.egg-info/requires.txt
|
|
9
|
+
src/sublinear.egg-info/top_level.txt
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
sublinear
|