onsense 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- onsense-0.1.0/.gitignore +10 -0
- onsense-0.1.0/LICENSE +21 -0
- onsense-0.1.0/PKG-INFO +222 -0
- onsense-0.1.0/PROTOCOL.md +103 -0
- onsense-0.1.0/README.md +193 -0
- onsense-0.1.0/THIRD_PARTY_NOTICES.md +54 -0
- onsense-0.1.0/onsense/__init__.py +14 -0
- onsense-0.1.0/onsense/__main__.py +7 -0
- onsense-0.1.0/onsense/auth.py +82 -0
- onsense-0.1.0/onsense/cli.py +87 -0
- onsense-0.1.0/onsense/clip.py +506 -0
- onsense-0.1.0/onsense/crypto.py +66 -0
- onsense-0.1.0/onsense/doctor.py +188 -0
- onsense-0.1.0/onsense/pair.py +153 -0
- onsense-0.1.0/onsense/server.py +165 -0
- onsense-0.1.0/pyproject.toml +46 -0
- onsense-0.1.0/tests/test_auth.py +41 -0
- onsense-0.1.0/tests/test_crypto.py +55 -0
- onsense-0.1.0/uv.lock +1126 -0
onsense-0.1.0/.gitignore
ADDED
onsense-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 shdev
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
onsense-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,222 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: onsense
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Turn your Android phone into a camera and sensor source for PC AI agents (onSense MCP broker)
|
|
5
|
+
Author: shdev
|
|
6
|
+
License: MIT
|
|
7
|
+
License-File: LICENSE
|
|
8
|
+
Keywords: ai-agent,android,camera,claude,mcp
|
|
9
|
+
Classifier: Environment :: Console
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
12
|
+
Classifier: Operating System :: OS Independent
|
|
13
|
+
Classifier: Programming Language :: Python :: 3
|
|
14
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Topic :: Multimedia :: Video :: Capture
|
|
18
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
19
|
+
Classifier: Topic :: Utilities
|
|
20
|
+
Requires-Python: >=3.10
|
|
21
|
+
Requires-Dist: cryptography>=42
|
|
22
|
+
Requires-Dist: httpx>=0.27
|
|
23
|
+
Requires-Dist: mcp>=1.2.0
|
|
24
|
+
Requires-Dist: qrcode>=7.4
|
|
25
|
+
Requires-Dist: zeroconf>=0.131
|
|
26
|
+
Provides-Extra: test
|
|
27
|
+
Requires-Dist: pytest>=7; extra == 'test'
|
|
28
|
+
Description-Content-Type: text/markdown
|
|
29
|
+
|
|
30
|
+
# onsense — Phone Camera & Sensors for PC AI Agents
|
|
31
|
+
|
|
32
|
+
**onsense** is the PC-side MCP broker for [onSense](https://play.google.com/store/apps/details?id=com.shdev.onsense).
|
|
33
|
+
Install the Android app on your phone, run `uvx onsense pair` on your PC, and your AI client (Claude, Cursor, or any MCP-compatible tool) can see through your phone's camera and read its sensors — all over local Wi-Fi, with no cloud relay.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Architecture
|
|
38
|
+
|
|
39
|
+
```text
|
|
40
|
+
Android phone (onSense app) PC (this package) AI client
|
|
41
|
+
HTTP provider :8080 ←→ stdio MCP broker Claude / Cursor / …
|
|
42
|
+
camera frames, photos, onsense serve natural-language or
|
|
43
|
+
sensors, QR pairing onsense clip :8770 /eye tool calls
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
- The **phone** runs an HTTP provider on port 8080. It exposes camera frames, recent photos, and sensor readings, all gated by a pairing token.
|
|
47
|
+
- The **PC package** (`onsense`) is a stdio MCP broker. It translates MCP tool calls from your AI client into HTTP requests to the phone.
|
|
48
|
+
- Discovery uses mDNS (`_onsense._tcp.local.`). If the phone's IP changes, the broker rediscovers it automatically — no manual reconfiguration needed.
|
|
49
|
+
- An optional **clip bridge** daemon on port 8770 lets the phone push camera frames, files, and clipboard content to the PC (and, if enabled, lets the phone pull from the PC clipboard).
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Requirements
|
|
54
|
+
|
|
55
|
+
- Python 3.10+ and [uv](https://docs.astral.sh/uv/) (provides `uvx`)
|
|
56
|
+
- The onSense Android app installed on a phone on the **same local Wi-Fi network** as your PC
|
|
57
|
+
- An MCP-compatible AI client such as [Claude Code](https://claude.ai/code) or Cursor
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Quick Start
|
|
62
|
+
|
|
63
|
+
### 1. Pair your phone
|
|
64
|
+
|
|
65
|
+
Run this once. The PC prints a QR code; scan it from the onSense app ("Scan PC QR"). The phone pushes its address and token to the PC, which registers the MCP server automatically.
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
uvx onsense pair
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
After pairing, restart your AI client once so it picks up the new `onsense` MCP server.
|
|
72
|
+
|
|
73
|
+
### 2. Use it
|
|
74
|
+
|
|
75
|
+
Ask your AI client naturally:
|
|
76
|
+
|
|
77
|
+
> "Take a photo of what's in front of my phone."
|
|
78
|
+
> "What are the current sensor readings?"
|
|
79
|
+
> "Show me the last 5 photos on my phone."
|
|
80
|
+
|
|
81
|
+
The MCP tools are called automatically. You can also invoke them directly as `/eye` or whichever slash command your client supports.
|
|
82
|
+
|
|
83
|
+
### 3. Diagnose problems
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
uvx onsense doctor # checks Python, uv, MCP, Claude registration, mDNS, phone reachability
|
|
87
|
+
uvx onsense doctor --base http://192.168.1.5:8080 --token YOUR_PAIRING_TOKEN
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## Subcommands
|
|
93
|
+
|
|
94
|
+
| Command | What it does |
|
|
95
|
+
| --- | --- |
|
|
96
|
+
| `uvx onsense pair` | Display a QR code on the PC; phone scans it and pushes `{base, token}`; registers MCP automatically |
|
|
97
|
+
| `uvx onsense pair "onsense://pair?base=...&token=..."` | Parse a pairing URI directly (phone-displayed QR → manual copy) |
|
|
98
|
+
| `uvx onsense pair --img screenshot.png` | Decode a pairing QR from a screenshot file (requires opencv) |
|
|
99
|
+
| `uvx onsense serve` | Run the MCP server (stdio). Also starts the clip daemon automatically unless `--no-clip` is passed |
|
|
100
|
+
| `uvx onsense serve --no-clip` | Run the MCP server only, without starting the clip daemon |
|
|
101
|
+
| `uvx onsense clip` | Run the clip bridge daemon standalone (port 8770) |
|
|
102
|
+
| `uvx onsense clip --allow-pull` | Enable phone→pull: phone can GET /clip to retrieve the PC clipboard |
|
|
103
|
+
| `uvx onsense clip --set-clipboard` | Auto-inject received content into the PC OS clipboard (off by default) |
|
|
104
|
+
| `uvx onsense doctor` | Diagnose installation, connectivity, and phone reachability |
|
|
105
|
+
|
|
106
|
+
---
|
|
107
|
+
|
|
108
|
+
## Other MCP clients (Claude Desktop, Cursor, Codex)
|
|
109
|
+
|
|
110
|
+
`uvx onsense pair` auto-registers the server with **Claude Code** (via `claude mcp add`). The MCP server itself is standard stdio MCP, so any MCP-compatible client works once configured — only the auto-registration is Claude Code-specific. For other clients, run `uvx onsense pair` once to obtain your phone's `PHONE_BASE` / `PHONE_TOKEN` (also saved to `~/.onsense/pair.json`), then add the server manually:
|
|
111
|
+
|
|
112
|
+
**Claude Desktop / Cursor** — add to the client's MCP config (`claude_desktop_config.json` or `.cursor/mcp.json`):
|
|
113
|
+
|
|
114
|
+
```json
|
|
115
|
+
{
|
|
116
|
+
"mcpServers": {
|
|
117
|
+
"onsense": {
|
|
118
|
+
"command": "uvx",
|
|
119
|
+
"args": ["onsense", "serve"],
|
|
120
|
+
"env": { "PHONE_BASE": "http://<phone-ip>:8080", "PHONE_TOKEN": "<token>" }
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
**Codex** — add to `~/.codex/config.toml`:
|
|
127
|
+
|
|
128
|
+
```toml
|
|
129
|
+
[mcp_servers.onsense]
|
|
130
|
+
command = "uvx"
|
|
131
|
+
args = ["onsense", "serve"]
|
|
132
|
+
env = { PHONE_BASE = "http://<phone-ip>:8080", PHONE_TOKEN = "<token>" }
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Restart the client once after adding the server.
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## MCP Tools
|
|
140
|
+
|
|
141
|
+
These tools are exposed to your AI client after pairing:
|
|
142
|
+
|
|
143
|
+
| Tool | Description |
|
|
144
|
+
| --- | --- |
|
|
145
|
+
| `get_live_frame()` | Capture the current camera frame from the phone (returns JPEG image) |
|
|
146
|
+
| `read_sensors()` | Return phone sensor readings as JSON: battery level/charging state, ambient light (lux), accelerometer (x/y/z) |
|
|
147
|
+
| `recent_photos(limit=10)` | List recent photos on the phone: id, name, date_added, size, width, height |
|
|
148
|
+
| `get_photo(id, max_width=1024)` | Fetch a specific photo by id (downscaled to max_width); use ids from `recent_photos` |
|
|
149
|
+
|
|
150
|
+
If the phone's IP changes, the broker retries using mDNS autodiscovery before surfacing an error.
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## Clip Bridge (Phone ↔ PC File & Clipboard)
|
|
155
|
+
|
|
156
|
+
`onsense serve` automatically starts a clip daemon on **port 8770**. You can also run it standalone with `onsense clip`.
|
|
157
|
+
|
|
158
|
+
### Phone → PC push (POST /clip)
|
|
159
|
+
|
|
160
|
+
The onSense Android app can push content to the PC:
|
|
161
|
+
|
|
162
|
+
- **Images** are saved to disk as `latest.jpg` (in `ONSENSE_CLIP_DIR`, default `<tempdir>/onsense/`).
|
|
163
|
+
- **Text files** are saved to disk.
|
|
164
|
+
- **Other files** (video, PDF, etc.) are saved to disk by filename.
|
|
165
|
+
- If `--set-clipboard` is active, images and text are also injected into the PC OS clipboard so you can paste with Ctrl+V immediately.
|
|
166
|
+
|
|
167
|
+
By default, `--set-clipboard` is **off** — files are saved to disk but the clipboard is not touched.
|
|
168
|
+
|
|
169
|
+
### PC → Phone pull (GET /clip)
|
|
170
|
+
|
|
171
|
+
Off by default. Enable with:
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
uvx onsense clip --allow-pull
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
When enabled, the phone can GET /clip to retrieve the current PC clipboard content (copied files first, then images, then text). Returns 204 if the clipboard is empty.
|
|
178
|
+
|
|
179
|
+
### Ports and environment variables
|
|
180
|
+
|
|
181
|
+
| Variable | Default | Purpose |
|
|
182
|
+
| --- | --- | --- |
|
|
183
|
+
| `PHONE_BASE` | (from pairing) | Phone HTTP base URL, e.g. `http://192.168.1.5:8080` |
|
|
184
|
+
| `PHONE_TOKEN` | (from pairing) | Pairing auth token |
|
|
185
|
+
| `ONSENSE_CLIP_DIR` | `<tempdir>/onsense` | Directory where pushed files are saved |
|
|
186
|
+
| `ONSENSE_CLIP_MAX_MB` | `200` | Maximum incoming file size in MB (0 = unlimited) |
|
|
187
|
+
| `ONSENSE_CLIP_ALLOW_PULL` | `0` | Set to `1` to enable PC→phone pull without the CLI flag |
|
|
188
|
+
| `ONSENSE_CLIP_SET_CLIPBOARD` | `0` | Set to `1` to auto-inject into OS clipboard without the CLI flag |
|
|
189
|
+
| `ONSENSE_TEST_FRAME` | (unset) | Local JPEG path returned by `get_live_frame` when phone is unreachable |
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
193
|
+
## Security
|
|
194
|
+
|
|
195
|
+
**Local network only.** The PC-side services bind to all interfaces but reject connections from non-private IP addresses at the application layer. They are not intended to be exposed to the internet.
|
|
196
|
+
|
|
197
|
+
**HMAC request signing.** The pairing token is never sent in cleartext. Every authenticated request is signed with HMAC-SHA256 over `METHOD\npath\ntimestamp\nnonce\nalgorithm`, keyed by a signing key derived from the pairing token via HKDF-SHA256 (a key separate from the encryption key), and carries `X-Ts` / `X-Nonce` / `X-Sig` / `X-Enc` headers. Servers verify in constant time, reject timestamps outside a ±300 s window, and reject reused nonces — so a sniffed request cannot be replayed and the token cannot be stolen off the wire. New Android installs generate a 128-bit random token, stored on the phone and in `~/.onsense/pair.json` (chmod 600) after pairing. (The `/health` endpoint is unauthenticated.) See [PROTOCOL.md](PROTOCOL.md) for the exact wire format.
|
|
198
|
+
|
|
199
|
+
**Pull and clipboard injection are off by default.** `GET /clip` (phone pulls PC clipboard) and OS clipboard auto-injection (phone push → Ctrl+V) are disabled unless you explicitly pass `--allow-pull` or `--set-clipboard`.
|
|
200
|
+
|
|
201
|
+
**Encrypted bodies (AES-256-GCM).** Sensitive payloads — camera frames, photos, sensor data, files, and clipboard content — are encrypted end-to-end with AES-256-GCM. The key is derived from the pairing token via HKDF-SHA256 (a key separate from the signing key); each message uses a fresh 96-bit nonce, and the GCM tag authenticates the body and binds it to its request (so tampering or response substitution is rejected). A passive sniffer on the same Wi-Fi sees only ciphertext. Low-sensitivity metadata stays plaintext: the open `/version` and `/health` endpoints, HTTP error responses, and the saved-file path returned after a push. (Note: large transfers currently buffer the body in memory while encrypting/decrypting — chunked streaming for very large files is a future step. Transport-level TLS is also a possible future hardening.)
|
|
202
|
+
|
|
203
|
+
**File size cap.** Incoming pushes are rejected if they exceed `ONSENSE_CLIP_MAX_MB` (default 200 MB). Set to `0` to remove the cap.
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
207
|
+
## License
|
|
208
|
+
|
|
209
|
+
MIT. See [LICENSE](LICENSE).
|
|
210
|
+
|
|
211
|
+
---
|
|
212
|
+
|
|
213
|
+
## 한국어 요약
|
|
214
|
+
|
|
215
|
+
**onsense**는 Android 폰(onSense 앱)을 PC AI 에이전트의 카메라·센서로 연결하는 PC측 MCP 브로커입니다.
|
|
216
|
+
|
|
217
|
+
- `uvx onsense pair` — 폰과 한 번 페어링하면 Claude·Cursor에 MCP 서버가 자동 등록됩니다.
|
|
218
|
+
- `uvx onsense doctor` — 연결·설치 문제를 진단합니다.
|
|
219
|
+
- `uvx onsense serve` — AI 클라이언트가 호출하는 MCP 서버(stdio)를 실행합니다. 동시에 클립 브리지 데몬(:8770)도 자동 기동합니다.
|
|
220
|
+
- `uvx onsense clip` — 폰↔PC 파일·클립보드 브리지를 단독으로 실행합니다.
|
|
221
|
+
|
|
222
|
+
폰과 PC는 **같은 로컬 Wi-Fi**에 있어야 하며, 개발자 운영 클라우드 서버는 없습니다. 통신은 현재 로컬 HTTP(비암호화)이므로 신뢰할 수 있는 네트워크에서 사용하세요.
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# onsense — Wire Protocol & Security
|
|
2
|
+
|
|
3
|
+
This document specifies the on-the-wire protocol between the **phone** (Android app,
|
|
4
|
+
HTTP provider on :8080) and the **PC** (this package: MCP broker + clip daemon on :8770),
|
|
5
|
+
so the cryptography can be reviewed and reimplemented. The reference implementations are
|
|
6
|
+
`pc/onsense/auth.py` + `pc/onsense/crypto.py` (Python) and `Auth.kt` + `Crypto.kt` (Kotlin);
|
|
7
|
+
they are byte-for-byte compatible (verified by a shared test vector).
|
|
8
|
+
|
|
9
|
+
## 1. Pairing
|
|
10
|
+
|
|
11
|
+
The phone displays (or scans) a pairing URI `onsense://pair?base=<http://ip:8080>&token=<token>`.
|
|
12
|
+
The token is a 128-bit random value, base64url-encoded (no padding). It is exchanged **out of
|
|
13
|
+
band** (QR on screen / direct entry) and never transmitted over the network in cleartext.
|
|
14
|
+
|
|
15
|
+
The token string is the shared secret. **It is used as UTF-8 bytes as-is** (no base64 decode) as
|
|
16
|
+
input keying material — this keeps old and new token formats and both language implementations
|
|
17
|
+
consistent.
|
|
18
|
+
|
|
19
|
+
## 2. Key derivation (HKDF-SHA256)
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
ikm = token.encode("utf-8")
|
|
23
|
+
auth_key = HKDF-SHA256(ikm, salt="", info="onsense/auth/v1", length=32) # request signing
|
|
24
|
+
aead_key = HKDF-SHA256(ikm, salt="", info="onsense/aead/v1", length=32) # body encryption
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
HKDF is RFC 5869 with an empty salt (32 zero bytes). The two `info` strings give independent
|
|
28
|
+
keys (domain separation).
|
|
29
|
+
|
|
30
|
+
## 3. Request authentication (HMAC-SHA256)
|
|
31
|
+
|
|
32
|
+
Every authenticated request carries:
|
|
33
|
+
|
|
34
|
+
| Header | Value |
|
|
35
|
+
|---|---|
|
|
36
|
+
| `X-Ts` | unix seconds (decimal string) |
|
|
37
|
+
| `X-Nonce` | 16 random bytes, hex |
|
|
38
|
+
| `X-Enc` | body encryption algorithm, currently `aes256gcm/v1` |
|
|
39
|
+
| `X-Sig` | `hex(HMAC-SHA256(auth_key, canonical))` |
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
canonical = "{METHOD}\n{path}\n{ts}\n{nonce}\n{enc}"
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
`METHOD` is uppercase; `path` is the request path **without** query string. Including `X-Enc`
|
|
46
|
+
in the signed canonical prevents an attacker from stripping/downgrading encryption.
|
|
47
|
+
|
|
48
|
+
Verification (constant-time):
|
|
49
|
+
1. `X-Enc` must equal the supported algorithm, else reject (downgrade protection).
|
|
50
|
+
2. `|now - ts|` ≤ 300 s, else reject (clock-skew / replay window).
|
|
51
|
+
3. recompute the signature and compare with `hmac.compare_digest`.
|
|
52
|
+
4. `X-Nonce` must not have been seen within the window (in-memory replay cache).
|
|
53
|
+
|
|
54
|
+
The `/health` (clip daemon) and `/version` (phone) endpoints are unauthenticated.
|
|
55
|
+
|
|
56
|
+
## 4. Body encryption (AES-256-GCM)
|
|
57
|
+
|
|
58
|
+
Sensitive bodies — camera frames, photos, sensor JSON, files, clipboard — are encrypted.
|
|
59
|
+
The wire body is:
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
nonce(12) || ciphertext || tag(16)
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
`nonce` is 12 random bytes per message (**never reused** with a key). `tag` is the 128-bit
|
|
66
|
+
GCM tag. The AEAD additional data (AAD) binds the body to its request/response:
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
request body AAD = "req\n{METHOD}\n{path}\n{X-Ts}\n{X-Nonce}\n{X-Enc}"
|
|
70
|
+
response body AAD = "resp\n{METHOD}\n{path}\n{request X-Nonce}\n{X-Enc}"
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
A recipient **must not** use any decrypted bytes before the GCM tag verifies (decrypt to a
|
|
74
|
+
temporary buffer/file, verify, then commit). A tag mismatch (tampering, wrong key, or
|
|
75
|
+
mismatched AAD) is rejected.
|
|
76
|
+
|
|
77
|
+
### What is encrypted
|
|
78
|
+
|
|
79
|
+
- Phone → PC: camera (`/shot.jpg`), sensors (`/sensors.json`), photo list (`/photos`),
|
|
80
|
+
photo (`/photo`) **response bodies**; clip push (`POST /clip`) **request body**;
|
|
81
|
+
clip pull (`GET /clip`) **response body**.
|
|
82
|
+
- Plaintext (low-sensitivity): `/version`, `/health`, HTTP error responses, and the saved-file
|
|
83
|
+
path string returned after a push. HTTP headers (filename, content-type, sizes) are plaintext.
|
|
84
|
+
|
|
85
|
+
## 5. Threat model & limitations
|
|
86
|
+
|
|
87
|
+
Protects against, on a shared/untrusted LAN: token theft, request forgery, replay, passive
|
|
88
|
+
eavesdropping of sensitive payloads, and body/response tampering or substitution.
|
|
89
|
+
|
|
90
|
+
Does **not** (yet) provide: full transport confidentiality of metadata/headers; protection of
|
|
91
|
+
the low-sensitivity plaintext endpoints. Current limitations:
|
|
92
|
+
|
|
93
|
+
- **Single-GCM buffering**: each message is encrypted/decrypted as one unit, so very large
|
|
94
|
+
transfers buffer the body in memory. Chunked AEAD streaming for large files is a future step.
|
|
95
|
+
- TLS (transport-level encryption) is a possible future hardening.
|
|
96
|
+
|
|
97
|
+
Servers bind to all interfaces but reject non-private (non-RFC1918/loopback) client IPs at the
|
|
98
|
+
application layer; the daemon is intended for trusted private networks only.
|
|
99
|
+
|
|
100
|
+
## 6. Versioning
|
|
101
|
+
|
|
102
|
+
`X-Enc` carries an explicit `…/v1` tag so the algorithm can be rotated without ambiguity.
|
|
103
|
+
`info` strings in §2 are likewise versioned.
|
onsense-0.1.0/README.md
ADDED
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
# onsense — Phone Camera & Sensors for PC AI Agents
|
|
2
|
+
|
|
3
|
+
**onsense** is the PC-side MCP broker for [onSense](https://play.google.com/store/apps/details?id=com.shdev.onsense).
|
|
4
|
+
Install the Android app on your phone, run `uvx onsense pair` on your PC, and your AI client (Claude, Cursor, or any MCP-compatible tool) can see through your phone's camera and read its sensors — all over local Wi-Fi, with no cloud relay.
|
|
5
|
+
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Architecture
|
|
9
|
+
|
|
10
|
+
```text
|
|
11
|
+
Android phone (onSense app) PC (this package) AI client
|
|
12
|
+
HTTP provider :8080 ←→ stdio MCP broker Claude / Cursor / …
|
|
13
|
+
camera frames, photos, onsense serve natural-language or
|
|
14
|
+
sensors, QR pairing onsense clip :8770 /eye tool calls
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
- The **phone** runs an HTTP provider on port 8080. It exposes camera frames, recent photos, and sensor readings, all gated by a pairing token.
|
|
18
|
+
- The **PC package** (`onsense`) is a stdio MCP broker. It translates MCP tool calls from your AI client into HTTP requests to the phone.
|
|
19
|
+
- Discovery uses mDNS (`_onsense._tcp.local.`). If the phone's IP changes, the broker rediscovers it automatically — no manual reconfiguration needed.
|
|
20
|
+
- An optional **clip bridge** daemon on port 8770 lets the phone push camera frames, files, and clipboard content to the PC (and, if enabled, lets the phone pull from the PC clipboard).
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Requirements
|
|
25
|
+
|
|
26
|
+
- Python 3.10+ and [uv](https://docs.astral.sh/uv/) (provides `uvx`)
|
|
27
|
+
- The onSense Android app installed on a phone on the **same local Wi-Fi network** as your PC
|
|
28
|
+
- An MCP-compatible AI client such as [Claude Code](https://claude.ai/code) or Cursor
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Quick Start
|
|
33
|
+
|
|
34
|
+
### 1. Pair your phone
|
|
35
|
+
|
|
36
|
+
Run this once. The PC prints a QR code; scan it from the onSense app ("Scan PC QR"). The phone pushes its address and token to the PC, which registers the MCP server automatically.
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
uvx onsense pair
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
After pairing, restart your AI client once so it picks up the new `onsense` MCP server.
|
|
43
|
+
|
|
44
|
+
### 2. Use it
|
|
45
|
+
|
|
46
|
+
Ask your AI client naturally:
|
|
47
|
+
|
|
48
|
+
> "Take a photo of what's in front of my phone."
|
|
49
|
+
> "What are the current sensor readings?"
|
|
50
|
+
> "Show me the last 5 photos on my phone."
|
|
51
|
+
|
|
52
|
+
The MCP tools are called automatically. You can also invoke them directly as `/eye` or whichever slash command your client supports.
|
|
53
|
+
|
|
54
|
+
### 3. Diagnose problems
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
uvx onsense doctor # checks Python, uv, MCP, Claude registration, mDNS, phone reachability
|
|
58
|
+
uvx onsense doctor --base http://192.168.1.5:8080 --token YOUR_PAIRING_TOKEN
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## Subcommands
|
|
64
|
+
|
|
65
|
+
| Command | What it does |
|
|
66
|
+
| --- | --- |
|
|
67
|
+
| `uvx onsense pair` | Display a QR code on the PC; phone scans it and pushes `{base, token}`; registers MCP automatically |
|
|
68
|
+
| `uvx onsense pair "onsense://pair?base=...&token=..."` | Parse a pairing URI directly (phone-displayed QR → manual copy) |
|
|
69
|
+
| `uvx onsense pair --img screenshot.png` | Decode a pairing QR from a screenshot file (requires opencv) |
|
|
70
|
+
| `uvx onsense serve` | Run the MCP server (stdio). Also starts the clip daemon automatically unless `--no-clip` is passed |
|
|
71
|
+
| `uvx onsense serve --no-clip` | Run the MCP server only, without starting the clip daemon |
|
|
72
|
+
| `uvx onsense clip` | Run the clip bridge daemon standalone (port 8770) |
|
|
73
|
+
| `uvx onsense clip --allow-pull` | Enable phone→pull: phone can GET /clip to retrieve the PC clipboard |
|
|
74
|
+
| `uvx onsense clip --set-clipboard` | Auto-inject received content into the PC OS clipboard (off by default) |
|
|
75
|
+
| `uvx onsense doctor` | Diagnose installation, connectivity, and phone reachability |
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
## Other MCP clients (Claude Desktop, Cursor, Codex)
|
|
80
|
+
|
|
81
|
+
`uvx onsense pair` auto-registers the server with **Claude Code** (via `claude mcp add`). The MCP server itself is standard stdio MCP, so any MCP-compatible client works once configured — only the auto-registration is Claude Code-specific. For other clients, run `uvx onsense pair` once to obtain your phone's `PHONE_BASE` / `PHONE_TOKEN` (also saved to `~/.onsense/pair.json`), then add the server manually:
|
|
82
|
+
|
|
83
|
+
**Claude Desktop / Cursor** — add to the client's MCP config (`claude_desktop_config.json` or `.cursor/mcp.json`):
|
|
84
|
+
|
|
85
|
+
```json
|
|
86
|
+
{
|
|
87
|
+
"mcpServers": {
|
|
88
|
+
"onsense": {
|
|
89
|
+
"command": "uvx",
|
|
90
|
+
"args": ["onsense", "serve"],
|
|
91
|
+
"env": { "PHONE_BASE": "http://<phone-ip>:8080", "PHONE_TOKEN": "<token>" }
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
**Codex** — add to `~/.codex/config.toml`:
|
|
98
|
+
|
|
99
|
+
```toml
|
|
100
|
+
[mcp_servers.onsense]
|
|
101
|
+
command = "uvx"
|
|
102
|
+
args = ["onsense", "serve"]
|
|
103
|
+
env = { PHONE_BASE = "http://<phone-ip>:8080", PHONE_TOKEN = "<token>" }
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Restart the client once after adding the server.
|
|
107
|
+
|
|
108
|
+
---
|
|
109
|
+
|
|
110
|
+
## MCP Tools
|
|
111
|
+
|
|
112
|
+
These tools are exposed to your AI client after pairing:
|
|
113
|
+
|
|
114
|
+
| Tool | Description |
|
|
115
|
+
| --- | --- |
|
|
116
|
+
| `get_live_frame()` | Capture the current camera frame from the phone (returns JPEG image) |
|
|
117
|
+
| `read_sensors()` | Return phone sensor readings as JSON: battery level/charging state, ambient light (lux), accelerometer (x/y/z) |
|
|
118
|
+
| `recent_photos(limit=10)` | List recent photos on the phone: id, name, date_added, size, width, height |
|
|
119
|
+
| `get_photo(id, max_width=1024)` | Fetch a specific photo by id (downscaled to max_width); use ids from `recent_photos` |
|
|
120
|
+
|
|
121
|
+
If the phone's IP changes, the broker retries using mDNS autodiscovery before surfacing an error.
|
|
122
|
+
|
|
123
|
+
---
|
|
124
|
+
|
|
125
|
+
## Clip Bridge (Phone ↔ PC File & Clipboard)
|
|
126
|
+
|
|
127
|
+
`onsense serve` automatically starts a clip daemon on **port 8770**. You can also run it standalone with `onsense clip`.
|
|
128
|
+
|
|
129
|
+
### Phone → PC push (POST /clip)
|
|
130
|
+
|
|
131
|
+
The onSense Android app can push content to the PC:
|
|
132
|
+
|
|
133
|
+
- **Images** are saved to disk as `latest.jpg` (in `ONSENSE_CLIP_DIR`, default `<tempdir>/onsense/`).
|
|
134
|
+
- **Text files** are saved to disk.
|
|
135
|
+
- **Other files** (video, PDF, etc.) are saved to disk by filename.
|
|
136
|
+
- If `--set-clipboard` is active, images and text are also injected into the PC OS clipboard so you can paste with Ctrl+V immediately.
|
|
137
|
+
|
|
138
|
+
By default, `--set-clipboard` is **off** — files are saved to disk but the clipboard is not touched.
|
|
139
|
+
|
|
140
|
+
### PC → Phone pull (GET /clip)
|
|
141
|
+
|
|
142
|
+
Off by default. Enable with:
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
uvx onsense clip --allow-pull
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
When enabled, the phone can GET /clip to retrieve the current PC clipboard content (copied files first, then images, then text). Returns 204 if the clipboard is empty.
|
|
149
|
+
|
|
150
|
+
### Ports and environment variables
|
|
151
|
+
|
|
152
|
+
| Variable | Default | Purpose |
|
|
153
|
+
| --- | --- | --- |
|
|
154
|
+
| `PHONE_BASE` | (from pairing) | Phone HTTP base URL, e.g. `http://192.168.1.5:8080` |
|
|
155
|
+
| `PHONE_TOKEN` | (from pairing) | Pairing auth token |
|
|
156
|
+
| `ONSENSE_CLIP_DIR` | `<tempdir>/onsense` | Directory where pushed files are saved |
|
|
157
|
+
| `ONSENSE_CLIP_MAX_MB` | `200` | Maximum incoming file size in MB (0 = unlimited) |
|
|
158
|
+
| `ONSENSE_CLIP_ALLOW_PULL` | `0` | Set to `1` to enable PC→phone pull without the CLI flag |
|
|
159
|
+
| `ONSENSE_CLIP_SET_CLIPBOARD` | `0` | Set to `1` to auto-inject into OS clipboard without the CLI flag |
|
|
160
|
+
| `ONSENSE_TEST_FRAME` | (unset) | Local JPEG path returned by `get_live_frame` when phone is unreachable |
|
|
161
|
+
|
|
162
|
+
---
|
|
163
|
+
|
|
164
|
+
## Security
|
|
165
|
+
|
|
166
|
+
**Local network only.** The PC-side services bind to all interfaces but reject connections from non-private IP addresses at the application layer. They are not intended to be exposed to the internet.
|
|
167
|
+
|
|
168
|
+
**HMAC request signing.** The pairing token is never sent in cleartext. Every authenticated request is signed with HMAC-SHA256 over `METHOD\npath\ntimestamp\nnonce\nalgorithm`, keyed by a signing key derived from the pairing token via HKDF-SHA256 (a key separate from the encryption key), and carries `X-Ts` / `X-Nonce` / `X-Sig` / `X-Enc` headers. Servers verify in constant time, reject timestamps outside a ±300 s window, and reject reused nonces — so a sniffed request cannot be replayed and the token cannot be stolen off the wire. New Android installs generate a 128-bit random token, stored on the phone and in `~/.onsense/pair.json` (chmod 600) after pairing. (The `/health` endpoint is unauthenticated.) See [PROTOCOL.md](PROTOCOL.md) for the exact wire format.
|
|
169
|
+
|
|
170
|
+
**Pull and clipboard injection are off by default.** `GET /clip` (phone pulls PC clipboard) and OS clipboard auto-injection (phone push → Ctrl+V) are disabled unless you explicitly pass `--allow-pull` or `--set-clipboard`.
|
|
171
|
+
|
|
172
|
+
**Encrypted bodies (AES-256-GCM).** Sensitive payloads — camera frames, photos, sensor data, files, and clipboard content — are encrypted end-to-end with AES-256-GCM. The key is derived from the pairing token via HKDF-SHA256 (a key separate from the signing key); each message uses a fresh 96-bit nonce, and the GCM tag authenticates the body and binds it to its request (so tampering or response substitution is rejected). A passive sniffer on the same Wi-Fi sees only ciphertext. Low-sensitivity metadata stays plaintext: the open `/version` and `/health` endpoints, HTTP error responses, and the saved-file path returned after a push. (Note: large transfers currently buffer the body in memory while encrypting/decrypting — chunked streaming for very large files is a future step. Transport-level TLS is also a possible future hardening.)
|
|
173
|
+
|
|
174
|
+
**File size cap.** Incoming pushes are rejected if they exceed `ONSENSE_CLIP_MAX_MB` (default 200 MB). Set to `0` to remove the cap.
|
|
175
|
+
|
|
176
|
+
---
|
|
177
|
+
|
|
178
|
+
## License
|
|
179
|
+
|
|
180
|
+
MIT. See [LICENSE](LICENSE).
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## 한국어 요약
|
|
185
|
+
|
|
186
|
+
**onsense**는 Android 폰(onSense 앱)을 PC AI 에이전트의 카메라·센서로 연결하는 PC측 MCP 브로커입니다.
|
|
187
|
+
|
|
188
|
+
- `uvx onsense pair` — 폰과 한 번 페어링하면 Claude·Cursor에 MCP 서버가 자동 등록됩니다.
|
|
189
|
+
- `uvx onsense doctor` — 연결·설치 문제를 진단합니다.
|
|
190
|
+
- `uvx onsense serve` — AI 클라이언트가 호출하는 MCP 서버(stdio)를 실행합니다. 동시에 클립 브리지 데몬(:8770)도 자동 기동합니다.
|
|
191
|
+
- `uvx onsense clip` — 폰↔PC 파일·클립보드 브리지를 단독으로 실행합니다.
|
|
192
|
+
|
|
193
|
+
폰과 PC는 **같은 로컬 Wi-Fi**에 있어야 하며, 개발자 운영 클라우드 서버는 없습니다. 통신은 현재 로컬 HTTP(비암호화)이므로 신뢰할 수 있는 네트워크에서 사용하세요.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Third-Party Notices
|
|
2
|
+
|
|
3
|
+
This file summarizes third-party open source components used by the PC-side
|
|
4
|
+
onSense MCP package.
|
|
5
|
+
|
|
6
|
+
The PC-side onSense MCP package is licensed under the MIT License. The Android
|
|
7
|
+
app, onSense brand assets, logos, store assets, and product documentation are
|
|
8
|
+
not covered by that license unless explicitly stated.
|
|
9
|
+
|
|
10
|
+
## Direct Dependencies
|
|
11
|
+
|
|
12
|
+
| Package | License | Notes |
|
|
13
|
+
|---|---|---|
|
|
14
|
+
| `mcp` | MIT | MCP Python SDK/runtime dependency |
|
|
15
|
+
| `httpx` | BSD-3-Clause | HTTP client |
|
|
16
|
+
| `zeroconf` | LGPL-2.1-or-later | mDNS discovery |
|
|
17
|
+
| `qrcode` | BSD | QR output for pairing |
|
|
18
|
+
| `cryptography` | Apache-2.0 OR BSD-3-Clause | AES-256-GCM body encryption (P2) |
|
|
19
|
+
|
|
20
|
+
## Notable Transitive Dependencies
|
|
21
|
+
|
|
22
|
+
| Package | License | Notes |
|
|
23
|
+
|---|---|---|
|
|
24
|
+
| `anyio` | MIT | Async compatibility layer |
|
|
25
|
+
| `httpcore` | BSD-3-Clause | HTTP transport used by `httpx` |
|
|
26
|
+
| `certifi` | MPL-2.0 | CA certificate bundle |
|
|
27
|
+
| `idna` | BSD-3-Clause | Internationalized domain name support |
|
|
28
|
+
| `h11` | MIT | HTTP/1.1 protocol library |
|
|
29
|
+
| `attrs` | MIT | Utility dependency used by dependency tree |
|
|
30
|
+
| `pydantic` | MIT | Data validation used by dependency tree |
|
|
31
|
+
| `starlette` | BSD-3-Clause | ASGI framework used by dependency tree |
|
|
32
|
+
| `uvicorn` | BSD-3-Clause | ASGI server used by dependency tree |
|
|
33
|
+
|
|
34
|
+
## LGPL Dependency Note
|
|
35
|
+
|
|
36
|
+
`zeroconf` is licensed under LGPL-2.1-or-later. onSense depends on it as an
|
|
37
|
+
external Python package and does not copy its source code into this package.
|
|
38
|
+
If onSense is redistributed as a bundled executable or packaged with vendored
|
|
39
|
+
dependencies, the distributor should review the LGPL-2.1-or-later obligations,
|
|
40
|
+
including license notice, license text availability, and user ability to
|
|
41
|
+
replace or relink the LGPL-covered component where applicable.
|
|
42
|
+
|
|
43
|
+
## Verification
|
|
44
|
+
|
|
45
|
+
Before publishing, re-check dependency metadata with the locked environment:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
uv run --project pc python - <<'PY'
|
|
49
|
+
from importlib.metadata import metadata
|
|
50
|
+
for name in ["mcp", "httpx", "zeroconf", "qrcode", "certifi"]:
|
|
51
|
+
m = metadata(name)
|
|
52
|
+
print(name, m.get("Version"), m.get("License") or m.get("License-Expression"))
|
|
53
|
+
PY
|
|
54
|
+
```
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
"""onSense — 폰을 AI의 눈·센서로 만드는 PC측 MCP 브로커.
|
|
2
|
+
|
|
3
|
+
서브커맨드: serve(MCP 서버) / pair(페어링+등록) / doctor(진단).
|
|
4
|
+
폰(Android 앱)은 HTTP 제공자, 이 패키지는 stdio MCP 브로커로 AI 클라이언트에 연결한다.
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
__version__ = "0.1.0"
|
|
8
|
+
|
|
9
|
+
# 폰 앱과 합의된 상수 (android: CameraService.PORT / Auth / NSD)
|
|
10
|
+
PHONE_PORT = 8080
|
|
11
|
+
MDNS_TYPE = "_onsense._tcp.local."
|
|
12
|
+
TOKEN_HEADER = "X-Token"
|
|
13
|
+
PAIR_PORT = 8765
|
|
14
|
+
CLIP_PORT = 8770 # 폰→PC 클립보드/파일 데몬 (clip.py). 양방향 GET 도 동일 포트.
|