friend-assistant 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +254 -3
- package/bin/friend.js +9 -0
- package/dist/BUILD.json +5 -0
- package/dist/friend.mjs +8521 -0
- package/dist/plugins/rubber-duck/plugin.json +13 -0
- package/dist/plugins/rubber-duck/skills/duck.md +14 -0
- package/dist/plugins/standup-writer/plugin.json +13 -0
- package/dist/plugins/standup-writer/skills/standup.md +12 -0
- package/dist/plugins/unit-converter/plugin.json +20 -0
- package/dist/plugins/unit-converter/tools/convert.py +40 -0
- package/dist/voice/friend_voice/__init__.py +3 -0
- package/dist/voice/friend_voice/__main__.py +54 -0
- package/dist/voice/friend_voice/engines.py +81 -0
- package/dist/voice/friend_voice/models.py +63 -0
- package/dist/voice/friend_voice/server.py +162 -0
- package/dist/voice/friend_voice/speaker.py +50 -0
- package/dist/voice/friend_voice/stt.py +68 -0
- package/dist/voice/friend_voice/tts.py +58 -0
- package/dist/web/assets/index-BXe8-2fW.js +4453 -0
- package/dist/web/assets/index-BoQanKF0.css +1 -0
- package/dist/web/index.html +14 -0
- package/package.json +30 -4
package/README.md
CHANGED
|
@@ -1,3 +1,254 @@
|
|
|
1
|
-
#
|
|
2
|
-
|
|
3
|
-
|
|
1
|
+
# Friend
|
|
2
|
+
|
|
3
|
+
A self-hosted personal AI assistant. It runs on your PC, talks to you from any screen over WebSocket, uses tools on your machine with risk-based approvals, and controls a full-screen sci-fi "stage" itself.
|
|
4
|
+
|
|
5
|
+
Phase 1: secure login, agent loop with Claude, CLI device node, approvals with PIN, and a minimal stage with a particle-sphere presence. Phase 2a: chart, gauge, timeline and graph objects, automatic layout, fade-out, orb and ring voice styles and mood intensity. Phase 5a: local voice. Hold Space and speak; Friend answers aloud. Phase 5b: only your voice commands it. Phase 2b: the agent can look through your camera, one approved picture at a time. Also built: memory (notes it keeps about you), tasks and reminders with a heartbeat, free-form pages in an isolated frame, a Docker code sandbox, and a plugin store. Models: Claude, or a local model through Ollama. See `docs/superpowers/specs` and `docs/superpowers/plans`.
|
|
6
|
+
|
|
7
|
+
## Install (npm)
|
|
8
|
+
|
|
9
|
+
Needs only **Node.js 20 or newer**. Everything else is checked, and offered, by the setup wizard.
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
npm install -g friend-assistant
|
|
13
|
+
friend onboard
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
(or without installing: `npx friend-assistant onboard`)
|
|
17
|
+
|
|
18
|
+
`friend onboard` is a guided setup, one step at a time. Press Enter to take the suggestion at any question; run it again and it carries on where it stopped.
|
|
19
|
+
|
|
20
|
+
1. **Welcome**: what is about to happen.
|
|
21
|
+
2. **What you want to use**: pick any of: Claude through your subscription (Claude Code), Claude with an API key, a model on this PC (Ollama), voice, safe code running (Docker).
|
|
22
|
+
3. **What this PC needs**: it checks Node.js, plus only what your choices need: Python (voice), Docker, Ollama, Claude Code, and whether the port is free. For each missing one it shows the exact command (`winget` on Windows, `brew` on a Mac, `apt` on Linux, `npm` for Claude Code), asks **before** installing, runs it one by one with the output visible, and checks again. If a program was installed but this terminal cannot see it yet, it tells you to open a new terminal and run `friend onboard` again.
|
|
23
|
+
4. **Your account**: password and PIN (each typed twice) and the assistant's name.
|
|
24
|
+
5. **How Friend thinks**: signs in to Claude if needed (`claude auth login`), asks for an API key, or downloads the local model, and sets the safety net (the local model behind Claude).
|
|
25
|
+
6. **Voice**: a female or male voice, then the speech models (about 250 MB).
|
|
26
|
+
7. **Your workspace folder**, 8. **Who can open Friend** (only this PC, or your network, and the port), 9. **How it looks** (a theme), 10. **Finish**: a summary, and it can start Friend and open the browser.
|
|
27
|
+
|
|
28
|
+
Handy options: `friend onboard --no-install` (never installs anything, only tells you what to do), `--yes` (take every suggestion), `--redo <step>` (run one step again, for example `--redo look`), `--reset` (start from the beginning). `friend doctor` shows what this PC has and how to fix what is missing, and changes nothing. Progress is kept in `onboarding.json` in the data folder; it never contains a password, PIN or key.
|
|
29
|
+
|
|
30
|
+
If `npm install` fails on `better-sqlite3` (it builds a small native part when no ready-made one fits your system), install the compiler tools it names (on Windows: the "Desktop development with C++" part of Visual Studio Build Tools) and run it again.
|
|
31
|
+
|
|
32
|
+
## Quick start (from the source code)
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
pnpm install
|
|
36
|
+
pnpm build # builds the web client
|
|
37
|
+
pnpm friend setup # password, PIN, assistant name, workspace folder, API key
|
|
38
|
+
pnpm friend start # gateway + local device node, http://127.0.0.1:8787
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Open the address in a browser, sign in, and press any key to talk. Press `Esc` (or long-press the top-left corner) for the owner menu: mute stage, log out, emergency lock.
|
|
42
|
+
|
|
43
|
+
Preview the whole look with no API key and no network: `pnpm build && pnpm demo`, then open the printed address. Each message you send shows the next scene (gauges and charts, a timeline, a graph, the orb and ring voice styles, a warm mood).
|
|
44
|
+
|
|
45
|
+
Other commands: `friend rules list|remove <id>`, `friend devices list|revoke <id>`, `friend status`, `friend config set <key> <value>`, `friend pair` (code to connect another PC), `friend node --url <ws-url> --code <code>`.
|
|
46
|
+
|
|
47
|
+
## Use your Claude subscription (Claude Code)
|
|
48
|
+
|
|
49
|
+
Instead of an API key you can let Friend think through the Claude Code app you already have signed in with your Claude plan. To the rest of Friend it is just another model: same tools, same approvals, same screen. Nothing else changes.
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
claude # once: sign in with your Claude account (if you have not already)
|
|
53
|
+
pnpm friend claude setup --use # checks it works and switches Friend over; restart `friend start`
|
|
54
|
+
pnpm friend claude status
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
- One Claude Code process stays running, so replies start in about a second after the first message (a fresh start takes 3 to 9 seconds). Friend's conversation is remembered across restarts.
|
|
58
|
+
- Claude Code has no tools of its own here. Friend's tools reach it through a private local bridge (127.0.0.1, random token), and every call still goes through Friend's own approval and PIN rules and audit log.
|
|
59
|
+
- Unrecognized voices (guests) get a separate, tool-less, nothing-saved session.
|
|
60
|
+
- If your plan's usage limit runs out, Claude Code is signed out or missing, Friend answers with the local Ollama model and tells you once on screen (`friend config set claudeCode.fallback none` to see errors instead). Other settings: `claudeCode.model` (default sonnet), `claudeCode.command`.
|
|
61
|
+
- Caveats: it uses your plan's limits; using a subscription this way is a grey area in Anthropic's terms, so treat it as experimental; if Friend answered through the fallback, Claude does not know about that exchange.
|
|
62
|
+
- Opt-in live test against the real CLI: `FRIEND_LIVE_CLAUDE=1 pnpm exec vitest run packages/server/src/provider/claude-code/live.test.ts`.
|
|
63
|
+
|
|
64
|
+
## Connectors (Gmail, Chrome control, any MCP service)
|
|
65
|
+
|
|
66
|
+
A connector gives Friend the tools of an outside program or service through the MCP standard. Open **Esc, then Connectors** (or type `/connectors`). Ready-made entries:
|
|
67
|
+
|
|
68
|
+
- **Chrome control**: your chrome-control extension. Friend starts `chrome-control-mcp` itself; install it once with `npm install -g chrome-control-mcp`, run `chrome-control-mcp setup` and load the extension (the panel lists the steps).
|
|
69
|
+
- **Gmail**: a community Gmail MCP server (`@gongrzhe/server-gmail-autoauth-mcp`). You create the Google OAuth credentials and sign in on Google's own page; **Friend never sees your Google password**. More mail and calendar connectors can be added the same way ("Something else": any program or any server by address).
|
|
70
|
+
|
|
71
|
+
How it is kept safe:
|
|
72
|
+
- **Adding or switching one on needs your PIN**, and the screen shows exactly which program will start (no shell is ever used) or which address it will reach. Plain `http://` is only allowed for a server on this PC.
|
|
73
|
+
- **Every call to a connector's tool asks for your approval** (the usual card; "Always allow" works for the ones you trust). Tools appear to the AI as `mcp.<connector>.<tool>`.
|
|
74
|
+
- **Secrets** (tokens, keys) are typed on the screen only. They are saved in `connectors.json` next to the database (readable only by you) and are never shown again, never put in the AI's context, never in the audit log. The AI cannot supply or change secrets.
|
|
75
|
+
- **What comes back is data, not orders.** The AI is told that email, web pages and documents from a connector are untrusted and that it must never send or delete anything unless you asked for exactly that.
|
|
76
|
+
- A connector that stops working shows why; it is started again on the next use. Friend does not wait for connectors when it starts.
|
|
77
|
+
- Turning one off removes its tools at once. Removing deletes its saved secrets.
|
|
78
|
+
|
|
79
|
+
**The AI can manage connectors too.** Ask it to "connect X", "switch Gmail off", "reload Chrome control" or "remove it". Adding or switching on shows the same PIN card with the exact program; turning off and reloading need nothing; removing asks. Opt-in live check of the real chrome-control: `FRIEND_LIVE_CHROME=1 pnpm exec vitest run packages/server/src/connectors/live.test.ts`.
|
|
80
|
+
|
|
81
|
+
## The AI writes its own skills and plugins
|
|
82
|
+
|
|
83
|
+
When it notices something you repeat, the AI may suggest a **skill** (instructions it loads later) or a **plugin tool** (a small python, node or shell program that runs in the code sandbox: no network, no access to your files, approval on every call). If you agree it calls `plugins.create`: the approval card shows **everything it wrote, untruncated**, and needs your PIN. It is then checked like any plugin, switched on and loaded straight away (`plugins.reload` re-reads everything; the Plugins panel has a **Reload plugins** button too). Creating the same id again installs a newer version. The AI can also list, switch off or on, and delete plugins.
|
|
84
|
+
|
|
85
|
+
## Commands
|
|
86
|
+
|
|
87
|
+
`/help` opens the list of every command with what it does.
|
|
88
|
+
|
|
89
|
+
## On-screen status, clock and settings
|
|
90
|
+
|
|
91
|
+
- **Status line (bottom-left):** always shows what Friend is doing in plain words: Ready, Listening, Understanding what you said, Thinking, Getting the weather for Pune, Creating a view, Running a command, Answering, Waiting for your approval, Reconnecting. It never shows command text or arguments.
|
|
92
|
+
- **Clock (bottom-right):** a small date and time, so the screen is never empty.
|
|
93
|
+
- **Settings (Esc, then Settings):** change the model provider (Claude subscription through Claude Code, Claude API key, or local Ollama), the model names, the fallback, the Ollama address, and the voice options without the terminal. A change is used straight away (no restart) except the speaking voice and listening model. Saving needs your PIN. The API key is write-only: it is never sent back to the screen. Network address, folders and safety rules are deliberately not in this screen; use `friend config set` for those.
|
|
94
|
+
- **Slash commands:** type `/` in the message bar for a list that narrows as you type (arrows or Tab to pick, Enter to run): /settings, /plugins, /voice, /camera on|off, /mute, /unmute, /clear, /status, /logout, /lock, /help. They run on the screen and never go to the AI, so they work instantly and even when the model is down. Start a line with // to send a message that begins with a slash.
|
|
95
|
+
- **PIN with the keyboard:** wherever a PIN is asked for, type the digits, Backspace to correct, Enter to confirm (Enter never deletes your voice; that needs a click). The on-screen keys still work for touch.
|
|
96
|
+
- **The AI knows the screen:** its instructions include a layout guide (what is always on screen, the five regions, how to compose one answer, how long things should stay), and every request tells it what is on screen, where, for how long, and which regions are free. If it puts something on top of another object or over the captions, status line or clock, Friend moves it to a free spot (or the tidy auto grid) and tells it why, so it learns within the conversation. Nothing is refused.
|
|
97
|
+
- **Voice presence look (Settings, tab "Voice presence", or `/look`):** the settings dialog is wide and has tabs: *Model*, *Voice* and *Voice presence*. On the last one you choose a **theme** (Aurora, Ocean, Ember, Violet storm, Matrix, Rose, Ghost) with one click, or set everything yourself: the **shape** (sphere, orb, ring), the **colour** (swatches or any colour), **brightness**, **size**, how strongly it **reacts to the voice**, the soft **glow** behind everything, whether the live **captions** show and how big they are, and the **view**. A live preview shows the result before you save. Saving a change of look needs **no PIN**, and every open screen changes at once. *Let the AI change the shape, colour and brightness to suit a topic* can be switched off to keep exactly the look you chose (you can still ask it "make it purple" yourself). Pressing the colour box also works; the colour must be a plain #rrggbb.
|
|
98
|
+
- **Views and panels:** Esc, then **View** (or `/view minimal|companion|workspace`, or Settings) chooses how much is on the screen besides the AI's pictures. *Minimal*: only the voice presence (this hides the status line and the clock too; the Esc menu's View button never switches to it by accident, only Settings or `/view minimal` do, and Friend tells you once when the view is Minimal). *Companion*: plus the status line and clock. *Workspace*: also a **Chat** panel (what was said and answered, with a box to type in) and a **Schedule** panel (open tasks and reminders, soonest first, late ones in red). Turn any panel on or off from the Esc menu or with `/chat`, `/tasks`. **Drag a panel by its title bar** (or the status line and clock anywhere on them) to move it; arrow keys nudge it, double-click puts it back, `/layout reset` puts all back. Panels remember their place in this browser. Changing the view starts from that view's own set of panels.
|
|
99
|
+
- **The AI can change settings too:** ask it to switch the model, the fallback or the voice options, or to show a cleaner or fuller screen. Changing settings is a PIN-approval action (the same card as for any risky action); changing only the view needs no PIN. It can never change the API key or the Claude Code command (those stay on the settings screen), and a change of model takes effect after its current reply. The AI is also told which panels you have open and keeps its pictures out of their way.
|
|
100
|
+
|
|
101
|
+
## Voice (local, optional)
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
pnpm friend voice setup # one time: private Python environment + speech models (about 250 MB)
|
|
105
|
+
pnpm friend start # starts the speech engine too
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Hold **Space** (or long-press the screen) and speak; release to send. Friend answers in captions and aloud, and the sphere follows the real audio. Pressing Space while it talks interrupts it. Typed messages are answered silently.
|
|
109
|
+
|
|
110
|
+
- **Female or male voice.** Twelve voices to choose from, six female and six male, American, British and Scottish (Amy, Kristin, Hannah, Jenny, Alba, Kathleen; Ryan, Joe, Norman, Henry, Alan, Tom), plus the original voice (Lessac), which stays the default until you choose. You are asked at `friend setup` ("female or male") and again at `friend voice setup` (a number, `female` or `male`; or skip the question with `--female`, `--male` or `--voice jenny`). `friend voice list` shows them. Later, in **Settings, Voice tab**, pick one from the list and press **Hear it** to listen before you decide; the change needs no restart: the voice loads in the background (the first time a voice is used it downloads about 60 MB, then it works offline), you get a message when it is ready, and if it cannot be loaded Friend goes back to the previous voice instead of going silent. Any other Piper voice works too: `friend voice setup --voice hi_IN-pratham-medium`.
|
|
111
|
+
- Everything runs on your PC: speech to text with faster-whisper, text to speech with Piper. Recordings are never stored or sent anywhere else.
|
|
112
|
+
- Needs Python 3.10 or newer. `friend voice status` shows the state; `friend config set voice.sttModel small` (or `voice.language hi`, `voice.enabled false`) changes it, then run `friend voice setup` again for new models. The speaking voice is chosen as described above.
|
|
113
|
+
- Recordings are limited to 90 seconds and 4 MB. While Friend is busy with a request, a new recording is refused (409) instead of being dropped silently. Emergency lock also cancels a recording that is still being transcribed.
|
|
114
|
+
- The browser needs microphone permission and a secure origin (localhost, or HTTPS through your domain).
|
|
115
|
+
- **Until you enroll your voice, anyone who can press Space on a logged-in browser speaks as you.** The PIN still guards risky actions.
|
|
116
|
+
|
|
117
|
+
### Only your voice commands Friend (speaker verification)
|
|
118
|
+
|
|
119
|
+
Press **Esc** (or long-press the top-left corner), choose **Enroll my voice**, read the phrases aloud and enter your PIN. Friend keeps only an averaged 256-number voiceprint, encrypted on disk (AES-256-GCM, key in `voice.key` next to the database). Recordings are never kept. After that every spoken request is scored against your voiceprint:
|
|
120
|
+
|
|
121
|
+
| Match | What happens |
|
|
122
|
+
|---|---|
|
|
123
|
+
| Your voice (0.72 or higher) | Runs as you, as before. |
|
|
124
|
+
| Close, or too short to tell (0.58 to 0.72) | Nothing runs. A card asks "Was that you?"; the right PIN runs the words, a wrong PIN (3 tries), Dismiss or 60 seconds drops them. |
|
|
125
|
+
| Another voice (below 0.58) | A guest: polite answer, no tools, none of your conversation or screen, nothing saved. You get a warning with what was said. |
|
|
126
|
+
|
|
127
|
+
- `friend voice status` shows whether a voice is enrolled; `friend voice forget` deletes it after the PIN (so does the enrollment panel). `friend config set voice.accept 0.75` and `voice.challenge 0.6` tune the two thresholds.
|
|
128
|
+
- Emergency lock drops any pending challenge and any enrollment in progress.
|
|
129
|
+
- A voice match is a convenience, not a lock: a recording or a cloned voice can fool any speaker model. That is why a borderline match needs the PIN and risky actions always ask for it.
|
|
130
|
+
- Python tests for the speech engine: `cd packages/voice && python -m venv .venv && .venv/Scripts/pip install -e .[dev] && .venv/Scripts/python -m pytest`.
|
|
131
|
+
|
|
132
|
+
## Pictures first: weather, stats and graphics
|
|
133
|
+
|
|
134
|
+
Friend is meant to show answers, not only say them. It controls the whole screen and picks the object that fits (gauge, chart, table, timeline, graph, or its own animated page).
|
|
135
|
+
|
|
136
|
+
- **Weather:** "what is the weather in Pune?" gets live data from Open-Meteo (no account or key) and puts temperature, humidity, wind and a 24-hour chart on screen.
|
|
137
|
+
- **Computer stats:** "how is my computer doing?" puts CPU load, memory and disk gauges, uptime and CPU on screen.
|
|
138
|
+
- For these two, Friend calls the tools **itself** as soon as the request is clear (a named city, or a computer word plus a status word), so the graphs appear immediately even with a small local model; the model then only says a short summary. Asked together, both sets stay on screen. Anything else is decided by the model, which is told never to invent numbers and never to announce an action without doing it (if a small model does, Friend asks it once to actually call the tool).
|
|
139
|
+
|
|
140
|
+
## Voice replies
|
|
141
|
+
|
|
142
|
+
Friend answers aloud to **both** spoken and typed messages (`friend config set voice.speakTyped false` to keep typed ones silent). The voice follows the text: each sentence is spoken as soon as it is written, and code blocks are never read aloud.
|
|
143
|
+
|
|
144
|
+
Browsers keep sound blocked until the page has had a click, tap or key press. Friend unlocks sound on the first one anywhere on the page, and shows a one-time hint if sound is still blocked. Needs `friend voice setup` once, and a restart of `friend start` afterwards.
|
|
145
|
+
|
|
146
|
+
## Memory
|
|
147
|
+
|
|
148
|
+
Tell Friend "remember that my sister is Priya" and it keeps a short note. The 20 newest notes are in front of the model in every conversation; older ones are found on demand ("what do you know about Priya?").
|
|
149
|
+
|
|
150
|
+
- Up to 200 notes of 500 characters each, stored in the local database. When full it refuses and asks you to forget something; it never drops notes silently.
|
|
151
|
+
- You control all of it: `friend memory list`, `friend memory forget <number>`, `friend memory clear` (asks for the PIN). You can also just say "forget note 7".
|
|
152
|
+
- Notes are treated as **data about you, never as instructions**, so text inside a note (or text a web page tricked the agent into saving) cannot give the agent orders. Guests and unrecognized voices never see your notes. The agent is told never to save passwords, PINs or keys.
|
|
153
|
+
|
|
154
|
+
## Plugins
|
|
155
|
+
|
|
156
|
+
Plugins add **skills** (instructions the agent loads when they fit, for example "write my stand-up") and **tools** (small programs the agent can call, for example an exact unit converter). Three examples are bundled in `plugins/`: `unit-converter`, `standup-writer`, `rubber-duck`.
|
|
157
|
+
|
|
158
|
+
- Manage them from the screen (Esc, then **Plugins**) or the terminal: `friend plugin catalog | list | install <name|folder> | enable <id> | disable <id> | remove <id>`.
|
|
159
|
+
- **Installing shows what is inside and needs your confirmation (the PIN in the browser).** Switching a plugin on needs the PIN too; switching off and removing do not.
|
|
160
|
+
- A plugin's tools can never be more powerful than the code sandbox: they run in a throwaway Docker container (no network, no access to your files, time and memory limits), get their arguments as `/work/args.json`, and **ask for your approval on every call**. Skills are text only: they give the model words, never power, and cannot override your rules or the approval system.
|
|
161
|
+
- Plugins are validated strictly (lowercase ids, size and file-count limits, no links, every file must be inside the plugin folder) and re-validated each time Friend loads; a broken plugin is switched off with a visible reason.
|
|
162
|
+
- Make your own: a folder with `plugin.json` plus the files it names, then `friend plugin install ./my-plugin`. See `docs/superpowers/specs/2026-10-08-friend-plugins-design.md` for the format. Downloading plugins from the internet is deliberately not supported yet (it needs signed packages to be safe).
|
|
163
|
+
|
|
164
|
+
## Running code safely (Docker)
|
|
165
|
+
|
|
166
|
+
The agent can write a program (Python, Node or shell), run it, and read the result: calculations, data crunching, quick experiments. It needs [Docker Desktop](https://www.docker.com/products/docker-desktop/) running; without it the tool says so and nothing runs. (Windows Sandbox is not needed, so Windows Home works.) The first use of each language downloads a small image (about 50 MB), so that one run needs internet; after that it works offline.
|
|
167
|
+
|
|
168
|
+
Every run asks for your approval, and the card shows the **whole program**. Each run is its own throwaway container:
|
|
169
|
+
|
|
170
|
+
- `--network none`: no internet and no DNS; it cannot send anything anywhere.
|
|
171
|
+
- It sees none of your files: only its own code is mounted, read-only. The root filesystem is read-only; only a 64 MB scratch `/tmp` is writable.
|
|
172
|
+
- Not root (user 65534), all capabilities dropped, `no-new-privileges`, 512 MB memory (no swap), 1 CPU, at most 128 processes.
|
|
173
|
+
- A time limit (10 s by default, 60 s at most); after that the container is killed and removed. Output is cut at 20,000 characters.
|
|
174
|
+
|
|
175
|
+
`pnpm test` includes a check against real containers that runs hostile code (network attempts, writes outside `/tmp`, memory hog, endless loop, fork bomb) and confirms each is stopped.
|
|
176
|
+
|
|
177
|
+
## Free-form pages (sandbox)
|
|
178
|
+
|
|
179
|
+
When none of the standard objects fits, the agent can draw its own small page on the stage: a custom visualization, a calculator, a timer, a game. It is HTML, CSS and JavaScript written inline (up to 100,000 characters) and runs in an isolated frame:
|
|
180
|
+
|
|
181
|
+
- `sandbox="allow-scripts"` without `allow-same-origin`, so the page has an empty origin: it cannot read your session, storage or the app, cannot call Friend's APIs, navigate the window, open popups or submit forms.
|
|
182
|
+
- A fixed Content-Security-Policy placed before anything the agent wrote: `default-src 'none'` (no network of any kind, no external scripts, fonts or frames), inline script and style only, images only as `data:` URLs.
|
|
183
|
+
- The app never listens for messages from the frame, and every object has a dismiss (×) button.
|
|
184
|
+
- Checked in a real browser: the demo's page tries `fetch`, `parent.document` and `localStorage` and reports each as blocked (`pnpm demo`, fifth message).
|
|
185
|
+
|
|
186
|
+
## Tasks and reminders
|
|
187
|
+
|
|
188
|
+
Say "remind me in 30 minutes to call the bank", "remind me tomorrow at 5pm to send the report", "every Monday at 9 remind me to back up" (hourly, daily or weekly repeats), or just "add milk to my list". "What is on my list?" shows your open tasks; "done with 3" or "remove 4" tidies up.
|
|
189
|
+
|
|
190
|
+
- A **heartbeat** inside the server checks every 15 seconds. When a reminder is due, an alert appears, a red card shows on the stage for 90 seconds and Friend says it aloud (when voice is set up).
|
|
191
|
+
- A reminder **only notifies you**. It never starts a conversation with the model and never runs a command on its own, so nothing happens on your PC while you are away.
|
|
192
|
+
- If no screen is open (or Friend is mid-conversation) the reminder waits and is shown as soon as you are back. A repeating task skips any slots it missed instead of replaying them.
|
|
193
|
+
- Times need no special format: Friend knows your local time and converts. Up to 200 open tasks.
|
|
194
|
+
|
|
195
|
+
## Camera (owner-controlled)
|
|
196
|
+
|
|
197
|
+
Ask "what is this?" while you hold something up to the camera, and Friend can look. It works one picture at a time:
|
|
198
|
+
|
|
199
|
+
1. The camera is **off** until you turn it on: press **Esc**, choose **Camera: off** (it becomes **Camera: on**). The switch is not remembered; a reload turns it off again.
|
|
200
|
+
2. When the agent wants to look it calls `camera.snap`, and an approval card appears: "Take one picture with the camera and show it to the AI model". There is no "Always allow" for the camera.
|
|
201
|
+
3. After you approve, the browser opens the camera, takes one frame (JPEG, longest side 1024 px), closes the camera and uploads the frame. While the camera is open a red **Camera on** badge with a small live preview shows in the corner; the agent cannot hide it.
|
|
202
|
+
|
|
203
|
+
- The picture is shown to the model for that turn only. It is never written to disk, never saved in the conversation history (history keeps "[camera picture, not kept]"), and the audit log records only its size. Guests can never use the camera.
|
|
204
|
+
- Emergency lock, logout, switching the camera off, or closing the page cancels a pending request and releases the camera.
|
|
205
|
+
- **With Claude as the provider the picture is sent to Anthropic's API.** With Ollama it stays on your PC, but only a vision model can use it: `ollama pull llava` (or another vision model), then `friend config set ollama.model llava` and `friend config set ollama.vision true`. Note a vision model may not support tools, so Claude is the practical choice today. Without vision the tool says so before it asks you.
|
|
206
|
+
- Like the microphone, the camera needs `localhost` or HTTPS.
|
|
207
|
+
- Text inside a picture is treated as data, never as instructions.
|
|
208
|
+
|
|
209
|
+
## Safety model
|
|
210
|
+
|
|
211
|
+
- Reading is free, writing inside your workspace is free, shell commands ask first, destructive commands ask and need your PIN.
|
|
212
|
+
- Approval cards and the PIN pad are drawn by the client, never by the agent.
|
|
213
|
+
- Everything the agent does is written to the audit log.
|
|
214
|
+
|
|
215
|
+
## Another port, and other devices on your network
|
|
216
|
+
|
|
217
|
+
By default Friend is reachable only on this PC, at http://127.0.0.1:8787.
|
|
218
|
+
|
|
219
|
+
```bash
|
|
220
|
+
pnpm friend lan # shows how it is set up and the addresses
|
|
221
|
+
pnpm friend lan on # also reachable from phones and other PCs on your network
|
|
222
|
+
pnpm friend lan on --port 9100 # the same, on another port
|
|
223
|
+
pnpm friend lan port 9100 # change only the port
|
|
224
|
+
pnpm friend lan off # this PC only again
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Restart `friend start` afterwards. It then prints the addresses, for example `Also on your network: http://192.168.1.20:9100`; open that on the other device and sign in. Things to know:
|
|
228
|
+
|
|
229
|
+
- Anyone on the network can open the sign-in page. The password and PIN still guard everything, so use a strong password. Creating the owner account is only possible on this PC.
|
|
230
|
+
- The connection is plain http, so other people on the network could read it. Use it on a network you trust, or put Friend behind HTTPS (`publicUrl`, below).
|
|
231
|
+
- **Browsers allow the microphone and camera only on localhost or HTTPS.** Over plain http on another device you can type and read, but voice and camera need HTTPS.
|
|
232
|
+
- Windows asks once whether to let Node.js through the firewall: allow it on *Private* networks.
|
|
233
|
+
- To open it by a name (`http://my-pc:9100`) add the name: `friend config set allowedHosts my-pc`.
|
|
234
|
+
|
|
235
|
+
## Using your own domain later
|
|
236
|
+
|
|
237
|
+
Run Friend behind a reverse proxy (for example Caddy) and set:
|
|
238
|
+
|
|
239
|
+
```bash
|
|
240
|
+
friend config set publicUrl https://friend.example.com
|
|
241
|
+
friend config set trustProxy true
|
|
242
|
+
friend config set allowedOrigins https://friend.example.com
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
Login, PIN, rate limits and lockout stay on in every mode. Phones need HTTPS to use the microphone and camera in later phases, which the domain setup provides.
|
|
246
|
+
|
|
247
|
+
## Tests
|
|
248
|
+
|
|
249
|
+
```bash
|
|
250
|
+
pnpm test # all packages
|
|
251
|
+
pnpm typecheck
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
`scripts/smoke.md` lists the manual end-to-end checks and their last results.
|
package/bin/friend.js
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// A friendly message instead of a syntax error on an old Node.js, before anything else is loaded.
|
|
3
|
+
const major = Number(process.versions.node.split(".")[0]);
|
|
4
|
+
if (major < 20) {
|
|
5
|
+
console.error(`Friend needs Node.js 20 or newer, and this is ${process.versions.node}.`);
|
|
6
|
+
console.error("Install a newer one from https://nodejs.org and run this again.");
|
|
7
|
+
process.exit(1);
|
|
8
|
+
}
|
|
9
|
+
await import("../dist/friend.mjs");
|