neoagent 2.3.1-beta.91 → 2.3.1-beta.93
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +2 -2
- package/docs/automation.md +82 -41
- package/docs/capabilities.md +85 -91
- package/docs/configuration.md +86 -103
- package/docs/getting-started.md +37 -35
- package/docs/index.md +24 -30
- package/docs/integrations.md +46 -63
- package/docs/migration.md +41 -146
- package/docs/operations.md +23 -20
- package/docs/skills.md +20 -20
- package/docs/why-neoagent.md +15 -13
- package/extensions/chrome-browser/background.mjs +10 -2
- package/flutter_app/DESIGN_SYSTEM.md +166 -0
- package/flutter_app/lib/main_app_shell.dart +27 -11
- package/flutter_app/lib/main_chat.dart +4 -2
- package/flutter_app/lib/main_controller.dart +5 -2
- package/flutter_app/lib/main_shared.dart +10 -10
- package/flutter_app/lib/main_spacing.dart +15 -0
- package/flutter_app/lib/main_theme.dart +10 -0
- package/package.json +1 -1
- package/server/db/database.js +1 -1
- package/server/public/.last_build_id +1 -1
- package/server/public/flutter_bootstrap.js +1 -1
- package/server/public/main.dart.js +8843 -8841
- package/server/routes/agents.js +4 -1
- package/server/routes/auth.js +1 -4
- package/server/routes/memory.js +3 -3
- package/server/routes/settings.js +11 -2
- package/server/routes/skills.js +12 -0
- package/server/routes/tasks.js +22 -2
- package/server/services/ai/toolRunner.js +12 -0
- package/server/services/memory/manager.js +18 -7
- package/server/services/messaging/manager.js +15 -1
- package/server/services/tasks/integration_runtime.js +1 -1
- package/server/services/tasks/task_repository.js +3 -6
package/README.md
CHANGED
|
@@ -34,7 +34,7 @@ npm install -g neoagent
|
|
|
34
34
|
neoagent install
|
|
35
35
|
```
|
|
36
36
|
|
|
37
|
-
Runs as a `launchd` user service on macOS and `systemd --user` on Linux.
|
|
37
|
+
Opens at **http://localhost:3333** when complete. Runs as a `launchd` user service on macOS and `systemd --user` on Linux.
|
|
38
38
|
|
|
39
39
|
## Manage
|
|
40
40
|
|
|
@@ -47,7 +47,7 @@ neoagent logs
|
|
|
47
47
|
|
|
48
48
|
## Links
|
|
49
49
|
|
|
50
|
-
[Docs](https://neolabs-systems.github.io/NeoAgent/docs/) | [Capabilities](docs/capabilities.md) | [Why NeoAgent](docs/why-neoagent.md) | [Issues](https://github.com/NeoLabs-Systems/NeoAgent/issues)
|
|
50
|
+
[Docs](https://neolabs-systems.github.io/NeoAgent/docs/) | [Getting Started](docs/getting-started.md) | [Configuration](docs/configuration.md) | [Capabilities](docs/capabilities.md) | [Why NeoAgent](docs/why-neoagent.md) | [Issues](https://github.com/NeoLabs-Systems/NeoAgent/issues)
|
|
51
51
|
|
|
52
52
|
---
|
|
53
53
|
|
package/docs/automation.md
CHANGED
|
@@ -1,53 +1,94 @@
|
|
|
1
1
|
# Automation
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Tasks run on a schedule or integration trigger, use the same tools as chat, and can deliver results through any connected messaging platform.
|
|
4
4
|
|
|
5
|
-
##
|
|
5
|
+
## Creating a Task
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Open **Tasks** in the UI and fill in:
|
|
8
8
|
|
|
9
|
-
| Field |
|
|
9
|
+
| Field | Description |
|
|
10
10
|
|---|---|
|
|
11
|
-
| Name | Human-readable
|
|
12
|
-
| Cron
|
|
13
|
-
| Prompt | Self-contained instruction for the future run |
|
|
14
|
-
| Enabled
|
|
15
|
-
| Model
|
|
11
|
+
| **Name** | Human-readable label |
|
|
12
|
+
| **Cron** | Five-field schedule expression |
|
|
13
|
+
| **Prompt** | Self-contained instruction for the future run |
|
|
14
|
+
| **Enabled** | Active or paused |
|
|
15
|
+
| **Model** | Optional per-task model override |
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
## Cron Expressions
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
```
|
|
20
|
+
┌───── minute (0–59)
|
|
21
|
+
│ ┌──── hour (0–23)
|
|
22
|
+
│ │ ┌─── day of month (1–31)
|
|
23
|
+
│ │ │ ┌── month (1–12)
|
|
24
|
+
│ │ │ │ ┌─ day of week (0–7, both 0 and 7 = Sunday)
|
|
25
|
+
│ │ │ │ │
|
|
26
|
+
* * * * *
|
|
27
|
+
```
|
|
20
28
|
|
|
21
|
-
|
|
29
|
+
Common patterns:
|
|
30
|
+
|
|
31
|
+
| Expression | Runs |
|
|
32
|
+
|---|---|
|
|
33
|
+
| `0 9 * * *` | Daily at 9:00 AM |
|
|
34
|
+
| `0 9 * * 1-5` | Weekdays at 9:00 AM |
|
|
35
|
+
| `0 8 * * 1` | Every Monday at 8:00 AM |
|
|
36
|
+
| `0 18 * * 5` | Every Friday at 6:00 PM |
|
|
37
|
+
| `0 9 1 * *` | First of every month at 9:00 AM |
|
|
38
|
+
| `0 */4 * * *` | Every 4 hours |
|
|
39
|
+
| `*/30 * * * *` | Every 30 minutes |
|
|
40
|
+
|
|
41
|
+
## Writing Good Task Prompts
|
|
42
|
+
|
|
43
|
+
Prompts run unattended. Be specific about what to check and when to notify — tasks that always send a message become noise, tasks that only notify on a condition are useful.
|
|
44
|
+
|
|
45
|
+
**Daily news digest**
|
|
46
|
+
```
|
|
47
|
+
Search Hacker News for the top 5 stories today. Send me a brief summary of each via Telegram, including the title and link.
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
**Price monitor**
|
|
51
|
+
```
|
|
52
|
+
Check the price of Bitcoin and Ethereum on CoinGecko. If either has changed more than 5% in the last 24 hours, send me a Telegram message with the current prices and percentage change. If there are no significant changes, do nothing.
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
**Server health check**
|
|
56
|
+
```
|
|
57
|
+
Run `df -h` and `free -m`. If disk usage on any partition is above 85% or available memory is below 500MB, send me a Telegram alert with the details. Otherwise do nothing.
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Weekly email digest**
|
|
61
|
+
```
|
|
62
|
+
Search my Gmail for unread emails from the last 7 days. Group them by sender domain and summarize the main topics. Send the summary to my Telegram.
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Tool Access
|
|
66
|
+
|
|
67
|
+
Automation can use everything available in chat:
|
|
22
68
|
|
|
23
69
|
| Capability | Examples |
|
|
24
70
|
|---|---|
|
|
25
|
-
| Browser | Navigate,
|
|
26
|
-
|
|
|
27
|
-
|
|
|
28
|
-
| Memory | Store
|
|
29
|
-
| Messaging | Send
|
|
30
|
-
| MCP |
|
|
31
|
-
|
|
|
32
|
-
| Recordings |
|
|
33
|
-
| Health | Read synced Android Health Connect metrics
|
|
34
|
-
| Android |
|
|
35
|
-
| Subagents | Spawn
|
|
36
|
-
| Outputs |
|
|
37
|
-
|
|
38
|
-
Prefer official integrations and
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
-
|
|
49
|
-
- Keep secrets in server config, not prompts or skills.
|
|
50
|
-
- Prefer read-only checks unless the task explicitly needs to mutate data.
|
|
51
|
-
- Ask for notification only when a condition is met to avoid noisy repeated messages.
|
|
52
|
-
- Review run history in **Runs** and service logs in **Logs** when behavior is surprising.
|
|
53
|
-
- Remember that browser, CLI, Android runtime, and local file tools run on the NeoAgent server or configured worker, not necessarily on your current laptop.
|
|
71
|
+
| **Browser** | Navigate, extract content, screenshot, evaluate JavaScript |
|
|
72
|
+
| **CLI** | Shell commands in a persistent terminal |
|
|
73
|
+
| **Files** | Read, write, search host files |
|
|
74
|
+
| **Memory** | Store and retrieve long-term facts |
|
|
75
|
+
| **Messaging** | Send results through any connected platform |
|
|
76
|
+
| **MCP** | Tools from configured remote MCP servers |
|
|
77
|
+
| **Integrations** | Gmail, Calendar, Notion, Slack, and other OAuth tools |
|
|
78
|
+
| **Recordings** | Search and read transcripts |
|
|
79
|
+
| **Health** | Read synced Android Health Connect metrics |
|
|
80
|
+
| **Android** | Control an emulator or device |
|
|
81
|
+
| **Subagents** | Spawn parallel helpers inside a longer run |
|
|
82
|
+
| **Outputs** | Artifacts, images, Mermaid graphs, markdown tables |
|
|
83
|
+
|
|
84
|
+
Prefer official integrations and MCP tools over browser automation when both can do the job — they are more reliable and easier to audit.
|
|
85
|
+
|
|
86
|
+
## Safety
|
|
87
|
+
|
|
88
|
+
NeoAgent runs on your server and can touch real files, messaging surfaces, connected accounts, and browser sessions. Keep prompts narrow.
|
|
89
|
+
|
|
90
|
+
- Use messaging allowlists to restrict which chats receive automated messages
|
|
91
|
+
- Keep secrets in server config, not in prompts or skill files
|
|
92
|
+
- Prefer read-only checks unless the task explicitly needs to write data
|
|
93
|
+
- Review run history in **Runs** and service logs in **Logs** when behavior is unexpected
|
|
94
|
+
- Browser, CLI, Android, and file tools run on the NeoAgent server — not necessarily your current laptop
|
package/docs/capabilities.md
CHANGED
|
@@ -1,124 +1,118 @@
|
|
|
1
1
|
# Capabilities
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
NeoAgent can operate real software on the machine it runs on — a browser, Android emulators and devices, and a terminal. It connects to your accounts, remembers context across sessions, and runs scheduled automations without you being present.
|
|
4
4
|
|
|
5
5
|
## Operator UI
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
| Section | What it is for |
|
|
7
|
+
| Section | What it's for |
|
|
10
8
|
|---|---|
|
|
11
|
-
| Chat |
|
|
12
|
-
| Runs | Live and historical
|
|
13
|
-
|
|
|
14
|
-
|
|
|
15
|
-
|
|
|
16
|
-
|
|
|
17
|
-
|
|
|
18
|
-
|
|
|
19
|
-
|
|
|
20
|
-
|
|
|
21
|
-
|
|
|
22
|
-
|
|
|
23
|
-
|
|
24
|
-
## Recordings
|
|
9
|
+
| **Chat** | Interactive agent runs with full tool access, memory, integrations, and messaging |
|
|
10
|
+
| **Runs** | Live and historical step-by-step execution — browser, Android, CLI, messaging, tasks, MCP, subagents |
|
|
11
|
+
| **Tasks** | Schedule-triggered and integration-triggered automations |
|
|
12
|
+
| **Skills** | Built-in and custom reusable workflows |
|
|
13
|
+
| **Integrations** | OAuth account connections for structured app tools |
|
|
14
|
+
| **MCP** | Remote MCP server registration and tool discovery |
|
|
15
|
+
| **Memory** | Long-term memory, core facts, and session search |
|
|
16
|
+
| **Devices** | Server-side browser and Android runtime controls |
|
|
17
|
+
| **Recordings** | Audio sessions, transcripts, segment playback, AI insights |
|
|
18
|
+
| **Health** | Android Health Connect sync status and synced metrics |
|
|
19
|
+
| **Settings** | AI providers, model routing, runtime settings, messaging credentials |
|
|
20
|
+
| **Logs** | Service logs and diagnostics |
|
|
25
21
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
Recording sessions support:
|
|
22
|
+
## Agent Tools
|
|
29
23
|
|
|
30
|
-
|
|
31
|
-
- Sources, chunks, transcript segments, session status, and playback URLs.
|
|
32
|
-
- Statuses for recording, processing, completed, failed, and cancelled sessions.
|
|
33
|
-
- Retry transcription and delete transcript segment actions.
|
|
34
|
-
- Full session deletion with storage cleanup.
|
|
35
|
-
- Agent tools for listing, opening, and searching transcripts: `recordings_list`, `recordings_get`, and `recordings_search`.
|
|
36
|
-
- A keyless social video extractor for public YouTube, TikTok, Instagram, and X URLs through `social_video_extract`, with metadata-first fetch, captions-first transcript strategy, STT fallback, and representative frame extraction.
|
|
24
|
+
What the agent can use in chat and automation runs:
|
|
37
25
|
|
|
38
|
-
|
|
26
|
+
| Area | Capabilities |
|
|
27
|
+
|---|---|
|
|
28
|
+
| **CLI** | PTY-capable `execute_command` with stdin, timeout, stdout/stderr, exit code |
|
|
29
|
+
| **Browser** | Navigate, click, type, extract, screenshot, evaluate JavaScript |
|
|
30
|
+
| **Android** | UI observation, input, screenshots, app launch, intent launch, APK install, `adb shell` |
|
|
31
|
+
| **Web search** | Brave Search API |
|
|
32
|
+
| **Files** | Read, write, edit, list, search |
|
|
33
|
+
| **HTTP** | Direct requests |
|
|
34
|
+
| **Memory** | Semantic search, session search, daily logs, core memory, API key name lookup |
|
|
35
|
+
| **Skills** | Create, list, update, delete persistent skills |
|
|
36
|
+
| **Tasks** | Create, update, delete, and one-time run automations |
|
|
37
|
+
| **MCP** | Add/remove MCP servers, use dynamic MCP tools |
|
|
38
|
+
| **Subagents** | Spawn, wait for, and cancel async helpers inside a run |
|
|
39
|
+
| **Images** | Generate with Grok, analyze with vision models |
|
|
40
|
+
| **Recordings** | List, inspect, search transcripts |
|
|
41
|
+
| **Social video** | Extract transcript and metadata from public YouTube, TikTok, Instagram, and X URLs |
|
|
42
|
+
| **Health** | Read synced mobile health metrics and summaries |
|
|
43
|
+
| **Outputs** | Markdown tables, Mermaid graphs, downloadable artifacts |
|
|
39
44
|
|
|
40
45
|
## Android Control
|
|
41
46
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
Android control supports:
|
|
45
|
-
|
|
46
|
-
- Starting and stopping the managed Android emulator.
|
|
47
|
-
- Listing ADB-connected devices and installed apps.
|
|
48
|
-
- Taking screenshots and UIAutomator XML dumps.
|
|
49
|
-
- Observing visible UI nodes.
|
|
50
|
-
- Opening apps and Android intents.
|
|
51
|
-
- Tapping, long pressing, typing, swiping, and pressing Android navigation keys.
|
|
52
|
-
- Waiting for text, resource IDs, descriptions, or classes to appear.
|
|
53
|
-
- Installing `.apk` and universal `.apks` bundles.
|
|
54
|
-
- Running `adb shell` commands when higher-level tools are not enough.
|
|
55
|
-
|
|
56
|
-
These actions run where the NeoAgent backend is running. If NeoAgent is deployed on a remote server, the AI controls the Android runtime attached to that server, not the laptop where you are reading the docs.
|
|
47
|
+
The agent operates Android — it is not an app that runs on Android. Controls run on a server-attached emulator or physical device over ADB.
|
|
57
48
|
|
|
58
|
-
|
|
49
|
+
- Start and stop the managed emulator
|
|
50
|
+
- List connected devices and installed apps
|
|
51
|
+
- Screenshot and UIAutomator XML dump
|
|
52
|
+
- Observe visible UI nodes
|
|
53
|
+
- Open apps and launch Android intents
|
|
54
|
+
- Tap, long press, type, swipe, press navigation keys
|
|
55
|
+
- Wait for text, resource IDs, descriptions, or classes to appear
|
|
56
|
+
- Install `.apk` and `.apks` bundles
|
|
57
|
+
- Run `adb shell` commands directly
|
|
59
58
|
|
|
60
|
-
|
|
59
|
+
These actions run on the NeoAgent server. If NeoAgent is deployed remotely, it controls the Android runtime on that machine — not your local device.
|
|
61
60
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
- `NEOAGENT_BACKEND_URL` build/run configuration for real devices.
|
|
65
|
-
- Health Connect permission flow and background sync.
|
|
66
|
-
- Microphone recording through an Android foreground service.
|
|
67
|
-
- Boot restore hooks for recording services when Android allows them.
|
|
61
|
+
## Recordings
|
|
68
62
|
|
|
69
|
-
|
|
63
|
+
Audio sessions are recorded server-side. The web client captures browser microphone and screen audio; the Android app records phone microphone audio via a foreground service.
|
|
70
64
|
|
|
71
|
-
|
|
65
|
+
- Chunked uploads with per-source sequence tracking
|
|
66
|
+
- Session statuses: recording, processing, completed, failed, cancelled
|
|
67
|
+
- Transcript segment retry and deletion
|
|
68
|
+
- Transcript search across sessions
|
|
69
|
+
- Agent tools: `recordings_list`, `recordings_get`, `recordings_search`
|
|
70
|
+
- Social video extraction via `social_video_extract` — title, description, transcript, and a representative frame from YouTube, TikTok, Instagram, and X URLs
|
|
72
71
|
|
|
73
|
-
|
|
72
|
+
Transcription uses Deepgram (`nova-3` model, multi-language by default). Enable `auto_recording_insights` in AI settings to generate summaries, action items, and events automatically after transcription.
|
|
74
73
|
|
|
75
|
-
##
|
|
74
|
+
## Runtime Modes
|
|
76
75
|
|
|
77
|
-
|
|
76
|
+
| Profile | What runs where |
|
|
77
|
+
|---|---|
|
|
78
|
+
| `trusted-host` | CLI and Android run on the host; browser runs in the VM or paired extension |
|
|
79
|
+
| `secure-vm` | CLI, browser, and Android all run inside the isolated VM |
|
|
78
80
|
|
|
79
|
-
|
|
80
|
-
- Messaging platforms let the agent talk through WhatsApp, Telegram, Discord, Slack, Google Chat, Teams, Matrix, Signal, iMessage/BlueBubbles, IRC, Twitch, LINE, Mattermost, configurable webhook bridges, and Telnyx Voice.
|
|
81
|
+
Production deployments can require `secure-vm` and a strong `NEOAGENT_VM_GUEST_TOKEN` (32+ characters).
|
|
81
82
|
|
|
82
|
-
|
|
83
|
+
The browser always runs in isolation — either the local VM or a paired Chrome extension on a remote machine. To pair an extension: download `/api/browser-extension/download` from NeoAgent, unzip it, enable Developer Mode in `chrome://extensions`, load the folder, then pair after signing in.
|
|
83
84
|
|
|
84
|
-
|
|
85
|
+
## Integrations and Messaging
|
|
85
86
|
|
|
86
|
-
|
|
87
|
+
NeoAgent has two separate layers:
|
|
87
88
|
|
|
88
|
-
|
|
89
|
+
**Official integrations** — structured OAuth-backed tools the agent can use:
|
|
89
90
|
|
|
90
|
-
|
|
|
91
|
+
| Provider | Tools |
|
|
91
92
|
|---|---|
|
|
92
|
-
|
|
|
93
|
-
|
|
|
94
|
-
|
|
|
95
|
-
|
|
|
96
|
-
|
|
|
97
|
-
|
|
|
98
|
-
|
|
|
99
|
-
|
|
|
100
|
-
|
|
|
101
|
-
|
|
|
102
|
-
| Subagents | Spawn, list, wait for, and cancel async subagents inside a run |
|
|
103
|
-
| Output | Generate markdown tables and Mermaid graphs |
|
|
104
|
-
| Images | Generate images with Grok and analyze local image files with a vision-capable model |
|
|
105
|
-
| Recordings | List, inspect, and search recording transcripts |
|
|
106
|
-
| Social video | Extract title, description, transcript, and a representative frame from public social video URLs |
|
|
107
|
-
| Health | Read synced mobile health metrics |
|
|
108
|
-
|
|
109
|
-
Generated binary or text artifacts can be promoted into user-scoped artifact storage under `~/.neoagent/data/artifacts` and served through authenticated `/api/artifacts/:id/content` URLs.
|
|
93
|
+
| Google Workspace | Gmail, Calendar, Drive, Docs, Sheets |
|
|
94
|
+
| Microsoft 365 | Outlook, Calendar, OneDrive, Teams |
|
|
95
|
+
| Notion | Pages, databases, blocks, search |
|
|
96
|
+
| Slack | Messages, conversations, search |
|
|
97
|
+
| Figma | Files, nodes, comments, rendered images |
|
|
98
|
+
| Home Assistant | Entity state, service calls |
|
|
99
|
+
| Trello | Boards, lists, cards, comments |
|
|
100
|
+
| Spotify | Playback, search, queue |
|
|
101
|
+
| Weather | Current conditions and forecasts (no API key needed) |
|
|
102
|
+
| Personal WhatsApp | Per-account read and send |
|
|
110
103
|
|
|
111
|
-
|
|
104
|
+
**Messaging platforms** — channels for communicating with the agent:
|
|
112
105
|
|
|
113
|
-
|
|
106
|
+
WhatsApp, Telegram, Discord, Slack, Google Chat, Teams, Matrix, Signal, iMessage/BlueBubbles, IRC, Twitch, LINE, Mattermost, Telnyx Voice, plus webhook bridges for Feishu, Nextcloud Talk, Nostr, Synology Chat, Tlon, Zalo, WeChat, and WebChat.
|
|
114
107
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
| `secure-vm` | CLI, browser, and Android tools run through the local VM backend |
|
|
108
|
+
Each official integration account can be set to **Read/Write** (default) or **Read Only**. Write tools are blocked server-side for read-only accounts.
|
|
109
|
+
|
|
110
|
+
## Android App and Health
|
|
119
111
|
|
|
120
|
-
|
|
112
|
+
The Flutter Android app connects to the same self-hosted backend and can:
|
|
121
113
|
|
|
122
|
-
|
|
114
|
+
- Run chat and operator UI
|
|
115
|
+
- Sync Android Health Connect data (steps, heart rate, sleep, exercise, weight)
|
|
116
|
+
- Record microphone audio via a foreground service
|
|
123
117
|
|
|
124
|
-
|
|
118
|
+
Synced health data is available through the `read_health_data` agent tool and the **Health** UI section.
|