@shinobiapps/shinobi 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -0
- package/dist/dashboard/server.d.ts.map +1 -1
- package/dist/dashboard/server.js +9 -0
- package/dist/dashboard/server.js.map +1 -1
- package/dist/lib/migrations.d.ts +2 -0
- package/dist/lib/migrations.d.ts.map +1 -1
- package/dist/lib/migrations.js +49 -4
- package/dist/lib/migrations.js.map +1 -1
- package/dist/lib/paths.d.ts +26 -0
- package/dist/lib/paths.d.ts.map +1 -0
- package/dist/lib/paths.js +68 -0
- package/dist/lib/paths.js.map +1 -0
- package/dist/lib/rate-limit.d.ts +25 -0
- package/dist/lib/rate-limit.d.ts.map +1 -0
- package/dist/lib/rate-limit.js +75 -0
- package/dist/lib/rate-limit.js.map +1 -0
- package/dist/models/decision_drafts.d.ts.map +1 -1
- package/dist/models/decision_drafts.js +2 -1
- package/dist/models/decision_drafts.js.map +1 -1
- package/dist/models/decisions.d.ts.map +1 -1
- package/dist/models/decisions.js +12 -3
- package/dist/models/decisions.js.map +1 -1
- package/dist/models/notes.d.ts.map +1 -1
- package/dist/models/notes.js +2 -1
- package/dist/models/notes.js.map +1 -1
- package/dist/models/subtasks.d.ts.map +1 -1
- package/dist/models/subtasks.js +2 -1
- package/dist/models/subtasks.js.map +1 -1
- package/dist/server/tools/projects.d.ts.map +1 -1
- package/dist/server/tools/projects.js +16 -2
- package/dist/server/tools/projects.js.map +1 -1
- package/dist/server/tools/workflow.d.ts.map +1 -1
- package/dist/server/tools/workflow.js +27 -3
- package/dist/server/tools/workflow.js.map +1 -1
- package/dist/services/auth/magic-link.d.ts.map +1 -1
- package/dist/services/auth/magic-link.js +3 -0
- package/dist/services/auth/magic-link.js.map +1 -1
- package/docs/configuration.md +21 -0
- package/docs/deploy-gcp-free.md +23 -1
- package/docs/recovery-runbook.md +162 -0
- package/package.json +3 -2
package/docs/configuration.md
CHANGED
|
@@ -37,6 +37,17 @@ Loopback binds skip the middleware entirely so local development stays zero-fric
|
|
|
37
37
|
|
|
38
38
|
Npm-tier plugins are discovered from `node_modules/@shinobi/plugin-*` in the current working directory at server start.
|
|
39
39
|
|
|
40
|
+
## Rate limits
|
|
41
|
+
|
|
42
|
+
Public-facing routes are rate limited per client IP (`CF-Connecting-IP` /
|
|
43
|
+
`X-Forwarded-For`, falling back to a shared local bucket). Both windows are
|
|
44
|
+
one minute.
|
|
45
|
+
|
|
46
|
+
| Variable | Default | Applies to |
|
|
47
|
+
|---|---|---|
|
|
48
|
+
| `SHINOBI_MCP_RATE_LIMIT` | `240` | `POST /mcp` tool calls |
|
|
49
|
+
| `SHINOBI_AUTH_RATE_LIMIT` | `10` | `/api/auth/magic-link`, `/api/auth/verify` |
|
|
50
|
+
|
|
40
51
|
## Embeddings (optional)
|
|
41
52
|
|
|
42
53
|
Shinobi falls back to FTS5 fulltext search by default. Configure a provider to enable semantic recall.
|
|
@@ -51,6 +62,16 @@ Shinobi falls back to FTS5 fulltext search by default. Configure a provider to e
|
|
|
51
62
|
|
|
52
63
|
When a provider is configured, write paths (`log_decision`, `log_dead_end`, `add_note`, `create_task`, `bulk_create_tasks`) generate and store embeddings synchronously after the row insert. Read paths (`recall`, `check_dead_ends`) prefer semantic match and fall back to FTS5 when no embedding is available.
|
|
53
64
|
|
|
65
|
+
## Session summaries (LLM optional)
|
|
66
|
+
|
|
67
|
+
`session_closeout` always refreshes the project's `recent_summary_md`. With
|
|
68
|
+
`SHINOBI_LLM_PROVIDER` configured (e.g. `groq` + `GROQ_API_KEY`, or `openai`),
|
|
69
|
+
the server compresses recent activity, decisions and dead ends into a terse
|
|
70
|
+
digest. Without a provider it persists the agent-authored closeout summary
|
|
71
|
+
instead (`recent_summary_provider = agent:closeout`) — the calling agent is an
|
|
72
|
+
LLM too, so no external key is required for the brain to stay fresh.
|
|
73
|
+
`agent_bootstrap` flags a stale summary via `summary_stale` either way.
|
|
74
|
+
|
|
54
75
|
## Recall mode
|
|
55
76
|
|
|
56
77
|
`recall` accepts an optional `mode` argument: `auto` (default), `semantic`, or `fulltext`. The `auto` mode picks semantic when an embedding provider is configured, otherwise fulltext.
|
package/docs/deploy-gcp-free.md
CHANGED
|
@@ -152,7 +152,25 @@ claude mcp add --transport http shinobi https://shinobi.yourdomain.com/mcp \
|
|
|
152
152
|
- **Upgrades**: `cd ~/shinobi && git pull && docker build -t shinobi . `
|
|
153
153
|
then `docker stop shinobi && docker rm shinobi` and re-run the
|
|
154
154
|
`docker run` command from step 3 (same token). The database lives on the
|
|
155
|
-
volume and survives.
|
|
155
|
+
volume and survives. Or skip the manual dance entirely — see
|
|
156
|
+
**Auto-deploy** below.
|
|
157
|
+
- **Auto-deploy (optional)**: `scripts/vm-autodeploy.sh` turns the upgrade
|
|
158
|
+
into a cron job on the VM host. Every tick it fetches `origin/main`; when
|
|
159
|
+
new commits land it builds the image **while the old container keeps
|
|
160
|
+
serving**, swaps containers carrying over the existing `SHINOBI_*` and
|
|
161
|
+
provider env vars, health-checks the new one, and rolls back to the
|
|
162
|
+
previous image if it doesn't come up. Concurrent ticks are guarded by a
|
|
163
|
+
lock file. Enable it with:
|
|
164
|
+
|
|
165
|
+
```bash
|
|
166
|
+
( crontab -l 2>/dev/null; echo "*/5 * * * * $HOME/shinobi/scripts/vm-autodeploy.sh >> $HOME/shinobi-autodeploy.log 2>&1" ) | crontab -
|
|
167
|
+
tail -f ~/shinobi-autodeploy.log # watch the first deploy go through
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
It runs on the **host**, never inside the container (a container cannot
|
|
171
|
+
`docker rm` itself mid-deploy). Worst-case deploy latency is one cron
|
|
172
|
+
interval + the e2-micro build time. Anything merged to `main` now reaches
|
|
173
|
+
production unattended — protect `main` accordingly (require PRs + green CI).
|
|
156
174
|
- **Backups**: the volume survives container replacement but not project
|
|
157
175
|
deletion. Wire `shinobi sync` to a private git repo (git is included in
|
|
158
176
|
the image):
|
|
@@ -172,6 +190,10 @@ claude mcp add --transport http shinobi https://shinobi.yourdomain.com/mcp \
|
|
|
172
190
|
The clone lives on the `/data` volume, so credentials and git config
|
|
173
191
|
survive container upgrades.
|
|
174
192
|
|
|
193
|
+
Backups are only real once you've restored from one — see the
|
|
194
|
+
[recovery runbook](recovery-runbook.md) for the tested container/VM/database
|
|
195
|
+
restore procedures and a drill to run before you depend on this setup.
|
|
196
|
+
|
|
175
197
|
> ⚠️ If the first push dies with `RPC failed; curl 55 ... unexpected
|
|
176
198
|
> disconnect` (common when sending multi-MB binary snapshots from small
|
|
177
199
|
> VMs over HTTP/2), pin the repo to HTTP/1.1 and raise the post buffer —
|
|
@@ -0,0 +1,162 @@
|
|
|
1
|
+
# Recovery runbook — when the brain goes down
|
|
2
|
+
|
|
3
|
+
The Shinobi server is the single shared brain: tasks, decisions, dead ends,
|
|
4
|
+
plans. This runbook is the tested path to bring it back when the container,
|
|
5
|
+
the VM, or the volume dies — written so it can be followed from a phone
|
|
6
|
+
(GCP browser SSH) on a bad beach connection.
|
|
7
|
+
|
|
8
|
+
It assumes the [GCP free-tier deploy](deploy-gcp-free.md): Docker container
|
|
9
|
+
`shinobi`, named volume `shinobi-data` mounted at `/data`, Cloudflare Tunnel
|
|
10
|
+
publishing `shinobi.yourdomain.com → 127.0.0.1:8765`, and nightly
|
|
11
|
+
`shinobi sync push` to a private GitHub snapshot repo.
|
|
12
|
+
|
|
13
|
+
## Before you leave (preflight checklist)
|
|
14
|
+
|
|
15
|
+
Recovery is only as good as what you can reach from a phone. Verify once,
|
|
16
|
+
at home:
|
|
17
|
+
|
|
18
|
+
- [ ] **Dashboard token** is in your password manager (every client and the
|
|
19
|
+
restore procedure need it).
|
|
20
|
+
- [ ] **Sync-repo PAT** (fine-grained, Contents read/write on the sync repo
|
|
21
|
+
only) is in your password manager — required to re-clone the backup.
|
|
22
|
+
- [ ] **GCP console** works on your phone (browser SSH included).
|
|
23
|
+
- [ ] **Cloudflare dashboard** login works on your phone.
|
|
24
|
+
- [ ] Last nightly backup is recent: `docker exec shinobi node /app/dist/cli.js sync status`
|
|
25
|
+
shows a `last push` within 24h, and the sync repo on GitHub shows the
|
|
26
|
+
commit.
|
|
27
|
+
- [ ] You ran the drill below at least once and know your restore time.
|
|
28
|
+
|
|
29
|
+
## Diagnose first
|
|
30
|
+
|
|
31
|
+
From any device: `https://shinobi.yourdomain.com/health`
|
|
32
|
+
|
|
33
|
+
| Symptom | Likely failure | Procedure |
|
|
34
|
+
| --- | --- | --- |
|
|
35
|
+
| `{"ok":true}` but MCP calls 503 from cloud sessions | Cloudflare min TLS raised above 1.2 | [D](#d-cloudflare-issues) |
|
|
36
|
+
| 502/530 from Cloudflare | Container or VM down, tunnel up | [A](#a-container-down-vm-alive), then [B](#b-vm-dead-or-deleted) |
|
|
37
|
+
| `ERR_NAME_NOT_RESOLVED` / tunnel shows `Routes: --` | Tunnel route lost | [D](#d-cloudflare-issues) |
|
|
38
|
+
| VM gone from GCP console / SSH unreachable | VM dead | [B](#b-vm-dead-or-deleted) |
|
|
39
|
+
| Health ok but data missing/corrupt | Volume damage | [C](#c-restore-database-from-backup) |
|
|
40
|
+
|
|
41
|
+
## A. Container down, VM alive
|
|
42
|
+
|
|
43
|
+
SSH to the VM (GCP console → VM instances → SSH):
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
docker ps -a # is the container there, just stopped?
|
|
47
|
+
docker start shinobi # if stopped
|
|
48
|
+
docker logs --tail 50 shinobi
|
|
49
|
+
curl http://127.0.0.1:8765/health
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
If the container is gone (e.g. after a failed upgrade), re-run it — the
|
|
53
|
+
database lives on the volume and survives:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
docker run -d --name shinobi --restart unless-stopped \
|
|
57
|
+
-p 127.0.0.1:8765:8765 \
|
|
58
|
+
-v shinobi-data:/data \
|
|
59
|
+
-e SHINOBI_DASHBOARD_TOKEN=<token from password manager> \
|
|
60
|
+
shinobi
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Expected time: **2–5 minutes**.
|
|
64
|
+
|
|
65
|
+
## B. VM dead or deleted
|
|
66
|
+
|
|
67
|
+
Full rebuild on a fresh e2-micro. This is the long path — the Docker build
|
|
68
|
+
alone takes 10–15 minutes on the free tier. Follow
|
|
69
|
+
[deploy-gcp-free.md](deploy-gcp-free.md) steps 1–3 (VM + swap + Docker +
|
|
70
|
+
clone + build + run), with one difference: **use your existing dashboard
|
|
71
|
+
token** from the password manager in the `docker run`, so every client keeps
|
|
72
|
+
working without reconfiguration.
|
|
73
|
+
|
|
74
|
+
Then restore the database — see [C](#c-restore-database-from-backup).
|
|
75
|
+
|
|
76
|
+
Then re-attach the tunnel: Cloudflare → Zero Trust → Networks → Connectors.
|
|
77
|
+
The old connector will show disconnected. Either reinstall `cloudflared` on
|
|
78
|
+
the new VM with the existing tunnel's install command (Connectors → your
|
|
79
|
+
tunnel → Configure → copy the `cloudflared service install eyJ...` command),
|
|
80
|
+
or create a fresh tunnel and re-add the published application route
|
|
81
|
+
(`shinobi.yourdomain.com` → HTTP `localhost:8765`). No DNS migration needed.
|
|
82
|
+
|
|
83
|
+
> ⚠️ Keep Cloudflare **Minimum TLS at 1.2** on the new setup — see
|
|
84
|
+
> [deploy-gcp-free.md](deploy-gcp-free.md) for why raising it to 1.3 kills
|
|
85
|
+
> all MCP calls from Claude cloud sessions.
|
|
86
|
+
|
|
87
|
+
Expected time: **30–45 minutes**, dominated by the Docker build.
|
|
88
|
+
|
|
89
|
+
## C. Restore database from backup
|
|
90
|
+
|
|
91
|
+
The nightly `sync push` snapshots the DB into the private sync repo. To pull
|
|
92
|
+
that snapshot back into a running container:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
# one-time on a fresh volume: clone the sync repo with the PAT
|
|
96
|
+
docker exec shinobi git clone https://YOUR_PAT@github.com/you/your-sync-repo.git /data/sync
|
|
97
|
+
docker exec shinobi git -C /data/sync config user.name "shinobi-server"
|
|
98
|
+
docker exec shinobi git -C /data/sync config user.email "shinobi@localhost"
|
|
99
|
+
docker exec shinobi node /app/dist/cli.js sync init /data/sync
|
|
100
|
+
|
|
101
|
+
# restore: pulls the repo and replaces the live DB with the snapshot
|
|
102
|
+
docker exec shinobi node /app/dist/cli.js sync pull
|
|
103
|
+
docker restart shinobi
|
|
104
|
+
curl http://127.0.0.1:8765/health
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Mind the RPO: the snapshot is up to 24h old (nightly cron at 03:00).
|
|
108
|
+
Anything logged between the last push and the failure is gone — accept it,
|
|
109
|
+
or check whether a laptop session from that window can re-log key decisions.
|
|
110
|
+
|
|
111
|
+
After a restore, re-arm the nightly cron if the VM is new:
|
|
112
|
+
|
|
113
|
+
```bash
|
|
114
|
+
( crontab -l 2>/dev/null; echo '0 3 * * * docker exec shinobi node /app/dist/cli.js sync push' ) | crontab -
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
## D. Cloudflare issues
|
|
118
|
+
|
|
119
|
+
- **MCP 503s from cloud sessions only** (laptop works): someone or something
|
|
120
|
+
raised Minimum TLS. Cloudflare → SSL/TLS → Edge Certificates → set
|
|
121
|
+
**Minimum TLS Version = 1.2**.
|
|
122
|
+
- **Route lost** (`Routes: --` on the tunnel): re-add the published
|
|
123
|
+
application route — Subdomain `shinobi`, Service Type HTTP,
|
|
124
|
+
URL `localhost:8765`.
|
|
125
|
+
- **Cloudflare outage**: nothing to fix on your side. The brain and backups
|
|
126
|
+
are unaffected; wait it out (degraded mode below).
|
|
127
|
+
|
|
128
|
+
## Verify recovery
|
|
129
|
+
|
|
130
|
+
From the phone, in order:
|
|
131
|
+
|
|
132
|
+
1. `https://shinobi.yourdomain.com/health` → `{"ok":true,...}`.
|
|
133
|
+
2. Dashboard loads with your token and shows your projects.
|
|
134
|
+
3. An MCP client (claude.ai chat with the Shinobi connector, or a Claude
|
|
135
|
+
Code session) can run `list_projects` and sees real data.
|
|
136
|
+
4. `docker exec shinobi node /app/dist/cli.js sync status` shows sync
|
|
137
|
+
configured, then run a manual `sync push` so a fresh snapshot exists.
|
|
138
|
+
|
|
139
|
+
## Degraded mode — what still works while the brain is offline
|
|
140
|
+
|
|
141
|
+
- Coding sessions (laptop, cloud) **work normally** — they just lose the
|
|
142
|
+
shared memory tools. Code, git, deploys are unaffected.
|
|
143
|
+
- The discipline that matters: note decisions/dead-ends somewhere durable
|
|
144
|
+
(commit messages, a scratch file) and re-log them via `log_decision` /
|
|
145
|
+
`log_dead_end` once the brain is back. Don't stand up a second brain from
|
|
146
|
+
a local copy — two diverging SQLite files are worse than a 24h gap.
|
|
147
|
+
- Approvals/push notifications are down with the server; fall back to
|
|
148
|
+
reviewing PRs directly on GitHub mobile.
|
|
149
|
+
|
|
150
|
+
## The drill (run at home, before the vacation)
|
|
151
|
+
|
|
152
|
+
1. `docker stop shinobi` → confirm `/health` fails → run procedure A. ⏱
|
|
153
|
+
2. `docker stop shinobi && docker rm shinobi` → procedure A's re-run path. ⏱
|
|
154
|
+
3. Simulate volume loss on a **throwaway copy**: create a scratch volume,
|
|
155
|
+
run a second container against it on another port, restore into it with
|
|
156
|
+
procedure C. Never drill C against the live volume. ⏱
|
|
157
|
+
4. From your phone only (laptop closed): GCP SSH login → `docker ps` →
|
|
158
|
+
Cloudflare dashboard login. If any login dance fails on the phone, fix
|
|
159
|
+
it now.
|
|
160
|
+
5. Log the measured times and any surprises as dead ends/decisions in
|
|
161
|
+
project "Shinobi remote MCP" — that's the institutional memory this tool
|
|
162
|
+
exists for.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@shinobiapps/shinobi",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.1",
|
|
4
4
|
"publishConfig": {
|
|
5
5
|
"access": "public"
|
|
6
6
|
},
|
|
@@ -21,6 +21,7 @@
|
|
|
21
21
|
"docs/github-integration.md",
|
|
22
22
|
"docs/plugin-development.md",
|
|
23
23
|
"docs/plugins.md",
|
|
24
|
+
"docs/recovery-runbook.md",
|
|
24
25
|
"docs/relay.md",
|
|
25
26
|
"docs/remote-access.md",
|
|
26
27
|
"docs/remote-mcp.md",
|
|
@@ -46,7 +47,7 @@
|
|
|
46
47
|
"smoke:mcp": "node --import tsx scripts/smoke-mcp-stdio.ts",
|
|
47
48
|
"smoke:dashboard": "node --import tsx scripts/smoke-dashboard.ts",
|
|
48
49
|
"smoke:mcp-http": "node --import tsx scripts/smoke-mcp-http.ts",
|
|
49
|
-
"test": "npm run build && npm run smoke:tools && npm run smoke:mcp && npm run smoke:mcp-http && npm run smoke:dashboard",
|
|
50
|
+
"test": "npm run build && npm run test:vitest && npm run smoke:tools && npm run smoke:mcp && npm run smoke:mcp-http && npm run smoke:dashboard",
|
|
50
51
|
"test:vitest": "vitest run",
|
|
51
52
|
"test:watch": "vitest"
|
|
52
53
|
},
|