@shinobiapps/shinobi 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. package/CHANGELOG.md +41 -0
  2. package/dist/dashboard/server.d.ts.map +1 -1
  3. package/dist/dashboard/server.js +9 -0
  4. package/dist/dashboard/server.js.map +1 -1
  5. package/dist/lib/migrations.d.ts +2 -0
  6. package/dist/lib/migrations.d.ts.map +1 -1
  7. package/dist/lib/migrations.js +49 -4
  8. package/dist/lib/migrations.js.map +1 -1
  9. package/dist/lib/paths.d.ts +26 -0
  10. package/dist/lib/paths.d.ts.map +1 -0
  11. package/dist/lib/paths.js +68 -0
  12. package/dist/lib/paths.js.map +1 -0
  13. package/dist/lib/rate-limit.d.ts +25 -0
  14. package/dist/lib/rate-limit.d.ts.map +1 -0
  15. package/dist/lib/rate-limit.js +75 -0
  16. package/dist/lib/rate-limit.js.map +1 -0
  17. package/dist/models/decision_drafts.d.ts.map +1 -1
  18. package/dist/models/decision_drafts.js +2 -1
  19. package/dist/models/decision_drafts.js.map +1 -1
  20. package/dist/models/decisions.d.ts.map +1 -1
  21. package/dist/models/decisions.js +12 -3
  22. package/dist/models/decisions.js.map +1 -1
  23. package/dist/models/notes.d.ts.map +1 -1
  24. package/dist/models/notes.js +2 -1
  25. package/dist/models/notes.js.map +1 -1
  26. package/dist/models/subtasks.d.ts.map +1 -1
  27. package/dist/models/subtasks.js +2 -1
  28. package/dist/models/subtasks.js.map +1 -1
  29. package/dist/server/tools/projects.d.ts.map +1 -1
  30. package/dist/server/tools/projects.js +16 -2
  31. package/dist/server/tools/projects.js.map +1 -1
  32. package/dist/server/tools/workflow.d.ts.map +1 -1
  33. package/dist/server/tools/workflow.js +27 -3
  34. package/dist/server/tools/workflow.js.map +1 -1
  35. package/dist/services/auth/magic-link.d.ts.map +1 -1
  36. package/dist/services/auth/magic-link.js +3 -0
  37. package/dist/services/auth/magic-link.js.map +1 -1
  38. package/docs/configuration.md +21 -0
  39. package/docs/deploy-gcp-free.md +23 -1
  40. package/docs/recovery-runbook.md +162 -0
  41. package/package.json +3 -2
@@ -37,6 +37,17 @@ Loopback binds skip the middleware entirely so local development stays zero-fric
37
37
 
38
38
  Npm-tier plugins are discovered from `node_modules/@shinobi/plugin-*` in the current working directory at server start.
39
39
 
40
+ ## Rate limits
41
+
42
+ Public-facing routes are rate limited per client IP (`CF-Connecting-IP` /
43
+ `X-Forwarded-For`, falling back to a shared local bucket). Both windows are
44
+ one minute.
45
+
46
+ | Variable | Default | Applies to |
47
+ |---|---|---|
48
+ | `SHINOBI_MCP_RATE_LIMIT` | `240` | `POST /mcp` tool calls |
49
+ | `SHINOBI_AUTH_RATE_LIMIT` | `10` | `/api/auth/magic-link`, `/api/auth/verify` |
50
+
40
51
  ## Embeddings (optional)
41
52
 
42
53
  Shinobi falls back to FTS5 fulltext search by default. Configure a provider to enable semantic recall.
@@ -51,6 +62,16 @@ Shinobi falls back to FTS5 fulltext search by default. Configure a provider to e
51
62
 
52
63
  When a provider is configured, write paths (`log_decision`, `log_dead_end`, `add_note`, `create_task`, `bulk_create_tasks`) generate and store embeddings synchronously after the row insert. Read paths (`recall`, `check_dead_ends`) prefer semantic match and fall back to FTS5 when no embedding is available.
53
64
 
65
+ ## Session summaries (LLM optional)
66
+
67
+ `session_closeout` always refreshes the project's `recent_summary_md`. With
68
+ `SHINOBI_LLM_PROVIDER` configured (e.g. `groq` + `GROQ_API_KEY`, or `openai`),
69
+ the server compresses recent activity, decisions and dead ends into a terse
70
+ digest. Without a provider it persists the agent-authored closeout summary
71
+ instead (`recent_summary_provider = agent:closeout`) — the calling agent is an
72
+ LLM too, so no external key is required for the brain to stay fresh.
73
+ `agent_bootstrap` flags a stale summary via `summary_stale` either way.
74
+
54
75
  ## Recall mode
55
76
 
56
77
  `recall` accepts an optional `mode` argument: `auto` (default), `semantic`, or `fulltext`. The `auto` mode picks semantic when an embedding provider is configured, otherwise fulltext.
@@ -152,7 +152,25 @@ claude mcp add --transport http shinobi https://shinobi.yourdomain.com/mcp \
152
152
  - **Upgrades**: `cd ~/shinobi && git pull && docker build -t shinobi . `
153
153
  then `docker stop shinobi && docker rm shinobi` and re-run the
154
154
  `docker run` command from step 3 (same token). The database lives on the
155
- volume and survives.
155
+ volume and survives. Or skip the manual dance entirely — see
156
+ **Auto-deploy** below.
157
+ - **Auto-deploy (optional)**: `scripts/vm-autodeploy.sh` turns the upgrade
158
+ into a cron job on the VM host. Every tick it fetches `origin/main`; when
159
+ new commits land it builds the image **while the old container keeps
160
+ serving**, swaps containers carrying over the existing `SHINOBI_*` and
161
+ provider env vars, health-checks the new one, and rolls back to the
162
+ previous image if it doesn't come up. Concurrent ticks are guarded by a
163
+ lock file. Enable it with:
164
+
165
+ ```bash
166
+ ( crontab -l 2>/dev/null; echo "*/5 * * * * $HOME/shinobi/scripts/vm-autodeploy.sh >> $HOME/shinobi-autodeploy.log 2>&1" ) | crontab -
167
+ tail -f ~/shinobi-autodeploy.log # watch the first deploy go through
168
+ ```
169
+
170
+ It runs on the **host**, never inside the container (a container cannot
171
+ `docker rm` itself mid-deploy). Worst-case deploy latency is one cron
172
+ interval + the e2-micro build time. Anything merged to `main` now reaches
173
+ production unattended — protect `main` accordingly (require PRs + green CI).
156
174
  - **Backups**: the volume survives container replacement but not project
157
175
  deletion. Wire `shinobi sync` to a private git repo (git is included in
158
176
  the image):
@@ -172,6 +190,10 @@ claude mcp add --transport http shinobi https://shinobi.yourdomain.com/mcp \
172
190
  The clone lives on the `/data` volume, so credentials and git config
173
191
  survive container upgrades.
174
192
 
193
+ Backups are only real once you've restored from one — see the
194
+ [recovery runbook](recovery-runbook.md) for the tested container/VM/database
195
+ restore procedures and a drill to run before you depend on this setup.
196
+
175
197
  > ⚠️ If the first push dies with `RPC failed; curl 55 ... unexpected
176
198
  > disconnect` (common when sending multi-MB binary snapshots from small
177
199
  > VMs over HTTP/2), pin the repo to HTTP/1.1 and raise the post buffer —
@@ -0,0 +1,162 @@
1
+ # Recovery runbook — when the brain goes down
2
+
3
+ The Shinobi server is the single shared brain: tasks, decisions, dead ends,
4
+ plans. This runbook is the tested path to bring it back when the container,
5
+ the VM, or the volume dies — written so it can be followed from a phone
6
+ (GCP browser SSH) on a bad beach connection.
7
+
8
+ It assumes the [GCP free-tier deploy](deploy-gcp-free.md): Docker container
9
+ `shinobi`, named volume `shinobi-data` mounted at `/data`, Cloudflare Tunnel
10
+ publishing `shinobi.yourdomain.com → 127.0.0.1:8765`, and nightly
11
+ `shinobi sync push` to a private GitHub snapshot repo.
12
+
13
+ ## Before you leave (preflight checklist)
14
+
15
+ Recovery is only as good as what you can reach from a phone. Verify once,
16
+ at home:
17
+
18
+ - [ ] **Dashboard token** is in your password manager (every client and the
19
+ restore procedure need it).
20
+ - [ ] **Sync-repo PAT** (fine-grained, Contents read/write on the sync repo
21
+ only) is in your password manager — required to re-clone the backup.
22
+ - [ ] **GCP console** works on your phone (browser SSH included).
23
+ - [ ] **Cloudflare dashboard** login works on your phone.
24
+ - [ ] Last nightly backup is recent: `docker exec shinobi node /app/dist/cli.js sync status`
25
+ shows a `last push` within 24h, and the sync repo on GitHub shows the
26
+ commit.
27
+ - [ ] You ran the drill below at least once and know your restore time.
28
+
29
+ ## Diagnose first
30
+
31
+ From any device: `https://shinobi.yourdomain.com/health`
32
+
33
+ | Symptom | Likely failure | Procedure |
34
+ | --- | --- | --- |
35
+ | `{"ok":true}` but MCP calls 503 from cloud sessions | Cloudflare min TLS raised above 1.2 | [D](#d-cloudflare-issues) |
36
+ | 502/530 from Cloudflare | Container or VM down, tunnel up | [A](#a-container-down-vm-alive), then [B](#b-vm-dead-or-deleted) |
37
+ | `ERR_NAME_NOT_RESOLVED` / tunnel shows `Routes: --` | Tunnel route lost | [D](#d-cloudflare-issues) |
38
+ | VM gone from GCP console / SSH unreachable | VM dead | [B](#b-vm-dead-or-deleted) |
39
+ | Health ok but data missing/corrupt | Volume damage | [C](#c-restore-database-from-backup) |
40
+
41
+ ## A. Container down, VM alive
42
+
43
+ SSH to the VM (GCP console → VM instances → SSH):
44
+
45
+ ```bash
46
+ docker ps -a # is the container there, just stopped?
47
+ docker start shinobi # if stopped
48
+ docker logs --tail 50 shinobi
49
+ curl http://127.0.0.1:8765/health
50
+ ```
51
+
52
+ If the container is gone (e.g. after a failed upgrade), re-run it — the
53
+ database lives on the volume and survives:
54
+
55
+ ```bash
56
+ docker run -d --name shinobi --restart unless-stopped \
57
+ -p 127.0.0.1:8765:8765 \
58
+ -v shinobi-data:/data \
59
+ -e SHINOBI_DASHBOARD_TOKEN=<token from password manager> \
60
+ shinobi
61
+ ```
62
+
63
+ Expected time: **2–5 minutes**.
64
+
65
+ ## B. VM dead or deleted
66
+
67
+ Full rebuild on a fresh e2-micro. This is the long path — the Docker build
68
+ alone takes 10–15 minutes on the free tier. Follow
69
+ [deploy-gcp-free.md](deploy-gcp-free.md) steps 1–3 (VM + swap + Docker +
70
+ clone + build + run), with one difference: **use your existing dashboard
71
+ token** from the password manager in the `docker run`, so every client keeps
72
+ working without reconfiguration.
73
+
74
+ Then restore the database — see [C](#c-restore-database-from-backup).
75
+
76
+ Then re-attach the tunnel: Cloudflare → Zero Trust → Networks → Connectors.
77
+ The old connector will show disconnected. Either reinstall `cloudflared` on
78
+ the new VM with the existing tunnel's install command (Connectors → your
79
+ tunnel → Configure → copy the `cloudflared service install eyJ...` command),
80
+ or create a fresh tunnel and re-add the published application route
81
+ (`shinobi.yourdomain.com` → HTTP `localhost:8765`). No DNS migration needed.
82
+
83
+ > ⚠️ Keep Cloudflare **Minimum TLS at 1.2** on the new setup — see
84
+ > [deploy-gcp-free.md](deploy-gcp-free.md) for why raising it to 1.3 kills
85
+ > all MCP calls from Claude cloud sessions.
86
+
87
+ Expected time: **30–45 minutes**, dominated by the Docker build.
88
+
89
+ ## C. Restore database from backup
90
+
91
+ The nightly `sync push` snapshots the DB into the private sync repo. To pull
92
+ that snapshot back into a running container:
93
+
94
+ ```bash
95
+ # one-time on a fresh volume: clone the sync repo with the PAT
96
+ docker exec shinobi git clone https://YOUR_PAT@github.com/you/your-sync-repo.git /data/sync
97
+ docker exec shinobi git -C /data/sync config user.name "shinobi-server"
98
+ docker exec shinobi git -C /data/sync config user.email "shinobi@localhost"
99
+ docker exec shinobi node /app/dist/cli.js sync init /data/sync
100
+
101
+ # restore: pulls the repo and replaces the live DB with the snapshot
102
+ docker exec shinobi node /app/dist/cli.js sync pull
103
+ docker restart shinobi
104
+ curl http://127.0.0.1:8765/health
105
+ ```
106
+
107
+ Mind the RPO: the snapshot is up to 24h old (nightly cron at 03:00).
108
+ Anything logged between the last push and the failure is gone — accept it,
109
+ or check whether a laptop session from that window can re-log key decisions.
110
+
111
+ After a restore, re-arm the nightly cron if the VM is new:
112
+
113
+ ```bash
114
+ ( crontab -l 2>/dev/null; echo '0 3 * * * docker exec shinobi node /app/dist/cli.js sync push' ) | crontab -
115
+ ```
116
+
117
+ ## D. Cloudflare issues
118
+
119
+ - **MCP 503s from cloud sessions only** (laptop works): someone or something
120
+ raised Minimum TLS. Cloudflare → SSL/TLS → Edge Certificates → set
121
+ **Minimum TLS Version = 1.2**.
122
+ - **Route lost** (`Routes: --` on the tunnel): re-add the published
123
+ application route — Subdomain `shinobi`, Service Type HTTP,
124
+ URL `localhost:8765`.
125
+ - **Cloudflare outage**: nothing to fix on your side. The brain and backups
126
+ are unaffected; wait it out (degraded mode below).
127
+
128
+ ## Verify recovery
129
+
130
+ From the phone, in order:
131
+
132
+ 1. `https://shinobi.yourdomain.com/health` → `{"ok":true,...}`.
133
+ 2. Dashboard loads with your token and shows your projects.
134
+ 3. An MCP client (claude.ai chat with the Shinobi connector, or a Claude
135
+ Code session) can run `list_projects` and sees real data.
136
+ 4. `docker exec shinobi node /app/dist/cli.js sync status` shows sync
137
+ configured, then run a manual `sync push` so a fresh snapshot exists.
138
+
139
+ ## Degraded mode — what still works while the brain is offline
140
+
141
+ - Coding sessions (laptop, cloud) **work normally** — they just lose the
142
+ shared memory tools. Code, git, deploys are unaffected.
143
+ - The discipline that matters: note decisions/dead-ends somewhere durable
144
+ (commit messages, a scratch file) and re-log them via `log_decision` /
145
+ `log_dead_end` once the brain is back. Don't stand up a second brain from
146
+ a local copy — two diverging SQLite files are worse than a 24h gap.
147
+ - Approvals/push notifications are down with the server; fall back to
148
+ reviewing PRs directly on GitHub mobile.
149
+
150
+ ## The drill (run at home, before the vacation)
151
+
152
+ 1. `docker stop shinobi` → confirm `/health` fails → run procedure A. ⏱
153
+ 2. `docker stop shinobi && docker rm shinobi` → procedure A's re-run path. ⏱
154
+ 3. Simulate volume loss on a **throwaway copy**: create a scratch volume,
155
+ run a second container against it on another port, restore into it with
156
+ procedure C. Never drill C against the live volume. ⏱
157
+ 4. From your phone only (laptop closed): GCP SSH login → `docker ps` →
158
+ Cloudflare dashboard login. If any login dance fails on the phone, fix
159
+ it now.
160
+ 5. Log the measured times and any surprises as dead ends/decisions in
161
+ project "Shinobi remote MCP" — that's the institutional memory this tool
162
+ exists for.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@shinobiapps/shinobi",
3
- "version": "0.2.0",
3
+ "version": "0.2.1",
4
4
  "publishConfig": {
5
5
  "access": "public"
6
6
  },
@@ -21,6 +21,7 @@
21
21
  "docs/github-integration.md",
22
22
  "docs/plugin-development.md",
23
23
  "docs/plugins.md",
24
+ "docs/recovery-runbook.md",
24
25
  "docs/relay.md",
25
26
  "docs/remote-access.md",
26
27
  "docs/remote-mcp.md",
@@ -46,7 +47,7 @@
46
47
  "smoke:mcp": "node --import tsx scripts/smoke-mcp-stdio.ts",
47
48
  "smoke:dashboard": "node --import tsx scripts/smoke-dashboard.ts",
48
49
  "smoke:mcp-http": "node --import tsx scripts/smoke-mcp-http.ts",
49
- "test": "npm run build && npm run smoke:tools && npm run smoke:mcp && npm run smoke:mcp-http && npm run smoke:dashboard",
50
+ "test": "npm run build && npm run test:vitest && npm run smoke:tools && npm run smoke:mcp && npm run smoke:mcp-http && npm run smoke:dashboard",
50
51
  "test:vitest": "vitest run",
51
52
  "test:watch": "vitest"
52
53
  },