kamal-dev 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.rspec +3 -0
- data/.standard.yml +3 -0
- data/CHANGELOG.md +508 -0
- data/LICENSE.txt +21 -0
- data/README.md +899 -0
- data/Rakefile +10 -0
- data/exe/kamal-dev +10 -0
- data/exe/plugin-kamal-dev +283 -0
- data/lib/kamal/cli/dev.rb +1192 -0
- data/lib/kamal/dev/builder.rb +332 -0
- data/lib/kamal/dev/compose_parser.rb +255 -0
- data/lib/kamal/dev/config.rb +359 -0
- data/lib/kamal/dev/devcontainer.rb +122 -0
- data/lib/kamal/dev/devcontainer_parser.rb +204 -0
- data/lib/kamal/dev/registry.rb +149 -0
- data/lib/kamal/dev/secrets_loader.rb +93 -0
- data/lib/kamal/dev/state_manager.rb +271 -0
- data/lib/kamal/dev/templates/dev-entrypoint.sh +44 -0
- data/lib/kamal/dev/templates/dev.yml +93 -0
- data/lib/kamal/dev/version.rb +7 -0
- data/lib/kamal/dev.rb +33 -0
- data/lib/kamal/providers/base.rb +121 -0
- data/lib/kamal/providers/upcloud.rb +299 -0
- data/lib/kamal-dev.rb +5 -0
- data/sig/kamal/dev.rbs +6 -0
- data/test_installer.sh +73 -0
- metadata +141 -0
data/README.md
ADDED
|
@@ -0,0 +1,899 @@
|
|
|
1
|
+
# Kamal::Dev
|
|
2
|
+
|
|
3
|
+
**Scale your development capacity horizontally with cloud-powered devcontainer workspaces.**
|
|
4
|
+
|
|
5
|
+
Kamal::Dev extends [Kamal](https://github.com/basecamp/kamal) to deploy and manage development container workspaces to cloud infrastructure. Deploy multiple parallel development environments for AI-assisted development, remote pair programming, or horizontal scaling of development tasks.
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
|
|
9
|
+
- 🚀 **One-command deployment** - Deploy devcontainers from `.devcontainer/devcontainer.json` specifications
|
|
10
|
+
- ☁️ **Multi-cloud support** - Pluggable provider architecture (UpCloud reference implementation)
|
|
11
|
+
- 💰 **Cost estimation** - Preview cloud costs before provisioning VMs
|
|
12
|
+
- 🔒 **Secrets injection** - Secure credential management via `.kamal/secrets` system
|
|
13
|
+
- 📦 **State tracking** - Atomic state file operations with file locking
|
|
14
|
+
- 🔄 **Lifecycle management** - Full control: deploy, list, stop, remove workspaces
|
|
15
|
+
- ⚙️ **Resource limits** - Enforce CPU/memory constraints per container
|
|
16
|
+
- 🎯 **Docker-native** - Direct Docker command generation from devcontainer specs
|
|
17
|
+
|
|
18
|
+
## Installation
|
|
19
|
+
|
|
20
|
+
Add to your application's Gemfile:
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
gem "kamal", "~> 2.0"
|
|
24
|
+
gem "kamal-dev"
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Then run:
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
bundle install
|
|
31
|
+
|
|
32
|
+
# Run the plugin installer to set up kamal dev commands
|
|
33
|
+
bundle exec plugin-kamal-dev
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
The installer will ask which method you prefer:
|
|
37
|
+
|
|
38
|
+
**Option 1 (Recommended): Patch gem executable**
|
|
39
|
+
- Patches the global `kamal` executable installed with the gem
|
|
40
|
+
- Creates a backup (`kamal.backup`) before patching
|
|
41
|
+
- Works with `kamal dev` and `bundle exec kamal dev`
|
|
42
|
+
- Global installation (available in all projects)
|
|
43
|
+
|
|
44
|
+
**Option 2: Create project binstub**
|
|
45
|
+
- Creates `bin/kamal` in your project directory
|
|
46
|
+
- Local to your project only
|
|
47
|
+
- Use with `bin/kamal dev`
|
|
48
|
+
|
|
49
|
+
**That's it!** After installation, you can use kamal dev commands.
|
|
50
|
+
|
|
51
|
+
### Alternative Setup Methods
|
|
52
|
+
|
|
53
|
+
If you prefer not to use the installer, you can manually set up kamal-dev:
|
|
54
|
+
|
|
55
|
+
<details>
|
|
56
|
+
<summary>Click to expand alternative setup options</summary>
|
|
57
|
+
|
|
58
|
+
**Option 1: Quick test (no installation)**
|
|
59
|
+
|
|
60
|
+
Use `bundle exec` with the `-r` flag:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
bundle exec ruby -rkamal-dev -S kamal dev deploy
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
**Option 2: Manual binstub edit**
|
|
67
|
+
|
|
68
|
+
Generate the binstub and edit it manually:
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
bundle binstubs kamal --force
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Then edit `bin/kamal` to add this line after the bundler setup:
|
|
75
|
+
|
|
76
|
+
```ruby
|
|
77
|
+
require "kamal-dev" # Add this line to load kamal-dev extension
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
**Option 3: Rails/Boot file require**
|
|
81
|
+
|
|
82
|
+
If your project has a boot file (e.g., Rails `config/boot.rb`), add:
|
|
83
|
+
|
|
84
|
+
```ruby
|
|
85
|
+
require "kamal-dev"
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Then use: `bundle exec kamal dev`
|
|
89
|
+
|
|
90
|
+
</details>
|
|
91
|
+
|
|
92
|
+
## Quick Start
|
|
93
|
+
|
|
94
|
+
**1. Generate configuration template:**
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
kamal dev init
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
This creates `config/dev.yml` with a complete template. Edit it to configure:
|
|
101
|
+
- Your cloud provider (currently UpCloud)
|
|
102
|
+
- VM size and region
|
|
103
|
+
- Number of workspaces
|
|
104
|
+
- Resource limits
|
|
105
|
+
|
|
106
|
+
**2. Set up secrets** (`.kamal/secrets`):
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
export UPCLOUD_USERNAME="your-username"
|
|
110
|
+
export UPCLOUD_PASSWORD="your-password"
|
|
111
|
+
export GITHUB_TOKEN="ghp_..."
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
**3. Deploy workspaces:**
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
# If you chose Option 1 (gem executable):
|
|
118
|
+
kamal dev deploy --count 3
|
|
119
|
+
# or: bundle exec kamal dev deploy --count 3
|
|
120
|
+
|
|
121
|
+
# If you chose Option 2 (binstub):
|
|
122
|
+
bin/kamal dev deploy --count 3
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
**4. List running workspaces:**
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
kamal dev list
|
|
129
|
+
|
|
130
|
+
# Output:
|
|
131
|
+
# NAME IP STATUS DEPLOYED AT
|
|
132
|
+
# myapp-dev-1 1.2.3.4 running 2025-11-16 10:30:00 UTC
|
|
133
|
+
# myapp-dev-2 2.3.4.5 running 2025-11-16 10:30:15 UTC
|
|
134
|
+
# myapp-dev-3 3.4.5.6 running 2025-11-16 10:30:30 UTC
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
**5. Stop/remove when done:**
|
|
138
|
+
|
|
139
|
+
```bash
|
|
140
|
+
kamal dev stop --all # Stop containers, keep VMs
|
|
141
|
+
kamal dev remove --all # Destroy VMs, cleanup state
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
## Configuration
|
|
145
|
+
|
|
146
|
+
### config/dev.yml Structure
|
|
147
|
+
|
|
148
|
+
```yaml
|
|
149
|
+
# Required fields
|
|
150
|
+
service: myapp-dev # Service name prefix
|
|
151
|
+
image: .devcontainer/devcontainer.json # Devcontainer spec or direct image
|
|
152
|
+
|
|
153
|
+
provider:
|
|
154
|
+
type: upcloud # Cloud provider (upcloud, hetzner, aws, gcp)
|
|
155
|
+
zone: us-nyc1 # Data center location
|
|
156
|
+
plan: 1xCPU-2GB # VM size/plan
|
|
157
|
+
|
|
158
|
+
# Optional fields
|
|
159
|
+
secrets: # Secrets to inject from .kamal/secrets
|
|
160
|
+
- GITHUB_TOKEN
|
|
161
|
+
- DATABASE_URL
|
|
162
|
+
|
|
163
|
+
secrets_file: .kamal/secrets # Custom secrets file path (default: .kamal/secrets)
|
|
164
|
+
|
|
165
|
+
ssh:
|
|
166
|
+
key_path: ~/.ssh/id_ed25519.pub # SSH public key (default: ~/.ssh/id_rsa.pub)
|
|
167
|
+
|
|
168
|
+
defaults:
|
|
169
|
+
cpus: 2 # Default CPU limit
|
|
170
|
+
memory: 4g # Default memory limit
|
|
171
|
+
memory_swap: 8g # Swap limit
|
|
172
|
+
|
|
173
|
+
vms:
|
|
174
|
+
count: 5 # Number of workspaces to deploy
|
|
175
|
+
spread: false # Colocate (false) or one per VM (true)
|
|
176
|
+
|
|
177
|
+
naming:
|
|
178
|
+
pattern: "{service}-{index}" # Container naming pattern
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
### Devcontainer.json Support
|
|
182
|
+
|
|
183
|
+
Kamal::Dev parses VS Code [devcontainer.json](https://containers.dev/) specifications and generates Docker run commands automatically:
|
|
184
|
+
|
|
185
|
+
**Supported properties:**
|
|
186
|
+
- `image` - Base Docker image
|
|
187
|
+
- `forwardPorts` - Port mappings (`-p 3000:3000`)
|
|
188
|
+
- `mounts` - Volume mounts (`-v source:target`)
|
|
189
|
+
- `containerEnv` - Environment variables (`-e KEY=value`)
|
|
190
|
+
- `runArgs` - Docker run flags (e.g., `--cpus=2`)
|
|
191
|
+
- `remoteUser` - Container user (`--user vscode`)
|
|
192
|
+
- `workspaceFolder` - Working directory (`-w /workspace`)
|
|
193
|
+
|
|
194
|
+
**Example devcontainer.json:**
|
|
195
|
+
|
|
196
|
+
```json
|
|
197
|
+
{
|
|
198
|
+
"image": "ruby:3.2",
|
|
199
|
+
"forwardPorts": [3000, 5432],
|
|
200
|
+
"containerEnv": {
|
|
201
|
+
"RAILS_ENV": "development"
|
|
202
|
+
},
|
|
203
|
+
"mounts": [
|
|
204
|
+
"source=${localWorkspaceFolder},target=/workspace,type=bind"
|
|
205
|
+
],
|
|
206
|
+
"remoteUser": "vscode",
|
|
207
|
+
"workspaceFolder": "/workspace"
|
|
208
|
+
}
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
## Docker Compose Support
|
|
212
|
+
|
|
213
|
+
Kamal::Dev supports deploying complex development stacks using Docker Compose, enabling multi-service deployments (app + database + cache + workers) with custom Dockerfiles.
|
|
214
|
+
|
|
215
|
+
### Registry Configuration
|
|
216
|
+
|
|
217
|
+
To build and push images, configure a container registry in `config/dev.yml`:
|
|
218
|
+
|
|
219
|
+
```yaml
|
|
220
|
+
service: myapp-dev
|
|
221
|
+
|
|
222
|
+
# Registry for image building and pushing
|
|
223
|
+
registry:
|
|
224
|
+
server: ghcr.io # or docker.io for Docker Hub
|
|
225
|
+
username: GITHUB_USER # ENV var name (not actual username)
|
|
226
|
+
password: GITHUB_TOKEN # ENV var name (not actual password)
|
|
227
|
+
|
|
228
|
+
provider:
|
|
229
|
+
type: upcloud
|
|
230
|
+
zone: us-nyc1
|
|
231
|
+
plan: 2xCPU-4GB
|
|
232
|
+
|
|
233
|
+
# Reference compose file from devcontainer
|
|
234
|
+
image: .devcontainer/devcontainer.json # which references compose.yaml
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
Then set credentials in `.kamal/secrets`:
|
|
238
|
+
|
|
239
|
+
```bash
|
|
240
|
+
export GITHUB_USER="your-github-username"
|
|
241
|
+
export GITHUB_TOKEN="ghp_your_personal_access_token"
|
|
242
|
+
export UPCLOUD_USERNAME="your-upcloud-username"
|
|
243
|
+
export UPCLOUD_PASSWORD="your-upcloud-password"
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
**Supported Registries:**
|
|
247
|
+
- GitHub Container Registry (GHCR): `server: ghcr.io`
|
|
248
|
+
- Docker Hub: `server: docker.io`
|
|
249
|
+
- Custom registries: `server: registry.example.com`
|
|
250
|
+
|
|
251
|
+
### Building and Pushing Images
|
|
252
|
+
|
|
253
|
+
**Build image from Dockerfile:**
|
|
254
|
+
|
|
255
|
+
```bash
|
|
256
|
+
kamal dev build
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
**Push image to registry:**
|
|
260
|
+
|
|
261
|
+
```bash
|
|
262
|
+
kamal dev push
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
**Build, push, and deploy in one command:**
|
|
266
|
+
|
|
267
|
+
```bash
|
|
268
|
+
kamal dev deploy --count 3
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
**Skip build or push:**
|
|
272
|
+
|
|
273
|
+
```bash
|
|
274
|
+
kamal dev deploy --skip-build # Use existing local image
|
|
275
|
+
kamal dev deploy --skip-push # Use local image, don't push to registry
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
**Tag strategies:**
|
|
279
|
+
|
|
280
|
+
Images are automatically tagged with:
|
|
281
|
+
- **Timestamp tag:** Unix timestamp (e.g., `1700000000`)
|
|
282
|
+
- **Git SHA tag:** Short commit hash (e.g., `abc123f`)
|
|
283
|
+
- **Custom tag:** Specify with `--tag` flag
|
|
284
|
+
|
|
285
|
+
### Multi-Service Deployment (Docker Compose)
|
|
286
|
+
|
|
287
|
+
Deploy full development stacks with multiple services:
|
|
288
|
+
|
|
289
|
+
**Example: Rails app with PostgreSQL**
|
|
290
|
+
|
|
291
|
+
`.devcontainer/devcontainer.json`:
|
|
292
|
+
```json
|
|
293
|
+
{
|
|
294
|
+
"dockerComposeFile": "compose.yaml",
|
|
295
|
+
"service": "app",
|
|
296
|
+
"workspaceFolder": "/workspace"
|
|
297
|
+
}
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
`.devcontainer/compose.yaml`:
|
|
301
|
+
```yaml
|
|
302
|
+
services:
|
|
303
|
+
app:
|
|
304
|
+
build:
|
|
305
|
+
context: ..
|
|
306
|
+
dockerfile: .devcontainer/Dockerfile
|
|
307
|
+
volumes:
|
|
308
|
+
- ../:/workspace:cached
|
|
309
|
+
environment:
|
|
310
|
+
DATABASE_URL: postgres://postgres:postgres@postgres:5432/myapp_dev
|
|
311
|
+
ports:
|
|
312
|
+
- "3000:3000"
|
|
313
|
+
depends_on:
|
|
314
|
+
- postgres
|
|
315
|
+
|
|
316
|
+
postgres:
|
|
317
|
+
image: postgres:16
|
|
318
|
+
volumes:
|
|
319
|
+
- postgres_data:/var/lib/postgresql/data
|
|
320
|
+
environment:
|
|
321
|
+
POSTGRES_PASSWORD: postgres
|
|
322
|
+
|
|
323
|
+
volumes:
|
|
324
|
+
postgres_data:
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
**Deployment workflow:**
|
|
328
|
+
|
|
329
|
+
1. **Build** - Builds app service image from Dockerfile
|
|
330
|
+
2. **Push** - Pushes image to registry (e.g., `ghcr.io/user/myapp-dev:abc123`)
|
|
331
|
+
3. **Transform** - Replaces `build:` with `image:` reference in compose.yaml
|
|
332
|
+
4. **Deploy** - Deploys full stack to each VM via `docker-compose up -d`
|
|
333
|
+
|
|
334
|
+
**Result:** Each VM gets an isolated stack (app + postgres + volumes)
|
|
335
|
+
|
|
336
|
+
```bash
|
|
337
|
+
# Deploy 3 isolated stacks
|
|
338
|
+
kamal dev deploy --count 3
|
|
339
|
+
|
|
340
|
+
# Each VM runs:
|
|
341
|
+
# - myapp-dev container (your app)
|
|
342
|
+
# - postgres container (isolated database)
|
|
343
|
+
# - Named volumes for persistence
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
**List all containers:**
|
|
347
|
+
|
|
348
|
+
```bash
|
|
349
|
+
kamal dev list
|
|
350
|
+
|
|
351
|
+
# Output includes all services:
|
|
352
|
+
# NAME IP STATUS DEPLOYED AT
|
|
353
|
+
# myapp-dev-1-app 1.2.3.4 running 2025-11-18 10:30:00
|
|
354
|
+
# myapp-dev-1-postgres 1.2.3.4 running 2025-11-18 10:30:00
|
|
355
|
+
# myapp-dev-2-app 2.3.4.5 running 2025-11-18 10:30:15
|
|
356
|
+
# myapp-dev-2-postgres 2.3.4.5 running 2025-11-18 10:30:15
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
### Compose File Requirements
|
|
360
|
+
|
|
361
|
+
**Supported features:**
|
|
362
|
+
- ✅ Services with `build:` sections (main app service)
|
|
363
|
+
- ✅ Services with `image:` references (postgres, redis, etc.)
|
|
364
|
+
- ✅ Build context (string or object format)
|
|
365
|
+
- ✅ Dockerfile path specification
|
|
366
|
+
- ✅ Environment variables, volumes, ports
|
|
367
|
+
- ✅ Service dependencies (`depends_on`)
|
|
368
|
+
- ✅ Named volumes
|
|
369
|
+
|
|
370
|
+
**Limitations (Phase 1):**
|
|
371
|
+
- ❌ Single architecture builds only (amd64)
|
|
372
|
+
- ❌ Advanced compose features (networks, configs, profiles)
|
|
373
|
+
- ❌ Shared databases across VMs (each VM gets isolated stack)
|
|
374
|
+
|
|
375
|
+
**Main service detection:**
|
|
376
|
+
- First service with `build:` section is treated as main app service
|
|
377
|
+
- Only main service image is built and pushed to registry
|
|
378
|
+
- Dependent services (postgres, redis) use pre-built images
|
|
379
|
+
|
|
380
|
+
### Example: Full Stack Rails Application
|
|
381
|
+
|
|
382
|
+
**Directory structure:**
|
|
383
|
+
```
|
|
384
|
+
.devcontainer/
|
|
385
|
+
├── Dockerfile
|
|
386
|
+
├── devcontainer.json
|
|
387
|
+
└── compose.yaml
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
**Dockerfile:**
|
|
391
|
+
```dockerfile
|
|
392
|
+
FROM ruby:3.2
|
|
393
|
+
|
|
394
|
+
RUN apt-get update && apt-get install -y \
|
|
395
|
+
build-essential \
|
|
396
|
+
libpq-dev \
|
|
397
|
+
nodejs \
|
|
398
|
+
yarn
|
|
399
|
+
|
|
400
|
+
WORKDIR /workspace
|
|
401
|
+
|
|
402
|
+
COPY Gemfile* ./
|
|
403
|
+
RUN bundle install
|
|
404
|
+
|
|
405
|
+
CMD ["bundle", "exec", "rails", "server", "-b", "0.0.0.0"]
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
**compose.yaml:**
|
|
409
|
+
```yaml
|
|
410
|
+
services:
|
|
411
|
+
app:
|
|
412
|
+
build:
|
|
413
|
+
context: ..
|
|
414
|
+
dockerfile: .devcontainer/Dockerfile
|
|
415
|
+
volumes:
|
|
416
|
+
- ../:/workspace:cached
|
|
417
|
+
environment:
|
|
418
|
+
DATABASE_URL: postgres://postgres:postgres@postgres:5432/myapp_dev
|
|
419
|
+
REDIS_URL: redis://redis:6379/0
|
|
420
|
+
ports:
|
|
421
|
+
- "3000:3000"
|
|
422
|
+
depends_on:
|
|
423
|
+
- postgres
|
|
424
|
+
- redis
|
|
425
|
+
|
|
426
|
+
postgres:
|
|
427
|
+
image: postgres:16
|
|
428
|
+
volumes:
|
|
429
|
+
- postgres_data:/var/lib/postgresql/data
|
|
430
|
+
environment:
|
|
431
|
+
POSTGRES_PASSWORD: postgres
|
|
432
|
+
|
|
433
|
+
redis:
|
|
434
|
+
image: redis:7-alpine
|
|
435
|
+
volumes:
|
|
436
|
+
- redis_data:/data
|
|
437
|
+
|
|
438
|
+
volumes:
|
|
439
|
+
postgres_data:
|
|
440
|
+
redis_data:
|
|
441
|
+
```
|
|
442
|
+
|
|
443
|
+
**Deploy:**
|
|
444
|
+
```bash
|
|
445
|
+
kamal dev deploy --count 2
|
|
446
|
+
|
|
447
|
+
# Builds app image
|
|
448
|
+
# Pushes to ghcr.io/user/myapp-dev:abc123
|
|
449
|
+
# Deploys 2 isolated stacks (each with app + postgres + redis)
|
|
450
|
+
```
|
|
451
|
+
|
|
452
|
+
### Troubleshooting Compose Deployments
|
|
453
|
+
|
|
454
|
+
**Build failures:**
|
|
455
|
+
- Check Dockerfile syntax
|
|
456
|
+
- Verify build context path
|
|
457
|
+
- Review build args and secrets
|
|
458
|
+
- Enable verbose mode: `VERBOSE=1 kamal dev build`
|
|
459
|
+
|
|
460
|
+
**Push failures:**
|
|
461
|
+
- Verify registry credentials in `.kamal/secrets`
|
|
462
|
+
- Check GHCR token has `write:packages` permission
|
|
463
|
+
- Ensure image name follows registry conventions
|
|
464
|
+
|
|
465
|
+
**Deploy failures:**
|
|
466
|
+
- Check transformed compose.yaml: `.kamal/dev_transformed_compose.yaml`
|
|
467
|
+
- Verify all service images are accessible
|
|
468
|
+
- Review volume mount paths
|
|
469
|
+
- Check for port conflicts between services
|
|
470
|
+
|
|
471
|
+
### Secrets Management
|
|
472
|
+
|
|
473
|
+
Secrets are loaded from `.kamal/secrets` (shell script with `export` statements) and injected into containers as Base64-encoded environment variables.
|
|
474
|
+
|
|
475
|
+
**.kamal/secrets:**
|
|
476
|
+
|
|
477
|
+
```bash
|
|
478
|
+
export GITHUB_TOKEN="ghp_your_token_here"
|
|
479
|
+
export DATABASE_URL="postgres://user:pass@host:5432/db"
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
**In container:**
|
|
483
|
+
|
|
484
|
+
```bash
|
|
485
|
+
# Secrets available as env vars
|
|
486
|
+
echo $GITHUB_TOKEN_B64 | base64 -d # Decode if needed
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
## Remote Code Sync (DevPod-Style)
|
|
490
|
+
|
|
491
|
+
Kamal-dev supports **DevPod-style remote development** where your code is cloned from a git repository into the container rather than mounted from your local machine. This is ideal for cloud-based development workflows.
|
|
492
|
+
|
|
493
|
+
### How It Works
|
|
494
|
+
|
|
495
|
+
When you configure the `git:` section in `config/dev.yml`:
|
|
496
|
+
|
|
497
|
+
1. **During image build**: A special entrypoint script (`dev-entrypoint.sh`) is injected into your Docker image
|
|
498
|
+
2. **On container startup**: The entrypoint clones your repository into the workspace folder
|
|
499
|
+
3. **For local development**: VS Code devcontainers work normally with mounted code (no git clone)
|
|
500
|
+
|
|
501
|
+
**Key benefits:**
|
|
502
|
+
- ✅ No local file mounting needed (pure cloud deployment)
|
|
503
|
+
- ✅ Code changes persist across container restarts
|
|
504
|
+
- ✅ Supports private repositories via GitHub Personal Access Token (PAT)
|
|
505
|
+
- ✅ Automatic credential caching for git operations (pull/push)
|
|
506
|
+
|
|
507
|
+
### Setup Instructions
|
|
508
|
+
|
|
509
|
+
**Step 1: Generate GitHub Personal Access Token**
|
|
510
|
+
|
|
511
|
+
1. Go to [GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)](https://github.com/settings/tokens)
|
|
512
|
+
2. Click **Generate new token (classic)**
|
|
513
|
+
3. Give it a name: "kamal-dev deployment"
|
|
514
|
+
4. Select scopes:
|
|
515
|
+
- ✅ `repo` (Full control of private repositories)
|
|
516
|
+
5. Click **Generate token**
|
|
517
|
+
6. **Copy the token** (starts with `ghp_...`) - you won't see it again
|
|
518
|
+
|
|
519
|
+
**Step 2: Add token to secrets file**
|
|
520
|
+
|
|
521
|
+
Add your token to `.kamal/secrets`:
|
|
522
|
+
|
|
523
|
+
```bash
|
|
524
|
+
export GITHUB_TOKEN="ghp_xxxxxxxxxxxxxxxxxxxx"
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
**Important**: Ensure the variable is **exported** so it's available to Ruby processes.
|
|
528
|
+
|
|
529
|
+
**Step 3: Configure git clone in config/dev.yml**
|
|
530
|
+
|
|
531
|
+
```yaml
|
|
532
|
+
git:
|
|
533
|
+
repository: https://github.com/yourorg/yourrepo.git # HTTPS URL (not SSH)
|
|
534
|
+
branch: main # Branch to checkout
|
|
535
|
+
workspace_folder: /workspaces/myapp # Where to clone code
|
|
536
|
+
token: GITHUB_TOKEN # Environment variable name
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
**Step 4: Deploy**
|
|
540
|
+
|
|
541
|
+
```bash
|
|
542
|
+
kamal dev deploy --count 2
|
|
543
|
+
```
|
|
544
|
+
|
|
545
|
+
The deployment process will:
|
|
546
|
+
1. Build your image with the entrypoint script injected
|
|
547
|
+
2. Push to registry
|
|
548
|
+
3. Deploy containers with git environment variables
|
|
549
|
+
4. On first boot, containers clone your repository
|
|
550
|
+
|
|
551
|
+
### Verification
|
|
552
|
+
|
|
553
|
+
**Check if code was cloned:**
|
|
554
|
+
|
|
555
|
+
```bash
|
|
556
|
+
# SSH into VM
|
|
557
|
+
ssh root@<vm-ip>
|
|
558
|
+
|
|
559
|
+
# Check container logs
|
|
560
|
+
docker logs myapp-dev-1-app
|
|
561
|
+
|
|
562
|
+
# Should see:
|
|
563
|
+
# [kamal-dev] Remote deployment detected
|
|
564
|
+
# [kamal-dev] Cloning https://github.com/yourorg/yourrepo.git (branch: main)
|
|
565
|
+
# [kamal-dev] Clone complete: /workspaces/myapp
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
**Verify git authentication is cached:**
|
|
569
|
+
|
|
570
|
+
```bash
|
|
571
|
+
# Exec into container
|
|
572
|
+
docker exec -it myapp-dev-1-app bash
|
|
573
|
+
|
|
574
|
+
# Try pulling
|
|
575
|
+
cd /workspaces/myapp
|
|
576
|
+
git pull
|
|
577
|
+
|
|
578
|
+
# Should succeed without prompting for credentials
|
|
579
|
+
```
|
|
580
|
+
|
|
581
|
+
### Important Notes
|
|
582
|
+
|
|
583
|
+
- **Use HTTPS URLs**: `https://github.com/user/repo.git` (NOT `git@github.com:user/repo.git`)
|
|
584
|
+
- **Token security**: The token is injected as an environment variable and used only at startup for cloning
|
|
585
|
+
- **Credential caching**: Git credentials are stored in `~/.git-credentials` inside the container for future git operations
|
|
586
|
+
- **Local development**: If you use VS Code with devcontainer.json, the git clone is skipped - your local code is mounted instead
|
|
587
|
+
- **Token scopes**: For private repos, you need the `repo` scope. For public repos, no token is needed.
|
|
588
|
+
|
|
589
|
+
### Troubleshooting
|
|
590
|
+
|
|
591
|
+
**"fatal: could not read Username for 'https://github.com'"**
|
|
592
|
+
- Verify `GITHUB_TOKEN` is in `.kamal/secrets`
|
|
593
|
+
- Ensure the variable is **exported** (`export GITHUB_TOKEN=...`)
|
|
594
|
+
- Check the token has `repo` scope for private repositories
|
|
595
|
+
|
|
596
|
+
**"Permission denied" when cloning**
|
|
597
|
+
- Check the token hasn't expired (GitHub tokens can have expiration dates)
|
|
598
|
+
- Verify the token has access to the repository (check repo permissions)
|
|
599
|
+
- Ensure you're using HTTPS URL, not SSH format
|
|
600
|
+
|
|
601
|
+
**Code not appearing in /workspaces**
|
|
602
|
+
- Check container logs: `docker logs <container-name>`
|
|
603
|
+
- Verify workspace_folder matches devcontainer.json `workspaceFolder`
|
|
604
|
+
- Ensure git repository URL is accessible
|
|
605
|
+
|
|
606
|
+
## Commands Reference
|
|
607
|
+
|
|
608
|
+
All commands below assume you've run `bundle exec plugin-kamal-dev` as described in the Installation section. If you're using an alternative setup method, adjust the commands accordingly (see Alternative Setup Methods in Installation).
|
|
609
|
+
|
|
610
|
+
### init
|
|
611
|
+
|
|
612
|
+
Generate a configuration template.
|
|
613
|
+
|
|
614
|
+
```bash
|
|
615
|
+
kamal dev init
|
|
616
|
+
```
|
|
617
|
+
|
|
618
|
+
**What it does:**
|
|
619
|
+
1. Creates `config/` directory if it doesn't exist
|
|
620
|
+
2. Copies template to `config/dev.yml`
|
|
621
|
+
3. Prompts before overwriting if file already exists
|
|
622
|
+
4. Displays next steps for configuration
|
|
623
|
+
|
|
624
|
+
**Example output:**
|
|
625
|
+
```
|
|
626
|
+
✅ Created config/dev.yml
|
|
627
|
+
|
|
628
|
+
Next steps:
|
|
629
|
+
|
|
630
|
+
1. Edit config/dev.yml with your cloud provider credentials
|
|
631
|
+
2. Create .kamal/secrets file with your secrets
|
|
632
|
+
3. Deploy your first workspace: kamal dev deploy --count 3
|
|
633
|
+
```
|
|
634
|
+
|
|
635
|
+
### deploy
|
|
636
|
+
|
|
637
|
+
Deploy devcontainer workspaces to cloud VMs.
|
|
638
|
+
|
|
639
|
+
```bash
|
|
640
|
+
kamal dev deploy [OPTIONS]
|
|
641
|
+
|
|
642
|
+
Options:
|
|
643
|
+
--count N Number of containers to deploy (default: from config)
|
|
644
|
+
--from PATH Path to devcontainer.json (default: from config)
|
|
645
|
+
--config PATH Path to config file (default: config/dev.yml)
|
|
646
|
+
```
|
|
647
|
+
|
|
648
|
+
**What it does:**
|
|
649
|
+
1. Loads configuration and devcontainer spec
|
|
650
|
+
2. Estimates cloud costs → prompts for confirmation
|
|
651
|
+
3. Provisions VMs via cloud provider API
|
|
652
|
+
4. Bootstraps Docker on VMs (if not installed)
|
|
653
|
+
5. Deploys containers with injected secrets
|
|
654
|
+
6. Saves state to `.kamal/dev_state.yml`
|
|
655
|
+
|
|
656
|
+
### list
|
|
657
|
+
|
|
658
|
+
List deployed devcontainer workspaces.
|
|
659
|
+
|
|
660
|
+
```bash
|
|
661
|
+
kamal dev list [OPTIONS]
|
|
662
|
+
|
|
663
|
+
Options:
|
|
664
|
+
--format FORMAT Output format: table (default), json, yaml
|
|
665
|
+
```
|
|
666
|
+
|
|
667
|
+
**Example output:**
|
|
668
|
+
|
|
669
|
+
```
|
|
670
|
+
NAME IP STATUS DEPLOYED AT
|
|
671
|
+
myapp-dev-1 1.2.3.4 running 2025-11-16 10:30:00 UTC
|
|
672
|
+
myapp-dev-2 2.3.4.5 stopped 2025-11-16 10:30:15 UTC
|
|
673
|
+
```
|
|
674
|
+
|
|
675
|
+
### stop
|
|
676
|
+
|
|
677
|
+
Stop devcontainer(s) without destroying VMs.
|
|
678
|
+
|
|
679
|
+
```bash
|
|
680
|
+
kamal dev stop [NAME] [OPTIONS]
|
|
681
|
+
|
|
682
|
+
Arguments:
|
|
683
|
+
NAME Container name to stop (optional)
|
|
684
|
+
|
|
685
|
+
Options:
|
|
686
|
+
--all Stop all containers
|
|
687
|
+
```
|
|
688
|
+
|
|
689
|
+
**What it does:**
|
|
690
|
+
- Executes `docker stop {container}` via SSH
|
|
691
|
+
- Updates state file: status → "stopped"
|
|
692
|
+
- VMs remain running (reduces restart time)
|
|
693
|
+
|
|
694
|
+
### remove
|
|
695
|
+
|
|
696
|
+
Destroy VMs and remove container state.
|
|
697
|
+
|
|
698
|
+
```bash
|
|
699
|
+
kamal dev remove [NAME] [OPTIONS]
|
|
700
|
+
|
|
701
|
+
Arguments:
|
|
702
|
+
NAME Container name to remove (optional)
|
|
703
|
+
|
|
704
|
+
Options:
|
|
705
|
+
--all Remove all containers
|
|
706
|
+
--force Skip confirmation prompt
|
|
707
|
+
```
|
|
708
|
+
|
|
709
|
+
**What it does:**
|
|
710
|
+
1. Prompts for confirmation (unless `--force`)
|
|
711
|
+
2. Stops containers via `docker stop`
|
|
712
|
+
3. Destroys VMs via provider API
|
|
713
|
+
4. Removes entries from state file
|
|
714
|
+
5. Deletes state file if empty
|
|
715
|
+
|
|
716
|
+
### status
|
|
717
|
+
|
|
718
|
+
Show detailed status of devcontainer(s).
|
|
719
|
+
|
|
720
|
+
```bash
|
|
721
|
+
kamal dev status [NAME] [OPTIONS]
|
|
722
|
+
|
|
723
|
+
Arguments:
|
|
724
|
+
NAME Container name to check (optional)
|
|
725
|
+
|
|
726
|
+
Options:
|
|
727
|
+
--all Show status for all containers
|
|
728
|
+
```
|
|
729
|
+
|
|
730
|
+
## Troubleshooting
|
|
731
|
+
|
|
732
|
+
### VM Provisioning Fails
|
|
733
|
+
|
|
734
|
+
**Problem:** `ProvisioningError: VM failed to reach running state`
|
|
735
|
+
|
|
736
|
+
**Solutions:**
|
|
737
|
+
- Check provider API credentials in `.kamal/secrets`
|
|
738
|
+
- Verify zone/region availability
|
|
739
|
+
- Check account quotas (VMs, storage, IPs)
|
|
740
|
+
- Try smaller VM plan (e.g., 1xCPU-1GB)
|
|
741
|
+
|
|
742
|
+
### Container Won't Start
|
|
743
|
+
|
|
744
|
+
**Problem:** Container status shows "failed"
|
|
745
|
+
|
|
746
|
+
**Solutions:**
|
|
747
|
+
- Check image name in devcontainer.json
|
|
748
|
+
- Verify secrets are valid (Base64 encoding issues)
|
|
749
|
+
- SSH to VM and check Docker logs: `ssh root@{vm_ip} docker logs {container}`
|
|
750
|
+
- Review resource limits (may be too restrictive)
|
|
751
|
+
|
|
752
|
+
### State File Corruption
|
|
753
|
+
|
|
754
|
+
**Problem:** "State file appears corrupted or locked"
|
|
755
|
+
|
|
756
|
+
**Solutions:**
|
|
757
|
+
```bash
|
|
758
|
+
# Check for lock file
|
|
759
|
+
ls -la .kamal/dev_state.yml.lock
|
|
760
|
+
|
|
761
|
+
# Remove stale lock (if no processes using it)
|
|
762
|
+
rm .kamal/dev_state.yml.lock
|
|
763
|
+
|
|
764
|
+
# Rebuild state from provider dashboard
|
|
765
|
+
kamal dev list --rebuild # (future feature)
|
|
766
|
+
```
|
|
767
|
+
|
|
768
|
+
### SSH Key Not Found
|
|
769
|
+
|
|
770
|
+
**Problem:** "SSH public key not found at ~/.ssh/id_rsa.pub"
|
|
771
|
+
|
|
772
|
+
**Solutions:**
|
|
773
|
+
```yaml
|
|
774
|
+
# Configure custom SSH key in config/dev.yml
|
|
775
|
+
ssh:
|
|
776
|
+
key_path: ~/.ssh/id_ed25519.pub
|
|
777
|
+
```
|
|
778
|
+
|
|
779
|
+
Or generate new SSH key:
|
|
780
|
+
```bash
|
|
781
|
+
ssh-keygen -t ed25519 -C "kamal-dev@example.com"
|
|
782
|
+
```
|
|
783
|
+
|
|
784
|
+
### Debug Mode
|
|
785
|
+
|
|
786
|
+
Enable verbose logging:
|
|
787
|
+
|
|
788
|
+
```bash
|
|
789
|
+
VERBOSE=1 kamal dev deploy --count 2
|
|
790
|
+
```
|
|
791
|
+
|
|
792
|
+
## Development
|
|
793
|
+
|
|
794
|
+
After checking out the repo:
|
|
795
|
+
|
|
796
|
+
```bash
|
|
797
|
+
# Install dependencies
|
|
798
|
+
bin/setup
|
|
799
|
+
|
|
800
|
+
# Run tests
|
|
801
|
+
bundle exec rspec
|
|
802
|
+
|
|
803
|
+
# Run linter
|
|
804
|
+
bundle exec standardrb
|
|
805
|
+
|
|
806
|
+
# Run full suite (tests + linter)
|
|
807
|
+
bundle exec rake
|
|
808
|
+
|
|
809
|
+
# Interactive console
|
|
810
|
+
bin/console
|
|
811
|
+
|
|
812
|
+
# Install locally for testing
|
|
813
|
+
bundle exec rake install
|
|
814
|
+
```
|
|
815
|
+
|
|
816
|
+
### Integration Tests
|
|
817
|
+
|
|
818
|
+
Integration tests provision real VMs (costs money). Set up test credentials:
|
|
819
|
+
|
|
820
|
+
```bash
|
|
821
|
+
# Create .kamal/secrets with UpCloud test account
|
|
822
|
+
export UPCLOUD_USERNAME="test-user"
|
|
823
|
+
export UPCLOUD_PASSWORD="test-password"
|
|
824
|
+
|
|
825
|
+
# Run integration tests
|
|
826
|
+
INTEGRATION_TESTS=1 bundle exec rspec
|
|
827
|
+
```
|
|
828
|
+
|
|
829
|
+
**⚠️ Warning:** Integration tests will provision and destroy VMs. Estimated cost: ~$0.01-0.05 per test run.
|
|
830
|
+
|
|
831
|
+
## Architecture
|
|
832
|
+
|
|
833
|
+
### Provider Adapter Pattern
|
|
834
|
+
|
|
835
|
+
```
|
|
836
|
+
┌─────────────┐
|
|
837
|
+
│ CLI │
|
|
838
|
+
│ Commands │
|
|
839
|
+
└──────┬──────┘
|
|
840
|
+
│
|
|
841
|
+
▼
|
|
842
|
+
┌──────────────────┐ ┌─────────────────┐
|
|
843
|
+
│ Provider::Base │◄─────┤ DevConfig │
|
|
844
|
+
│ (interface) │ │ (configuration) │
|
|
845
|
+
└────────┬─────────┘ └─────────────────┘
|
|
846
|
+
│
|
|
847
|
+
├──► Provider::Upcloud
|
|
848
|
+
├──► Provider::Hetzner (future)
|
|
849
|
+
├──► Provider::AWS (future)
|
|
850
|
+
└──► Provider::GCP (future)
|
|
851
|
+
```
|
|
852
|
+
|
|
853
|
+
### State Management
|
|
854
|
+
|
|
855
|
+
State is tracked in `.kamal/dev_state.yml` with file locking to prevent corruption:
|
|
856
|
+
|
|
857
|
+
```yaml
|
|
858
|
+
deployments:
|
|
859
|
+
myapp-dev-1:
|
|
860
|
+
vm_id: "00abc123-def4-5678-90ab-cdef12345678"
|
|
861
|
+
vm_ip: "1.2.3.4"
|
|
862
|
+
container_name: "myapp-dev-1"
|
|
863
|
+
status: running
|
|
864
|
+
deployed_at: "2025-11-16T14:30:00Z"
|
|
865
|
+
```
|
|
866
|
+
|
|
867
|
+
**File locking:**
|
|
868
|
+
- Uses `File.flock(File::LOCK_EX)` for exclusive writes
|
|
869
|
+
- Uses `File.flock(File::LOCK_SH)` for shared reads
|
|
870
|
+
- NFS-compatible dotlock fallback
|
|
871
|
+
|
|
872
|
+
## Roadmap
|
|
873
|
+
|
|
874
|
+
- [ ] Hetzner Cloud provider adapter
|
|
875
|
+
- [ ] AWS EC2 provider adapter
|
|
876
|
+
- [ ] GCP Compute Engine provider adapter
|
|
877
|
+
- [ ] Multi-project workspace sharing
|
|
878
|
+
- [ ] Automatic workspace hibernation (cost optimization)
|
|
879
|
+
- [ ] Devcontainer features support (via devcontainer CLI)
|
|
880
|
+
- [ ] Web UI for workspace management
|
|
881
|
+
|
|
882
|
+
## Contributing
|
|
883
|
+
|
|
884
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/ljuti/kamal-dev.
|
|
885
|
+
|
|
886
|
+
**Before submitting a PR:**
|
|
887
|
+
1. Run full test suite: `bundle exec rake`
|
|
888
|
+
2. Ensure linter passes: `bundle exec standardrb`
|
|
889
|
+
3. Add tests for new features
|
|
890
|
+
4. Update CHANGELOG.md
|
|
891
|
+
5. Update documentation (README, YARD comments)
|
|
892
|
+
|
|
893
|
+
## License
|
|
894
|
+
|
|
895
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
|
896
|
+
|
|
897
|
+
## Credits
|
|
898
|
+
|
|
899
|
+
Built as an extension to [Kamal](https://github.com/basecamp/kamal) by Basecamp.
|