sandcms 0.6.3 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -11
- package/dist/admin/Account.svelte +2 -0
- package/dist/admin/Admin.svelte +21 -6
- package/dist/admin/AssetAttachment.svelte +9 -6
- package/dist/admin/AssetAttachment.svelte.d.ts +3 -1
- package/dist/admin/BlueprintFields.svelte +0 -3
- package/dist/admin/BlueprintFields.svelte.d.ts +0 -1
- package/dist/admin/Editor.svelte +5 -10
- package/dist/admin/Fields.svelte +2 -8
- package/dist/admin/Fields.svelte.d.ts +0 -1
- package/dist/admin/Gallery.svelte +221 -80
- package/dist/admin/Gallery.svelte.d.ts +0 -1
- package/dist/admin/Inbox.svelte +3 -2
- package/dist/admin/LinkField.svelte +0 -3
- package/dist/admin/LinkField.svelte.d.ts +0 -1
- package/dist/admin/MediaLibrary.svelte +10 -7
- package/dist/admin/MediaPicker.svelte +43 -35
- package/dist/admin/MediaPicker.svelte.d.ts +0 -1
- package/dist/admin/Navigation.svelte +13 -2
- package/dist/admin/Pages.svelte +195 -0
- package/dist/admin/Pages.svelte.d.ts +7 -0
- package/dist/admin/Shell.svelte +27 -5
- package/dist/admin/SortableList.svelte +82 -61
- package/dist/admin/SortableList.svelte.d.ts +3 -1
- package/dist/admin/ThemePicker.svelte +72 -0
- package/dist/admin/ThemePicker.svelte.d.ts +18 -0
- package/dist/admin/UploadButton.svelte +2 -1
- package/dist/admin/UploadProgress.svelte +24 -0
- package/dist/admin/UploadProgress.svelte.d.ts +7 -0
- package/dist/admin/Users.svelte +8 -3
- package/dist/admin/client.d.ts +3 -670
- package/dist/admin/client.js +2 -1
- package/dist/admin/components/ui/alert/alert.svelte.d.ts +3 -13
- package/dist/admin/components/ui/attachment/attachment-media.svelte.d.ts +3 -13
- package/dist/admin/components/ui/attachment/attachment.svelte.d.ts +6 -26
- package/dist/admin/components/ui/badge/badge.svelte.d.ts +7 -25
- package/dist/admin/components/ui/button/button.svelte.d.ts +15 -53
- package/dist/admin/components/ui/toggle/toggle.svelte.d.ts +6 -26
- package/dist/admin/generated/admin.css +823 -115
- package/dist/admin/navigation.d.ts +9 -0
- package/dist/admin/navigation.js +20 -1
- package/dist/admin/theme-context.d.ts +7 -0
- package/dist/admin/theme-context.js +3 -0
- package/dist/admin/theme.d.ts +7 -0
- package/dist/admin/theme.js +14 -0
- package/dist/admin/types.d.ts +3 -1
- package/dist/admin/uploads.d.ts +8 -1
- package/dist/admin/uploads.js +94 -16
- package/dist/blueprints/files.js +1 -0
- package/dist/blueprints/labels.d.ts +6 -0
- package/dist/blueprints/labels.js +14 -0
- package/dist/blueprints/load.js +16 -2
- package/dist/blueprints/types.d.ts +4 -1
- package/dist/cli/admin.d.ts +39 -0
- package/dist/cli/admin.js +95 -0
- package/dist/cli/backup.js +4 -4
- package/dist/cli/doctor.d.ts +10 -1
- package/dist/cli/doctor.js +98 -33
- package/dist/cli/email-test.d.ts +3 -0
- package/dist/cli/email-test.js +20 -0
- package/dist/cli/files.d.ts +2 -6
- package/dist/cli/files.js +12 -2
- package/dist/cli/index.js +401 -95
- package/dist/cli/rebaseline.d.ts +10 -0
- package/dist/cli/rebaseline.js +117 -0
- package/dist/cli/scaffold.js +8 -8
- package/dist/cli/setup.js +16 -26
- package/dist/cli/upgrade.js +34 -7
- package/dist/env.d.ts +26 -2
- package/dist/env.js +48 -6
- package/dist/notification-fields.js +2 -1
- package/dist/server/admin-load.d.ts +85 -5
- package/dist/server/admin-load.js +26 -1
- package/dist/server/api-access.d.ts +4 -0
- package/dist/server/api-access.js +15 -2
- package/dist/server/api-keys.d.ts +96 -0
- package/dist/server/api-keys.js +115 -0
- package/dist/server/api-pagination.d.ts +3 -0
- package/dist/server/api.js +21 -63
- package/dist/server/auth.d.ts +10 -9
- package/dist/server/cms.d.ts +63 -21
- package/dist/server/cms.js +64 -7
- package/dist/server/db/auto-migrate.d.ts +0 -4
- package/dist/server/db/auto-migrate.js +12 -3
- package/dist/server/db/legacy.d.ts +13 -0
- package/dist/server/db/legacy.js +60 -0
- package/dist/server/db/migration-data.js +2 -34
- package/dist/server/db/migrations/{0000_foundation.sql → 0000_baseline.sql} +120 -4
- package/dist/server/db/migrations/meta/0000_snapshot.json +578 -3
- package/dist/server/db/migrations/meta/_journal.json +2 -58
- package/dist/server/db/schema.d.ts +119 -2
- package/dist/server/db/schema.js +23 -9
- package/dist/server/db/store.d.ts +25 -2
- package/dist/server/db/store.js +90 -26
- package/dist/server/forms.js +2 -1
- package/dist/server/hooks.js +155 -81
- package/dist/server/media/contracts.d.ts +115 -0
- package/dist/server/media/contracts.js +12 -0
- package/dist/server/media/deletions.d.ts +2 -1
- package/dist/server/media/deletions.js +3 -3
- package/dist/server/media/delivery.js +7 -11
- package/dist/server/media/processor-child.d.ts +2 -0
- package/dist/server/media/processor-child.js +236 -0
- package/dist/server/media/processor.d.ts +51 -0
- package/dist/server/media/processor.js +135 -0
- package/dist/server/media/render-math.d.ts +24 -0
- package/dist/server/media/render-math.js +50 -0
- package/dist/server/media/runner.d.ts +30 -1
- package/dist/server/media/runner.js +418 -165
- package/dist/server/media/service.d.ts +53 -11
- package/dist/server/media/service.js +307 -98
- package/dist/server/media/storage.d.ts +5 -0
- package/dist/server/media/storage.js +279 -0
- package/dist/server/media/upload-ticket.d.ts +12 -0
- package/dist/server/media/upload-ticket.js +35 -0
- package/dist/server/media/verify.d.ts +64 -0
- package/dist/server/media/verify.js +201 -0
- package/dist/server/platform/http.d.ts +27 -0
- package/dist/server/platform/http.js +77 -0
- package/dist/server/platform/index.d.ts +52 -0
- package/dist/server/platform/index.js +98 -0
- package/dist/server/platform/snapshots.d.ts +18 -0
- package/dist/server/platform/snapshots.js +110 -0
- package/dist/server/platform/volume.d.ts +20 -0
- package/dist/server/platform/volume.js +60 -0
- package/docs/acceptance.md +7 -5
- package/docs/api.md +56 -4
- package/docs/cloudflare.md +26 -152
- package/docs/content.md +20 -5
- package/docs/deployment.md +222 -57
- package/docs/email.md +6 -0
- package/docs/forms.md +2 -2
- package/docs/media.md +94 -21
- package/docs/package.md +21 -18
- package/docs/plans/0.7-self-hosted.md +264 -0
- package/docs/releases.md +37 -1
- package/docs/upgrades.md +72 -9
- package/package.json +37 -44
- package/templates/Dockerfile +32 -0
- package/templates/integration.ts +26 -5
- package/templates/openship.json +9 -0
- package/templates/railpack.json +1 -1
- package/templates/sandcms/blueprints/files/image.yaml +1 -1
- package/templates/sandcms/config.yaml +1 -1
- package/dist/cli/cloudflare.d.ts +0 -8
- package/dist/cli/cloudflare.js +0 -135
- package/dist/server/db/migrations/0001_independent_password_resets.sql +0 -3
- package/dist/server/db/migrations/0002_email_failure_reason.sql +0 -1
- package/dist/server/db/migrations/0003_direct_r2_uploads.sql +0 -10
- package/dist/server/db/migrations/0004_agent_api_keys.sql +0 -38
- package/dist/server/db/migrations/0005_api_retention_and_pagination.sql +0 -3
- package/dist/server/db/migrations/0006_durable_media_deletion.sql +0 -7
- package/dist/server/db/migrations/0007_content_and_job_indexes.sql +0 -2
- package/dist/server/db/migrations/0008_search_text_cache.sql +0 -45
- package/dist/server/db/migrations/meta/0001_snapshot.json +0 -1197
- package/dist/server/db/migrations/meta/0002_snapshot.json +0 -1204
- package/dist/server/db/migrations/meta/0003_snapshot.json +0 -1281
- package/dist/server/db/migrations/meta/0004_snapshot.json +0 -1546
- package/dist/server/db/migrations/meta/0005_snapshot.json +0 -1572
- package/dist/server/db/migrations/meta/0006_snapshot.json +0 -1627
- package/dist/server/db/migrations/meta/0007_snapshot.json +0 -1643
- package/dist/server/db/migrations/meta/0008_snapshot.json +0 -1711
- package/dist/worker/index.d.ts +0 -10
- package/dist/worker/index.js +0 -460
- package/dist/worker/local.d.ts +0 -5
- package/dist/worker/local.js +0 -57
- package/dist/worker/source.d.ts +0 -2
- package/dist/worker/source.js +0 -31
- package/dist/worker/uploads.d.ts +0 -22
- package/dist/worker/uploads.js +0 -48
- package/templates/sandcms/worker/index.ts +0 -1
- package/templates/sandcms/worker/local.ts +0 -1
- package/templates/sandcms/worker/wrangler.jsonc +0 -6
- /package/dist/{worker → server/media}/svg.d.ts +0 -0
- /package/dist/{worker → server/media}/svg.js +0 -0
package/docs/media.md
CHANGED
|
@@ -1,16 +1,46 @@
|
|
|
1
1
|
# Media
|
|
2
2
|
|
|
3
|
-
The website's Bun process owns SQLite
|
|
3
|
+
Images are stored and processed by the site itself. The website's Bun process owns SQLite, the durable media job runner and the files under the media directory; a child process decodes and encodes images with [sharp](https://sharp.pixelplumbing.com) (libvips). Public images use the website's media route, which supports browser and CDN caching. No external service is involved.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## Storage
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Media lives in `SANDCMS_MEDIA_DIR`, by default `media/` next to the database (`/data/media` in the starter image), on the same volume:
|
|
8
|
+
|
|
9
|
+
```text
|
|
10
|
+
media/originals/{id} uploaded files, never modified or served
|
|
11
|
+
media/variants/{id}/{version}/{preset}-{size}.{format} encoded variants, served at /_sandcms/media/{key}
|
|
12
|
+
media/tmp/{id}/… upload chunks and processing scratch files
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Every file is written to a temporary path, flushed to disk and renamed before the database records it, so a crash never leaves a half-written original or variant behind a database row. An existing original is never replaced. Width, height and each variant's ETag are stored in SQLite. Temporary files older than a day are removed at start and hourly.
|
|
16
|
+
|
|
17
|
+
Back up the media directory with the rest of the volume; see [backups](deployment.md#backups-and-restore). Media usage and free disk space are shown in the admin and by `sandcms doctor`.
|
|
18
|
+
|
|
19
|
+
## Formats and limits
|
|
20
|
+
|
|
21
|
+
Accepted input: JPEG, PNG, WebP, AVIF, GIF, TIFF and SVG. Output is WebP, AVIF and JPEG; SVG is served as a sanitized SVG. TIFF originals are never served directly, only their variants. HEIC is not supported.
|
|
22
|
+
|
|
23
|
+
### Limits
|
|
24
|
+
|
|
25
|
+
| Setting | Default | Range | Applies to |
|
|
26
|
+
| ------------------- | ----------- | ------------------------- | --------------------------------------------------------- |
|
|
27
|
+
| `media.uploadLimit` | 100,000,000 | 1 – 500,000,000 bytes | Each uploaded file |
|
|
28
|
+
| `media.maxPixels` | 100,000,000 | 1,000,000 – 1,000,000,000 | Decoded raster area (width × height, after EXIF) |
|
|
29
|
+
| `media.maxStorage` | none | bytes | Originals plus variants; uploads are refused when reached |
|
|
30
|
+
|
|
31
|
+
SVG is limited to 1 MB. An upload is also refused (507) unless the disk has free space of at least three times its size, plus the size of other uploads in progress, plus 5 GB. Memory use grows with `maxPixels`: the container needs room for one decode at that size, see [memory](deployment.md#memory-and-image-limits).
|
|
32
|
+
|
|
33
|
+
```yaml
|
|
34
|
+
media:
|
|
35
|
+
uploadLimit: 200000000 # 200 MB
|
|
36
|
+
maxPixels: 150000000 # 150 MP
|
|
37
|
+
maxStorage: 50000000000 # 50 GB
|
|
38
|
+
```
|
|
8
39
|
|
|
9
40
|
## Presets
|
|
10
41
|
|
|
11
42
|
```yaml
|
|
12
43
|
media:
|
|
13
|
-
workerUrl: https://your-project-media.example.workers.dev
|
|
14
44
|
defaultFileBlueprint: image
|
|
15
45
|
uploadLimit: 100000000
|
|
16
46
|
presets:
|
|
@@ -22,7 +52,7 @@ media:
|
|
|
22
52
|
formats: { webp: 85 }
|
|
23
53
|
```
|
|
24
54
|
|
|
25
|
-
Sizes are longest-side pixels. Outputs preserve aspect ratio and do not upscale unless an optional `aspectRatio` (width divided by height, 0.01–100) requests a crop. Cropped outputs also never upscale; small originals produce smaller variants with the requested ratio, subject to pixel rounding. Formats are WebP, AVIF, and JPEG, each with quality 1–100.
|
|
55
|
+
Sizes are longest-side pixels. Outputs preserve aspect ratio and do not upscale unless an optional `aspectRatio` (width divided by height, 0.01–100) requests a crop. Cropped outputs also never upscale; small originals produce smaller variants with the requested ratio, subject to pixel rounding. Formats are WebP, AVIF, and JPEG, each with quality 1–100. SVG is sanitized using a strict XML parser and allowlists; scripts, foreign content, external references, event handlers, and styles are removed. Document types/entities are rejected. The sanitized derivative receives restrictive CSP and nosniff headers, and is not resized or encoded as WebP.
|
|
26
56
|
|
|
27
57
|
## Focal points
|
|
28
58
|
|
|
@@ -45,21 +75,23 @@ card:
|
|
|
45
75
|
formats: { webp: 80, avif: 65 }
|
|
46
76
|
```
|
|
47
77
|
|
|
48
|
-
Crops
|
|
78
|
+
Crops resize the image to cover the requested ratio, then cut out the region centred as close to the focal point as the edges allow. Presets without `aspectRatio` preserve the full image. Focus changes automatically queue new immutable variant keys when cropping is configured; current variants stay available until the complete new set is ready. Unrelated metadata edits do not rebuild images. The reserved `sandcms-preview` preset supplies an uncropped WebP for the picker on raster files with a focus field.
|
|
49
79
|
|
|
50
|
-
Existing project-owned blueprints/configuration are preserved during upgrades: add the field and ratio explicitly,
|
|
80
|
+
Existing project-owned blueprints/configuration are preserved during upgrades: add the field and ratio explicitly, then run `sandcms media rebuild` to prepare existing files and previews. No database migration is required.
|
|
51
81
|
|
|
52
82
|
## Privacy and publication
|
|
53
83
|
|
|
54
|
-
Originals
|
|
84
|
+
Originals stay private on the volume; no route serves them. A media asset becomes publicly accessible only when at least one **published snapshot** references it through a media field, media list, or gallery (including nested fields). Uploading to the library, saving a draft, and preparing variants grant no public access. Sharing an asset with published content makes its active variants public even if it is also used in drafts.
|
|
55
85
|
|
|
56
|
-
`cmsHandle` serves public variants at `/_sandcms/media/{key}`. It checks the current published references and active variant set
|
|
86
|
+
`cmsHandle` serves public variants at `/_sandcms/media/{key}`. It checks the current published references and the active variant set, then streams the file from disk. Removing the last published reference—by unpublishing, deleting content, or publishing a replacement—revokes access at the origin. Cached copies can remain available. Draft edits leave the existing published snapshot and its images live. Requests reaching the origin cannot access historical variant sets after replacement; previously cached versions can remain available.
|
|
57
87
|
|
|
58
|
-
Editor previews use `/api/cms/media/{key}` and require an active admin/editor session with no pending password change.
|
|
88
|
+
Editor previews use `/api/cms/media/{key}` and require an active admin/editor session with no pending password change. Query-based transformations are rejected; only named variants exist.
|
|
59
89
|
|
|
60
|
-
|
|
90
|
+
Variant responses carry `Content-Type`, `Content-Length`, a strong `ETag`, `Content-Security-Policy: default-src 'none'; style-src 'none'; sandbox`, `Cross-Origin-Resource-Policy: cross-origin` and `X-Content-Type-Options: nosniff`. A file missing on disk returns 404.
|
|
61
91
|
|
|
62
|
-
|
|
92
|
+
Successful public responses use `Cache-Control: public, max-age=86400, s-maxage=2592000`: one day of browser freshness and 30 days in shared caches. Public responses preserve ETags and support `If-None-Match` revalidation for GET and HEAD, including weak validators, lists, and `*`. The CMS checks current access before returning a body or a cacheable `304`. Cache hits bypass these checks. Cached availability after unpublishing, deletion, or variant replacement is intentional; immediate revocation is not promised. Cache retention is not guaranteed, so a cold location can return 404 while a warm cache still serves the image.
|
|
93
|
+
|
|
94
|
+
Editor previews and all delivery errors use `Cache-Control: private, no-store` and omit ETags. Preview responses vary by cookie; public responses are independent of sessions. Never cache private previews. Already downloaded images cannot be recalled. For CDN cache rules in front of the site, see [optional CDN caching](cloudflare.md#optional-cdn-caching).
|
|
63
95
|
|
|
64
96
|
Publication does not move files or require cache registration. Uploads get distinct asset IDs and preset changes produce distinct variant URLs. Rebuilds reuse existing completed variants rather than overwriting them. Do not replace bytes under an existing URL: processing changes that alter output must introduce a new variant version. The public policy deliberately does not advertise `immutable`.
|
|
65
97
|
|
|
@@ -71,8 +103,7 @@ Import the Svelte component from the package root and pass the asset returned by
|
|
|
71
103
|
`cms.media.publicAsset(id)`:
|
|
72
104
|
|
|
73
105
|
Each variant includes a `url` using the CMS public or authenticated preview route.
|
|
74
|
-
The component needs no URL configuration.
|
|
75
|
-
`media.workerUrl` in YAML) configures only the server's private Worker connection.
|
|
106
|
+
The component needs no URL configuration.
|
|
76
107
|
|
|
77
108
|
```svelte
|
|
78
109
|
<script lang="ts">
|
|
@@ -111,24 +142,66 @@ The component emits a `<picture>` with AVIF, WebP, then JPEG sources when availa
|
|
|
111
142
|
and adds no styling or admin dependencies. Consumers control layout with their own CSS.
|
|
112
143
|
`imageSources(asset, preset)` remains available for custom markup.
|
|
113
144
|
|
|
114
|
-
##
|
|
145
|
+
## Uploads
|
|
146
|
+
|
|
147
|
+
Files upload to the site itself in 8 MB chunks, so no request is large or long enough to hit a reverse proxy's default limits:
|
|
148
|
+
|
|
149
|
+
1. The admin (or an [API client](api.md#media-uploads)) asks to start an upload with the file's name, type and size. The CMS checks the blueprint's accepted types, `uploadLimit`, disk space and quota, and returns a chunk URL with a signed upload ticket.
|
|
150
|
+
2. Chunks are sent one at a time with `PUT {url}?index=N`. Each chunk is stored on disk as it streams in; re-sending a chunk replaces it. A `GET {url}` lists the chunks already stored, so an interrupted upload resumes where it stopped, also after a site restart.
|
|
151
|
+
3. Completion assembles the chunks and checks the total size. A background job then validates the file and stores the original: SVGs go through the sanitizer; raster files must decode as the declared type within `maxPixels`. Variant jobs are queued after that.
|
|
152
|
+
|
|
153
|
+
The ticket is bound to the asset, the upload attempt, the declared type and size, and the user who started it (their session or any of their API keys may send chunks). An upload with no chunk for 15 minutes expires; the asset is marked failed and its chunks are removed. Upload bytes never enter SQLite. Existing BLOB uploads from earlier releases still finish through the legacy runner.
|
|
115
154
|
|
|
116
|
-
|
|
155
|
+
`cms.media.upload(file)` stores a file from server code through the same validation, without HTTP.
|
|
156
|
+
|
|
157
|
+
## Processing
|
|
158
|
+
|
|
159
|
+
Images are processed one at a time per site by a child process running sharp, so a huge decode cannot take down the web server. Each batch decodes the original once, auto-rotates it from EXIF orientation, converts it to sRGB, and encodes every requested variant from that master. Variants never upscale. Each finished variant is written and recorded before the next starts, so a retry skips finished variants. Jobs have durable attempts, leases, retry delays and failure state; restarting the website resumes them. If the child exits, for example when the container's memory limit is reached, the batch fails at once and each unfinished variant counts as a failed attempt; a variant that fails four times marks the asset failed. The media library shows processing/ready/failed and permits retries. An asset's old active set stays available throughout a rebuild.
|
|
160
|
+
|
|
161
|
+
The child runs at low CPU priority (`nice 10`), so page requests always win over background processing within a site. It uses one libvips thread by default. Set `SANDCMS_MEDIA_THREADS` (for example `2`–`4`) to let a site finish large batches sooner when the server is idle; Docker's default CPU weights still split CPU evenly between busy containers, so leave the host's CPU limit unset or generous rather than low. Processing stays one image at a time, so memory use does not grow with the thread count beyond libvips' per-thread buffers.
|
|
162
|
+
|
|
163
|
+
Only the JPEG, PNG, WebP, AVIF, GIF and TIFF decoders are enabled, and the decoded format must match the declared type, so an SVG named `.png` is rejected rather than rasterized.
|
|
117
164
|
|
|
118
165
|
```sh
|
|
119
166
|
bun x --no-install sandcms media rebuild
|
|
120
167
|
```
|
|
121
168
|
|
|
122
|
-
Rebuild compares the current preset hash, schedules missing sets, shows progress, and exits nonzero for failures. It does not delete old variants.
|
|
169
|
+
Rebuild compares the current preset hash, schedules missing sets, shows progress, and exits nonzero for failures. It does not delete old variants. `media rebuild --force` also re-encodes the active variants of every ready raster image under the same version and URLs; use it after the 0.7.x [R2 import](cloudflare.md) to replace variants encoded by Cloudflare. Rebuilds wait while the website's processor is busy, and never start the website's background tasks. Configure a later reviewed cleanup operation if you need to remove unreferenced historical variant keys.
|
|
170
|
+
|
|
171
|
+
### Changes from Cloudflare processing (0.7.0)
|
|
172
|
+
|
|
173
|
+
- Animated GIF and WebP inputs are flattened to their first frame.
|
|
174
|
+
- Metadata is stripped from variants except the EXIF copyright and artist fields.
|
|
175
|
+
- HEIC input is not supported.
|
|
176
|
+
- Inputs are no longer capped at 100 MB / 100 MP or a 20 MB output; `uploadLimit` and `maxPixels` set the limits.
|
|
177
|
+
- Imported variants keep their Cloudflare encoding until `media rebuild --force`; new variants are encoded by sharp under the same preset version.
|
|
178
|
+
|
|
179
|
+
### Checking and repairing files
|
|
180
|
+
|
|
181
|
+
```sh
|
|
182
|
+
bun x --no-install sandcms media verify
|
|
183
|
+
bun x --no-install sandcms media verify --repair
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
`media verify` lists media rows whose original or active variants are missing on disk, for example after restoring a database snapshot that is newer than the media in a backup, and exits nonzero when any are missing. `--repair` queues missing variants to be rendered again when the original exists, and marks an image failed with "Original file is missing. Upload this image again." when it does not. Uploads in progress are not checked.
|
|
187
|
+
|
|
188
|
+
### Freeing unused files
|
|
189
|
+
|
|
190
|
+
```sh
|
|
191
|
+
bun x --no-install sandcms media prune
|
|
192
|
+
bun x --no-install sandcms media prune --delete
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
`media prune` lists originals and variant folders that no media row uses: files of assets missing from the database (for example after restoring an older database snapshot) and variant versions left behind by preset changes or rebuilds. A version is kept while it is active, targeted, or still being rendered, and anything changed in the last hour is skipped so a running upload is never touched. Without `--delete` nothing is changed. `--delete` asks for confirmation (`--yes` skips it) and checks each entry against the database again before removing it. Removal is permanent: if you later restore an older database backup that still used those files, run `media verify --repair` and re-upload images whose originals are gone. Upload staging in `tmp/` is cleaned by the running website and is not touched.
|
|
123
196
|
|
|
124
|
-
|
|
197
|
+
## Deletion
|
|
125
198
|
|
|
126
|
-
|
|
199
|
+
Editable asset fields come from file blueprints. If a blueprint declares a photographer relation, it references that field's configured collection. Publish a photographer's rename to update public credits; deletion clears the credit. Deletion displays known draft/published usages and allows confirmation without additional permission escalation.
|
|
127
200
|
|
|
128
|
-
|
|
201
|
+
Deletion clears draft/published references and revokes active variants immediately, then records cleanup in the durable `media_deletions` queue. Assets remain `deleting` until their files are removed. Cleanup waits for active uploads and processing to finish or their leases to expire; processing cannot restore the asset or schedule more work, and a file written after the deletion is removed again. Failures retry automatically, including after a website restart. The library polls pending deletions and excludes them from image pickers.
|
|
129
202
|
|
|
130
203
|
## Bulk uploads and galleries
|
|
131
204
|
|
|
132
|
-
The library and multi-image pickers accept multiple files in one selection. Files upload sequentially
|
|
205
|
+
The library and multi-image pickers accept multiple files in one selection. Files upload sequentially in selection order, each in chunks with a progress bar; a failed chunk is retried before the file is reported as failed. Errors name each failed file, and later files still upload. Successful uploads remain available when another file fails. Use the library's processing retry for files already accepted by the server.
|
|
133
206
|
|
|
134
207
|
Use a `gallery` field for ordered images. Its file blueprint defines available metadata; `overrides` explicitly enables per-placement editing of selected fields. A gallery without overrides displays images and ordering controls only. See [gallery configuration](content.md#galleries-and-project-specific-metadata). Both galleries and `media-list` fields support drag handles and arrow controls for ordering.
|
package/docs/package.md
CHANGED
|
@@ -4,46 +4,49 @@
|
|
|
4
4
|
|
|
5
5
|
## Publish from macOS
|
|
6
6
|
|
|
7
|
-
Use Bun **1.4.
|
|
7
|
+
Use Bun **1.4.4**; Node and npm are not needed. Bun reads npm credentials from `~/.npmrc` or `NPM_CONFIG_TOKEN`. Create a granular access token at npmjs.com with publish rights for `sandquist-group` and keep it out of the repository:
|
|
8
|
+
|
|
9
|
+
```sh
|
|
10
|
+
echo '//registry.npmjs.org/:_authToken=${NPM_TOKEN}' >> ~/.npmrc
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Run these commands from the repository root after pulling the changes:
|
|
8
14
|
|
|
9
15
|
```sh
|
|
10
16
|
bun --version
|
|
11
|
-
|
|
12
|
-
npm login --registry=https://registry.npmjs.org/
|
|
13
|
-
npm whoami --registry=https://registry.npmjs.org/
|
|
17
|
+
bun pm whoami
|
|
14
18
|
bun install --frozen-lockfile
|
|
15
19
|
bun run release:check
|
|
16
|
-
|
|
17
|
-
|
|
20
|
+
bun pm pack --dry-run
|
|
21
|
+
bun pm pack --destination /tmp
|
|
18
22
|
```
|
|
19
23
|
|
|
20
|
-
This release produces `/tmp/sandcms-0.
|
|
24
|
+
This release produces `/tmp/sandcms-0.8.0.tgz`. `bun pm pack` builds the package through `prepack`; it packages only the allowlisted distribution, templates, documentation, and package metadata. The repository's example website, tests, databases, and environment files are excluded. `release:check` checks types, runs unit tests, inspects the packed archive, and installs/builds that archive in both a fresh starter and an existing SvelteKit site.
|
|
21
25
|
|
|
22
26
|
Upload the archive you just built:
|
|
23
27
|
|
|
24
28
|
```sh
|
|
25
|
-
|
|
26
|
-
|
|
29
|
+
bun publish /tmp/sandcms-0.8.0.tgz --access restricted --registry=https://registry.npmjs.org/
|
|
30
|
+
bun pm view sandcms@0.8.0 version
|
|
27
31
|
```
|
|
28
32
|
|
|
29
|
-
Complete
|
|
33
|
+
Complete the 2FA prompt (`bun publish` opens the browser, or pass `--otp`). The account must have publishing permission in `sandquist-group`. The package also defaults to restricted access and the npmjs registry in `publishConfig`. Do not set `private: true` on this package: that prevents publication entirely. Generated websites do set it.
|
|
30
34
|
|
|
31
|
-
For later releases, increment `version` in `package.json
|
|
35
|
+
For later releases, increment `version` in `package.json`, repeat the checks and packing, and publish the resulting archive filename. npm versions cannot be overwritten. The CLI reads its package version when creating starter dependencies and integration manifests.
|
|
32
36
|
|
|
33
37
|
## Create a starter from npm
|
|
34
38
|
|
|
35
|
-
After publishing,
|
|
39
|
+
After publishing, add a read token for the package to `~/.npmrc` (as above), then run:
|
|
36
40
|
|
|
37
41
|
```sh
|
|
38
|
-
|
|
39
|
-
bunx --bun --package sandcms@0.6.3 sandcms setup --cwd ./my-site
|
|
42
|
+
bunx --bun --package sandcms@0.8.0 sandcms setup --cwd ./my-site
|
|
40
43
|
cd my-site
|
|
41
44
|
bun run dev
|
|
42
45
|
```
|
|
43
46
|
|
|
44
|
-
Setup asks for the project name and initial local admin email. The generated site includes a minimal public home page, `/admin/[...path]`, `/api/cms/[...path]`, the server hook/integration, YAML blueprints, and a local database.
|
|
47
|
+
Setup asks for the project name and initial local admin email. The generated site includes a minimal public home page, `/admin/[...path]`, `/api/cms/[...path]`, the server hook/integration, YAML blueprints, a `Dockerfile`, and a local database. Uploads and image processing work locally with no extra service; media is stored next to the database. This is a Bun application; Node alone cannot run the CMS.
|
|
45
48
|
|
|
46
|
-
Setup also adds commented `SANDCMS_TRUSTED_PROXY_HEADER
|
|
49
|
+
Setup also adds commented `SANDCMS_TRUSTED_PROXY_HEADER` and `ADDRESS_HEADER` examples to `.env.local`. They stay disabled for local development. In production, set both to the client IP header your reverse proxy sets; follow the [client IP setup and check](deployment.md#client-ip-behind-a-reverse-proxy). Rerunning setup adds missing examples without duplicating existing ones or overwriting configured proxy settings.
|
|
47
50
|
|
|
48
51
|
For an existing SvelteKit project, run the same setup command with `--cwd` pointing to that project and review `sandcms/setup-report.md`. Inside an installed project, use the local CLI:
|
|
49
52
|
|
|
@@ -67,11 +70,11 @@ cd /tmp/my-sandcms-site
|
|
|
67
70
|
bun run dev
|
|
68
71
|
```
|
|
69
72
|
|
|
70
|
-
`setup --dry-run --name "Example"` only prints the integration plan. `--yes` skips optional prompts
|
|
73
|
+
`setup --dry-run --name "Example"` only prints the integration plan. `--yes` skips optional prompts; `--skip-install` is useful in fixture tests. `--email` optionally supplies a local bootstrap email. Never pass passwords as command arguments.
|
|
71
74
|
|
|
72
75
|
A fresh site receives SvelteKit, TypeScript, the pinned Bun adapter, a public starter page, CMS route glue, starter YAML, environment values, and a local database. An existing site must already be a SvelteKit project. Setup checks reserved routes and leaves custom hooks, adapters, config, and public routes intact. Read `sandcms/setup-report.md` and apply the hook composition patch if needed. Existing JavaScript hooks are recognized; setup does not shadow them with a TypeScript file. Tailwind, ESLint, Prettier, Vitest, and Playwright are individually optional website add-ons.
|
|
73
76
|
|
|
74
|
-
The built package contains Svelte admin components, compiled CSS/fonts, server code, CLI, reviewed migrations, Worker source, and templates. `bun run test:package` installs and builds it in a clean fixture without Tailwind and verifies a customized existing site.
|
|
77
|
+
The built package contains Svelte admin components, compiled CSS/fonts, server code, the media processor entry (`sandcms/media-processor`), CLI, reviewed migrations, the deprecated Worker source, and templates. `sharp` is a dependency; its prebuilt binary is installed for the build platform, so install dependencies inside the image (as the starter `Dockerfile` does), not on a different OS. `bun run test:package` installs and builds it in a clean fixture without Tailwind and verifies a customized existing site.
|
|
75
78
|
|
|
76
79
|
## Private registry use
|
|
77
80
|
|
|
@@ -0,0 +1,264 @@
|
|
|
1
|
+
# SandCMS 0.7.0 plan: self-contained media on any Docker host
|
|
2
|
+
|
|
3
|
+
Status: **proposed** (2026-11-17).
|
|
4
|
+
|
|
5
|
+
0.7.0 removes the Cloudflare Worker and R2 from SandCMS. A site becomes one container with one persistent volume, and runs on any Docker host. We recommend [Dokploy](https://dokploy.com) or [Openship](https://openship.io) as the host platform; nothing in the package depends on either.
|
|
6
|
+
|
|
7
|
+
## 1. Goals
|
|
8
|
+
|
|
9
|
+
- **One container, one volume.** SQLite, image originals, variants and database snapshots live under `/data`. No external service is needed to run a site.
|
|
10
|
+
- **Full-resolution originals.** Editors upload camera originals of 70 MB and more, including TIFF, without compressing first. The site serves optimized variants.
|
|
11
|
+
- **Runs behind any reverse proxy with default settings.** No raised timeouts or body limits on the proxy.
|
|
12
|
+
- **Preview first, launch later.** A site can start on a temporary hostname. Changing its domain is an environment change and a restart, with no rebuild.
|
|
13
|
+
- **Safe with plain volume backups.** A file-level copy of the volume, taken while the site runs, is always restorable.
|
|
14
|
+
|
|
15
|
+
Non-goals for 0.7.0: an S3 media backend (the storage interface leaves room for one), multiple instances per site, video, HEIC input, platform-specific automation.
|
|
16
|
+
|
|
17
|
+
## 2. Decisions
|
|
18
|
+
|
|
19
|
+
| # | Decision | Why |
|
|
20
|
+
| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
21
|
+
| D1 | Media is stored on the site volume (`/data/media`), behind a storage interface. | One volume to back up. An S3 backend can be added later without touching callers. |
|
|
22
|
+
| D2 | Images are processed with sharp (libvips) in a child process, one image at a time. | Removes Cloudflare's 100 MP, 100 MB and TIFF limits. A huge decode can't take down the web server. |
|
|
23
|
+
| D3 | Uploads go to the site itself in **8 MB chunks**, resumable, authorized by a signed ticket. A session or an API key may upload. | No request is long or large enough to hit a proxy's default limits. Survives restarts and bad connections, and keeps the API-key flow in `docs/api.md`. |
|
|
24
|
+
| D4 | `BODY_SIZE_LIMIT=10M`; `cmsHandle` enforces 1 MB on every route except the chunk route. | The Bun adapter's limit applies to all routes at once (`adapter-bun/src/index.js:51`). |
|
|
25
|
+
| D5 | **Overlapping deploys are allowed.** Old and new containers may share the volume briefly; background work coordinates through database leases (processor, snapshot). One long-running replica per site. | Openship deploys run-new-then-swap on Docker servers. SQLite handles two processes on a local volume, and migrations are additive. (An earlier draft refused a second server with an instance lock; that broke every overlapping deploy.) |
|
|
26
|
+
| D6 | The app writes a verified database snapshot **every hour** to `/data/backups/latest.sqlite`. | A file copy of a live SQLite database can be torn. With the snapshot inside the volume, any volume backup contains a consistent database. |
|
|
27
|
+
| D7 | Variants live on the same volume as originals. | Restores need no re-encoding. |
|
|
28
|
+
| D8 | The starter ships a `Dockerfile` with a `HEALTHCHECK`. | Works on any Docker host. `railpack.json` stays for existing sites. |
|
|
29
|
+
| D9 | The site derives its public origin from the request, and redirects other hostnames to `SANDCMS_ORIGIN`. | A domain change needs no rebuild; preview and extra domains redirect without proxy rules. |
|
|
30
|
+
| D10 | Existing R2 media is imported once with `sandcms media import-r2`, keeping asset IDs, variant keys and the preset version. Images are unavailable during the import. | Public URLs and published content are unchanged. A short maintenance window is simpler than a fallback path. |
|
|
31
|
+
| D11 | The Worker options, env variables and package exports stay, deprecated, through 0.7.x. | Existing integrations pass them, and the old Worker must stay deployable until the import is verified. |
|
|
32
|
+
|
|
33
|
+
## 3. Architecture
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Visitor ──HTTPS──▶ reverse proxy (TLS, default settings)
|
|
37
|
+
│
|
|
38
|
+
▼
|
|
39
|
+
SandCMS container (Bun, single instance, health-checked)
|
|
40
|
+
├─ public site + /admin + /api/cms
|
|
41
|
+
├─ cmsHandle: canonical host · body guard · media delivery · chunk upload
|
|
42
|
+
├─ job runner ──spawn──▶ media processor child (sharp)
|
|
43
|
+
└─ snapshot scheduler (hourly)
|
|
44
|
+
│
|
|
45
|
+
▼
|
|
46
|
+
/data (one persistent volume per site)
|
|
47
|
+
sandcms.sqlite (+wal/shm, guard) site-id instance.lock
|
|
48
|
+
media/originals/{id}
|
|
49
|
+
media/variants/{id}/{version}/{preset}-{size}.{fmt}
|
|
50
|
+
media/tmp/{id}/{attempt}/
|
|
51
|
+
backups/latest.sqlite · previous.sqlite
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## 4. Media
|
|
55
|
+
|
|
56
|
+
The Worker is reached through one closure, `worker()` in `src/lib/server/media/service.ts:61-74`, with 8 call sites. It is replaced by two modules. The job, lease, version and publication model stays.
|
|
57
|
+
|
|
58
|
+
### 4.1 Storage (`src/lib/server/media/storage.ts`)
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
interface MediaStorage {
|
|
62
|
+
writeChunk(id, attempt, index, body: ReadableStream, bytes): Promise<void>;
|
|
63
|
+
assemble(id, attempt, totalBytes): Promise<string>; // path of the staged file
|
|
64
|
+
promoteOriginal(id, stagedPath): Promise<void>; // fails if an original exists
|
|
65
|
+
original(id): BunFile | null;
|
|
66
|
+
writeVariant(key, tmpPath): Promise<{ etag: string }>;
|
|
67
|
+
variant(key): BunFile | null;
|
|
68
|
+
removeAsset(id): Promise<void>;
|
|
69
|
+
sweepTmp(olderThan: Date): Promise<number>;
|
|
70
|
+
usage(): Promise<{ bytes: number; free: number }>;
|
|
71
|
+
}
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
- 0.7.0 ships the local-disk implementation only. The root is `SANDCMS_MEDIA_DIR`, defaulting to `<dirname(SANDCMS_DATABASE)>/media`.
|
|
75
|
+
- **Durability.** Files are written to a temporary path, `fsync`ed, renamed, and the directory `fsync`ed, before the database says they exist.
|
|
76
|
+
- **No overwrite.** `promoteOriginal` uses `link()` + `unlink()`, which fails if the original exists, matching the Worker's conditional put (`worker/index.ts:338, 362`).
|
|
77
|
+
- **Metadata in SQLite.** Original `width`, `height` and `uploadAttempt` become nullable `media` columns in retained migration `0009_local_media` (`bun run db:generate --name local_media`). Each variant's `etag` (short sha256) goes into the existing `variants` and `jobs.result` JSON.
|
|
78
|
+
- **Deletion.** After every promote or variant write, the parent process re-checks the media row and removes the file if the row is `deleting` or missing. `media_deletions` rows cascade away with the media row (`schema.ts:333`), so a missing row counts as deleted.
|
|
79
|
+
- **Disk guard.** In-flight upload bytes are reserved in `upload_sessions`. An upload is refused unless free space ≥ `3 × bytes` + reserved bytes + 5 GB, and within the optional `media.maxStorage` quota. Usage and free space appear in the admin and in `doctor`.
|
|
80
|
+
- `sweepTmp` runs at start and hourly.
|
|
81
|
+
|
|
82
|
+
### 4.2 Upload
|
|
83
|
+
|
|
84
|
+
1. `POST /api/cms/media/upload` keeps its validation (blueprint `accept`, `uploadLimit`, MIME) and returns an absolute URL:
|
|
85
|
+
```json
|
|
86
|
+
{
|
|
87
|
+
"id": "…",
|
|
88
|
+
"url": "https://site/api/cms/media/upload/{id}",
|
|
89
|
+
"chunkSize": 8388608,
|
|
90
|
+
"headers": { "x-sandcms-upload": "<ticket>" }
|
|
91
|
+
}
|
|
92
|
+
```
|
|
93
|
+
The ticket is an HMAC (key derived from `SANDCMS_SECRET`) over `id`, `attempt`, `mime`, `bytes`, the user or API-key ID, and an expiry. It travels in a header, so it stays out of access logs.
|
|
94
|
+
2. `PUT {url}?index=N` stores one chunk. `cmsHandle` handles it before SvelteKit reads the body.
|
|
95
|
+
- Authorized by the ticket plus an admin/editor session (with the origin check) or an API key with `cms:write`. `hooks.ts:30-49` and `api-access.ts` gain this route.
|
|
96
|
+
- Bytes are counted while streaming, so a missing `Content-Length` (HTTP/2, HTTP/3) is fine. A chunk over `chunkSize` is rejected.
|
|
97
|
+
- Each chunk is a file `tmp/{id}/{attempt}/{index}`; re-sending replaces it.
|
|
98
|
+
- The first chunk moves the session from `uploading` to `receiving` and takes its lease; later chunks extend it. The expiry branch (`runner.ts:28-63`) and the deletion runner (`deletions.ts:43`) respect that lease.
|
|
99
|
+
- `GET {url}` lists stored chunk indexes, for resume.
|
|
100
|
+
3. `POST /api/cms/media/upload/complete` assembles the chunks, verifies the total length, and sets the session to `pending`. The existing runner branch (`runner.ts:79-128`) validates and promotes asynchronously: SVGs through the sanitizer (moved from `src/lib/worker/svg.ts` to `src/lib/server/media/svg.ts`), raster files through the processor.
|
|
101
|
+
4. `cms.media.upload(file)` (`service.ts:209-219`) writes through the storage module directly.
|
|
102
|
+
5. **Body guard.** Any other request over `SANDCMS_REQUEST_LIMIT` (default 1 MB) gets 413 before its body is read; bodies without a length are counted and cut off. `cmsHandle` must run first in `sequence()`.
|
|
103
|
+
6. The admin client (`src/lib/admin/uploads.ts`) sends chunks sequentially with per-chunk retry and a progress bar.
|
|
104
|
+
|
|
105
|
+
Accepted input: JPEG, PNG, WebP, AVIF, GIF, **TIFF**, SVG. Output stays WebP, AVIF and JPEG. Defaults: `uploadLimit` 100 MB (up to 500 MB), `media.maxPixels` 100 MP.
|
|
106
|
+
|
|
107
|
+
### 4.3 Processing (`src/lib/server/media/processor.ts` + child entry)
|
|
108
|
+
|
|
109
|
+
- The runner keeps `jobs`, leases, retries and batching (`runner.ts:182-293`). The Worker batch call becomes `processor.render(id, version, specs)`.
|
|
110
|
+
- **One processor per site.** Render work is claimed only when the child is idle. A database lease stops the server and a CLI command (`media rebuild`, `import-r2`) from processing at once.
|
|
111
|
+
- **Child process.** Spawned per batch over Bun's `ipc` channel, with `sharp.cache(false)` and `oom_score_adj` 1000, so a memory limit kills the child and not the web server. On exit the parent fails the batch at once. Two kills in a row mark the asset failed with a clear message.
|
|
112
|
+
- **Loader allowlist.** Block all libvips loaders, then unblock JPEG, PNG, WebP, AVIF, GIF and TIFF. `metadata().format` must match the declared MIME, with `failOn: 'error'`. Otherwise an SVG claiming to be a PNG would be rasterized.
|
|
113
|
+
- **Decode once.** Each batch decodes the original into a master image at the largest requested size (EXIF-rotated, sRGB, temporary lossless file), derives each spec from it sequentially, and reports each variant as it finishes. The parent `fsync`s, renames and commits it, and extends the lease. A retry skips finished variants. (Measured: sharp's `clone()` re-decodes per pipeline, and `sharp.concurrency(1)` does not serialize pipelines.)
|
|
114
|
+
- `inspect` ports `dimensions()` (`worker/index.ts:103-128`): orientation-corrected size and the max-pixels rejection. Stored dimensions come from sharp's output.
|
|
115
|
+
- `render` reuses the size math from `renderVariant` (`worker/index.ts:191-251`) as a shared function. Never upscale. Focal-point crops are a resize followed by `extract`.
|
|
116
|
+
- **Behaviour changes from Cloudflare**, for the release notes: animated GIF and WebP inputs are flattened to their first frame; metadata is stripped except copyright and artist EXIF.
|
|
117
|
+
- Originals are never modified, and TIFF originals are never served.
|
|
118
|
+
- sharp becomes a direct dependency. The child entry must survive the consumer's Vite SSR bundle.
|
|
119
|
+
|
|
120
|
+
### 4.4 Delivery
|
|
121
|
+
|
|
122
|
+
`deliver()` (`service.ts:410-440`):
|
|
123
|
+
|
|
124
|
+
- keeps the variant-membership and `isPublic()` checks;
|
|
125
|
+
- returns the `Bun.file` response with `Content-Length`, and maps a missing file to 404;
|
|
126
|
+
- answers `If-None-Match` from the stored `etag`, porting `worker/index.ts:395-404`;
|
|
127
|
+
- sets the headers the Worker used to set (`worker/index.ts:386-392`): `Content-Type`, `Content-Security-Policy: default-src 'none'; style-src 'none'; sandbox`, `Cross-Origin-Resource-Policy`, `X-Content-Type-Options: nosniff`.
|
|
128
|
+
|
|
129
|
+
`delivery.ts` stops deleting `Content-Length` and re-wrapping the body (`delivery.ts:23-25`). Cache headers and URLs (`/_sandcms/media/{id}/{version}/{preset}-{size}.{fmt}`) are unchanged.
|
|
130
|
+
|
|
131
|
+
### 4.5 Configuration and compatibility
|
|
132
|
+
|
|
133
|
+
- New config: `media.maxPixels`, `media.maxStorage`, a higher allowed `media.uploadLimit`. `media.workerUrl` becomes optional and ignored (`load.ts:222`).
|
|
134
|
+
- New env: `SANDCMS_MEDIA_DIR`, `SANDCMS_REQUEST_LIMIT`, `SANDCMS_INDEXING`, `SANDCMS_BOOTSTRAP_EMAIL`.
|
|
135
|
+
- **Deprecated, kept through 0.7.x (D11):** `createCMS({ workerToken, workerUrl })`, the `SANDCMS_WORKER_*` declarations, and the `./worker` and `./worker/local` exports.
|
|
136
|
+
- **Removed:** `src/lib/cli/cloudflare.ts`, `sandcms worker update`, the Cloudflare prompt in setup, `templates/sandcms/worker/*` for new sites, the `wrangler` dev dependency, the dead `workerUrl` prop in the admin (`client.ts:44-46`).
|
|
137
|
+
- Setup and upgrade leave project-owned `sandcms/worker/*` files alone and report them as removable.
|
|
138
|
+
- New background work starts from `media.start()`, which existing integrations already call, and never from CLI commands.
|
|
139
|
+
|
|
140
|
+
### 4.6 Import from R2 (`sandcms media import-r2`)
|
|
141
|
+
|
|
142
|
+
Run once per existing site, in a maintenance window, while the old Worker is still deployed.
|
|
143
|
+
|
|
144
|
+
- Refuses to start while `upload_sessions` or legacy `uploads` rows are pending.
|
|
145
|
+
- Holds the processor lease, so nothing renders during the import.
|
|
146
|
+
- Raster originals come through the Worker's `/internal/source/{id}`, with URLs minted locally from `SANDCMS_WORKER_TOKEN` (`src/lib/worker/source.ts`).
|
|
147
|
+
- SVG originals are not served by that route (`worker/index.ts:278`); the already sanitized `svg-1.svg` variant becomes both original and variant.
|
|
148
|
+
- Variants come through `/internal/media/{key}` for every ready `jobs.result`. Assets in `deleting` are skipped.
|
|
149
|
+
- Fills `width`, `height` and `etag`; keeps `activeVersion` and the `'remote-transform-v2:'` salt (`presets.ts:6`).
|
|
150
|
+
- Resumable and idempotent; ends by verifying every original and variant (existence and size).
|
|
151
|
+
|
|
152
|
+
`sandcms media rebuild --force` re-encodes everything with sharp on request.
|
|
153
|
+
|
|
154
|
+
## 5. Platform features
|
|
155
|
+
|
|
156
|
+
- **Overlapping deploys (D5).** No instance lock: a second server on the same volume serves normally. The processor and the hourly snapshot each take a database lease, so only one server does that work at a time.
|
|
157
|
+
- **Site identity.** On first start the app writes `/data/site-id` (a random ID plus the project name) and later refuses to run if the project name differs. In production it refuses to create a new database when the data directory is not a mount. Both catch a shared or missing volume before content is written.
|
|
158
|
+
- **Request origin (D9).** The starter stops baking `paths.origin` into the build (`cli/scaffold.ts:55`). The Bun adapter then derives the origin from the `Host` header (`adapter-bun/src/handler.js:44-52`). `sandcms upgrade` reports the change for existing `vite.config.ts` files.
|
|
159
|
+
- **Canonical host.** A request whose `Host` (lowercased, default port stripped) isn't `SANDCMS_ORIGIN`'s host gets a 308 to the same URL on the origin, with `Cache-Control: no-store`. Exceptions: the health route, and development. Prerendered pages and `static/` files bypass hooks; `doctor` warns when a site has them.
|
|
160
|
+
- **Indexing.** `SANDCMS_INDEXING=false` sends `X-Robots-Tag: noindex` and a disallow-all `robots.txt`. Unset means indexed.
|
|
161
|
+
- **Health.** `GET /_sandcms/health` returns 200 when the process serves and the database opens. Liveness only: disk and media problems show in the admin and `doctor`, so a full disk doesn't restart the site in a loop.
|
|
162
|
+
- **Snapshots (D6).** Hourly: rotate `latest.sqlite` to `previous.sqlite`, then write a verified snapshot with `backup()` (`src/lib/cli/backup.ts:27-44`). Catches up at start. `before-migrate` snapshots are pruned to the newest 3.
|
|
163
|
+
- **Bootstrap.** With `SANDCMS_BOOTSTRAP_EMAIL` set and no users, the first start creates the admin and logs a one-time password. No shell access is needed for a new site.
|
|
164
|
+
- **Dockerfile (D8).** The starter gets a multi-stage `Dockerfile` on the pinned Bun image: install with the frozen lockfile, build, run `bun build/index.js`, `HEALTHCHECK` on `/_sandcms/health`, no `VOLUME` instruction.
|
|
165
|
+
- **Doctor.** Drops the Worker and Railpack-only checks. Adds: media directory, disk usage and quota, snapshot age, `BODY_SIZE_LIMIT` ≥ chunk size, trusted-header configuration, site-id.
|
|
166
|
+
|
|
167
|
+
## 6. Hosting
|
|
168
|
+
|
|
169
|
+
### 6.1 What the host must provide
|
|
170
|
+
|
|
171
|
+
| Requirement | Detail |
|
|
172
|
+
| ------------- | -------------------------------------------------------------------------------------------------- |
|
|
173
|
+
| Container | Built from the site's `Dockerfile` (or Railpack), one replica |
|
|
174
|
+
| Volume | One persistent volume mounted at `/data`, on a local filesystem, **used by this site only** |
|
|
175
|
+
| Deploys | Replace the container: stop the old one, then start the new one. Stop timeout ≥ 30 s |
|
|
176
|
+
| Reverse proxy | TLS, routes the site's hostnames, passes `Host` unchanged, overwrites the client-IP header it sets |
|
|
177
|
+
| Health check | The image's `HEALTHCHECK`, or `GET /_sandcms/health` |
|
|
178
|
+
| Memory | A container memory limit that covers one decode at `media.maxPixels` (measured in §8) |
|
|
179
|
+
| Backups | A scheduled copy of the whole volume to off-server storage |
|
|
180
|
+
|
|
181
|
+
Environment:
|
|
182
|
+
|
|
183
|
+
```dotenv
|
|
184
|
+
SANDCMS_ORIGIN=https://www.example.com
|
|
185
|
+
SANDCMS_SECRET=<32+ random characters>
|
|
186
|
+
SANDCMS_DATABASE=/data/sandcms.sqlite
|
|
187
|
+
SANDCMS_AUTO_MIGRATE=true
|
|
188
|
+
BODY_SIZE_LIMIT=10M
|
|
189
|
+
SANDCMS_TRUSTED_PROXY_HEADER=x-real-ip # the header your proxy sets
|
|
190
|
+
ADDRESS_HEADER=x-real-ip
|
|
191
|
+
# SANDCMS_INDEXING=false # preview sites only
|
|
192
|
+
# SANDCMS_BOOTSTRAP_EMAIL=you@example.com # first start only
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
### 6.2 Recommended platforms
|
|
196
|
+
|
|
197
|
+
Dokploy and Openship both provide everything in §6.1: Git deploys, a reverse proxy with Let's Encrypt, named volumes, scheduled volume backups to S3-compatible storage, and rollback. Platform notes:
|
|
198
|
+
|
|
199
|
+
- **Dokploy.** Volume names are global on a server, so give each site its own (`sandcms-<slug>-data`). The default update order starts the new container first; set it to stop-first. Rollback needs a registry configured. Volume backups are full archives.
|
|
200
|
+
- **Openship.** Supports incremental volume backups and validates a restore before replacing data. Not yet tested by us with SandCMS.
|
|
201
|
+
|
|
202
|
+
### 6.3 Site lifecycle
|
|
203
|
+
|
|
204
|
+
1. **Create** the app with its own volume and the environment above, on a preview hostname, with `SANDCMS_INDEXING=false` and `SANDCMS_BOOTSTRAP_EMAIL`. First start migrates the database and creates the admin.
|
|
205
|
+
2. **Launch:** point the domain's DNS at the host, add the hostnames on the platform, set `SANDCMS_ORIGIN`, remove `SANDCMS_INDEXING` and `SANDCMS_BOOTSTRAP_EMAIL`, restart. Other hostnames now redirect to the origin.
|
|
206
|
+
3. **Roll back** a bad deploy with the platform's rollback. If that deploy applied a migration, the old code refuses the newer database (`store.ts:91-92`); also restore the `before-migrate` snapshot with `sandcms restore`.
|
|
207
|
+
|
|
208
|
+
### 6.4 Backup and restore
|
|
209
|
+
|
|
210
|
+
- Back up the whole volume on a schedule. The site may keep running.
|
|
211
|
+
- **Restore:** stop the site, restore the volume, run `sandcms restore --from /data/backups/latest.sqlite`, start the site. Always restore the database from the snapshot: the copy of the live database inside a volume backup can be torn.
|
|
212
|
+
- Up to one hour of content edits can be lost relative to the media files in the same backup. `sandcms media verify` reports media rows without files after a restore.
|
|
213
|
+
|
|
214
|
+
## 7. Upgrading a site from 0.6
|
|
215
|
+
|
|
216
|
+
1. Check that no uploads are pending, and take a backup.
|
|
217
|
+
2. Make sure the volume has free space of at least 3 × the site's current media.
|
|
218
|
+
3. Install `sandcms@0.7.0`, run `sandcms upgrade --check`, then `sandcms upgrade`. Confirm `cmsHandle` runs first in `sequence()`, and remove `paths.origin` from `vite.config.ts`.
|
|
219
|
+
4. Set `BODY_SIZE_LIMIT=10M` and switch the host to stop-then-start deploys.
|
|
220
|
+
5. Deploy with `SANDCMS_WORKER_URL` and `SANDCMS_WORKER_TOKEN` still set. Migration `0009_local_media` applies; existing data is preserved.
|
|
221
|
+
6. Run `sandcms media import-r2` and check the site.
|
|
222
|
+
7. Remove the Worker env and redeploy. Delete the Worker and its R2 bucket once a restore from backup has been verified to contain originals.
|
|
223
|
+
|
|
224
|
+
Going back to 0.6 after step 5 means restoring the `before-migrate` snapshot along with the old image. `docs/upgrades.md`, `docs/deployment.md`, `docs/media.md` and the README are rewritten for this release; `docs/cloudflare.md` is reduced to the legacy import.
|
|
225
|
+
|
|
226
|
+
## 8. Testing
|
|
227
|
+
|
|
228
|
+
- **Unit:**
|
|
229
|
+
- storage: durable writes, no-overwrite promote, sweep, usage, reservations and quota, deletion with a missing row;
|
|
230
|
+
- upload: ticket signature, expiry and user binding; session and API-key authorization; chunk retry, resume, oversize chunk, missing `Content-Length`; session expiry and deletion during an upload;
|
|
231
|
+
- processor fixtures: JPEG with EXIF orientation, 16-bit TIFF, CMYK JPEG, PNG with alpha, animated GIF, an SVG declared as PNG, rejection above `maxPixels`, corrupt file;
|
|
232
|
+
- processor lifecycle: child crash mid-batch, kill by signal, retry skipping finished variants, the processor lease against a CLI command;
|
|
233
|
+
- render dimensions matching current Worker results;
|
|
234
|
+
- body guard on non-upload routes;
|
|
235
|
+
- delivery: ETag/304, security headers, `Content-Length`, missing file, publication revoked;
|
|
236
|
+
- import against a fake Worker: SVG, originals without stored dimensions, non-active ready versions, errors with resume;
|
|
237
|
+
- instance lock (second process refused, released on crash), site-id mismatch, snapshots (rotation, catch-up, not started by CLI), canonical host, indexing unset vs `false`.
|
|
238
|
+
- **Migrations, per AGENTS.md:** fresh initialization, upgrade from 0.6.3 with data, repeated migration, rollback after a failed migration, rejection of incompatible databases, and a 0.6.3 build rejecting a 0.7 database.
|
|
239
|
+
- **e2e:** remove wrangler from `scripts/e2e-server.ts`. Cover a 70 MB chunked upload with progress, an interrupted upload that resumes, TIFF → variants, an API-key upload, and that a draft image isn't public.
|
|
240
|
+
- **Package and container:** `scripts/test-package.ts` builds a consumer site and processes an image through the bundled child. `scripts/test-container.ts` builds the starter `Dockerfile`, processes a large image under a memory limit, and checks the web process survives a child out-of-memory kill.
|
|
241
|
+
- **Load:** process real 70 MB originals on a production-sized server; record peak memory and time to set the documented memory limit and the `maxPixels` default.
|
|
242
|
+
- **Release gate:** `bun run release:check`, then `bun run build` and `bun run test:e2e`.
|
|
243
|
+
|
|
244
|
+
## 9. Work order
|
|
245
|
+
|
|
246
|
+
| Step | Scope |
|
|
247
|
+
| ---- | ----------------------------------------------------------------------- |
|
|
248
|
+
| 1 | Storage module, migration `0009`, delivery from disk |
|
|
249
|
+
| 2 | Processor child, runner integration, deletion and lease changes |
|
|
250
|
+
| 3 | Chunked upload: server routes, body guard, admin client, API-key access |
|
|
251
|
+
| 4 | Platform features (§5), starter `Dockerfile`, doctor |
|
|
252
|
+
| 5 | `media import-r2`, `media verify`, `media rebuild --force` |
|
|
253
|
+
| 6 | Remove Cloudflare setup; deprecations; setup/upgrade templates |
|
|
254
|
+
| 7 | Tests (§8), docs, release notes, load test |
|
|
255
|
+
|
|
256
|
+
## 10. Accepted risks and open items
|
|
257
|
+
|
|
258
|
+
- **Single volume, single server per site.** Recovery depends on the host's volume backups.
|
|
259
|
+
- **No CDN or request filtering** in front of sites unless the operator adds one.
|
|
260
|
+
- **Images are unavailable during the one-time R2 import** (D10).
|
|
261
|
+
- **Mixed encoders after import.** Imported variants were rendered by Cloudflare and later ones by sharp, under the same version, until a forced rebuild.
|
|
262
|
+
- **HEIC is not supported**: sharp's prebuilt binaries can't decode HEVC.
|
|
263
|
+
- **Open:** the memory limit and `maxPixels` default, set by the load test.
|
|
264
|
+
- **Later:** S3 media backend; removal of the deprecated Worker surface; optional host automation.
|