@coywolf/emdash 0.0.0-stage → 0.35.14
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +203 -2
- package/docs/README.md +45 -0
- package/docs/ai.md +39 -0
- package/docs/backups.md +88 -0
- package/docs/breadcrumbs.md +59 -0
- package/docs/code-blocks.md +21 -0
- package/docs/content-urls.md +41 -0
- package/docs/custom-blocks.md +27 -0
- package/docs/discovery.md +32 -0
- package/docs/files.md +44 -0
- package/docs/form-uploads.md +59 -0
- package/docs/headings.md +37 -0
- package/docs/images.md +141 -0
- package/docs/internals.md +13 -0
- package/docs/links.md +34 -0
- package/docs/performance.md +72 -0
- package/docs/redirects.md +84 -0
- package/docs/releasing.md +21 -0
- package/docs/reviews.md +89 -0
- package/docs/robots.md +81 -0
- package/docs/schema.md +79 -0
- package/docs/search.md +114 -0
- package/docs/videos.md +48 -0
- package/docs/wordpress-import.md +198 -0
- package/package.json +79 -5
- package/src/admin/admin-css.ts +57 -0
- package/src/admin/ai.tsx +915 -0
- package/src/admin/backups.tsx +613 -0
- package/src/admin/breadcrumbs.tsx +542 -0
- package/src/admin/codeBlocks.tsx +204 -0
- package/src/admin/customBlocks.tsx +321 -0
- package/src/admin/dirty.ts +20 -0
- package/src/admin/discovery.tsx +463 -0
- package/src/admin/download.ts +28 -0
- package/src/admin/feature-switches.tsx +124 -0
- package/src/admin/features.tsx +142 -0
- package/src/admin/files.tsx +791 -0
- package/src/admin/formUploads.tsx +445 -0
- package/src/admin/guides.tsx +18 -0
- package/src/admin/headings.tsx +262 -0
- package/src/admin/images.tsx +825 -0
- package/src/admin/index.tsx +62 -0
- package/src/admin/links.tsx +980 -0
- package/src/admin/performance.tsx +330 -0
- package/src/admin/preview.tsx +63 -0
- package/src/admin/redirects-removed.tsx +266 -0
- package/src/admin/redirects.tsx +470 -0
- package/src/admin/reviews.tsx +232 -0
- package/src/admin/robots-bots.tsx +647 -0
- package/src/admin/robots-shared.tsx +109 -0
- package/src/admin/robots-wizard.tsx +748 -0
- package/src/admin/robots.tsx +911 -0
- package/src/admin/save-bar.tsx +134 -0
- package/src/admin/schema.tsx +1187 -0
- package/src/admin/search.tsx +355 -0
- package/src/admin/settings-ui.tsx +87 -0
- package/src/admin/trash-prompt-core.ts +135 -0
- package/src/admin/trash-prompt.tsx +156 -0
- package/src/admin/usage.tsx +301 -0
- package/src/admin/videos-display.tsx +248 -0
- package/src/admin/videos.tsx +1202 -0
- package/src/admin/wpImport.tsx +1516 -0
- package/src/ai/entities.ts +59 -0
- package/src/ai/export.ts +61 -0
- package/src/ai/logic.ts +606 -0
- package/src/ai/pack.ts +121 -0
- package/src/ai/prompts.ts +87 -0
- package/src/ai/providers.ts +276 -0
- package/src/ai/routes.ts +359 -0
- package/src/ai/store.ts +301 -0
- package/src/ai/wikidata.ts +87 -0
- package/src/ai/worker.ts +530 -0
- package/src/astro/astro-modules.d.ts +6 -0
- package/src/astro/breadcrumbs/Breadcrumbs.astro +129 -0
- package/src/astro/breadcrumbs/BreadcrumbsBlock.astro +12 -0
- package/src/astro/codeBlocks/CodeBlock.astro +62 -0
- package/src/astro/codeBlocks/EmDashCode.astro +61 -0
- package/src/astro/customBlocks/CustomBlock.astro +64 -0
- package/src/astro/customBlocks/Details.astro +8 -0
- package/src/astro/customBlocks/Disclosure.astro +8 -0
- package/src/astro/customBlocks/Note.astro +8 -0
- package/src/astro/customBlocks/Podcast.astro +8 -0
- package/src/astro/customBlocks/Quote.astro +8 -0
- package/src/astro/customBlocks/Testimonial.astro +8 -0
- package/src/astro/files/FileCard.astro +103 -0
- package/src/astro/headings/HeadingBlock.astro +129 -0
- package/src/astro/headings/TableOfContents.astro +45 -0
- package/src/astro/headings/request.ts +19 -0
- package/src/astro/images/ResponsiveImage.astro +60 -0
- package/src/astro/index.ts +84 -0
- package/src/astro/reviews/CoywolfReview.astro +27 -0
- package/src/astro/reviews/Review.astro +49 -0
- package/src/astro/search/SearchBox.astro +61 -0
- package/src/astro/search/SearchBoxForm.astro +509 -0
- package/src/astro/tables/CaptionedTable.astro +72 -0
- package/src/astro/videos/CoywolfVideo.astro +448 -0
- package/src/backups/download.ts +112 -0
- package/src/backups/dump.ts +288 -0
- package/src/backups/module.ts +276 -0
- package/src/backups/pack.ts +33 -0
- package/src/backups/restore.ts +209 -0
- package/src/backups/store.ts +409 -0
- package/src/breadcrumbs/pack.ts +113 -0
- package/src/breadcrumbs/settings.ts +102 -0
- package/src/breadcrumbs/stamp.ts +40 -0
- package/src/breadcrumbs/trail.ts +123 -0
- package/src/codeBlocks/contrast.ts +179 -0
- package/src/codeBlocks/hljs.d.ts +5 -0
- package/src/codeBlocks/pack.ts +110 -0
- package/src/codeBlocks/render.ts +304 -0
- package/src/codeBlocks/settings.ts +18 -0
- package/src/codeBlocks/themes.generated.ts +304 -0
- package/src/codeBlocks/themes.ts +114 -0
- package/src/core/compose.ts +244 -0
- package/src/core/content-url.ts +888 -0
- package/src/core/csv.ts +119 -0
- package/src/core/d1-batch.ts +65 -0
- package/src/core/entry-docs.ts +175 -0
- package/src/core/features-module.ts +48 -0
- package/src/core/features.ts +392 -0
- package/src/core/generation.ts +88 -0
- package/src/core/guide-content.ts +166 -0
- package/src/core/module.ts +104 -0
- package/src/core/read-failures.ts +57 -0
- package/src/core/request-memo.ts +71 -0
- package/src/core/secrets.ts +83 -0
- package/src/core/settings-enhance.ts +74 -0
- package/src/core/site.ts +36 -0
- package/src/core/storage.ts +27 -0
- package/src/customBlocks/pack.ts +260 -0
- package/src/customBlocks/render.ts +424 -0
- package/src/customBlocks/rich.ts +313 -0
- package/src/customBlocks/settings.ts +46 -0
- package/src/discovery/build.ts +252 -0
- package/src/discovery/feed-build.ts +217 -0
- package/src/discovery/feed.ts +172 -0
- package/src/discovery/indexnow.ts +114 -0
- package/src/discovery/llms.ts +79 -0
- package/src/discovery/markdown.ts +294 -0
- package/src/discovery/memory-cache.ts +97 -0
- package/src/discovery/middleware.ts +186 -0
- package/src/discovery/module.ts +353 -0
- package/src/discovery/news.ts +99 -0
- package/src/discovery/pack.ts +58 -0
- package/src/discovery/settings.ts +91 -0
- package/src/discovery/urls.ts +35 -0
- package/src/env.d.ts +10 -0
- package/src/files/card.ts +231 -0
- package/src/files/counts.ts +60 -0
- package/src/files/export.ts +44 -0
- package/src/files/format.ts +163 -0
- package/src/files/module.ts +596 -0
- package/src/files/pack.ts +92 -0
- package/src/files/r2-rules.ts +41 -0
- package/src/files/range.ts +50 -0
- package/src/files/s3.ts +114 -0
- package/src/files/serve.ts +141 -0
- package/src/files/sigv4.ts +160 -0
- package/src/files/site.ts +137 -0
- package/src/files/walker.ts +58 -0
- package/src/formUploads/lib.ts +266 -0
- package/src/formUploads/pack.ts +55 -0
- package/src/formUploads/wrap.ts +592 -0
- package/src/forms.ts +16 -0
- package/src/headings/pack.ts +154 -0
- package/src/headings/render.ts +139 -0
- package/src/headings/settings.ts +117 -0
- package/src/headings/slug.ts +68 -0
- package/src/headings/stamp.ts +142 -0
- package/src/headings/test-hooks.mjs +14 -0
- package/src/headings/toc.ts +48 -0
- package/src/images/cloudflare.ts +329 -0
- package/src/images/lib.ts +221 -0
- package/src/images/module.ts +372 -0
- package/src/images/pack.ts +804 -0
- package/src/images/pending.ts +37 -0
- package/src/images/settings.ts +48 -0
- package/src/images/setup.ts +192 -0
- package/src/images/variants-estimate.ts +162 -0
- package/src/images/variants-store.ts +475 -0
- package/src/images/variants.ts +311 -0
- package/src/index.ts +143 -0
- package/src/links/check.ts +101 -0
- package/src/links/classify.ts +270 -0
- package/src/links/export.ts +59 -0
- package/src/links/module.ts +652 -0
- package/src/links/pack.ts +43 -0
- package/src/links/pt.ts +325 -0
- package/src/links/store.ts +704 -0
- package/src/middleware.ts +287 -0
- package/src/modules.ts +53 -0
- package/src/options.ts +66 -0
- package/src/pageCache/lib.ts +274 -0
- package/src/pageCache/pack.ts +240 -0
- package/src/pageCache/usage-core.ts +152 -0
- package/src/pageCache/usage.ts +262 -0
- package/src/pageCache/warm.ts +722 -0
- package/src/redirects/middleware.ts +169 -0
- package/src/redirects/module.ts +124 -0
- package/src/redirects/pack.ts +89 -0
- package/src/redirects/removed-core.ts +151 -0
- package/src/redirects/removed.ts +293 -0
- package/src/redirects/rules.ts +355 -0
- package/src/redirects/slug-changes.ts +207 -0
- package/src/redirects/slug-core.ts +49 -0
- package/src/redirects/transfer.ts +92 -0
- package/src/reviews/lib.ts +624 -0
- package/src/reviews/pack.ts +135 -0
- package/src/reviews/schema.ts +128 -0
- package/src/reviews/settings.ts +26 -0
- package/src/robots/bots.ts +321 -0
- package/src/robots/data/bots.json +696 -0
- package/src/robots/data/verified.json +582 -0
- package/src/robots/directory.ts +26 -0
- package/src/robots/explain.ts +191 -0
- package/src/robots/guided.ts +300 -0
- package/src/robots/importer.ts +263 -0
- package/src/robots/middleware.ts +151 -0
- package/src/robots/module.ts +446 -0
- package/src/robots/pack.ts +69 -0
- package/src/robots/radar.ts +129 -0
- package/src/robots/rep.ts +362 -0
- package/src/robots/rules.ts +883 -0
- package/src/robots/transfer.ts +45 -0
- package/src/robots/validate.ts +524 -0
- package/src/schema/catalog.ts +146 -0
- package/src/schema/graph.ts +791 -0
- package/src/schema/module.ts +1138 -0
- package/src/schema/og-image.ts +100 -0
- package/src/schema/pack.ts +58 -0
- package/src/search/engine.ts +578 -0
- package/src/search/fallback.ts +80 -0
- package/src/search/live-cache.ts +116 -0
- package/src/search/live-client.ts +770 -0
- package/src/search/live-serve.ts +282 -0
- package/src/search/module.ts +261 -0
- package/src/search/pack.ts +57 -0
- package/src/search/query.ts +132 -0
- package/src/search/ratelimit-core.ts +118 -0
- package/src/search/ratelimit.ts +79 -0
- package/src/search/snippet.ts +135 -0
- package/src/search/title-index.ts +139 -0
- package/src/search/title-match.ts +192 -0
- package/src/shared.ts +71 -0
- package/src/version.ts +2 -0
- package/src/videos/date.ts +40 -0
- package/src/videos/engagement.ts +75 -0
- package/src/videos/lib.ts +679 -0
- package/src/videos/module.ts +1160 -0
- package/src/videos/pack.ts +187 -0
- package/src/videos/pause.ts +42 -0
- package/src/videos/poster.ts +324 -0
- package/src/videos/render.ts +336 -0
- package/src/videos/store.ts +551 -0
- package/src/videos/stream.ts +134 -0
- package/src/wpImport/convert.ts +702 -0
- package/src/wpImport/guests.ts +329 -0
- package/src/wpImport/gutenberg.ts +157 -0
- package/src/wpImport/markers.ts +66 -0
- package/src/wpImport/pack.ts +451 -0
- package/src/wpImport/parents.ts +202 -0
- package/src/wpImport/prepare.ts +592 -0
- package/src/wpImport/redirects.ts +358 -0
- package/src/wpImport/sources.ts +67 -0
- package/src/wpImport/stream.ts +160 -0
- package/src/wpImport/urls.ts +354 -0
package/docs/reviews.md
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
# Reviews
|
|
2
|
+
|
|
3
|
+
[Coywolf Pack documentation](README.md) · [Getting started](../README.md#getting-started)
|
|
4
|
+
|
|
5
|
+
The review box Coywolf uses on WordPress (Coywolf Custom Blocks' review block): a rating badge with "4.5 out of 5" under it, then **What I liked most** and **Could be better** lists side by side, stacking on narrow screens. It adds Review schema that Google can read for review snippets and product pros and cons.
|
|
6
|
+
|
|
7
|
+
- **Coywolf Review block** (slash menu → Content): item name, item type (the types Google accepts for reviews: Product, Software application, Book, Course, Movie, Game, Event, Recipe, Local business, Organization, and a few more), brand, item URL, item image (for schema only), book details (author, ISBN, publisher, genre, copyright year) and software details (operating systems, app category) for schema, rating (0–5 in tenths, e.g. 4.7), the two headings (defaults above), pros and cons (one per line), an optional summary or verdict, and the heading level (H2 by default, like WordPress). It's server-rendered plain HTML with no script. Its CSS is about 2 KB, inlined once per page before the first box, so nothing shifts.
|
|
8
|
+
- **Ratings** are picked from a menu of 5.0 down to 0.0 in steps of 0.1 (EmDash's number field only takes whole numbers). The badge and caption show the rating as stored ("4.7 out of 5", "5 out of 5"), and Review schema's `ratingValue` is the same number. Reviews saved before 0.11.0 with half steps keep working; saving the entry stores them in the new format (`"4"` becomes `"4.0"`) so the menu shows them.
|
|
9
|
+
- **Reviews page** (**Plugins → Coywolf Pack → Reviews**): the badge's accent color (default `#2C8452`; badge text is white), **Custom CSS**, and a live preview of a sample review, wide or at phone width. The preview runs in a sandboxed frame, so your CSS can't restyle the admin.
|
|
10
|
+
- **Accessible**: the box is a `<section>` labeled "Review of <item>". The badge number is hidden from screen readers, which read "Rated 4.5 out of 5" instead. The lists are real headings and lists.
|
|
11
|
+
|
|
12
|
+
| Feature | Default | What it adds |
|
|
13
|
+
| --- | --- | --- |
|
|
14
|
+
| `reviews` | off | The block on the site, and the Reviews page |
|
|
15
|
+
| `reviews.schema` | off | Review JSON-LD for each review, in Schema & Social's graph when it's on, otherwise one standalone script per review (`page:metadata`, ids `coywolf-review-N`) |
|
|
16
|
+
|
|
17
|
+
## Customizing the look
|
|
18
|
+
|
|
19
|
+
Every part has a stable class: `.cw-review` (the card), `__rating`, `__badge`, `__caption`, `__columns`, `__col` (plus `__pros` / `__cons`), `__heading`, `__list`, `__item` and `__summary`. Every design value is a CSS custom property. The built-in CSS only reads them, with fallbacks, so you can set them on `.cw-review`, on a wrapper, or on `:root`:
|
|
20
|
+
|
|
21
|
+
| Property | Default | |
|
|
22
|
+
| --- | --- | --- |
|
|
23
|
+
| `--cw-review-accent` | the Reviews page color | Badge background |
|
|
24
|
+
| `--cw-review-badge-color` | `#fff` | Badge text |
|
|
25
|
+
| `--cw-review-badge-size` / `-badge-padding` / `-badge-radius` | `3.5rem` / `.5rem 1rem` / `10%` | |
|
|
26
|
+
| `--cw-review-bg` / `--cw-review-bg-dark` | `#fff` / `#1d1f23` | Card background (light / dark); `--cw-review-bg` alone sets both |
|
|
27
|
+
| `--cw-review-border` | `1px solid` + the border color | Whole border shorthand (e.g. `0`); overrides the colors below |
|
|
28
|
+
| `--cw-review-border-color` / `--cw-review-border-color-dark` | `#dfe0e3` / `#3a3d44` | |
|
|
29
|
+
| `--cw-review-radius` / `-padding` / `-margin` | `12px` / `2rem 1rem` / `0 0 1.5rem` | |
|
|
30
|
+
| `--cw-review-color` / `-font` | `inherit` | Text color and font |
|
|
31
|
+
| `--cw-review-heading-color` / `-heading-size` / `-heading-transform` | `inherit` / `1.2rem` / `uppercase` | |
|
|
32
|
+
| `--cw-review-list-color` / `--cw-review-list-color-dark` / `-list-size` / `-list-style` | `#555` / `#c9ccd1` / `1rem` / `square` | `--cw-review-list-color` alone sets both |
|
|
33
|
+
| `--cw-review-caption-color` / `-caption-size` | `inherit` / `1rem` | The "4.5 out of 5" line |
|
|
34
|
+
| `--cw-review-gap` / `-column-gap` | `1rem 1.5rem` | Space between the badge and lists, and between the lists |
|
|
35
|
+
| `--cw-review-rating-width` | `7rem` | Badge column's minimum width |
|
|
36
|
+
| `--cw-review-stack-at` | `26rem` | Narrowest the lists area gets beside the badge before the badge moves above it |
|
|
37
|
+
| `--cw-review-column-min` | `13rem` | Narrowest a list column gets before the lists stack; `100%` keeps them in one column |
|
|
38
|
+
|
|
39
|
+
Light and dark colors use CSS `light-dark()`, so the box follows the page's own `color-scheme`: on a theme that declares `color-scheme: light dark` it turns dark with the page, and on a light-only theme (no `color-scheme`) it stays light whatever the visitor's system setting. The `-dark` properties are the dark half.
|
|
40
|
+
|
|
41
|
+
The layout uses flex and grid wrapping instead of a media query, so it adapts to the box's own width (a sidebar or a phone alike) and the "breakpoints" are the custom properties above.
|
|
42
|
+
|
|
43
|
+
Put overrides in **Custom CSS** on the Reviews page. It's added after the built-in styles on pages with a review, so it wins. Start every rule with `.cw-review` so it only reaches review boxes; it's printed as-is (not rewritten), with `</style`, `<!--` and `-->` removed and a 20 KB cap. For example, a crimson badge and one column:
|
|
44
|
+
|
|
45
|
+
```css
|
|
46
|
+
.cw-review {
|
|
47
|
+
--cw-review-accent: #b22d47;
|
|
48
|
+
--cw-review-column-min: 100%;
|
|
49
|
+
}
|
|
50
|
+
.cw-review .cw-review__heading { text-transform: none; }
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Review schema
|
|
54
|
+
|
|
55
|
+
With Schema & Social's graph on, each review joins the page's `@graph` like videos do. The reviews come from the entry's `coywolf-review` blocks (any Portable Text field), followed by any the theme passes as `page.coywolf.reviews`, de-duplicated by item type and name, up to 20.
|
|
56
|
+
|
|
57
|
+
- **Products** become a top-level `Product` (`<page>#review-N-item`, with name, `brand` as a `Brand`, url and image). The `Review` (`<page>#review-N`) is nested in it as `review`. Google reads pros and cons (`positiveNotes` / `negativeNotes`, each an `ItemList` of `ListItem`s with position and name) only from a Review nested in a Product, and a nested review needs no `itemReviewed`.
|
|
58
|
+
- **Other types** become a top-level `Review` with the item in `itemReviewed`. That's the shape Google's review-snippet docs show, and it avoids validating, say, an Event or SoftwareApplication as its own rich result.
|
|
59
|
+
- **Pros and cons** are emitted only for Product reviews with at least two statements in all (Google's minimum); they still show in the box either way.
|
|
60
|
+
- **Books and software** get their details on the item: a Book's `author` (Person), `isbn`, `publisher` (Organization), `genre` and `copyrightYear`; a SoftwareApplication's `operatingSystem` and `applicationCategory`.
|
|
61
|
+
- **Every review** has a `reviewRating` (`Rating`, `ratingValue`, `bestRating` 5, `worstRating` 0, because the scale starts at 0), the Article's author(s) as `author` (else the publisher), the graph's publisher, the Article's `datePublished`, `mainEntityOfPage` → the WebPage, and the summary as `reviewBody`. The item is added to the Article's (or WebPage's) `about`, alongside AI Enrichment's entities.
|
|
62
|
+
- **Not emitted**: reviews without an item name or a rating, and "self-serving" reviews, which Google ignores. Those are an Organization or Local business whose URL is on your own site or whose name is the publisher's.
|
|
63
|
+
|
|
64
|
+
With the graph off, each review is a standalone JSON-LD script in the same shape. The author is the page's byline name (else the site), and the publisher is the site. Some item types need more properties for their own rich results (an Event's date and location, for example). The block doesn't collect those, so validate important pages in Google's Rich Results Test. Google's product snippets focus on one product per page: several Product reviews on one page are all output, but Google may use only one of them.
|
|
65
|
+
|
|
66
|
+
## Legacy WordPress reviews
|
|
67
|
+
|
|
68
|
+
Themes that render imported WordPress review markers (`{ name, brand, rating, strengths, shortcomings }`, with the lists as `<ul>` HTML) can use the same box and schema:
|
|
69
|
+
|
|
70
|
+
```astro
|
|
71
|
+
---
|
|
72
|
+
import { Review, reviewFromAttrs } from "@coywolf/emdash/astro";
|
|
73
|
+
---
|
|
74
|
+
<Review attrs={markerAttrs} /> <!-- or <Review {...markerAttrs} /> -->
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
and pass the same reviews to the page context for schema:
|
|
78
|
+
|
|
79
|
+
```astro
|
|
80
|
+
---
|
|
81
|
+
const reviews = reviewMarkers(post.data.content).map(reviewFromAttrs); // your marker parser
|
|
82
|
+
const page = { ...createPublicPageContext({ /* … */ }), coywolf: { videos, reviews } };
|
|
83
|
+
---
|
|
84
|
+
<EmDashHead page={page} />
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`reviewFromAttrs` reduces the `<li>` HTML to plain text (tags dropped, entities decoded) and maps the marker attributes to the block's fields. You can also pass `itemType`, `url`, `image` and `summary` attributes, and the WordPress book (`author`, `isbn`, `publisher`, `genre`, `copyright`) and software (`os`, `category`) fields. To turn imported markers into Coywolf Review blocks instead, see [Migrating from WordPress](wordpress-import.md). `<Review>` isn't gated by the feature switch, because a theme that uses it has opted in, but it uses the Reviews page's accent and custom CSS. Schema still needs `reviews.schema`. Remove any review JSON-LD the theme prints itself.
|
|
88
|
+
|
|
89
|
+
`reviews: false` leaves the module out.
|
package/docs/robots.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Robots.txt Rules
|
|
2
|
+
|
|
3
|
+
[Coywolf Pack documentation](README.md) · [Getting started](../README.md#getting-started)
|
|
4
|
+
|
|
5
|
+
Manage `robots.txt` in plain English instead of a text box. The page opens with a one-paragraph summary of the whole file (**Search engines can crawl everything. AI training crawlers are blocked from the whole site.**), then three tabs:
|
|
6
|
+
|
|
7
|
+
- **Rules**: each rule as a sentence (“Block AI training crawlers (10) from the /private/ section and everything in it”), with an on/off switch, Edit (the same guided dialog), Duplicate and Delete. Order doesn't matter: crawlers follow the most specific matching line, and the file is written most-specific-first so older top-to-bottom crawlers agree. Below the list: a URL tester (**Can GPTBot fetch /2026/my-post/?**) and **What's being served**, the live file with Copy and Download, plus EmDash's original file for reference.
|
|
8
|
+
- **Bots**: the crawler directory (about 700 bots) with search and filters for category, purpose, operator, verification status and “used in rules”. Mark a bot verified (source URL and note, stored with the date and your name), rename it (the token never changes), add bots the directory lacks (token checked against `[A-Za-z0-9._-]+`, with a purpose), edit or delete them, and sync from Cloudflare Radar.
|
|
9
|
+
- **Settings and history**: keep EmDash's admin private, keep media crawlable, “crawlers named in a rule also keep the rules for all crawlers”, sitemaps, rule-name comments, Extra lines (for experts), the last 20 saved versions with Restore, **Reset to EmDash's original**, and **Export rules** / **Import rules…**: the rules and settings as a JSON file, to keep a copy or move them to another site. An import goes through the same checks as any change (with a confirmation listing the effects), replaces the current rules and settings, and keeps the current version in history.
|
|
10
|
+
|
|
11
|
+
**Templates** (header) replace the rule list in one click, through the same checks: *Block AI training, allow AI search*, *Allow everything*, *Block everything except search engines*.
|
|
12
|
+
|
|
13
|
+
## Taking over from EmDash
|
|
14
|
+
|
|
15
|
+
Turning the feature on takes over seamlessly. The first `/robots.txt` request or the first visit to the page (whichever comes first) converts the robots.txt EmDash was serving (its SEO setting, or its built-in default) into rules: groups become rules (identical lines for several bots become one rule, and known crawler groups are recognized), `Sitemap:` lines become sitemap settings, other lines (Crawl-delay, Content-Signal…) become Extra lines with their `User-agent` lines. The import is checked: for every crawler named in either file plus a few well-known ones and a stand-in for “any other crawler”, and every path the files' lines target, both files must give the same allow/block verdict (Google's matcher). If no combination of settings reproduces the file exactly, the original is kept as Extra lines, so crawlers always see the same rules. The one deliberate change: media in EmDash's library (`/_emdash/api/media/`) is opened to crawlers so images can appear in image search, and the summary under **What's being served** says so. The import is saved with an insert-if-absent, so concurrent first requests can't import twice. Turning the feature off serves EmDash's robots.txt again and keeps your rules for next time.
|
|
16
|
+
|
|
17
|
+
## Adding a rule
|
|
18
|
+
|
|
19
|
+
**Add rule** opens a four-step dialog:
|
|
20
|
+
|
|
21
|
+
1. **What do you want to do?** Keep crawlers out, or let crawlers in.
|
|
22
|
+
2. **Which part of your site?** Everything · A section (with “also wherever this folder name appears deeper”) · One page (with “only this exact address”) · A kind of file (PDF, Word, spreadsheets, images, ZIP, video… or other extensions, optionally only inside a section) · Links with tracking or extra parameters (utm_…, ref, fbclid, gclid, session IDs, sort and filters, site search, any `?`, or one you name) · Advanced pattern. Paste any address from your site: it's trimmed to its path, encoded, and read (“We read this as the /recipes/ section”), with a one-click switch when another choice fits better. Your collections' sections are offered as chips. A live box shows addresses the rule **will match** and nearby ones it **won't**, from the real matcher.
|
|
23
|
+
3. **Which crawlers?** Everyone · Search engines · AI training · AI search and assistants · SEO tools · Pick specific bots (search, categories with tick-all, verified badges, or a token that isn't listed). Each preset lists its members. Presets are curated by **purpose** from the operator's own documentation (`purposes` in `src/robots/data/verified.json`, with source URLs), never by Radar category, and only include verified tokens; bots that only come from Radar never join a preset, while verified custom bots with a matching purpose do. AI training is GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Bytespider, Amazonbot, MistralAI-Training and Webzio-Extended. Rules made from a preset follow it: when a preset changes, the rule is updated on the next page load, with a one-time note listing crawlers added and removed.
|
|
24
|
+
4. **Review**: consequences first (“Search engines can fetch this section. 10 AI training crawlers can't.”), **Except…** for allowed items inside a blocked area, the name and note (prefilled), **Try a URL**, the exact lines under **Show robots.txt lines**, and the checks.
|
|
25
|
+
|
|
26
|
+
## Checks
|
|
27
|
+
|
|
28
|
+
Every change is checked in the browser as you type and again on the server before saving. Errors block saving; warnings need “I understand, add it anyway”.
|
|
29
|
+
|
|
30
|
+
- **Errors**: invalid crawler tokens, exceptions outside the blocked area (or ones that reopen all of it), line breaks, control characters or `#` in a value, patterns not starting with `/` or `*`, values over 2,083 characters, bad extensions or parameter names, an exact duplicate of another rule, the same crawlers both allowed and blocked on the same path, and a failed self-check.
|
|
31
|
+
- **Warnings**: a block on a discovery file that's kept readable, an Allow that nothing blocks, a block already covered by a wider block (naming it), a rule for everyone that named crawlers won't follow (with “Also apply it to these crawlers”), naming a crawler that then ignores the rules for all crawlers (RFC 9309 group selection; with the setting off, offers to turn it on), search engines blocked from your pages, CSS/JavaScript/images that renderers need, the media library, the sitemap, opening EmDash's admin, and blocks that the “Keep media crawlable” setting overrides.
|
|
32
|
+
- **Notes**: an Allow inside a block (valid; explained), where a more specific rule still decides, a partial duplicate (with **Merge**), unknown or unverified tokens, and case-sensitive paths.
|
|
33
|
+
- **Self-check**: after generating the file, every rule's crawlers are tested on addresses the rule targets, and the parsed file must give the verdict the rules say; crawlers the rule doesn't name must get the same verdict with and without it. A failure names the rule, crawler, address and deciding line.
|
|
34
|
+
|
|
35
|
+
## Automatic lines and the sitemap
|
|
36
|
+
|
|
37
|
+
**Discovery files stay readable.** Any crawler that your rules would block from `/.well-known/` (security.txt, ai-plugin.json and future machine-discovery manifests), from `/llms.txt` while Discovery's llms.txt feature is on (`/llms-full.txt` holds your full text, so add it yourself if you want it), or from discovery paths you add, gets an `Allow` line for them. The Rules tab lists these as **Automatic** lines with the reason. Turn the allowance off, or add paths, under **Settings and history**. Feature switches are read when the file is generated (middleware and admin preview), the self-check confirms each allowance, and a rule that targets a discovery file gets a warning.
|
|
38
|
+
|
|
39
|
+
**List the site's sitemap** sits at the top of the Rules tab. It's on for new and imported setups. A saved choice to turn it off is kept (for example when you submit sitemaps in Search Console), with a small note you can dismiss.
|
|
40
|
+
|
|
41
|
+
## How the file is written
|
|
42
|
+
|
|
43
|
+
One group per set of crawlers that follow the same lines. With “keep the rules for all crawlers” on (the default), a named crawler's group also gets the general rules, except where its own rules cover the same addresses (so “block GPTBot from the whole site” beats a general Allow). Every crawler not blocked from the whole site gets `Allow: /_emdash/api/media/` and `Disallow: /_emdash/`. We serve the file rather than writing EmDash's **SEO → robots.txt** setting because that setting is capped at 5,000 characters and couldn't keep EmDash's lines in every group.
|
|
44
|
+
|
|
45
|
+
Extra lines that would block search engines or the whole site are flagged on the page (they don't block saving). If EmDash's robots.txt settings can't be read during the takeover, nothing is imported and EmDash's file keeps being served until a later request succeeds.
|
|
46
|
+
|
|
47
|
+
Rules saved before 0.7.0 keep working unchanged: the stored config is read with defaults for the new fields, and “keep the rules for all crawlers” is turned on only when that changes no verdict.
|
|
48
|
+
|
|
49
|
+
Feature switches: **Robots.txt Rules** (`robots`) and **Weekly crawler list from Cloudflare Radar** (`robots.radarSync`). Both default to off.
|
|
50
|
+
|
|
51
|
+
## Setup
|
|
52
|
+
|
|
53
|
+
Uses the same `coywolfPack()` middleware as Redirects, and the `DB` binding. A static `public/robots.txt` in the site would be served by Workers static assets before the Worker runs, so remove it. Rules are read at most once a minute per Worker isolate; history and bot changes live in plugin storage (`robots_history`, `robots_bot_overrides`), the dismissed sitemap note in KV; the response is cached for an hour (`Cache-Control: public, max-age=3600`).
|
|
54
|
+
|
|
55
|
+
## The crawler directory
|
|
56
|
+
|
|
57
|
+
A robots.txt `User-agent:` line takes a product token (`GPTBot`, `Claude-SearchBot`, `meta-externalagent`), which often isn't the bot's name or full user-agent string. The bundled directory (`src/robots/data/bots.json`) starts from Cloudflare Radar's bot directory and Coywolf SEO's curated tokens, and records for every bot how its token was confirmed:
|
|
58
|
+
|
|
59
|
+
- **verified, operator docs**: the operator's own documentation names the token (`sourceUrl`, `verifiedAt`). The major AI crawlers and search engines were re-checked on 2026-10-03: OpenAI, Anthropic, Google (including Google-Extended), Bing, Apple (Applebot-Extended), Perplexity, Common Crawl, Amazon, Meta, DuckDuckGo, Yandex, Baidu, Mistral, Webz.io and ImageSift.
|
|
60
|
+
- **verified, user agent**: the token appears in the user-agent string Radar publishes for the bot.
|
|
61
|
+
- **unverified**: derived from a pattern or name, or a legacy token the operator no longer documents (for example `anthropic-ai`, `Claude-Web`, `FacebookBot`, `cohere-ai`). These still work in robots.txt if the crawler uses them, and are flagged in the picker.
|
|
62
|
+
|
|
63
|
+
Signed agents that identify only cryptographically (Web Bot Auth) and have no token, such as ChatGPT agent, are left out. Bots that share a token (for example the nine regional Amazon Bedrock AgentCore browsers) are one entry, and hash-like identifiers that aren't product tokens are flagged unverified.
|
|
64
|
+
|
|
65
|
+
To rebuild the list, run `scripts/build-bots.mjs`. It reads Cloudflare Radar's directory (`RADAR_API_TOKEN`, or a snapshot saved earlier with `--save-snapshot` and passed back with `--radar-snapshot`), Coywolf SEO's curated tokens (`--wp <dir>`, or fetched with `gh api` from coywolf-llc/coywolf-seo), and the hand-verified tokens in `src/robots/data/verified.json`. To verify a token, add it to `verified.json` with the operator's documentation URL and the date you checked it, then rerun:
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
RADAR_API_TOKEN=… node scripts/build-bots.mjs
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Weekly refresh from Cloudflare Radar
|
|
72
|
+
|
|
73
|
+
With `robots.radarSync` on and a token set, the `robots-refresh-bots` task runs weekly (and **Refresh from Radar** runs it now). It makes one request, `GET https://api.cloudflare.com/client/v4/radar/bots?limit=1000`, and stores only differences from the bundled list in plugin storage (`robots_bots`), so a quiet week writes nothing. Radar publishes user-agent patterns, not robots.txt tokens, so bots that first appear this way get a token derived from their pattern and stay **unverified** until the bundled list is updated with a source. Bots that leave Radar stay usable and are marked.
|
|
74
|
+
|
|
75
|
+
Create a Cloudflare API token with **Account → Radar → Read** and either paste it into the setup card on the Robots.txt page's **Bots** tab (stored encrypted; it's also on the plugin's generic Settings page) or set it as a Worker secret:
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
npx wrangler secret put RADAR_API_TOKEN
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
The module declares `network:request` for `api.cloudflare.com` only.
|
package/docs/schema.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Schema & Social
|
|
2
|
+
|
|
3
|
+
[Coywolf Pack documentation](README.md) · [Getting started](../README.md#getting-started)
|
|
4
|
+
|
|
5
|
+
The structured data and social tags from Coywolf SEO for WordPress, configured on **Plugins → Schema**. Everything is off until you turn it on under **Plugins → Coywolf Pack → Schema & Social**:
|
|
6
|
+
|
|
7
|
+
| Feature | What it does |
|
|
8
|
+
| --- | --- |
|
|
9
|
+
| `schema` | The module's main switch. |
|
|
10
|
+
| `schema.graph` | One JSON-LD `@graph` per page in place of EmDash's built-in JSON-LD (it uses the same `primary` id, so EmDash's is replaced, not duplicated): WebSite (with a SearchAction), the publisher Organization or Person, a typed WebPage, the primary ImageObject (with dimensions and alt), author Person nodes, and a typed Article. It keeps everything EmDash's own JSON-LD has: headline, description, image, dates, author and publisher. |
|
|
11
|
+
| `schema.breadcrumbs` | A BreadcrumbList. Uses the theme's `breadcrumbs` from the page context when it passes them (`[]` means none), otherwise it's derived from the URL path (parent segments, then the page title). **If your theme prints its own BreadcrumbList, remove it when you turn this on**, or pages will have two. |
|
|
12
|
+
| `schema.robots` | A robots meta tag: `index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1` by default, merged with each entry's **No index** (which stays `noindex, nofollow`). |
|
|
13
|
+
| `schema.openGraph` | `og:locale`, plus `og:image:width`, `og:image:height`, `og:image:type`, `og:image:alt` and `twitter:image:alt` when the image is from the media library. |
|
|
14
|
+
| `schema.authors` | Schema.org Person properties per byline (job title, sameAs profiles, image, description…), used as article authors. |
|
|
15
|
+
|
|
16
|
+
The Schema page has:
|
|
17
|
+
|
|
18
|
+
- **Site details**: whether the publisher is an Organization or a Person (a byline), and the publisher's properties, picked from the same property list as on WordPress (name, legalName, url, logo, sameAs, address, contactPoint, founder, and so on; logo and image can come from the media library). Anything left empty falls back to EmDash's site title, URL and logo.
|
|
19
|
+
- **Types**: the WebPage subtype (AboutPage, ContactPage, CollectionPage…) and Article subtype (BlogPosting, NewsArticle, TechArticle… or none) for the home page, other non-entry pages, and each collection. By default, article pages get a BlogPosting, as with EmDash's own JSON-LD, and everything else a WebPage.
|
|
20
|
+
- **Authors**: Person properties per byline. Bylines without saved properties use their name, website (or the author page URL), bio and avatar. Tick **Profile page only** on a property (birthDate, birthPlace, email, gender, nationality…) to output it only on the page whose main subject is that person, not on every article they wrote.
|
|
21
|
+
- **Overrides**: for a single entry, a different page or article type, and its **main subject**: what the page is about. Pick a byline for a profile page, or the site publisher for an About or Contact page. See [Main subject](#main-subject).
|
|
22
|
+
- **Robots & social**: the search URL template (`/?s={search_term_string}` by default; empty for no SearchAction), the author page URL pattern (empty by default: set it to match your site's author pages, e.g. `/author/{slug}/`, so authors get a `url` and `@id` there; without it, authors without a byline website get no `url` and an `@id` on the site root), the breadcrumb home label, the robots directives (a blank max-snippet or max-video-preview means -1, no limit), and an `og:locale` override (derived from the site locale otherwise, e.g. `en` → `en_US`).
|
|
23
|
+
- **Preview**: the tags and JSON-LD the module would add to the home page or an entry.
|
|
24
|
+
|
|
25
|
+
## Setup
|
|
26
|
+
|
|
27
|
+
Nothing to configure beyond the Schema page. The module reads the site database (`DB` by default; set `schema: { database: "MY_DB" }` otherwise) to look up image dimensions and alt text, cached per Worker isolate for 10 minutes. Its settings come with the feature switches' query; its Site Details, types and author properties are cached until a pack settings save (here or in another isolate, see [Database reads](internals.md)) or ten minutes. It needs the `content:read`, `schema:read`, `bylines:read` and `media:read` capabilities, which it declares.
|
|
28
|
+
|
|
29
|
+
### Pass the entry to save reads
|
|
30
|
+
|
|
31
|
+
A theme that renders an entry already has it (from `getEmDashEntry()`), with its bylines hydrated by EmDash. Pass it on the page context as `coywolf.entry`, and Schema & Social takes the credited bylines (`entry.data.bylines`) and the entry's blocks (for review schema) from it instead of reading them again: two to five fewer queries per page. It's only used when `entry.data.id` is the page's `content.id`. `coywolf.bylines` (just the credits) works too.
|
|
32
|
+
|
|
33
|
+
```astro
|
|
34
|
+
---
|
|
35
|
+
const { entry } = await getEmDashEntry("posts", slug);
|
|
36
|
+
const page = { ...createPublicPageContext({ Astro, kind: "content", content: { collection: "posts", id: entry.data.id, slug: entry.id }, /* … */ }), coywolf: { entry } };
|
|
37
|
+
---
|
|
38
|
+
<EmDashHead page={page} />
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
## Notes
|
|
42
|
+
|
|
43
|
+
- Page types come from the page context your theme passes to `EmDashHead`: an entry page is matched to its collection through `content`, and `pageType: "article"` is what makes a page an article by default. Pages without `content` use the home page or "other pages" types.
|
|
44
|
+
- Derived breadcrumbs name parent segments from the URL (`/health-tips/` → "Health tips"); pass `breadcrumbs` in the page context for exact names.
|
|
45
|
+
- A content page costs a few extra database reads per render (the entry override, its bylines, and saved author properties); image lookups and settings are cached.
|
|
46
|
+
- **Default bylines: link your byline to your user account under Bylines** (Content → Bylines → your byline → user; that also turns off **Guest**). EmDash then credits every post or page you create without picking a byline to your byline: the editor shows it as **Automatic**, and EmDash's reads (`entry.data.bylines`, `getEntriesBylines`) return it like a picked one, so the article's author schema names you with no change here. EmDash stores no credit for it, so `where: { byline }` (and `getEntriesByByline`) don't list those entries: an author page that should has to add the user's own (`where: { author_id }`) entries credited to the byline, as the Coywolf themes' `src/lib/byline-entries.ts` does. To link from the command line: `node scripts/byline-link-user.mjs --url=https://coywolf.com` (a dry run that finds the site's admin and the byline with their name and lists the entries that will start showing it; `--byline=<slug>` and `--user=<id>` pick others, `--apply` saves, `--purge-cache` then clears the page cache). It uses EmDash's admin API with your `npx emdash login` (or `EMDASH_TOKEN`) and sends the byline's bio, avatar and website back with it, since EmDash's byline save clears them when they're left out.
|
|
47
|
+
|
|
48
|
+
## Main subject
|
|
49
|
+
|
|
50
|
+
An override's **Main subject** makes the page's WebPage node point at what the page is about with `mainEntity`, and puts that entity in the graph once:
|
|
51
|
+
|
|
52
|
+
- **A byline**: the Person from the Authors tab, with the same `@id` it has as an article author, plus its **Profile page only** properties. The entry's own credited bylines are listed first.
|
|
53
|
+
- **Site publisher**: the publisher Organization (or Person) from Site details.
|
|
54
|
+
|
|
55
|
+
For an author's profile page, set the page type to **Profile page**, the article type to **None**, and the main subject to the person:
|
|
56
|
+
|
|
57
|
+
```json
|
|
58
|
+
{ "@type": "ProfilePage", "@id": "https://coywolf.com/jon-henshaw/#webpage", "mainEntity": { "@id": "https://coywolf.com/jon-henshaw/#person" }, "dateCreated": "…", "dateModified": "…" },
|
|
59
|
+
{ "@type": "Person", "@id": "https://coywolf.com/jon-henshaw/#person", "name": "Jon Henshaw", "birthDate": "1973-09-09", "…": "…" }
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
A ProfilePage also gets `dateCreated` (the entry's publish date) next to `dateModified`. When the main subject is the publisher, or is also credited as the article's author, it's still one node: the copies are merged. `birthPlace`, `homeLocation` and `workLocation` are output as a `Place` and `nationality` as a `Country` (named by the value).
|
|
63
|
+
|
|
64
|
+
## Videos in the graph
|
|
65
|
+
|
|
66
|
+
With the graph on, embedded videos are VideoObject nodes inside it, linked from the Article (or WebPage) as `video`: videos from the Videos module (when Video schema is on) and any a theme passes on the page context:
|
|
67
|
+
|
|
68
|
+
```astro
|
|
69
|
+
---
|
|
70
|
+
const page = { ...createPublicPageContext({ /* … */ }), coywolf: { videos: [
|
|
71
|
+
{ name, description, thumbnailUrl, embedUrl, contentUrl, uploadDate, duration: "PT18S" },
|
|
72
|
+
] } };
|
|
73
|
+
---
|
|
74
|
+
<EmDashHead page={page} />
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
Theme videos are validated (absolute http(s) URLs, ISO 8601 duration) and de-duplicated by embed or content URL. When the graph is on, the Videos module doesn't print separate VideoObject blocks.
|
|
78
|
+
|
|
79
|
+
A video's details (metadata, plays and likes, whether it's in a published entry) are read once per request: the schema and every Coywolf Video block on the page share one D1 batch.
|
package/docs/search.md
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# Search
|
|
2
|
+
|
|
3
|
+
[Coywolf Pack documentation](README.md) · [Getting started](../README.md#getting-started)
|
|
4
|
+
|
|
5
|
+
EmDash has full-text search built in: SQLite FTS5 with BM25 ranking, English stemming, prefix matching, highlighted snippets, a public API (`/_emdash/api/search`), and a `LiveSearch` component. This module adds what it leaves out. Turn on **Search** and its parts on the Coywolf Pack page. Search is off by default; once it's on, **Live results** is on too, and the other parts are off until you turn them on.
|
|
6
|
+
|
|
7
|
+
- **Search settings** (`search.settings`): a **Search** admin page to choose which collections are searchable, set field weights, pick the tokenizer (English stemming, exact words, or trigram substrings), and rebuild indexes with a progress readout and per-collection entry counts. It calls EmDash's own search API, so it needs EmDash's `search:manage` permission (admins). Fields are made searchable in each collection's schema.
|
|
8
|
+
- **Live results** (`search.live`, on by default with Search): matching entries in a dropdown as visitors type into the site's search box, like the Coywolf Search WordPress plugin. See [Live results](#live-results).
|
|
9
|
+
- **Search box** (`search.box`): a `SearchBox` component with as-you-type suggestions while **Live results** is on (titles first, then full text, with excerpts), arrow keys, Enter and Escape, a clear button, content-type labels, highlighted matches, screen-reader announcements, and a fade that respects reduced motion. When nothing matches every word, it shows results for any of the words, ranked by how many they contain. Without JavaScript it's a plain search form that submits to your search page.
|
|
10
|
+
- **Search rate limit** (`search.rateLimit`): limits each visitor to 120 searches a minute (configurable) on EmDash's public search and suggestion endpoints and the pack's search and live results routes, answering `429` with `Retry-After`. Visitors are keyed by a salted hash of their IP address, never the address itself.
|
|
11
|
+
|
|
12
|
+
## Live results
|
|
13
|
+
|
|
14
|
+
With **Live results** on, every public page gets a small deferred script at the end of the body (via EmDash's page fragments, so the layout needs `<EmDashBodyEnd>`): `<script src="/_emdash/api/plugins/coywolf-pack/search/live-client?v=<version>" defer data-cw-live="…">`, with the page's settings in the attribute. The file (about 21 KB, 6.5 KB compressed) has a versioned URL, so browsers download it once and keep it for a year (`immutable`); a new pack version changes the URL. Only the pack middleware serves that file (EmDash's plugin routes can't answer with a JavaScript content type or set its cache headers), so on sites without the middleware the same client is added inline at the end of the body instead. It attaches to any GET form with a search field (`type="search"`, or a field named `s` or `q`), so a theme's own search form works as is; the pack's `SearchBox` has the same dropdown built in. Pages without a search form do nothing with it, and its styles are added only when a form is found.
|
|
15
|
+
|
|
16
|
+
- **Instant title matches.** The first time a visitor focuses a search field, the script loads a compact title index (every published entry's title, URL and type, newest first, up to 5,000) and sends a tiny warm-up request so the server is ready. From then on, title matches appear as soon as a key is pressed, with no server round trip. Matching ignores case and accents ("ecole" finds "École"); titles that start with what was typed come first, then titles with every word at the start of a word, then titles that contain the words (3+ letters).
|
|
17
|
+
- From 2 characters, 120 ms after the last keystroke, it also asks the server, which answers with up to 8 entries: title matches first, then full-text matches (with the any-word fallback). The answer merges into the list without reshuffling it: rows the server also found gain their excerpt in place, its other results are added below, and when the list is full, title matches the server didn't confirm make room. Each row has the title, with the typed words underlined, and an excerpt of about 180 characters around the first match, with the matches in bold and "…" where it was cut. A final **View all results** row opens the search page, the same URL the form submits.
|
|
18
|
+
- The first result is selected as results appear, so Enter opens it. Arrow keys move (wrapping, and through View all results), Escape closes the list and a second Escape clears the field, Tab or a click elsewhere closes it, and submitting the form searches as before. Results are real links, so middle-click and "open in new tab" work.
|
|
19
|
+
- Accessible as an ARIA combobox: `role="combobox"` with `aria-expanded`, `aria-controls` and `aria-activedescendant` on the field, a `listbox` of `option`s, and a polite live region announcing the result count (with a one-time keyboard hint). When the list fills in twice (instant matches, then the server's answer), the count is announced once, for the final list, unless the server takes longer than 0.7 seconds. A result picked with the arrow keys or the pointer stays picked when the server's answer arrives. A clear (×) button sits inside the field for pointer and touch; Escape does the same from the keyboard. Forced colors are respected, and the fade is skipped for reduced motion.
|
|
20
|
+
- It takes the theme's font and text color from the form and its background from the nearest opaque ancestor, and tints with `currentColor`, so it fits light and dark themes without configuration.
|
|
21
|
+
- In-flight requests are cancelled as you type, answers are cached per query for the page view, and a slow earlier answer never replaces a newer one. Without JavaScript the form works exactly as before.
|
|
22
|
+
|
|
23
|
+
Results come from `GET /_emdash/api/plugins/coywolf-pack/search/live?q=…` (optional `limit` up to 20, `collections`, `locale`): published entries only, each with `title`, `titleHtml`, `url`, `type` and `snippet`. `titleHtml` and `snippet` are escaped HTML whose only tags are `<mark>`. Excerpts come from the text EmDash indexed (the searchable fields other than the title). `?warm=1` does nothing but load the search setup into a server isolate (answers `204`). The title index is `GET /_emdash/api/plugins/coywolf-pack/search/index?v=…` (optional `locale`, `collections`): `{ v, types, entries: [[title, url, typeIndex], …] }`.
|
|
24
|
+
|
|
25
|
+
**Speed and caching.** With the pack middleware installed (`coywolfPack()` in `src/middleware.ts`), the middleware answers both URLs itself, before EmDash's plugin routing:
|
|
26
|
+
|
|
27
|
+
- Every answer is stored in Cloudflare's cache (the Cache API, per location; live answers for 5 minutes, title indexes for a day) and in a small per-isolate memory cache. The key is the query (trimmed, spaces collapsed, lower-cased), `limit`, `collections`, `locale` and the **search content version**. Browsers keep live answers for 60 seconds, and the title index for a year when the page's version matches (its URL changes with every version), 5 minutes otherwise.
|
|
28
|
+
- The content version changes whenever published content does (publish, unpublish, update, delete, restore), and when search settings change on the **Search** admin page. New keys miss, so visitors never see results from before an edit; old copies just expire. Other Worker isolates pick up a new version within five minutes (it's read with the feature switches).
|
|
29
|
+
- Cache hits skip the database and the rate limit. Misses count against **Search rate limit** when it's on.
|
|
30
|
+
- A miss reads the database in a few batched round trips: search settings (cached per isolate for a minute), then title and full-text matches for every collection at once, then excerpts and URL terms at once. The any-word fallback adds one more. That's 2–3 D1 round trips (about 7 statements) where it was 25–31 queries.
|
|
31
|
+
- Responses carry `Server-Timing` (`cw-search` says `hit-memory`, `hit-edge`, `miss` or `warm`; `cw-d1` counts statements and round trips) and `X-Coywolf-Cache: HIT` or `MISS`.
|
|
32
|
+
|
|
33
|
+
Without the middleware, the plugin routes run the same batched search, without the Cache API layer.
|
|
34
|
+
|
|
35
|
+
Tune it in `astro.config.mjs` (defaults shown):
|
|
36
|
+
|
|
37
|
+
```js
|
|
38
|
+
coywolfPlugin({ search: { live: { limit: 8, minChars: 2, debounce: 120, enterOpensTop: true, instant: true, indexMax: 5000 } } });
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
`enterOpensTop: false` leaves nothing selected until the visitor arrows to a result, so Enter runs a full search. `instant: false` skips the title index (server results only). `indexMax` caps the title index (newest entries kept); at 5,000 entries it's roughly 100–150 KB compressed, loaded once per content version.
|
|
42
|
+
|
|
43
|
+
To measure on a live site:
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
# A miss, then a hit (same query, any case or spacing).
|
|
47
|
+
curl -s -o /dev/null -D - "https://coywolf.com/_emdash/api/plugins/coywolf-pack/search/live?q=wolf%20$RANDOM" | grep -iE "server-timing|x-coywolf-cache|cache-control"
|
|
48
|
+
curl -s -o /dev/null -D - "https://coywolf.com/_emdash/api/plugins/coywolf-pack/search/live?q=wolf" | grep -iE "server-timing|x-coywolf-cache"
|
|
49
|
+
# The title index and the warm-up.
|
|
50
|
+
curl -s --compressed -o /dev/null -w "%{size_download} bytes, %{time_total}s\n" "https://coywolf.com/_emdash/api/plugins/coywolf-pack/search/index?v=x"
|
|
51
|
+
curl -s -o /dev/null -D - "https://coywolf.com/_emdash/api/plugins/coywolf-pack/search/live?warm=1" | grep -i server-timing
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Search box
|
|
55
|
+
|
|
56
|
+
```astro
|
|
57
|
+
---
|
|
58
|
+
import { SearchBox } from "@coywolf/emdash/astro";
|
|
59
|
+
---
|
|
60
|
+
<SearchBox action="/search" collections={["posts", "pages"]} placeholder="Search articles" />
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Props: `action` (your search page, default `/search`), `name` (`q`), `label`, `showLabel`, `placeholder`, `collections`, `locale`, `minChars` (2), `debounce` (200 ms; its answers come from the same edge-cached live results route), `limit` (8), `showType`, `showSnippets`, `submitButton`, `value`, `class`, and `id` (set a different one for each box on a page). Its script (about 4 KB minified, 2 KB compressed) and styles load only on pages that render it, and nothing at all is sent while **Search box** is off. Suggestions are real links, so middle-click and "open in new tab" work.
|
|
64
|
+
|
|
65
|
+
Colors are CSS custom properties with light and dark defaults. Override them on `.cw-search`:
|
|
66
|
+
|
|
67
|
+
```css
|
|
68
|
+
.cw-search {
|
|
69
|
+
--cw-search-bg: #fff;
|
|
70
|
+
--cw-search-fg: #1a1a1a;
|
|
71
|
+
--cw-search-muted: #5c5c5c;
|
|
72
|
+
--cw-search-border: #d4d4d4;
|
|
73
|
+
--cw-search-active-bg: #f1f1f1;
|
|
74
|
+
--cw-search-active-border: currentColor;
|
|
75
|
+
--cw-search-active-fg: inherit;
|
|
76
|
+
--cw-search-mark: rgb(255 214 0 / 0.35);
|
|
77
|
+
--cw-search-radius: 0.5rem;
|
|
78
|
+
}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Suggestions come from the live results route (above); with **Live results** off the SearchBox is a plain search form with a clear button. `GET /_emdash/api/plugins/coywolf-pack/search/query?q=…&mode=suggest` still returns EmDash's results plus each entry's URL (from the collection's URL pattern) and type label. Use the same OR fallback on your search page:
|
|
82
|
+
|
|
83
|
+
```astro
|
|
84
|
+
---
|
|
85
|
+
import { searchWithFallback } from "@coywolf/emdash/astro";
|
|
86
|
+
const q = Astro.url.searchParams.get("q")?.trim() ?? "";
|
|
87
|
+
const { items, fallback } = q ? await searchWithFallback(q, { limit: 20 }) : { items: [], fallback: false };
|
|
88
|
+
---
|
|
89
|
+
{fallback && <p>No results for all of your words. Showing results for some of them.</p>}
|
|
90
|
+
{items.map((item) => <a href={item.url}>{item.title}</a>)}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Fallback results aren't paginated (the best 50 are ranked together).
|
|
94
|
+
|
|
95
|
+
## Rate limit setup
|
|
96
|
+
|
|
97
|
+
The limit is enforced by a [Workers Rate Limiting](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/) binding when you add one. Its limit and period (10 or 60 seconds) come from `wrangler.jsonc`:
|
|
98
|
+
|
|
99
|
+
```jsonc
|
|
100
|
+
"ratelimits": [
|
|
101
|
+
{ "name": "SEARCH_RATE_LIMITER", "namespace_id": "1001", "simple": { "limit": 120, "period": 60 } }
|
|
102
|
+
]
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
```js
|
|
106
|
+
coywolfPlugin({ search: { rateLimiter: "SEARCH_RATE_LIMITER" } });
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
- `rateLimiter`: the binding's name. Counting happens at Cloudflare (per location), so it holds across Worker isolates. Blocked requests get `Retry-After: 60`.
|
|
110
|
+
- Without a binding, each Worker isolate counts on its own (`requestsPerMinute`, default 120; `0` turns it off). That's best effort: it stops one client hammering one isolate, but it isn't a hard limit.
|
|
111
|
+
- Optional Worker secret `SEARCH_RATE_LIMIT_SALT` for the IP hash (recommended; a fixed built-in salt is used otherwise, so keys agree across isolates).
|
|
112
|
+
- Requires the pack middleware (`coywolfPack()` in `src/middleware.ts`). The admin endpoints (enable, rebuild, stats) aren't limited.
|
|
113
|
+
|
|
114
|
+
For a hard limit that never reaches your Worker, use a Cloudflare WAF rate limiting rule instead of (or as well as) this feature. The free plan includes one: **Security → WAF → Rate limiting rules**, match `URI Path starts with /_emdash/api/search` or `URI Path starts with /_emdash/api/plugins/coywolf-pack/search/`, counted per IP, for example 60 requests per 10 seconds with a 10-second block.
|
package/docs/videos.md
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Videos
|
|
2
|
+
|
|
3
|
+
[Coywolf Pack documentation](README.md) · [Getting started](../README.md#getting-started)
|
|
4
|
+
|
|
5
|
+
The Coywolf Video Manager for EmDash, on Cloudflare Stream.
|
|
6
|
+
|
|
7
|
+
EmDash 1.2.0 added its own **Video** block (`video`): a media-library video in the browser's player, or a media provider's embed, with a caption. It's a different block from **Coywolf Video** (`coywolf-video`), which plays Cloudflare Stream videos with the light embed, posters, views and likes, schema and the sitemap below. Both can be used side by side; the pack doesn't render or index EmDash's.
|
|
8
|
+
|
|
9
|
+
- **Coywolf Video block** (slash menu → Media): pick a video from your Stream library, then set a title, description, poster (a frame time or an image), start time, controls/autoplay/loop/muted/preload, full or maximum width, and whether to show the name, description, views, a like button, the like count and the upload date: **Site default** (follow **Videos → Settings**), **Show** or **Hide**. The **Loop like a GIF** style plays muted, autoplaying and looping with no controls. Server-rendered in a `<figure>` sized by the video's aspect ratio, so nothing shifts.
|
|
10
|
+
- **Light embed** (the default; **Videos → Player → Load the player only when it's needed**): pages show the video's poster as a responsive image (Stream thumbnails at 480, 800 and 1200 px) with a play button, and load Stream's player (about 350 KB of JavaScript) when someone presses play, already playing. Autoplaying and GIF-style videos show the poster until the visitor's first scroll, touch, pointer move or key press, then load their player once on screen (loading it on idle still ran Stream's ~1.5 s of player script inside PageSpeed's measuring window). Without it, a video near the top of a page loads with the page and, on a slow phone connection, becomes its largest paint several seconds late (coywolf.com's mobile Performance score was 70 on such a page, 100 on the home page). Without JavaScript the player loads as before. Turn the setting off to load the player with the page.
|
|
11
|
+
- **Pause and reduced motion** (WCAG 2.2.2): autoplaying videos without controls (including the GIF style) show a pause/play button in the corner, so the motion can always be stopped; pausing before the player has loaded keeps the poster (also when that press is the visitor's first interaction), and a pause pressed while the player is still starting holds once it's ready. Visitors whose system asks for reduced motion get the poster and a play button instead of autoplay.
|
|
12
|
+
- **Posters from your media host**: with **Clean Image URLs** on and a media host set, each video's poster is copied from Stream into the media bucket the first time it's shown (`cwposter-<uid>-<hash>.jpg`, one per video and poster time), then served as resized WebP/AVIF from the media host like any other image. Stream makes thumbnails on request: even cached ones took 120–450 ms to start arriving (over a second for a new one), always as JPEG, from a host the browser had to connect to first. Rendering never waits for the copy: the copied posters are listed in one plugin setting (`videosMirroredPosters`, read with the feature switches, so it adds no query), and a poster that isn't listed yet is shown from Stream for that page view while it's copied and listed in the background. A page shown that way is cached for 5 minutes (plus 1 minute served while refreshing) instead of the Page cache lifetime, so the next render uses the media host; when cache warming rendered it, it isn't cached at all and the warmer visits it again once the poster is copied (see **Cache warming**). This needs the pack middleware: since Astro streams pages and a video renders after the page's headers are ready, the middleware reads each cacheable HTML page in full before returning it (it does for every cacheable HTML page, so it can also keep a page rendered after a failed read out of the cache; see [Page cache](performance.md#page-cache)). Pass `Astro.locals` as the last argument of `hostedPosterImage()` in your own components so the page is marked exactly; without it the middleware still notices, but may also shorten another page rendered at the same moment. Without a media host, or if a copy fails, Stream's thumbnails are used and the copy is retried later. Themes can use `hostedPosterImage()` from `@coywolf/emdash/astro` for their own video markup. A newly copied poster is resized at each size right away (AVIF and WebP), so the first visitor doesn't wait for it. Posters load lazily; when a video is the first thing on the page, set `Astro.locals.coywolfPriorityKey` to its block's `_key` and its poster loads at once with high priority.
|
|
13
|
+
- **Views & likes** (**Videos → Settings**, like Video Manager's): site-wide defaults for showing the name, description, views, like button, like count and upload date (all on), the date style (**Oct 6, 2026** or **7 months ago**; a relative date is refreshed in the browser, since pages are cached). Blocks set to **Site default** follow them. Blocks saved before these settings existed (with the old on/off toggles) and blocks imported from WordPress with an earlier pack keep their own choices; **Make all video blocks follow the site defaults** ignores those older choices without editing any content (turn it off to bring them back). Blocks set to Show or Hide always win. The WordPress importer now sets these only when the WordPress block did (or when the WordPress site had the option off), as Video Manager inherited them, and always as Show or Hide. Older blocks store those choices as true/false, which the block editor can't show (it says **Site default** while the page follows the stored choice). `node scripts/video-show-migrate.mjs --url=https://coywolf.com` lists them; add `--apply` to rewrite them as Show/Hide (or, with **Make all video blocks follow the site defaults** on, as Site default), so pages render the same. It uses EmDash's content API with your `npx emdash login` (or `EMDASH_TOKEN`), backs up each entry to a local folder first, republishes published entries (keeping their published date), saves drafts as drafts, and skips published entries with pending draft changes. Republishing sets each entry's updated date to now (EmDash's API can't save without it), which a theme may show as **Updated** and use for `article:modified_time`, sitemap `lastmod` and feed dates; `node scripts/video-show-migrate.mjs --url=https://coywolf.com --restore-updated-at --backup-dir=<the backup folder>` lists the original dates and writes the guarded SQL that puts them back, and with `--apply --d1=<database>` (run from the folder with the site's wrangler config) runs it through `wrangler d1 execute --remote` and clears the page cache. An entry edited since the migration keeps its new date.
|
|
14
|
+
- **Appearance** (**Videos → Settings**, with a live preview of unsaved changes): color scheme (auto, always light, always dark, or off to use the theme's text color), alignment of the name and of the like/views/date row, corner radius, an optional border, player background, name and description color, size and weights, the like icon (heart, thumbs up or star, Video Manager's icons), like colors (unclicked, clicked and their backgrounds; hovering previews the other state, and the clicked state holds until the pointer leaves), and the views/date color and size. The like icon pops when clicked (not with reduced motion). Each Coywolf Video block can override the alignments, corner radius and border (on/off, width, color) for that video; left empty, it uses these settings (WordPress imports keep Video Manager's per-block values where they were set). Only settings changed from their defaults are printed (as `--cw-video-*` custom properties on the figure), so a site that changes nothing looks as before, and theme CSS that targets `.cw-video__*` still wins. The figure's CSS is printed once per page.
|
|
15
|
+
- **Admin** (**Plugins → Coywolf Pack → Videos**): the library with thumbnails, tags, length, upload date, plays, likes and the number of entries each video is used in; search (with `#tag`) and a filter (with plays, with likes, used, not used); Stream storage used; a notice for embedded videos no longer in Stream; edit name, description, tags, creator, poster, allowed origins and MP4 downloads; delete (from Stream and the site, with a warning listing the entries that use it); captions; and uploads (tagged with the default tags from **Settings → Uploads** unless you enter others) that go straight from the browser to Stream (tus for files over 200 MB), so they never pass through the Worker.
|
|
16
|
+
- **Embed index**: saving, publishing, unpublishing, restoring or deleting an entry records which videos it embeds (any Portable Text field, at any depth). **Rebuild embed index** scans existing content. WordPress `cloudflare-stream` marker blocks (as imported to wellbeing.io) are indexed too, for usage counts and the sitemap.
|
|
17
|
+
|
|
18
|
+
| Feature | Default | What it adds |
|
|
19
|
+
| --- | --- | --- |
|
|
20
|
+
| `videos` | off | The module: library, uploads, block, index |
|
|
21
|
+
| `videos.schema` | off | A VideoObject JSON-LD script per embedded video (`page:metadata`, one script per video, never `primary`): name, description, 1200px thumbnail, ISO 8601 duration, upload date, embed URL, the MP4 as `contentUrl` when downloads are on, view and like counts when plays and likes are on, and caption tracks plus a transcript (up to 10,000 characters) when captions are on |
|
|
22
|
+
| `videos.sitemap` | off | A Google video sitemap at `/coywolf-video-sitemap.xml` for published entries (cached 10 minutes; rebuilt when content changes) |
|
|
23
|
+
| `videos.engagement` | off | Plays (counted once per browser session after 2 seconds of playback, not for autoplaying videos) and likes (one per visitor per day, using a daily-salted hash of the IP address; no cookies, nothing personal stored) |
|
|
24
|
+
| `videos.captions` | off | Upload WebVTT, generate captions with Stream, delete; ready tracks are copied to the site and served at `/coywolf-video-captions/<id>/<lang>.vtt` |
|
|
25
|
+
| `videos.webhook` | off | A Stream webhook (HMAC-verified) that refreshes a video's details as soon as Stream finishes processing it; **Subscribe** on the Videos page |
|
|
26
|
+
|
|
27
|
+
## Setup
|
|
28
|
+
|
|
29
|
+
1. Create an API token with **Account → Stream → Edit**.
|
|
30
|
+
2. On the **Videos** page, enter the account ID and the token in the **Connect** card (or later under **Videos → Settings**; the token is a secret setting, which needs EmDash's `EMDASH_ENCRYPTION_KEY`). Until Stream is connected, the page shows only that card: the library, uploads, captions and the webhook appear once it's connected. Or set `CF_ACCOUNT_ID` and the `CF_STREAM_TOKEN` Worker secret, the same variables EmDash's `cloudflareStream()` media provider reads. Optional settings (**Videos → Settings**): the customer subdomain (`customer-….cloudflarestream.com`, learned from the library if empty), the player accent color, Views & likes, Appearance (including the background color) and default upload tags.
|
|
31
|
+
3. Turn on **Videos** (and any sub-features) under **Plugins → Coywolf Pack**, open the **Videos** page, and click **Test connection**. Then **Rebuild embed index** once.
|
|
32
|
+
4. For the sitemap and caption files, add `coywolfPack()` to the site middleware (see Redirects), and list the sitemap in `robots.txt`:
|
|
33
|
+
|
|
34
|
+
```text
|
|
35
|
+
Sitemap: https://coywolf.com/coywolf-video-sitemap.xml
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Option: `videos: { maxUploadDurationSeconds: 3600 }` (the longest upload Stream accepts; Stream reserves that much quota while an upload is in progress). `videos: false` leaves the module out.
|
|
39
|
+
|
|
40
|
+
Use the block from theme code too: `import { CoywolfVideo } from "@coywolf/emdash/astro"` and `<CoywolfVideo node={{ uid, preset: "gif" }} />`. It also accepts the WordPress marker attributes (`id`, `host`, `aspect`, `name`, `description`, `seconds`), which default to the GIF style, so it can replace a theme's own Stream component. Those marker blocks don't get schema from this module, because a theme rendering them usually emits its own.
|
|
41
|
+
|
|
42
|
+
## Limits
|
|
43
|
+
|
|
44
|
+
- Stream lists up to 1,000 videos per call; the library is cached for 5 minutes (plugin KV and isolate memory). **Refresh** reloads it.
|
|
45
|
+
- Page views make no Stream API calls: player sizes, names and counts come from the site's own copy, filled when the library loads, when an entry with a new video is saved, and by the webhook.
|
|
46
|
+
- Plays aren't counted for autoplaying videos (including the GIF style), so muted previews don't inflate them.
|
|
47
|
+
- Plays and likes count per IP address (not per browser), and only for videos embedded in published entries: drafts' videos get no public data. Each address may send 10 plays and 20 likes a minute and add 100 likes a day; over that the routes answer 429. Likes must come from the Coywolf Video script (it sends an `X-Coywolf-Video: 1` header), so other sites can't like on a visitor's behalf; plays without the header still count, since pages cached before the header was added don't send it. People sharing an address (an office, a mobile carrier) share one like per video. Requests without an address (EmDash reads it from Cloudflare's request data, missing in local dev or behind some proxies) count no plays or likes, rather than all counting as one visitor. Visitors who block `embed.cloudflarestream.com` (the player SDK, loaded only where plays are counted) aren't counted.
|
|
48
|
+
- Signed URLs (`requireSignedURLs`) aren't supported.
|