@ketecode/loom 0.1.0 → 0.2.0

package/skills/ui-flow-audit/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: ui-flow-audit
+description: Static trace of every user-facing control in a frontend, following each click, toggle or submit through its event handler, state setters, effects, store actions, network calls and their responses to the final rendered state, to find interactions that each work alone but cancel each other, race, double-fire or leave the screen in the wrong state, reported flow by flow. Use when a feature behaves correctly in isolation yet breaks when actions are combined or repeated, a toggle snaps back, a filter or form resets unexpectedly, or someone wants an audit of how the UI's state changes fit together before a release.
+license: MIT
+compatibility: Any Agent Skills host. Framework-neutral (React, Vue, Svelte, Angular, vanilla); reading code is enough, running the app and its tests confirms findings.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: coding
+  kete-loom/risk: runs-commands
+  kete-loom/owner: "Kete-org"
+---
+
+# UI flow audit
+
+Bugs in interactive UIs often live between handlers, not inside them: a click sets state, an effect reacts and resets it, a response from an earlier request lands last and overwrites the newer one. Each piece passes its unit test. This audit traces the whole chain for each control and checks what the user finally sees.
+
+The audit does not change source files. It may run the app or tests to confirm a finding, with the user's permission. It produces a report; fixing is a separate step the user approves. Neighbours: `browser-qa` clicks through a running app; `react-reviewer` reviews a diff; `react-patterns` says how to structure state. This skill maps cause and effect across a whole screen or feature.
+
+## 0. Scope
+
+1. Agree the scope with the user: one screen, one feature, or a list of routes. A whole app at once gives a shallow report; **ask** to narrow it if there are more than ~30 controls.
+2. Identify the framework and the state tools in use (component state, context, Redux/Zustand/Pinia/NgRx stores, a query cache such as TanStack Query or SWR, URL/search params, form libraries, local storage).
+3. Note the expected behaviour for each flow. If there is a spec, PRD or test describing it, read it. If not, write your assumption next to each flow and flag it for confirmation.
+
+## 1. Inventory controls
+
+Search the scoped components for every user-facing trigger and list them:
+
+- Event props: `onClick`, `onChange`, `onSubmit`, `onBlur`, `onKeyDown`, `@click`, `v-model`, `(click)`, `bind:value`, `on:submit`.
+- Indirect triggers: route changes, URL param edits, drag/drop, keyboard shortcuts, timers and polling, websocket or SSE messages, window focus refetch, `storage` events from other tabs.
+
+Give each an ID (`F1`, `F2`...) with file and line.
+
+## 2. Trace each flow
+
+For each control, follow the chain until nothing else changes. Write it as a sequence:
+
+```
+F3 "Apply filter" button (Filters.tsx:42)
+ 1. onClick -> setFilters({...filters, status})            Filters.tsx:44
+ 2. filters is a dependency of useEffect -> setPage(1)     ResultsList.tsx:18
+ 3. [filters, page] in query key -> GET /api/items?status&page   useItems.ts:9
+ 4. URL sync effect writes ?status=... but not page        useUrlState.ts:31
+ 5. Response -> cache -> list renders
+ Final state: filter applied, page 1, URL missing page
+```
+
+At each step record: what changed, who reads it, and what that reader does. Follow:
+
+- Every setter call and every place that state is read (search for its name).
+- Effects/watchers/computed values that depend on it, and what they set in turn.
+- Store actions, reducers and middleware/sagas/listeners they trigger.
+- Network calls: request, success path, error path, and what happens if two are in flight.
+- Derived or mirrored state (props copied into state, state copied to URL or storage), the commonest source of fights.
+
+## 3. Cross-check pairs and sequences
+
+The bugs are in combinations. For each flow, check against the others that touch the same state:
+
+| Check | Question | Typical bug |
+|---|---|---|
+| Shared writes | Do two flows write the same state, URL param or cache key? | Second handler or an effect silently undoes the first |
+| Effect loops | Does an effect write state that triggers another effect that writes back? | Value snaps back, infinite render loop, flicker |
+| Order | Does the final state depend on which handler or effect runs first? | `onBlur` save vs `onClick` cancel; parent and child both handle the event |
+| Stale closure | Does a handler, timer or subscription capture an old value? | Increment uses the value from mount; interval sees initial filters |
+| Races | Can responses arrive out of order? Is there cancellation (`AbortController`, query keys, ignore flag)? | Typing "ab" shows results for "a" |
+| Repeat | What if the user clicks twice, double submits, or toggles fast? | Duplicate POST, toggle ends in wrong state, optimistic update not rolled back |
+| Optimistic updates | Is rollback on error correct, and does a refetch overwrite a newer local change? | Item reappears after delete; like count jumps |
+| Reset paths | What resets this state (route change, logout, form reset, key change)? Does it reset everything it should, and nothing else? | Old filter survives navigation; modal reopens with last user's data |
+| Remount | Does a changing `key`, conditional render or route remount the component and lose state? | Input clears while typing |
+| Disabled/loading | Is the control disabled while its action runs, and re-enabled on error? | Stuck spinner after a failed request |
+| External sync | Back/forward buttons, a second tab, a websocket event: do they agree with local state? | URL says page 3, list shows page 1 |
+
+## 4. Confirm
+
+For each suspected problem, try to confirm it before reporting:
+
+1. Re-read the exact lines; check the framework's semantics (for example, React 18+ batches all setters in the same tick, including in promises and timeouts; `useEffect` normally runs after paint, but after a discrete event such as a click React flushes it before the browser paints; Vue watchers are batched and by default run before the next component update, `flush: 'pre'`).
+2. If tests exist, look for one covering the combination. If the user allows running things, run the app and reproduce, or write a throwaway test outside the source tree (or delete it afterwards), and note the result. Do not commit anything.
+3. Mark each finding **confirmed** (reproduced or provable from the code alone) or **suspected** (depends on timing or data you could not check). Drop anything you cannot point to a line for.
+
+## Never
+
+- Never edit source files during the audit. Suggested fixes go in the report.
+- Never report a hypothetical without the sequence of steps that produces it.
+- Never assume a library's behaviour; check its docs or source when a finding depends on it (cache invalidation, batching, debounce semantics).
+
+## Report
+
+```
+## UI flow audit: <scope>
+Assumptions to confirm: <list>
+
+### Flow map
+F1 <control> (file:line): <one-line chain> -> final state OK | PROBLEM
+F2 ...
+
+### Findings (most severe first)
+1. [confirmed|suspected] <title>  Flows: F3 + F7
+   Steps: <user steps that trigger it>
+   Chain: <file:line -> file:line -> ...>
+   Expected: <...>  Actual: <...>
+   Fix: <smallest change, e.g. derive instead of mirror, cancel stale request, single owner for the URL param>
+   Test to add: <the combination test that would catch it>
+2. ...
+
+### Clean flows
+<IDs that were traced and are fine>
+
+### Not traced
+<controls or triggers skipped and why>
+```

package/skills/ui-flow-audit/evals/triggers.json

@@ -0,0 +1,17 @@
+{
+  "should_trigger": [
+    "When I change the sort order and then the filter, the sort snaps back to default; trace what every control on the search page actually does",
+    "Each button on the cart screen works on its own but combinations leave the wrong total, audit how the handlers and effects interact",
+    "Map every click handler, effect and store update on the booking form and find where they fight each other",
+    "The dark mode toggle flips back after a second, follow the state through the context, the effect and localStorage to see why",
+    "Before release, audit the kanban board's interactions for races, double submits and stale state between actions"
+  ],
+  "should_not_trigger": [
+    "Click through the staging build and write up any bugs you find before the demo",
+    "Review my React pull request for missing useEffect dependencies",
+    "Should this modal's open state live in the URL or in component state?",
+    "Write React Testing Library tests for the cart total component",
+    "Draw the user journey for onboarding as a flowchart for the product team",
+    "Our dashboard renders slowly when filters change, profile it"
+  ]
+}

package/skills/ui-motion/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: ui-motion
+description: Interface animation for the web with the Motion library (motion/react, formerly Framer Motion) and plain CSS transitions, keyframes and the View Transitions API - motion duration and easing tokens, spring physics, hover, tap and drag gestures, AnimatePresence exit animations, layout and shared-element transitions, route changes, prefers-reduced-motion fallbacks, and keeping animation on transform and opacity within the frame budget. Use when adding or fixing a micro-interaction, modal or list animation, page transition or draggable element, an animation stutters or causes layout shift, or motion must respect reduced-motion settings.
+license: MIT
+compatibility: Any Agent Skills host. Motion examples need React 18+ and the motion npm package; CSS and View Transitions sections apply to any framework.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: coding
+  kete-loom/risk: writes-files
+  kete-loom/owner: "Kete-org"
+---
+
+# UI motion
+
+Animation in a product UI has a job: show where something came from, confirm an action, or keep the user oriented during a change. If it does none of these, leave it out. Good UI motion is short, consistent, interruptible and optional.
+
+Neighbours: `remotion-video` and `manim-animations` render video; `ui-design-polish` sets the visual direction; `react-performance` covers render cost. This skill is live interface animation.
+
+## 0. Before you start
+
+1. Check what the project has: `motion` or `framer-motion` in `package.json`, a CSS animation convention, or existing duration/easing variables. Use what is there. Do not add Motion for something a CSS transition does.
+2. Note the import path in use. Current Motion: `import { motion, AnimatePresence } from "motion/react"`. Older code imports the same names from `"framer-motion"`; both work, do not mix them in one codebase. Migrating is a separate change; **ask first**.
+3. Ask what the motion is for if it is not obvious. "Make it feel alive" needs a concrete target: which element, which change, what it should communicate.
+
+## 1. Decide: CSS or Motion
+
+| Need | Use |
+|---|---|
+| Hover, focus, colour or simple show/hide state | CSS `transition` |
+| Looping or decorative keyframes (spinner, shimmer) | CSS `@keyframes` |
+| Exit animation when React unmounts an element | Motion `AnimatePresence` |
+| Element changes size/position because layout changed (reorder, expand) | Motion `layout` |
+| Same thing moves between two components (thumbnail to detail) | Motion `layoutId`, or View Transitions |
+| Drag, swipe, pan with momentum | Motion `drag` |
+| Interruptible, velocity-aware motion | Motion springs |
+| Page or route change in a multi-page or SPA app | View Transitions API where supported, else Motion `AnimatePresence` around the route |
+
+## 2. Motion tokens
+
+Define once and reuse (values to start from in `references/tokens.md`):
+
+- Durations: `--duration-fast` 100-150ms (hover, press), `--duration-base` 200-250ms (small enter/exit), `--duration-slow` 300-400ms (modals, page-level). UI animation over 500ms feels slow.
+- Easing: ease-out for entering (`cubic-bezier(0.2, 0, 0, 1)` style curves), ease-in for leaving, ease-in-out for moving on screen. Never linear except for progress and spinners.
+- Springs: two or three named presets (`snappy`, `gentle`, `bouncy`) as Motion transition objects.
+- Exits are faster than entrances (about 75% of the enter duration).
+
+## 3. Build it
+
+Core patterns with code in `references/patterns.md`:
+
+- **Enter/exit**: wrap conditionally rendered elements in `<AnimatePresence>`, give each child a stable `key`, set `initial`, `animate` and `exit`. Use `mode="wait"` when one element replaces another; `initial={false}` on `AnimatePresence` to skip the animation on first page load.
+- **Gestures**: `whileHover`, `whileTap`, `whileFocus` for feedback. Tap feedback is a small scale (0.97-0.98), not a bounce.
+- **Drag**: `drag` (`true`, `"x"` or `"y"`), `dragConstraints` (pixel object or a ref to a container), `dragElastic` (0-1), `dragMomentum={false}` to stop on release, `onDragEnd={(e, info) => ...}` using `info.offset` and `info.velocity` to decide a swipe. Always provide a keyboard or button alternative to a drag action.
+- **Layout**: `layout` on elements whose position or size changes; `layout="position"` when only position should animate (stops text squashing); `layoutId` for shared elements. Wrap siblings that affect each other in `<LayoutGroup>`.
+- **Lists**: stagger with `transition={{ delay: index * 0.03 }}` or variants with `staggerChildren`; cap total stagger at about 300ms so long lists do not take seconds.
+- **Springs**: `transition={{ type: "spring", visualDuration: 0.3, bounce: 0.2 }}` is easiest to reason about; or `stiffness`/`damping`/`mass` (these override `bounce` and `duration`). Bounce 0 for UI chrome, 0.1-0.3 for playful elements.
+- **Bundle size**: in bundle-sensitive apps use `LazyMotion` with `domAnimation` (no drag or layout) or `domMax` and the `m` component from `"motion/react-m"`.
+
+## 4. Reduced motion
+
+Required, not optional (WCAG 2.2 SC 2.3.3 Animation from Interactions is AAA, but vestibular disorders are common and the OS setting is the user's explicit request).
+
+- App-wide in Motion: `<MotionConfig reducedMotion="user">` at the root. It disables transform and layout animations when the OS setting is on and keeps opacity and colour changes.
+- Per component: `const reduce = useReducedMotion();` then swap movement for a fade.
+- CSS: `@media (prefers-reduced-motion: reduce) { ... }` removing translation, parallax, auto-play and large scale changes. Keep short opacity fades so state changes are still visible.
+- Auto-playing or looping motion longer than 5 seconds needs a pause control (WCAG 2.2.2).
+- Nothing should flash more than three times per second (WCAG 2.3.1).
+
+## 5. Performance
+
+- Animate `transform` (`x`, `y`, `scale`, `rotate`) and `opacity`. These can run on the compositor. Animating `width`, `height`, `top`, `left`, `margin` or `box-shadow` triggers layout or paint every frame; use `layout` animations or `transform: scale` instead.
+- Frame budget: 16.7ms at 60Hz, 8.3ms at 120Hz. Check with the browser's Performance panel (look for long tasks and layout/paint during the animation) and the Rendering panel's paint flashing.
+- `will-change: transform` only on elements about to animate, removed after; not on everything.
+- Do not drive animation from React state set every frame; use Motion values (`useMotionValue`, `useTransform`, `useScroll`) which update without re-rendering.
+- Scroll-linked effects: prefer CSS scroll-driven animations (`animation-timeline: scroll()` / `view()`) where browser support fits, or Motion `useScroll`; never a `scroll` event listener that sets state.
+- Test on a throttled CPU (4x slowdown) and a real low-end phone if the audience uses them.
+
+## 6. Verify
+
+1. Trigger each animation, then interrupt it (click again mid-animation, navigate away). It must not jump, stack or leave an element half-visible.
+2. Turn on the OS reduced-motion setting (or emulate `prefers-reduced-motion` in browser devtools) and repeat.
+3. Record a performance profile of the heaviest animation; no dropped frames on the target device class.
+4. Check keyboard: focus lands correctly after a modal opens or a route changes, and is not lost to an exiting element.
+5. Check no layout shift is introduced (elements entering should not push content after load).
+
+## Never
+
+- Never block input until an animation finishes, or make users wait for motion to reach content.
+- Never animate on page load just because you can; entrance animations on every scroll section get tiring.
+- Never use motion as the only signal of a state change.
+- Never mix `framer-motion` and `motion/react` imports, or two animation libraries on one element.
+
+## Report back
+
+```
+## Motion: <scope>
+Added/changed: <element: trigger -> animation, duration/spring, file>
+Tokens: <defined or reused>
+Reduced motion: <how each animation degrades>
+Performance: <properties animated; profile result or "not profiled">
+Interruptions tested: <yes/no, what>
+Not done: <items and why>
+```

package/skills/ui-motion/evals/triggers.json

@@ -0,0 +1,18 @@
+{
+  "should_trigger": [
+    "Make the notification toasts slide in and fade out when dismissed using motion/react",
+    "Our accordion animates height and it janks on mobile, make it smooth",
+    "Add a shared-element transition so the product thumbnail grows into the detail page image",
+    "Let users swipe cards left or right to dismiss them, with a spring back if they don't swipe far enough",
+    "How do I make all our Framer Motion animations respect prefers-reduced-motion?",
+    "Define duration and easing tokens so every hover and modal transition in the app feels consistent"
+  ],
+  "should_not_trigger": [
+    "Render a 30 second promo MP4 from React components with spring animations",
+    "Animate a visual proof of the chain rule for a maths video",
+    "Pick a colour palette and type scale so the admin panel stops looking generic",
+    "The product grid re-renders on every filter change, profile and fix it",
+    "Check that our modal traps focus and is announced by screen readers",
+    "Write Playwright tests for the drag-and-drop kanban board"
+  ]
+}

package/skills/ui-motion/references/patterns.md

@@ -0,0 +1,138 @@
+# Motion patterns
+
+Motion for React (`npm i motion`, imports from `"motion/react"`). Check the project's installed version; the API below matches current Motion docs.
+
+## Enter and exit
+
+```tsx
+import { AnimatePresence, motion } from "motion/react";
+
+<AnimatePresence>
+  {open && (
+    <motion.div
+      key="panel"
+      initial={{ opacity: 0, y: 8 }}
+      animate={{ opacity: 1, y: 0 }}
+      exit={{ opacity: 0, y: 4, transition: { duration: 0.12 } }}
+      transition={spring.gentle}
+    />
+  )}
+</AnimatePresence>
+```
+
+The conditional must be inside `AnimatePresence`; if the whole `AnimatePresence` unmounts, there is no exit animation.
+
+## Replace one element with another
+
+```tsx
+<AnimatePresence mode="wait" initial={false}>
+  <motion.span key={status} initial={{ opacity: 0 }} animate={{ opacity: 1 }} exit={{ opacity: 0 }}>
+    {label}
+  </motion.span>
+</AnimatePresence>
+```
+
+Changing `key` is what triggers exit then enter.
+
+## Reorderable list
+
+```tsx
+<ul>
+  <AnimatePresence initial={false}>
+    {items.map((item) => (
+      <motion.li key={item.id} layout initial={{ opacity: 0 }} animate={{ opacity: 1 }} exit={{ opacity: 0 }}>
+        {item.name}
+      </motion.li>
+    ))}
+  </AnimatePresence>
+</ul>
+```
+
+Use `item.id` keys, never the array index, or the wrong items animate. For drag-to-reorder, Motion has `Reorder.Group` / `Reorder.Item`.
+
+## Shared element
+
+```tsx
+// In the grid
+<motion.img layoutId={`photo-${id}`} src={thumb} />
+// In the detail view
+<motion.img layoutId={`photo-${id}`} src={full} />
+```
+
+If text inside a `layout` element distorts, add `layout` to the child as well, or use `layout="position"`.
+
+## Swipe to dismiss
+
+```tsx
+<motion.div
+  drag="x"
+  dragConstraints={{ left: 0, right: 0 }}
+  dragElastic={0.6}
+  onDragEnd={(_, info) => {
+    if (Math.abs(info.offset.x) > 120 || Math.abs(info.velocity.x) > 500) onDismiss();
+  }}
+/>
+```
+
+Also render a visible dismiss button: drag is not available to keyboard or switch users.
+
+## Scroll-linked progress bar
+
+```tsx
+const { scrollYProgress } = useScroll();
+return <motion.div style={{ scaleX: scrollYProgress, transformOrigin: "0 50%" }} className="progress" />;
+```
+
+CSS-only alternative where supported:
+
+```css
+.progress { transform-origin: 0 50%; animation: grow linear both; animation-timeline: scroll(); }
+@keyframes grow { from { transform: scaleX(0); } to { transform: scaleX(1); } }
+```
+
+## Reduced motion
+
+```tsx
+import { MotionConfig, useReducedMotion } from "motion/react";
+
+<MotionConfig reducedMotion="user">{app}</MotionConfig>   // "user" | "always" | "never"
+
+function Card() {
+  const reduce = useReducedMotion();
+  return <motion.div initial={{ opacity: 0, y: reduce ? 0 : 16 }} animate={{ opacity: 1, y: 0 }} />;
+}
+```
+
+## View Transitions API (no library)
+
+Same-document (SPA):
+
+```js
+if (!document.startViewTransition) updateDom();
+else document.startViewTransition(() => updateDom());
+```
+
+```css
+.hero-image { view-transition-name: hero; }          /* unique per page */
+::view-transition-old(root), ::view-transition-new(root) { animation-duration: 200ms; }
+@media (prefers-reduced-motion: reduce) {
+  ::view-transition-group(*), ::view-transition-old(*), ::view-transition-new(*) { animation: none; }
+}
+```
+
+Cross-document (multi-page sites): add `@view-transition { navigation: auto; }` to both pages' CSS. Check current browser support on MDN or caniuse before relying on it; it degrades to a normal navigation where unsupported.
+
+In frameworks, use their wrapper when one exists (for example Astro's `<ClientRouter />` or the router's view-transition option) rather than calling the API by hand around router internals.
+
+## Bundle-size mode
+
+```tsx
+import { LazyMotion, domAnimation } from "motion/react";
+import * as m from "motion/react-m";
+
+<LazyMotion features={domAnimation} strict>
+  <m.div animate={{ opacity: 1 }} />
+</LazyMotion>
+```
+
+`domAnimation` covers animations, variants, exit and tap/hover/focus gestures; `domMax` adds drag, pan and layout animations. `strict` throws if a full `motion` component sneaks in.

package/skills/ui-motion/references/tokens.md

@@ -0,0 +1,52 @@
+# Motion tokens
+
+Starting values; tune to the product's tone. Keep CSS and JS in sync by defining them once.
+
+```css
+:root {
+  --duration-instant: 75ms;
+  --duration-fast: 150ms;
+  --duration-base: 220ms;
+  --duration-slow: 350ms;
+
+  --ease-out: cubic-bezier(0.2, 0, 0, 1);       /* enter */
+  --ease-in: cubic-bezier(0.4, 0, 1, 1);        /* exit */
+  --ease-in-out: cubic-bezier(0.4, 0, 0.2, 1);  /* move on screen */
+}
+
+@media (prefers-reduced-motion: reduce) {
+  :root {
+    --duration-fast: 1ms;
+    --duration-base: 1ms;
+    --duration-slow: 1ms;
+  }
+}
+```
+
+Setting durations to `1ms` rather than `0` keeps `transitionend` / `animationend` events firing for code that waits on them. Only do this blanket reduction if every animation is decorative; otherwise keep opacity fades and remove movement per component.
+
+```ts
+// motion-tokens.ts
+import type { Transition } from "motion/react";
+
+export const spring = {
+  snappy: { type: "spring", visualDuration: 0.2, bounce: 0 },
+  gentle: { type: "spring", visualDuration: 0.35, bounce: 0.1 },
+  bouncy: { type: "spring", visualDuration: 0.45, bounce: 0.3 },
+} satisfies Record<string, Transition>;
+
+export const fade = { duration: 0.15, ease: [0.2, 0, 0, 1] } satisfies Transition;
+```
+
+## What to use where
+
+| Element | Enter | Exit | Notes |
+|---|---|---|---|
+| Button press | scale 0.97, `instant` | back, `fast` | `whileTap` |
+| Tooltip | opacity + 4px y, `fast` | opacity, `instant` | delay show 300-500ms, hide immediately |
+| Dropdown / popover | opacity + scale 0.96 from trigger origin, `fast` | opacity, `instant` | `transform-origin` at the trigger |
+| Toast | slide + fade, `gentle` spring | fade, `fast` | |
+| Modal | backdrop fade `base`; panel scale 0.96 + fade | faster than enter | return focus to the trigger |
+| Drawer / sheet | translate from edge, `snappy` spring | translate out | drag to dismiss with velocity threshold |
+| List add/remove | height via `layout`, fade | fade, then siblings close gap | `AnimatePresence` + `layout` |
+| Route change | crossfade `base` or shared element | | keep under 300ms |

package/skills/unicode-text-processing/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: unicode-text-processing
+description: Code-level correctness for text in complex and non-Latin scripts such as Bengali, Devanagari, Tamil, Arabic, Hebrew, Thai, Khmer, Chinese, Japanese and Korean - choosing NFC versus NFKC normalization and case folding, counting and truncating by grapheme cluster instead of code unit, detecting scripts from the Unicode Script property, dictionary-based word segmentation for languages written without spaces, ICU collation, locale-aware sorting and search folding, bidirectional text isolation, display width, and confusable or invisible characters. Use when string length, slicing, sorting, search matching, tokenization or rendering breaks on non-English text, when a database or search index must compare such text correctly, or when user input mixes scripts.
+license: MIT
+compatibility: Any Agent Skills host. Examples use Python 3 (unicodedata, the regex package, PyICU), JavaScript (Intl.Segmenter, Intl.Collator, String.prototype.normalize), Java (java.text, ICU4J) and PostgreSQL ICU collations; use the equivalent ICU binding in other languages.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: writes-files
+  kete-loom/owner: "Kete-org"
+---
+
+# Unicode text processing
+
+A "character" is not a byte, not a UTF-16 code unit and often not a code point. Most bugs with non-Latin text come from code that assumes one of those. Fix them with the Unicode algorithms and ICU, never with hand-written character tables. This skill is about string handling in code; translating an app's UI and locale formatting is the `i18n-localization` skill.
+
+Standards to cite in comments and reviews: UAX #15 (normalization), UAX #29 (text segmentation), UAX #24 (script property), UTS #10 (collation), UAX #9 (bidi), UAX #11 (East Asian width), UTS #39 (security and confusables). Library calls per language are in `references/apis.md`; script-specific pitfalls in `references/script-notes.md`.
+
+## 0. Reproduce with real text
+
+1. Get a failing sample in the real script from the user or the data, as text and as code points (`[hex(ord(c)) for c in s]` in Python, `[...s].map(c => c.codePointAt(0).toString(16))` in JS). Screenshots hide the difference between look-alike sequences.
+2. Confirm the encoding end to end: files, HTTP `Content-Type; charset=utf-8`, database encoding and connection charset (MySQL `utf8mb4`, not `utf8`). Mojibake like `ব` means UTF-8 bytes were decoded as Latin-1 or Windows-1252 somewhere; fix the boundary, never "repair" the data with replace tables.
+3. Write a test with that sample before changing code.
+
+## 1. Normalize at the boundary
+
+| Form | Use for | Avoid for |
+|---|---|---|
+| NFC | Storing and comparing user text; default for the web and most APIs | |
+| NFD | Stripping combining marks for accent-insensitive search keys (Latin) | Storing text |
+| NFKC | Identifiers, usernames, search keys where full-width and compatibility forms should match (`Ａ` vs `A`, `fi` vs `fi`) | Storing the user's text: it is lossy |
+| NFKC_Casefold / `casefold()` after NFKC | Case-insensitive identifier matching | Display |
+
+Rules:
+- Normalize once when text enters the system (API input, file import), store NFC, and normalize again before comparing with outside data.
+- NFC and NFKC give different results for the same input: compare strings normalized to the same form.
+- Case-insensitive matching uses case folding (`str.casefold()`, ICU `foldCase`), not `lower()`. For Turkish and Azerbaijani, dotted and dotless i need the locale-specific rules.
+- For search keys, keep the original string for display and store a separate folded key.
+
+## 2. Count, slice and truncate by grapheme cluster
+
+What a user sees as one character can be several code points: Bengali `ক্ষ` (conjunct), Devanagari `कि` (consonant plus vowel sign), emoji with skin tones or ZWJ families, flags, Hangul jamo sequences.
+
+- Length limits shown to users ("max 50 characters") count extended grapheme clusters (UAX #29): Python `regex.findall(r"\X", s)`, JS `Intl.Segmenter(undefined, { granularity: "grapheme" })`, ICU `BreakIterator.getCharacterInstance`.
+- Storage limits (database column bytes, SMS segments, protocol fields) count bytes or code units; check them separately and **truncate at a grapheme boundary**, never mid-cluster, then verify the byte length.
+- Never reverse, split or slice user-visible strings by index without segmenting them first.
+- Note: a grapheme cluster is not always one Indic syllable (aksara). If the product needs syllable-level units (for example a cursor or a typing tutor for Bengali), check with the user. Unicode 15.1 added rule GB9c, so conjuncts such as `ক্ষ` in Bengali, Devanagari, Gujarati, Malayalam, Oriya and Telugu are one extended grapheme cluster; libraries on older Unicode data split them after the virama, so check the library's Unicode version.
+
+## 3. Detect scripts
+
+1. Use the Unicode `Script` and `Script_Extensions` properties (UAX #24): Python `regex` `\p{Script=Bengali}` / `\p{scx=Beng}`, JS `/\p{Script=Bengali}/u`, ICU `UScript`.
+2. Ignore characters whose script is `Common` or `Inherited` (digits, punctuation, combining marks) when deciding the dominant script.
+3. Script is not language: Bengali script covers Bengali and Assamese; Devanagari covers Hindi, Marathi, Nepali and Sanskrit; Arabic script covers Arabic, Persian and Urdu. Use a language-ID model only if the language matters, and report its confidence.
+4. Mixed-script identifiers (Latin plus Cyrillic in one username) are a spoofing signal; see step 7.
+
+## 4. Segment words and tokenize
+
+- Languages written without spaces between words (Thai, Lao, Khmer, Burmese, Chinese, Japanese) need dictionary-based word segmentation: ICU `BreakIterator.getWordInstance(locale)` or `Intl.Segmenter(locale, { granularity: "word" })` and its `isWordLike` flag. Splitting on whitespace returns whole sentences.
+- For Chinese and Japanese search, character bigrams or a dedicated analyzer (for example a morphological analyzer for Japanese) usually beat ICU word breaks; choose with the user and measure on their queries.
+- Splitting on `\w` or `[a-zA-Z]` breaks Indic and Arabic words, because combining vowel signs and virama are marks (`\p{M}`), not letters. Use `[\p{L}\p{M}\p{N}]+` with a Unicode-aware regex engine, or a word segmenter.
+- For ML tokenizers, normalize (NFC or the model's documented form) before tokenizing, and check the tokenizer does not strip ZWJ or ZWNJ, which change meaning and shape in Indic and Persian text.
+
+## 5. Sort and search with collation
+
+- Never sort user-visible text by code point. Use ICU collation for the user's locale (UTS #10 plus CLDR tailoring): Python `icu.Collator.createInstance(icu.Locale("bn"))`, JS `Intl.Collator("bn")`, Java `Collator.getInstance(Locale.forLanguageTag("bn"))`.
+- Choose strength deliberately: primary ignores accents and case, secondary ignores case, tertiary compares both.
+- In the database, use ICU collations (PostgreSQL `CREATE COLLATION bn_icu (provider = icu, locale = 'bn-BD')`; nondeterministic collations for case- or accent-insensitive equality) and index with the same collation the queries use.
+- For search, build a folded key: NFKC, case fold, and the script-specific foldings the user agrees to (for example removing Arabic tatweel and optional diacritics, unifying alef forms). Document every folding; each one makes some distinct words match.
+- Collation rules change with ICU and CLDR versions: rebuild indexes that depend on collation after an ICU upgrade (PostgreSQL warns about collation version mismatches).
+
+## 6. Bidirectional text
+
+- Set `dir="auto"` on user-generated content and wrap embedded values in `<bdi>` or Unicode isolates (FSI U+2068 ... PDI U+2069) so a Hebrew or Arabic name does not reorder the surrounding text.
+- Use logical order in storage; never reverse strings to "fix" display.
+- Use CSS logical properties (`margin-inline-start`) for layout; layout mirroring belongs to the `i18n-localization` skill.
+
+## 7. Security and invisible characters
+
+- Strip or reject bidi control characters (U+202A to U+202E, U+2066 to U+2069) in source code, filenames and identifiers; they enable "Trojan Source" style attacks (CVE-2021-42574).
+- Check identifiers for confusables and mixed scripts with UTS #39 (ICU `SpoofChecker`).
+- Keep ZWJ (U+200D) and ZWNJ (U+200C) in natural-language text where they are meaningful; remove them only in identifiers where policy says so.
+
+## 8. Display width and rendering
+
+- Terminal and fixed-width layouts: use East Asian Width and `wcwidth`, not `len`, and measure per grapheme.
+- Rendering needs a font that covers the script and a shaping engine (HarfBuzz in browsers and most toolkits). Boxes (tofu) or detached vowel signs mean a missing font or no shaping, not bad data.
+
+## Never
+
+- Never compare or deduplicate user text without normalizing both sides to the same form.
+- Never store NFKC-folded text as the only copy of the user's input.
+- Never hand-roll tables of letters, vowels or word lists for a script; use Unicode properties and ICU.
+- Never silently strip characters you do not recognise; log and report them.
+
+## Report back
+
+```
+Symptom: ... (sample as text and code points)
+Cause: which assumption failed (code units, no normalization, whitespace tokenization, code-point sort, ...)
+Fix: files changed, library calls used, form chosen (NFC/NFKC), collation and strength
+Tests: samples per script covered, command run, results
+Follow-ups: indexes to rebuild, data to re-normalize (with a migration the user approves)
+```

package/skills/unicode-text-processing/evals/triggers.json

@@ -0,0 +1,17 @@
+{
+  "should_trigger": [
+    "Our 50-character username limit cuts Bengali names in half and leaves a broken vowel sign",
+    "Searching for a Hindi word misses results because some rows store the nukta letter differently",
+    "Split Thai product descriptions into words for our search index, there are no spaces",
+    "Sort a list of Bengali and Arabic customer names correctly instead of by code point",
+    "Detect whether each support message is written in Devanagari, Arabic or Latin script",
+    "An Arabic name inside an English sentence flips the punctuation around in our notification text"
+  ],
+  "should_not_trigger": [
+    "Translate the checkout screen into Hindi and Bengali and add the message catalogs",
+    "Format prices and dates for Japanese users in the dashboard",
+    "Review this UI diff for hard-coded strings before it ships in other languages",
+    "OCR these scanned Bengali land records into searchable PDFs",
+    "Write a style guide for our Arabic marketing copy"
+  ]
+}