wuchale 0.2.0 → 0.3.0

package/README.md

@@ -26,6 +26,10 @@ simplicity in mind.
 No extra imports or annotations. `wuchale` extracts and compiles everything
 automatically. In the spirit of Svelte itself.
 
+> They say "i18n is too costly to add later"
+
+✨ **Not anymore**. wuchale brings i18n to existing Svelte projects — the UX-first way.
+
 ### 🧠 Compiler-Powered
 
 Built on the Svelte compiler and powered by AST
@@ -127,7 +131,7 @@ Then finally you write your Svelte files naturally:
 <script>
 import WuchaleTrans, { wuchaleTrans } from 'wuchale/runtime.svelte'
 </script>
-<h1>{wuchaleTrans(0)}</h1> <!-- Extracted "Hello" as index 0 -->
+<p>{wuchaleTrans(0)}</p> <!-- Extracted "Hello" as index 0 -->
 ```
 
 Full example below.
@@ -236,30 +240,88 @@ This is what you import when you set it up above in `App.svelte`.
 
 ## Supported syntax
 
-Text can be in three places: markup, script and attributes. Script means not
-just the part inside the `<script>` tags, but also inside interpolations inside
-the markup, such as `<p>{'This string'}</p>` and also in other places such as
-`{#if 'this string' == ''}`. And each can have their own rules. While these
-rules can be configured, the default is:
-
-- Markup:
-    - All text should be extracted unless prefixed with `-`. Example: `<p>-
-    This will be ignored.</p>`
-- Attributes:
-    - All attributes starting with upper case letters are extracted unless
-    prefixed with `-` like `label="-Ignore"`.
-    - All attributes starting with lower case letters are ignored, unless
-    prefixed with `+` like `alt="+extract"`.
-- Script:
-    - Strings: same rules as attributes above, but:
-    - If they are used inside the `<script>` tags, there is the additional
-      restriction that they must be inside a `$derived` variable declaration.
-      This is to make the behavior less magic and being more explicit.
-
-## Where does it look?
-
-All files that can contain reactive logic. This means `*.svelte` and
-`*.svelte.js` files specifically.
+Text can be in three places and the probability of the text inside them being
+intended is different for each of them. Therefore, a global heuristic function
+is applied to check whether the text should be extracted depending on each
+case, discussed below. And for specific granular control, comments can be used,
+like for typescript: `@wc-ignore` and `@wc-include`.
+
+### Markup
+
+Markup means text that we write inside tags like in paragraphs. This is almost
+always intended to be shown to the user. Therefore, the default global rule is
+to extract all text inside the markup, with the ability to force ignore with a
+comment:
+
+```svelte
+<p>This is extracted</p>
+<!-- @wc-ignore -->
+<p>This is ignored</p>
+```
+
+### Attributes
+
+Attributes are the text that are literally written like `class="this"`. Dynamic
+attributes are not considered for this rule, instead they follow the script
+rule below, because they are JS expressions. For text attributes, the default
+rule is that all text starting with a capital letter is extracted.
+Example:
+
+```svelte
+<p class="not-extracted" title="Extracted">
+    This is extracted
+</p>
+```
+
+For ignoring or force-including, convert them to expressions and follow the
+script rule below.
+
+### Script
+
+This includes all JS/TS code that is:
+    - Inside `<script>` tags
+    - Inside dynamic expressions inside the markup or attributes, anything in curly braces like `{call('this guy')}`
+    - In `*.svelte.[js|ts]` files.
+
+The rule for this is that all strings and template strings that start with
+capital letters are extracted. Additionally, if they are used inside the
+`<script>` tags and in their own files (third case above), there is the
+additional restriction that they must be inside a `$derived` variable
+declaration. This is to make the behavior less magic and being more explicit.
+Example:
+
+```svelte
+<script>
+    const a = 'Not extracted'
+    const b = $derived('not extracted either')
+    const c = $derived('This one is extracted')
+    const d = $derived(`This one as well ${a} - ${b}`)
+    const d = $derived(/* @wc-include */ `${a} - ${b} this is force extracted`)
+</script>
+
+<p class={/* @wc-ignore */ `Ignore${3}`} title={'Included'} >Normal text</p>
+
+```
+
+## Context?
+
+Sometimes we need to have different translations that are the same text in the
+source language. For that, the comment directive `@wc-context:` is provided and
+they will be separate.
+
+```svelte
+<b>
+    <!-- @wc-context: machine -->
+    Maintenance
+</b>
+<i>Is different from</i>
+<b>
+    <!-- @wc-context: beauty -->
+    Maintenance
+</b>
+```
+
+Excuse my poor example choice.
 
 ## Plurals?
 
@@ -292,17 +354,14 @@ export const defaultOptions = {
 ```
 
 While the others are self explanatory, the `heuristic` is a function that
-decides what text to extract and what not to. The `defaultHeuristic` is the
-implementation of the above rules, but you can roll your own and provide it
-here. The function should receive the following arguments:
+globally decides what text to extract and what not to. The `defaultHeuristic`
+is the implementation of the above rules, but you can roll your own and provide
+it here. The function should receive the following arguments:
 
 - `text`: The candidate text
 - `scope`: Where the text is located, i.e. it can be one of `markup`, `script`, and `attribute`
 
-And it should return an object with two properties:
-
-- `extract` (boolean): Whether to extract it or not
-- `replace` (`string`): The string to replace it with. This is how you specify how to remove parts such as the prefixes above (`+` and `-`).
+And it should return boolean to indicate whether to extract it.
 
 ## 🧹 Cleaning
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "wuchale",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "i18n for svelte without turning your codebase upside down",
   "main": "index.js",
   "scripts": {

package/preprocess/gemini.js

@@ -1,33 +1,40 @@
+// $$ cd .. && npm run test
 // $$ node %f
 
+import PO from 'pofile'
+
 const baseURL = 'https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key='
 const h = {'Content-Type': 'application/json'}
 
 /**
- * @param {string[]} fragments
+ * @param {import('pofile').Item[]} fragments
  * @param {string} sourceLocale
  * @param {string} targetLocale
  */
 function prepareData(fragments, sourceLocale, targetLocale) {
     const instruction = `
-        You will be given text fragments for a web app separated by new line characters.
-        Translate each of the fragments from the source to the target language.
+        You will be given the contents of a gettext .po file for a web app.
+        Translate each of the items from the source to the target language.
         You have to find out the languages using their ISO 639-1 codes.
         The source language is: ${sourceLocale}.
         The target language is: ${targetLocale}.
-        Preserve any placeholders and provide the translations in the target language only,
-        separated by new line characters in the same order.
+        You can read all of the information for the items including contexts,
+        comments and references to get the appropriate context about each item.
+        Provide the translated fragments in the in the same order, preserving
+        all placeholders.
         The placeholder format is like the following examples:
             - {0}: means arbitrary values.
             - <0>something</0>: means something enclosed in some tags, like HTML tags
             - <0/>: means a self closing tag, like in HTML
         In all of the examples, 0 is an example for any integer.
     `
+    const po = new PO()
+    po.items = fragments
     return {
         system_instruction: {
             parts: [{ text: instruction }]
         },
-        contents: [{parts: [{text: fragments.join('\n')}]}]
+        contents: [{parts: [{text: po.toString()}]}]
     }
 }
 
@@ -43,18 +50,16 @@ function setupGemini(sourceLocale = 'en', targetLocale, apiKey) {
         return
     }
     const url = `${baseURL}${apiKey}`
-    return async (/** @type {string[]} */ fragments) => {
+    return async (/** @type {import('pofile').Item[]} */ fragments) => {
         const data = prepareData(fragments, sourceLocale, targetLocale)
         const res = await fetch(url, {method: 'POST', headers: h, body: JSON.stringify(data)})
         const json = await res.json()
         const resText = json.candidates[0].content.parts[0].text
-        const trans = {}
-        for (const [i, text] of resText.split('\n').entries()) {
-            if (text.trim()) {
-                trans[fragments[i]] = text
+        for (const [i, item] of PO.parse(resText).items.entries()) {
+            if (item.msgstr[0]) {
+                fragments[i].msgstr = item.msgstr
             }
         }
-        return trans
     }
 }
 

package/preprocess/index.js

@@ -1,10 +1,10 @@
-import Preprocess, { defaultHeuristic, IndexTracker } from "./prep.js"
+import Preprocess, { defaultHeuristic, IndexTracker, NestText } from "./prep.js"
 import { parse } from "svelte/compiler"
 import {writeFile} from 'node:fs/promises'
 import compileTranslation from "./compile.js"
 import setupGemini from "./gemini.js"
 import PO from "pofile"
-import { relative } from "node:path"
+import { normalize, relative } from "node:path"
 
 export const defaultOptions = {
     sourceLocale: 'en',
@@ -35,7 +35,8 @@ async function loadPONoFail(filename) {
                     if (!item.msgstr[0]) {
                         untranslated++
                     }
-                    translations[item.msgid] = item
+                    const nTxt = new NestText(item.msgid, null, item.msgctxt)
+                    translations[nTxt.toKey()] = item
                 }
             }
             res({translations, total, obsolete, untranslated})
@@ -102,12 +103,12 @@ export default async function wuchale(options = defaultOptions) {
         // startup compile
         for (const loc of locales) {
             compiled[loc] = []
-            for (const txt in translations[loc]) {
-                const poItem = translations[loc][txt]
+            for (const key in translations[loc]) {
+                const poItem = translations[loc][key]
                 if (forProduction) {
                     poItem.references = []
                 }
-                const index = indexTracker.get(txt)
+                const index = indexTracker.get(key)
                 compiled[loc][index] = compileTranslation(poItem.msgstr[0], compiled[options.sourceLocale][index])
             }
             await writeFile(compiledFname[loc], JSON.stringify(compiled[loc], null, 2))
@@ -128,38 +129,40 @@ export default async function wuchale(options = defaultOptions) {
         for (const loc of locales) {
             const newTxts = []
             for (const nTxt of txts) {
-                const txt = nTxt.toString()
-                let translated = translations[loc][txt]
+                let key = nTxt.toKey()
+                let translated = translations[loc][key]
                 if (translated == null) {
                     translated = new PO.Item()
-                    translated.msgid = txt
-                    translations[loc][txt] = translated
+                    translated.msgid = nTxt.toString()
+                    translations[loc][key] = translated
+                }
+                if (nTxt.context) {
+                    translated.msgctxt = nTxt.context
                 }
                 if (!translated.references.includes(filename)) {
                     translated.references.push(filename)
                 }
                 if (loc === options.sourceLocale) {
+                    const txt = nTxt.toString()
                     if (translated.msgstr[0] !== txt) {
                         translated.msgstr = [txt]
-                        newTxts.push(txt)
+                        newTxts.push(translated)
                     }
                 } else if (!translated.msgstr[0]) {
-                    newTxts.push(txt)
+                    newTxts.push(translated)
                 }
             }
             if (loc !== options.sourceLocale && newTxts.length) {
                 const geminiT = setupGemini(options.sourceLocale, loc, options.geminiAPIKey)
                 if (geminiT) {
-                    const gTrans = await geminiT(newTxts)
-                    for (const txt of newTxts) {
-                        translations[loc][txt].msgstr = [gTrans[txt]]
-                    }
+                    console.info('Gemini translate', newTxts.length, 'items...')
+                    await geminiT(newTxts) // will update because it's by reference
                 }
             }
             for (const nTxt of txts) {
-                const txt = nTxt.toString()
-                const index = indexTracker.get(txt)
-                compiled[loc][index] = compileTranslation(translations[loc][txt].msgstr[0], compiled[options.sourceLocale][index])
+                const key = nTxt.toKey()
+                const index = indexTracker.get(key)
+                compiled[loc][index] = compileTranslation(translations[loc][key].msgstr[0], compiled[options.sourceLocale][index])
             }
             for (const [i, c] of compiled[loc].entries()) {
                 if (c == null) {
@@ -193,10 +196,10 @@ export default async function wuchale(options = defaultOptions) {
              * @param {string} id
             */
             handler: async function(code, id) {
-                if (!id.startsWith(projectRoot)) {
+                if (!id.startsWith(projectRoot) || id.startsWith(normalize(projectRoot + '/node_modules'))) {
                     return
                 }
-                const isModule = id.endsWith('.svelte.js')
+                const isModule = id.endsWith('.svelte.js') || id.endsWith('.svelte.ts')
                 if (!id.endsWith('.svelte') && !isModule) {
                     return
                 }
@@ -223,8 +226,8 @@ export default async function wuchale(options = defaultOptions) {
                 return
             }
             for (const loc of locales) {
-                for (const txt in translations[loc]) {
-                    const poItem = translations[loc][txt]
+                for (const key in translations[loc]) {
+                    const poItem = translations[loc][key]
                     poItem.obsolete = poItem.references.length === 0
                 }
                 await savePO(translations[loc], translationsFname[loc])

package/preprocess/prep.js

@@ -8,7 +8,7 @@ const rtFunc = 'wuchaleTrans'
 
 /**
  * @typedef {"script" | "markup" | "attribute"} TxtScope
- * @typedef {(text: string, scope: TxtScope) => {extract: boolean; replace: string}} HeuristicFunc
+ * @typedef {(text: string, scope: TxtScope) => boolean} HeuristicFunc
  */
 
 /**
@@ -16,32 +16,29 @@ const rtFunc = 'wuchaleTrans'
  */
 export function defaultHeuristic(text, scope = 'markup') {
     if (scope === 'markup') {
-        if (text.startsWith('-')) {
-            return {extract: false, replace: text.slice(1).trim()}
-        }
-        return {extract: true, replace: text}
+        return true
     }
     // script and attribute
-    if (text.startsWith('+')) {
-        return {extract: true, replace: text.slice(1).trim()}
-    }
     const range = 'AZ'
     const startCode = text.charCodeAt(0)
-    if (startCode >= range.charCodeAt(0) && startCode <= range.charCodeAt(1)) {
-        return {extract: true, replace: text}
-    }
-    return {extract: false, replace: text}
+    return startCode >= range.charCodeAt(0) && startCode <= range.charCodeAt(1)
 }
 
-class NestText extends String {
+export class NestText extends String {
     /**
      * @param {string} txt
      * @param {TxtScope} scope
+     * @param {string | null} [context]
      */
-    constructor(txt, scope) {
+    constructor(txt, scope, context) {
         super(txt)
         this.scope = scope
+        /** @type {string} */
+        this.context = context ?? null
     }
+
+    toKey = () => `${this.toString()}\n${this.context ?? ''}`.trim()
+
 }
 
 export class IndexTracker {
@@ -75,34 +72,32 @@ export default class Preprocess {
      * @param {HeuristicFunc} heuristic
      * @param {string} importFrom
      */
-    constructor(index, heuristic, importFrom) {
+    constructor(index, heuristic = defaultHeuristic, importFrom) {
         this.index = index
         this.importFrom = importFrom
         this.heuristic = heuristic
         this.content = ''
         /** @type {MagicString} */
         this.mstr = null
+        /** @type {boolean | null} */
+        this.forceInclude = null
+        /** @type {string} */
+        this.context = null
     }
 
     /**
-     * @param {number} start
-     * @param {number} end
      * @param {string} text
      * @param {TxtScope} scope
      * @returns {Array<*> & {0: boolean, 1: NestText}}
      */
-    modifyCheck = (start, end, text, scope) => {
+    checkHeuristic = (text, scope) => {
         text = text.replace(/\s+/g, ' ').trim()
         if (text === '') {
             // nothing to ask
             return [false, null]
         }
-        let {extract, replace} = this.heuristic(text, scope)
-        replace = replace.trim()
-        if (!extract && text !== replace) {
-            this.mstr.update(start, end, replace)
-        }
-        return [extract, new NestText(replace, scope)]
+        const extract = this.forceInclude || this.heuristic(text, scope)
+        return [extract, new NestText(text, scope)]
     }
 
     // visitComment = () => []
@@ -118,11 +113,12 @@ export default class Preprocess {
             return []
         }
         const { start, end } = node
-        const [pass, txt] = this.modifyCheck(start, end, node.value, 'script')
+        const [pass, txt] = this.checkHeuristic(node.value, 'script')
         if (!pass) {
             return []
         }
-        this.mstr.update(start, end, `${rtFunc}(${this.index.get(txt.toString())})`)
+        txt.context = this.context
+        this.mstr.update(start, end, `${rtFunc}(${this.index.get(txt.toKey())})`)
         return [txt]
     }
 
@@ -176,6 +172,17 @@ export default class Preprocess {
         return txts
     }
 
+    /**
+     * @param {import('estree').BinaryExpression} node
+     * @returns {NestText[]}
+     */
+    visitBinaryExpression = node => {
+        return [
+            ...this.visit(node.left),
+            ...this.visit(node.right),
+        ]
+    }
+
     /**
      * @param {import('estree').VariableDeclaration} node
      * @returns {NestText[]}
@@ -214,15 +221,15 @@ export default class Preprocess {
         const quasi0 = node.quasis[0]
         // @ts-ignore
         const {start: start0, end: end0} = quasi0
-        const [pass, txt] = this.modifyCheck(start0, end0, quasi0.value.cooked, 'script')
+        const [pass, txt0] = this.checkHeuristic(quasi0.value.cooked, 'script')
         if (!pass) {
             return txts
         }
-        let nTxt = txt.toString()
+        let txt = txt0.toString()
         for (const [i, expr] of node.expressions.entries()) {
             txts.push(...this.visit(expr))
             const quasi = node.quasis[i + 1]
-            nTxt += `{${i}}${quasi.value.cooked}`
+            txt += `{${i}}${quasi.value.cooked}`
             // @ts-ignore
             const {start, end} = quasi
             this.mstr.remove(start - 1, end)
@@ -231,14 +238,15 @@ export default class Preprocess {
             }
             this.mstr.update(end, end + 2, ', ')
         }
-        let repl = `${rtFunc}(${this.index.get(txt.toString())}`
+        const nTxt = new NestText(txt, txt0.scope, this.context)
+        let repl = `${rtFunc}(${this.index.get(nTxt.toKey())}`
         if (node.expressions.length) {
             repl += ', '
         }
         this.mstr.update(start0 - 1, end0 + 2, repl)
         // @ts-ignore
         this.mstr.update(node.end - 1, node.end, ')')
-        txts.push(new NestText(nTxt, 'script'))
+        txts.push(nTxt)
         return txts
     }
 
@@ -290,8 +298,7 @@ export default class Preprocess {
         const textNodesToModify = {}
         for (const [i, child] of node.fragment.nodes.entries()) {
             if (child.type === 'Text') {
-                const { start, end } = child
-                const [pass, modify] = this.modifyCheck(start, end, child.data, 'markup')
+                const [pass, modify] = this.checkHeuristic(child.data, 'markup')
                 if (pass) {
                     hasTextChild = true
                     textNodesToModify[i] = modify
@@ -374,7 +381,7 @@ export default class Preprocess {
         if (!txt) {
             return txts
         }
-        const nTxt = new NestText(txt, 'markup')
+        const nTxt = new NestText(txt, 'markup', this.context)
         txts.push(nTxt)
         if (iTag > 0) {
             const snippets = []
@@ -386,7 +393,7 @@ export default class Preprocess {
             if (node.inCompoundText) {
                 begin += `ctx={ctx}`
             } else {
-                begin += `id={${this.index.get(txt)}}`
+                begin += `id={${this.index.get(nTxt.toKey())}}`
             }
             let end = ' />\n'
             if (iArg > 0) {
@@ -396,7 +403,7 @@ export default class Preprocess {
             this.mstr.appendLeft(lastChildEnd, begin)
             this.mstr.appendRight(lastChildEnd, end)
         } else if (!node.inCompoundText) {
-            this.mstr.appendLeft(lastChildEnd, `{${rtFunc}(${this.index.get(txt)}, `)
+            this.mstr.appendLeft(lastChildEnd, `{${rtFunc}(${this.index.get(nTxt.toKey())}, `)
             this.mstr.appendRight(lastChildEnd, ')}')
         }
         return txts
@@ -409,12 +416,12 @@ export default class Preprocess {
      * @returns {NestText[]}
      */
     visitText = node => {
-        const { start, end } = node
-        const [pass, txt] = this.modifyCheck(start, end, node.data, 'markup')
+        const [pass, txt] = this.checkHeuristic(node.data, 'markup')
         if (!pass) {
             return []
         }
-        this.mstr.update(node.start, node.end, `{${rtFunc}(${this.index.get(txt.toString())})}`)
+        txt.context = this.context
+        this.mstr.update(node.start, node.end, `{${rtFunc}(${this.index.get(txt.toKey())})}`)
         return [txt]
     }
 
@@ -448,12 +455,13 @@ export default class Preprocess {
             }
             // Text
             const {start, end} = value
-            const [pass, txt] = this.modifyCheck(start, end, value.data, 'attribute')
+            const [pass, txt] = this.checkHeuristic(value.data, 'attribute')
             if (!pass) {
                 continue
             }
+            txt.context = this.context
             txts.push(txt)
-            this.mstr.update(value.start, value.end, `{${rtFunc}(${this.index.get(txt.toString())})}`)
+            this.mstr.update(value.start, value.end, `{${rtFunc}(${this.index.get(txt.toKey())})}`)
             if (!`'"`.includes(this.content[start - 1])) {
                 continue
             }
@@ -586,16 +594,49 @@ export default class Preprocess {
     }
 
     /**
-     * @param {import("svelte/compiler").AST.SvelteNode} node
+     * @param {string} data
+     */
+    processCommentDirectives = data => {
+        if (data === '@wc-ignore') {
+            this.forceInclude = false
+        }
+        if (data === '@wc-include') {
+            this.forceInclude = true
+        }
+        const ctxStart = '@wc-context:'
+        if (data.startsWith(ctxStart)) {
+            this.context = data.slice(ctxStart.length).trim()
+        }
+    }
+
+    /**
+     * @param {import("svelte/compiler").AST.SvelteNode & import('estree').BaseNode} node
      * @returns {NestText[]}
      */
     visit = node => {
-        const methodName = `visit${node.type}`
-        if (methodName in this) {
-            return this[methodName](node)
+        if (node.type === 'Comment') {
+            this.processCommentDirectives(node.data.trim())
+            return []
+        }
+        // for estree
+        for (const comment of node.leadingComments ?? []) {
+            this.processCommentDirectives(comment.value.trim())
+        }
+        let txts = []
+        if (this.forceInclude !== false) {
+            const methodName = `visit${node.type}`
+            if (methodName in this) {
+                txts = this[methodName](node)
+            }
         }
-        // console.log(node)
-        return []
+        this.forceInclude = null
+        if (this.context != null) {
+            for (const txt of txts) {
+                txt.context = this.context
+            }
+        }
+        this.context = null
+        return txts
     }
 
     /**