@vocab/core 1.0.3 → 1.1.1

package/CHANGELOG.md

@@ -1,5 +1,33 @@
 # @vocab/core
 
+## 1.1.1
+
+### Patch Changes
+
+- [`09a698a`](https://github.com/seek-oss/vocab/commit/09a698af6aff86a851e4f829916b8f1f6beaca58) [#89](https://github.com/seek-oss/vocab/pull/89) Thanks [@mikebarkmin](https://github.com/mikebarkmin)! - Add exports to packages with multiple entry points. This fixes
+  `ERR_UNSUPPORTED_DIR_IMPORT` issues e.g. with NextJS or other setups, which
+  rely on the new node resolver when using ESM packages.
+
+## 1.1.0
+
+### Minor Changes
+
+- [`87333d7`](https://github.com/seek-oss/vocab/commit/87333d79c4a883b07d7d8f2c272b16e2243c49bd) [#80](https://github.com/seek-oss/vocab/pull/80) Thanks [@askoufis](https://github.com/askoufis)! - Enable the creation of generated languages via the `generatedLanguages` config.
+  See [the docs] for more information and examples.
+
+  [the docs]: https://github.com/seek-oss/vocab#generated-languages
+
+### Patch Changes
+
+- Updated dependencies [[`87333d7`](https://github.com/seek-oss/vocab/commit/87333d79c4a883b07d7d8f2c272b16e2243c49bd)]:
+  - @vocab/types@1.1.0
+
+## 1.0.4
+
+### Patch Changes
+
+- [`206c0fa`](https://github.com/seek-oss/vocab/commit/206c0fa36b05f23da593ebed801197c523477af6) [#78](https://github.com/seek-oss/vocab/pull/78) Thanks [@askoufis](https://github.com/askoufis)! - Fix incorrect language hierarchy when an extended language extends another language
+
 ## 1.0.3
 
 ### Patch Changes

package/README.md

@@ -184,6 +184,14 @@ Configuration can either be passed into the Node API directly or be gathered fro
 **vocab.config.js**
 
 ```js
+function capitalize(element) {
+  return element.toUpperCase();
+}
+
+function pad(message) {
+  return '[' + message + ']';
+}
+
 module.exports = {
   devLanguage: 'en',
   languages: [
@@ -192,11 +200,25 @@ module.exports = {
     { name: 'en-US', extends: 'en' },
     { name: 'fr-FR' }
   ],
+  /**
+   * An array of languages to generate based off translations for existing languages
+   * Default: []
+   */
+  generatedLanguages: [
+    {
+      name: 'generatedLangauge',
+      extends: 'en',
+      generator: {
+        transformElement: capitalize,
+        transformMessage: pad
+      }
+    }
+  ],
   /**
    * The root directory to compile and validate translations
    * Default: Current working directory
    */
-  projectRoot: './example/';
+  projectRoot: './example/',
   /**
    * A custom suffix to name vocab translation directories
    * Default: '.vocab'
@@ -209,6 +231,131 @@ module.exports = {
 };
 ```
 
+## Generated languages
+
+Vocab supports the creation of generated languages via the `generatedLanguages` config.
+
+Generated languages are created by running a message `generator` over every translation message in an existing translation.
+A `generator` may contain a `transformElement` function, a `transformMessage` function, or both.
+Both of these functions accept a single string parameter and return a string.
+
+`transformElement` is applied to string literal values contained within `MessageFormatElement`s.
+A `MessageFormatElement` is an object representing a node in the AST of a compiled translation message.
+Simply put, any text that would end up being translated by a translator, i.e. anything that is not part of the [ICU Message syntax], will be passed to `transformElement`.
+An example of a use case for this function would be adding [diacritics] to every letter in order to stress your UI from a vertical line-height perspective.
+
+`transformMessage` receives the entire translation message _after_ `transformElement` has been applied to its individual elements.
+An example of a use case for this function would be adding padding text to the start/end of your messages in order to easily identify which text in your app has not been extracted into a `translations.json` file.
+
+By default, a generated language's messages will be based off the `devLanguage`'s messages, but this can be overridden by providing an `extends` value that references another language.
+
+**vocab.config.js**
+
+```js
+function capitalize(message) {
+  return message.toUpperCase();
+}
+
+function pad(message) {
+  return '[' + message + ']';
+}
+
+module.exports = {
+  devLanguage: 'en',
+  languages: [{ name: 'en' }, { name: 'fr' }],
+  generatedLanguages: [
+    {
+      name: 'generatedLanguage',
+      extends: 'en',
+      generator: {
+        transformElement: capitalize,
+        transformMessage: pad
+      }
+    }
+  ]
+};
+```
+
+Generated languages are consumed the same way as regular languages.
+Any Vocab API that accepts a `language` parameter will work with a generated language as well as a regular language.
+
+**App.tsx**
+
+```tsx
+const App = () => (
+  <VocabProvider language="generatedLanguage">
+    ...
+  </VocabProvider>
+);
+```
+
+[icu message syntax]: https://formatjs.io/docs/intl-messageformat/#message-syntax
+[diacritics]: https://en.wikipedia.org/wiki/Diacritic
+
+## Pseudo-localization
+
+The `@vocab/pseudo-localize` package exports low-level functions that can be used for pseudo-localization of translation messages.
+
+```ts
+import {
+  extendVowels,
+  padString,
+  pseudoLocalize,
+  substituteCharacters
+} from '@vocab/pseudo-localize';
+
+const message = 'Hello';
+
+// [Hello]
+const paddedMessage = padString(message);
+
+// Ḩẽƚƚö
+const substitutedMessage = substituteCharacters(message);
+
+// Heelloo
+const extendedMessage = extendVowels(message);
+
+// Extend the message and then substitute characters
+// Ḩẽẽƚƚöö
+const pseudoLocalizedMessage = pseudoLocalize(message);
+```
+
+Pseudo-localization is a transformation that can be applied to a translation message.
+Vocab's implementation of this transformation contains the following elements:
+
+- _Start and end markers (`padString`):_ All strings are encapsulated in `[` and `]`. If a developer doesn’t see these characters they know the string has been clipped by an inflexible UI element.
+- _Transformation of ASCII characters to extended character equivalents (`substituteCharacters`):_ Stresses the UI from a vertical line-height perspective, tests font and encoding support, and weeds out strings that haven’t been externalized correctly (they will not have the pseudo-localization applied to them).
+- _Padding text (`extendVowels`):_ Simulates translation-induced expansion. Vocab's implementation of this involves repeating vowels (and `y`) to simulate a 40% expansion in the message's length.
+
+This Netflix technology [blog post][blog post] inspired Vocab's implementation of this
+functionality.
+
+### Generating a pseudo-localized language using Vocab
+
+Vocab can generate a pseudo-localized language via the [`generatedLanguages` config][generated languages config], either via the webpack plugin or your `vocab.config.js` file.
+`@vocab/pseudo-localize` exports a `generator` that can be used directly in your config.
+
+**vocab.config.js**
+
+```js
+const { generator } = require('@vocab/pseudo-localize');
+
+module.exports = {
+  devLanguage: 'en',
+  languages: [{ name: 'en' }, { name: 'fr' }],
+  generatedLanguages: [
+    {
+      name: 'pseudo',
+      extends: 'en',
+      generator
+    }
+  ]
+};
+```
+
+[blog post]: https://netflixtechblog.com/pseudo-localization-netflix-12fff76fbcbe
+[generated languages config]: #generated-languages
+
 ## Use without React
 
 If you need to use Vocab outside of React, you can access the returned Vocab file directly. You'll then be responsible for when to load translations and how to update on translation load.

package/dist/declarations/src/generate-language.d.ts

@@ -0,0 +1,5 @@
+import { MessageGenerator, TranslationsByKey } from '@vocab/types';
+export declare function generateLanguageFromTranslations({ baseTranslations, generator, }: {
+    baseTranslations: TranslationsByKey<string>;
+    generator: MessageGenerator;
+}): TranslationsByKey<string>;

package/dist/declarations/src/load-translations.d.ts

@@ -1,9 +1,25 @@
-import { UserConfig, LoadedTranslation, LanguageTarget } from '@vocab/types';
+import { TranslationsByKey, UserConfig, LoadedTranslation, LanguageTarget } from '@vocab/types';
 import { Fallback } from './utils';
 export declare function getUniqueKey(key: string, namespace: string): string;
-export declare function getLanguageHierarcy({ languages, }: {
+export declare function mergeWithDevLanguageTranslation({ translation, devTranslation, }: {
+    translation: TranslationsByKey;
+    devTranslation: TranslationsByKey;
+}): TranslationsByKey<string>;
+export declare function getLanguageHierarchy({ languages, }: {
     languages: Array<LanguageTarget>;
 }): Map<string, string[]>;
+export declare function getFallbackLanguageOrder({ languages, languageName, devLanguage, fallbacks, }: {
+    languages: LanguageTarget[];
+    languageName: string;
+    devLanguage: string;
+    fallbacks: Fallback;
+}): string[];
+export declare function loadAltLanguageFile({ filePath, languageName, devTranslation, fallbacks, }: {
+    filePath: string;
+    languageName: string;
+    devTranslation: TranslationsByKey;
+    fallbacks: Fallback;
+}, { devLanguage, languages }: UserConfig): TranslationsByKey;
 export declare function loadTranslation({ filePath, fallbacks, }: {
     filePath: string;
     fallbacks: Fallback;

package/dist/declarations/src/validate/index.d.ts

@@ -1,3 +1,3 @@
 import { UserConfig, LoadedTranslation, LanguageName } from '@vocab/types';
-export declare function findMissingKeys(loadedTranslation: LoadedTranslation, devLanguageName: LanguageName, altLangauges: Array<LanguageName>): readonly [boolean, Record<string, string[]>];
+export declare function findMissingKeys(loadedTranslation: LoadedTranslation, devLanguageName: LanguageName, altLanguages: Array<LanguageName>): readonly [boolean, Record<string, string[]>];
 export declare function validate(config: UserConfig): Promise<boolean>;

package/dist/vocab-core.cjs.dev.js

@@ -10,6 +10,8 @@ var chokidar = require('chokidar');
 var chalk = require('chalk');
 var debug = require('debug');
 var glob = require('fast-glob');
+var IntlMessageFormat = require('intl-messageformat');
+var printer = require('@formatjs/icu-messageformat-parser/printer');
 var findUp = require('find-up');
 var Validator = require('fastest-validator');
 
@@ -21,6 +23,7 @@ var chokidar__default = /*#__PURE__*/_interopDefault(chokidar);
 var chalk__default = /*#__PURE__*/_interopDefault(chalk);
 var debug__default = /*#__PURE__*/_interopDefault(debug);
 var glob__default = /*#__PURE__*/_interopDefault(glob);
+var IntlMessageFormat__default = /*#__PURE__*/_interopDefault(IntlMessageFormat);
 var findUp__default = /*#__PURE__*/_interopDefault(findUp);
 var Validator__default = /*#__PURE__*/_interopDefault(Validator);
 
@@ -108,11 +111,78 @@ function getTranslationMessages(translations) {
   return mapValues(translations, v => v.message);
 }
 
+function generateLanguageFromTranslations({
+  baseTranslations,
+  generator
+}) {
+  if (!generator.transformElement && !generator.transformMessage) {
+    return baseTranslations;
+  }
+
+  const translationKeys = Object.keys(baseTranslations);
+  const generatedTranslations = {};
+
+  for (const translationKey of translationKeys) {
+    const translation = baseTranslations[translationKey];
+    let transformedMessage = translation.message;
+
+    if (generator.transformElement) {
+      const messageAst = new IntlMessageFormat__default['default'](translation.message).getAst();
+      const transformedAst = messageAst.map(transformMessageFormatElement(generator.transformElement));
+      transformedMessage = printer.printAST(transformedAst);
+    }
+
+    if (generator.transformMessage) {
+      transformedMessage = generator.transformMessage(transformedMessage);
+    }
+
+    generatedTranslations[translationKey] = {
+      message: transformedMessage
+    };
+  }
+
+  return generatedTranslations;
+}
+
+function transformMessageFormatElement(transformElement) {
+  return messageFormatElement => {
+    const transformedMessageFormatElement = { ...messageFormatElement
+    };
+
+    switch (transformedMessageFormatElement.type) {
+      case icuMessageformatParser.TYPE.literal:
+        const transformedValue = transformElement(transformedMessageFormatElement.value);
+        transformedMessageFormatElement.value = transformedValue;
+        break;
+
+      case icuMessageformatParser.TYPE.select:
+      case icuMessageformatParser.TYPE.plural:
+        const transformedOptions = { ...transformedMessageFormatElement.options
+        };
+
+        for (const key of Object.keys(transformedOptions)) {
+          transformedOptions[key].value = transformedOptions[key].value.map(transformMessageFormatElement(transformElement));
+        }
+
+        break;
+
+      case icuMessageformatParser.TYPE.tag:
+        const transformedChildren = transformedMessageFormatElement.children.map(transformMessageFormatElement(transformElement));
+        transformedMessageFormatElement.children = transformedChildren;
+        break;
+    }
+
+    return transformedMessageFormatElement;
+  };
+}
+
 function getUniqueKey(key, namespace) {
   return `${key}.${namespace}`;
 }
-
-function mergeWithDevLanguage(translation, devTranslation) {
+function mergeWithDevLanguageTranslation({
+  translation,
+  devTranslation
+}) {
   // Only use keys from the dev translation
   const keys = Object.keys(devTranslation);
   const newLanguage = {};
@@ -143,7 +213,7 @@ function getLanguageFallbacks({
   return languageFallbackMap;
 }
 
-function getLanguageHierarcy({
+function getLanguageHierarchy({
   languages
 }) {
   const hierarchyMap = new Map();
@@ -152,19 +222,45 @@ function getLanguageHierarcy({
   });
 
   for (const lang of languages) {
-    const langHierachy = [];
+    const langHierarchy = [];
     let currLang = lang.extends;
 
     while (currLang) {
-      langHierachy.push(currLang);
+      langHierarchy.push(currLang);
       currLang = fallbacks.get(currLang);
     }
 
-    hierarchyMap.set(lang.name, langHierachy);
+    hierarchyMap.set(lang.name, langHierarchy);
   }
 
   return hierarchyMap;
 }
+function getFallbackLanguageOrder({
+  languages,
+  languageName,
+  devLanguage,
+  fallbacks
+}) {
+  const languageHierarchy = getLanguageHierarchy({
+    languages
+  }).get(languageName);
+
+  if (!languageHierarchy) {
+    throw new Error(`Missing language hierarchy for ${languageName}`);
+  }
+
+  const fallbackLanguageOrder = [languageName];
+
+  if (fallbacks !== 'none') {
+    fallbackLanguageOrder.unshift(...languageHierarchy.reverse());
+
+    if (fallbacks === 'all' && fallbackLanguageOrder[0] !== devLanguage) {
+      fallbackLanguageOrder.unshift(devLanguage);
+    }
+  }
+
+  return fallbackLanguageOrder;
+}
 
 function getNamespaceByFilePath(relativePath, {
   translationsDirectorySuffix = defaultTranslationDirSuffix
@@ -208,17 +304,17 @@ function getTranslationsFromFile(translations, {
 
   for (const [translationKey, translation] of Object.entries(keys)) {
     if (typeof translation === 'string') {
-      printValidationError(`Found string for a translation "${translationKey}" in ${filePath}. Translation must be an object of the format {mesage: string}.`);
+      printValidationError(`Found string for a translation "${translationKey}" in ${filePath}. Translation must be an object of the format {message: string}.`);
       continue;
     }
 
     if (!translation) {
-      printValidationError(`Found empty translation "${translationKey}" in ${filePath}. Translation must be an object of the format {mesage: string}.`);
+      printValidationError(`Found empty translation "${translationKey}" in ${filePath}. Translation must be an object of the format {message: string}.`);
       continue;
     }
 
     if (!translation.message || typeof translation.message !== 'string') {
-      printValidationError(`No message found for translation "${translationKey}" in ${filePath}. Translation must be an object of the format {mesage: string}.`);
+      printValidationError(`No message found for translation "${translationKey}" in ${filePath}. Translation must be an object of the format {message: string}.`);
       continue;
     }
 
@@ -240,54 +336,44 @@ function loadAltLanguageFile({
   devLanguage,
   languages
 }) {
-  const result = {};
-  const languageHierarchy = getLanguageHierarcy({
-    languages
-  }).get(languageName);
-
-  if (!languageHierarchy) {
-    throw new Error(`Missing language hierarchy for ${languageName}`);
-  }
-
-  const fallbackLanguages = [languageName];
-
-  if (fallbacks !== 'none') {
-    fallbackLanguages.unshift(...languageHierarchy);
-
-    if (fallbacks === 'all' && fallbackLanguages[0] !== devLanguage) {
-      fallbackLanguages.unshift(devLanguage);
-    }
-  }
-
-  trace(`Loading alt language file with precendence: ${fallbackLanguages.slice().reverse().join(' -> ')}`);
+  const altLanguageTranslation = {};
+  const fallbackLanguageOrder = getFallbackLanguageOrder({
+    languages,
+    languageName,
+    devLanguage,
+    fallbacks
+  });
+  trace(`Loading alt language file with precedence: ${fallbackLanguageOrder.slice().reverse().join(' -> ')}`);
 
-  for (const fallbackLang of fallbackLanguages) {
-    if (fallbackLang !== devLanguage) {
+  for (const fallbackLanguage of fallbackLanguageOrder) {
+    if (fallbackLanguage !== devLanguage) {
       try {
-        const altFilePath = getAltLanguageFilePath(filePath, fallbackLang);
+        const altFilePath = getAltLanguageFilePath(filePath, fallbackLanguage);
         delete require.cache[altFilePath];
 
         const translationFile = require(altFilePath);
 
         const {
-          keys
+          keys: fallbackLanguageTranslation
         } = getTranslationsFromFile(translationFile, {
           filePath: altFilePath,
           isAltLanguage: true
         });
-        Object.assign(result, mergeWithDevLanguage(keys, devTranslation));
+        Object.assign(altLanguageTranslation, mergeWithDevLanguageTranslation({
+          translation: fallbackLanguageTranslation,
+          devTranslation
+        }));
       } catch (e) {
-        trace(`Missing alt language file ${getAltLanguageFilePath(filePath, fallbackLang)}
+        trace(`Missing alt language file ${getAltLanguageFilePath(filePath, fallbackLanguage)}
         `);
       }
     } else {
-      Object.assign(result, devTranslation);
+      Object.assign(altLanguageTranslation, devTranslation);
     }
   }
 
-  return result;
+  return altLanguageTranslation;
 }
-
 function loadTranslation({
   filePath,
   fallbacks
@@ -320,6 +406,19 @@ function loadTranslation({
     }, userConfig);
   }
 
+  for (const generatedLanguage of userConfig.generatedLanguages || []) {
+    const {
+      name: generatedLanguageName,
+      generator
+    } = generatedLanguage;
+    const baseLanguage = generatedLanguage.extends || userConfig.devLanguage;
+    const baseTranslations = languageSet[baseLanguage];
+    languageSet[generatedLanguageName] = generateLanguageFromTranslations({
+      baseTranslations,
+      generator
+    });
+  }
+
   return {
     filePath,
     keys: Object.keys(devTranslation),
@@ -603,7 +702,7 @@ async function writeIfChanged(filepath, contents) {
 }
 
 /* eslint-disable no-console */
-function findMissingKeys(loadedTranslation, devLanguageName, altLangauges) {
+function findMissingKeys(loadedTranslation, devLanguageName, altLanguages) {
   const devLanguage = loadedTranslation.languages[devLanguageName];
 
   if (!devLanguage) {
@@ -615,7 +714,7 @@ function findMissingKeys(loadedTranslation, devLanguageName, altLangauges) {
   const requiredKeys = Object.keys(devLanguage);
 
   if (requiredKeys.length > 0) {
-    for (const altLanguageName of altLangauges) {
+    for (const altLanguageName of altLanguages) {
       var _loadedTranslation$la;
 
       const altLanguage = (_loadedTranslation$la = loadedTranslation.languages[altLanguageName]) !== null && _loadedTranslation$la !== void 0 ? _loadedTranslation$la : {};
@@ -691,6 +790,35 @@ const schema = {
       }
     }
   },
+  generatedLanguages: {
+    type: 'array',
+    items: {
+      type: 'object',
+      props: {
+        name: {
+          type: 'string'
+        },
+        extends: {
+          type: 'string',
+          optional: true
+        },
+        generator: {
+          type: 'object',
+          props: {
+            transformElement: {
+              type: 'function',
+              optional: true
+            },
+            transformMessage: {
+              type: 'function',
+              optional: true
+            }
+          }
+        }
+      }
+    },
+    optional: true
+  },
   translationsDirectorySuffix: {
     type: 'string',
     optional: true
@@ -728,13 +856,12 @@ function validateConfig(c) {
 
       return v.message;
     }).join(' \n'));
-  } // Dev Language should exist in languages
-
+  }
 
-  const languageStrings = c.languages.map(v => v.name);
+  const languageStrings = c.languages.map(v => v.name); // Dev Language should exist in languages
 
   if (!languageStrings.includes(c.devLanguage)) {
-    throw new ValidationError('InvalidDevLanguage', `InvalidDevLanguage: The dev language "${chalk__default['default'].bold.cyan(c.devLanguage)}" was not found in languages ${languageStrings.join(', ')}.`);
+    throw new ValidationError('InvalidDevLanguage', `The dev language "${chalk__default['default'].bold.cyan(c.devLanguage)}" was not found in languages ${languageStrings.join(', ')}.`);
   }
 
   const foundLanguages = [];
@@ -752,6 +879,26 @@ function validateConfig(c) {
     }
   }
 
+  const foundGeneratedLanguages = [];
+
+  for (const generatedLang of c.generatedLanguages || []) {
+    // Generated languages must only exist once
+    if (foundGeneratedLanguages.includes(generatedLang.name)) {
+      throw new ValidationError('DuplicateGeneratedLanguage', `The generated language "${chalk__default['default'].bold.cyan(generatedLang.name)}" was defined multiple times.`);
+    }
+
+    foundGeneratedLanguages.push(generatedLang.name); // Generated language names must not conflict with language names
+
+    if (languageStrings.includes(generatedLang.name)) {
+      throw new ValidationError('InvalidGeneratedLanguage', `The generated language "${chalk__default['default'].bold.cyan(generatedLang.name)}" is already defined as a language.`);
+    } // Any extends must be in languages
+
+
+    if (generatedLang.extends && !languageStrings.includes(generatedLang.extends)) {
+      throw new ValidationError('InvalidExtends', `The generated language "${chalk__default['default'].bold.cyan(generatedLang.name)}"'s extends of ${chalk__default['default'].bold.cyan(generatedLang.extends)} was not found in languages ${languageStrings.join(', ')}.`);
+    }
+  }
+
   trace('Configuration file is valid');
   return true;
 }