gptrans 1.0.0

package/.gitattributes

@@ -0,0 +1,2 @@
+# Auto detect text files and perform LF normalization
+* text=auto

package/README.md

@@ -0,0 +1,115 @@
+# 🚆 GPTrans
+
+The smarter way to translate: AI-powered, cache-optimized, globally ready.
+
+It intelligently batches and caches translation requests, ensuring blazing-fast results and reducing API calls.
+
+Whether you're building a multilingual website, a mobile app, or a localization tool, GPTrans delivers top-tier performance with minimal setup.
+
+## ✨ Features
+
+- **AI-Powered Translations:** Harness advanced models like OpenAI's GPT and Anthropic's Sonnet for high-quality translations
+- **Smart Batching & Debouncing:** Automatically groups translation requests to optimize API usage
+- **Caching with DeepBase:** Quickly retrieves cached translations to boost performance
+- **Parameter Substitution:** Dynamically replace placeholders in your translations
+- **Flexible Configuration:** Customize source and target locales, model keys, and batching settings to fit your needs
+
+## 📦 Installation
+
+```bash
+npm install gptrans
+```
+
+## 🚀 Quick Start
+
+Here's a simple example to get you started:
+
+```javascript
+import GPTrans from 'gptrans';
+
+const gptrans = new GPTrans({
+  target: 'es-AR',
+  from: 'en-US',
+  model: 'claude-3-5-sonnet-20241022'
+});
+
+// Translate text with parameter substitution
+console.log(gptrans.t('Hello, {name}!', { name: 'John' }));
+
+// Set context for gender-aware translations
+console.log(gptrans.setContext('Message is for a woman').t('You are very good'));
+
+// Other translation examples
+console.log(gptrans.t('Withdraw'));
+console.log(gptrans.t('Top-up'));
+console.log(gptrans.t('Transfer'));
+console.log(gptrans.t('Deposit'));
+console.log(gptrans.t('Balance'));
+console.log(gptrans.t('Transaction'));
+console.log(gptrans.t('Account'));
+console.log(gptrans.t('Card'));
+```
+
+## ⚙️ Configuration Options
+
+When creating a new instance of Gptrans, you can customize:
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `target` | Target language locale | `'en-US'` |
+| `from` | Source language locale | `'es-AR'` |
+| `model` | Translation model key | - |
+| `batchThreshold` | Maximum number of characters to accumulate before triggering batch processing | `1000` |
+| `debounceTimeout` | Time in milliseconds to wait before processing translations | `500` |
+
+## 🔍 How It Works
+
+1. **First-Time Translation Behavior:** On the first request, Gptrans will return the original text while processing the translation in the background. This ensures your application remains responsive without waiting for API calls.
+2. **Translation Caching:** Once processed, translations are stored in `db/gptrans_<iso>.json`. Subsequent requests for the same text will be served instantly from the cache.
+3. **Smart Batch Processing:** Translations are processed in batches, providing better context for more accurate results.
+4. **Dynamic Model Integration:** Easily plug in multiple AI translation providers with the ModelMix library.
+5. **Customizable Prompts:** Load and modify translation prompts (see the `prompt/translate.md` file) to fine-tune the translation output.
+6. **Manual Corrections:** A JSON file stores key-translation pairs, allowing you to override specific translations and make manual corrections when needed. Simply edit the `db/gptrans_<iso>.json` file:
+
+```json
+{
+{
+    "balanc_pephba": "Saldo",
+    "transa_m1wmv2": "Transacción",
+    "accoun_rtvnkg": "Cuenta",
+    "card_yis1pt": "Tarjeta",
+    "hello_name_1vhllz3": "¡Hola, {name}!",
+    ...
+}
+```
+
+## 🌐 Environment Setup
+
+Gptrans uses dotenv for environment configuration. Create a `.env` file in your project root and add your API keys:
+
+```env
+OPENAI_API_KEY=your_openai_api_key
+ANTHROPIC_API_KEY=your_anthropic_api_key
+```
+
+## 🎉 Why Choose Gptrans?
+
+Gptrans stands out by combining advanced AI capabilities with efficient batching and caching. This means:
+
+- **Speed:** Reduced API calls and instant retrieval of cached translations
+- **Quality:** Leverage state-of-the-art models for precise and context-aware translations
+- **Flexibility:** Tailor the tool to your specific localization needs with minimal effort
+- **Zero Maintenance:** Set it up once and forget about it - automatic updates and self-healing capabilities keep everything running smoothly
+
+If you're looking to streamline your translation workflow and bring your applications to a global audience effortlessly, Gptrans is the perfect choice!
+
+## Contributing
+
+Contributions are welcome! Please open an issue or submit a pull request on GitHub to contribute improvements or fixes.
+
+## License
+
+GPTrans is released under the MIT License.
+
+Happy translating! 🌍✨
+

package/db/gptrans_es-AR.json

@@ -0,0 +1,13 @@
+{
+    "eres_muy_bueno_26czme": "Sos muy bueno",
+    "eres_muy_bueno_k3ml5b": "Sos muy buena",
+    "hello_name_1987p1n": "¡Hola, {name}!",
+    "topup_uzdh5y": "Recargar",
+    "transf_176pc1a": "Transferir",
+    "deposi_wg2ec5": "Depositar",
+    "balanc_1rv8if7": "Saldo",
+    "transa_1wtqm5d": "Transacción",
+    "accoun_x1y0v8": "Cuenta",
+    "card_yis1ox": "Tarjeta",
+    "tienes_fuego_1i2o3ok": "¿Tenés fuego?"
+}

package/db/gptrans_es-ES.json

@@ -0,0 +1,3 @@
+{
+    "tenes_fuego_1fs98im": "¿Tienes fuego?"
+}

package/db/gptrans_from_en-US.json

@@ -0,0 +1,10 @@
+{
+    "hello_name_1987p1n": "Hello, {name}!",
+    "topup_uzdh5y": "Top-up",
+    "transf_176pc1a": "Transfer",
+    "deposi_wg2ec5": "Deposit",
+    "balanc_1rv8if7": "Balance",
+    "transa_1wtqm5d": "Transaction",
+    "accoun_x1y0v8": "Account",
+    "card_yis1ox": "Card"
+}

package/db/gptrans_from_es-AR.json

@@ -0,0 +1,3 @@
+{
+    "tenes_fuego_1fs98im": "¿Tenés fuego?"
+}

package/db/gptrans_from_es-ES.json

@@ -0,0 +1,5 @@
+{
+    "eres_muy_bueno_26czme": "Eres muy bueno",
+    "eres_muy_bueno_k3ml5b": "Eres muy bueno",
+    "tienes_fuego_1i2o3ok": "Tienes fuego?"
+}

package/demo/case_1.js

@@ -0,0 +1,36 @@
+import GPTrans from '../index.js';
+
+const gptrans = new GPTrans({
+    model: 'claude-3-5-sonnet-20241022',
+});
+
+console.log(gptrans.t('Hello, {name}!', { name: 'Anya' }));
+
+console.log(gptrans.t('Top-up'));
+console.log(gptrans.t('Transfer'));
+console.log(gptrans.t('Deposit'));
+console.log(gptrans.t('Balance'));
+console.log(gptrans.t('Transaction'));
+console.log(gptrans.t('Account'));
+console.log(gptrans.t('Card'));
+
+// Case 2: Translate from Spanish Spain to Spanish Argentina
+const es2ar = new GPTrans({
+    from: 'es-ES',
+    target: 'es-AR',
+    model: 'claude-3-5-sonnet-20241022',
+});
+
+console.log(es2ar.t('Eres muy bueno'));
+console.log(es2ar.setContext('El mensaje es para una mujer').t('Eres muy bueno'));
+console.log(es2ar.setContext().t('Tienes fuego?'));
+
+// Case 3
+const ar2es = new GPTrans({
+    from: 'es-AR',
+    target: 'es-ES',
+    model: 'claude-3-5-sonnet-20241022',
+});
+
+console.log(ar2es.t('¿Tenés fuego?'));
+

package/index.js

@@ -0,0 +1,152 @@
+import DeepBase from 'deepbase';
+import stringHash from 'string-hash';
+import { ModelMix, MixOpenAI, MixAnthropic } from 'modelmix';
+import dotenv from 'dotenv';
+
+import { isoAssoc } from './isoAssoc.js';
+dotenv.config();
+
+class Gptrans {
+    static #mmixInstance = null;
+
+    static get mmix() {
+        if (!this.#mmixInstance) {
+            const mmix = new ModelMix();
+
+            mmix.attach(new MixOpenAI());
+            mmix.attach(new MixAnthropic());
+
+            this.#mmixInstance = mmix;
+        }
+        return this.#mmixInstance;
+    }
+
+    constructor({ from = 'en-US', target = 'es-AR', model = 'gpt-4o-mini', batchThreshold = 1000, debounceTimeout = 500, promptFile = './prompt/translate.md', context = '' }) {
+        this.target = target;
+        this.from = from;
+        this.dbTarget = new DeepBase({ name: 'gptrans_' + this.target });
+        this.dbFrom = new DeepBase({ name: 'gptrans_from_' + this.from });
+        this.batchThreshold = batchThreshold; // Now represents character count threshold
+        this.debounceTimeout = debounceTimeout;
+        this.pendingTranslations = new Map(); // [key, text]
+        this.pendingCharCount = 0; // Add character count tracker
+        this.debounceTimer = null;
+        this.modelKey = model;
+        this.promptFile = promptFile;
+        this.context = context;
+        this.modelConfig = {
+            config: {
+                max_history: 1,
+                debug: false,
+                bottleneck: {
+                    maxConcurrent: 5,
+                }
+            },
+            options: { max_tokens: batchThreshold }
+        };
+    }
+
+    setContext(context = '') {
+        if (this.context !== context && this.pendingTranslations.size > 0) {
+            clearTimeout(this.debounceTimer);
+            this._processBatch();
+        }
+        this.context = context;
+        return this;
+    }
+
+    t(text, params = {}) {
+        const key = this._textToKey(text);
+        const translation = this.get(key, text) || text;
+
+        return Object.entries(params).reduce(
+            (text, [key, value]) => text.replace(`{${key}}`, value),
+            translation
+        );
+    }
+
+    get(key, text) {
+        const translation = this.dbTarget.get(key);
+        if (!translation) {
+            this.pendingTranslations.set(key, text);
+            this.pendingCharCount += text.length; // Update character count
+
+            if (!this.dbFrom.get(key)) {
+                this.dbFrom.set(key, text);
+            }
+
+            // Clear existing timer
+            if (this.debounceTimer) {
+                clearTimeout(this.debounceTimer);
+            }
+
+            // Set new timer
+            this.debounceTimer = setTimeout(() => {
+                if (this.pendingTranslations.size > 0) {
+                    this._processBatch();
+                }
+            }, this.debounceTimeout);
+
+            // Process if we hit the character count threshold
+            if (this.pendingCharCount >= this.batchThreshold) {
+                clearTimeout(this.debounceTimer);
+                this._processBatch();
+            }
+        }
+        return translation;
+    }
+
+    async _processBatch() {
+        const batch = Array.from(this.pendingTranslations.entries());
+
+        // Clear pending translations and character count before awaiting translation
+        this.pendingTranslations.clear();
+        this.modelConfig.options.max_tokens = this.pendingCharCount + 1000;
+        this.pendingCharCount = 0;
+
+        const textsToTranslate = batch.map(([_, text]) => text).join('\n---\n');
+        const translations = await this._translate(textsToTranslate);
+
+        const translatedTexts = translations.split('\n---\n');
+
+        batch.forEach(([key], index) => {
+            this.dbTarget.set(key, translatedTexts[index].trim());
+        });
+    }
+
+    async _translate(text) {
+        const model = Gptrans.mmix.create(this.modelKey, this.modelConfig);
+
+        model.setSystem("You are an expert translator specialized in literary translation between FROM_LANG and TARGET_DENONYM TARGET_LANG.");
+
+        model.addTextFromFile(this.promptFile);
+
+        model.replace({ INPUT: text, CONTEXT: this.context });
+        model.replace(isoAssoc(this.target, 'TARGET_'));
+        model.replace(isoAssoc(this.from, 'FROM_'));
+
+        const response = await model.message();
+
+        const codeBlockRegex = /```(?:\w*\n)?([\s\S]*?)```/;
+        const match = response.match(codeBlockRegex);
+        const translatedText = match ? match[1].trim() : response;
+
+        return translatedText;
+    }
+
+    _textToKey(text, tokens = 5, maxlen = 6) {
+        const words = text
+            .toLowerCase()
+            .replace(/[áàâäéèêëíìîïóòôöúùûüñ]/g, c => 'aeioun'['áéíóúñ'.indexOf(c.toLowerCase())] || c)
+            .replace(/[^a-z0-9\s]+/g, "")
+            .split(" ")
+            .slice(0, tokens);
+
+        let key = words.map((x) => x.slice(0, maxlen)).join("_");
+        key += key ? '_' : '';
+        key += stringHash(text + this.context).toString(36);
+        return key;
+    }
+}
+
+export default Gptrans;

package/isoAssoc.js

@@ -0,0 +1,178 @@
+const countryName = {
+    'ar': 'Argentina',
+    'us': 'United States',
+    'es': 'Spain',
+    'pt': 'Portugal',
+    'br': 'Brazil',
+    'gb': 'United Kingdom',
+    'au': 'Australia',
+    'ca': 'Canada',
+    'cn': 'China',
+    'tw': 'Taiwan',
+    'hk': 'Hong Kong',
+    'sg': 'Singapore',
+    'mx': 'Mexico',
+    'in': 'India',
+    'sa': 'Saudi Arabia',
+    'bd': 'Bangladesh',
+    'ru': 'Russia',
+    'jp': 'Japan',
+    'fr': 'France',
+    'de': 'Germany',
+    'at': 'Austria',
+    'ch': 'Switzerland',
+    'kr': 'South Korea',
+    'it': 'Italy',
+    'tr': 'Turkey',
+    'vn': 'Vietnam',
+    'pl': 'Poland',
+    'nl': 'Netherlands',
+    'be': 'Belgium',
+    'id': 'Indonesia',
+    'th': 'Thailand',
+    'ph': 'Philippines',
+    'ir': 'Iran',
+    'ua': 'Ukraine',
+    'il': 'Israel',
+    'se': 'Sweden',
+    'no': 'Norway',
+    'fi': 'Finland',
+    'cz': 'Czech Republic',
+    'hu': 'Hungary',
+    'ro': 'Romania',
+    'bg': 'Bulgaria',
+    'co': 'Colombia',
+    'cl': 'Chile',
+    'pe': 'Peru',
+    've': 'Venezuela',
+    'ec': 'Ecuador',
+    'uy': 'Uruguay',
+    'py': 'Paraguay',
+    'bo': 'Bolivia',
+    'cr': 'Costa Rica',
+    'nz': 'New Zealand',
+    'gr': 'Greece',
+    'dk': 'Denmark'
+};
+
+const countryDenonym = {
+    'ar': 'Argentinian',
+    'es': 'Spanish',
+    'pt': 'Portuguese',
+    'br': 'Brazilian',
+    'us': 'American',
+    'gb': 'British',
+    'au': 'Australian',
+    'ca': 'Canadian',
+    'cn': 'Chinese',
+    'tw': 'Taiwanese',
+    'hk': 'Hong Kongese',
+    'sg': 'Singaporean',
+    'mx': 'Mexican',
+    'in': 'Indian',
+    'sa': 'Saudi Arabian',
+    'bd': 'Bangladeshi',
+    'ru': 'Russian',
+    'jp': 'Japanese',
+    'fr': 'French',
+    'de': 'German',
+    'at': 'Austrian',
+    'ch': 'Swiss',
+    'kr': 'Korean',
+    'it': 'Italian',
+    'tr': 'Turkish',
+    'vn': 'Vietnamese',
+    'pl': 'Polish',
+    'nl': 'Dutch',
+    'be': 'Belgian',
+    'id': 'Indonesian',
+    'th': 'Thai',
+    'ph': 'Filipino',
+    'ir': 'Iranian',
+    'ua': 'Ukrainian',
+    'il': 'Israeli',
+    'se': 'Swedish',
+    'no': 'Norwegian',
+    'fi': 'Finnish',
+    'cz': 'Czech',
+    'hu': 'Hungarian',
+    'ro': 'Romanian',
+    'bg': 'Bulgarian',
+    'co': 'Colombian',
+    'cl': 'Chilean',
+    'pe': 'Peruvian',
+    've': 'Venezuelan',
+    'ec': 'Ecuadorian',
+    'uy': 'Uruguayan',
+    'py': 'Paraguayan',
+    'bo': 'Bolivian',
+    'cr': 'Costa Rican',
+    'nz': 'New Zealander',
+    'gr': 'Greek',
+    'dk': 'Danish'
+};
+
+const langName = {
+    'es': 'Spanish',
+    'pt': 'Portuguese',
+    'en': 'English',
+    'zh': 'Chinese',
+    'hi': 'Hindi',
+    'ar': 'Arabic',
+    'bn': 'Bengali',
+    'ru': 'Russian',
+    'ja': 'Japanese',
+    'fr': 'French',
+    'de': 'German',
+    'ko': 'Korean',
+    'it': 'Italian',
+    'tr': 'Turkish',
+    'vi': 'Vietnamese',
+    'pl': 'Polish',
+    'nl': 'Dutch',
+    'id': 'Indonesian',
+    'th': 'Thai',
+    'tl': 'Tagalog',
+    'fa': 'Persian',
+    'uk': 'Ukrainian',
+    'he': 'Hebrew',
+    'sv': 'Swedish',
+    'no': 'Norwegian',
+    'fi': 'Finnish',
+    'cs': 'Czech',
+    'hu': 'Hungarian',
+    'ro': 'Romanian',
+    'bg': 'Bulgarian',
+    'ca': 'Catalan',
+    'gl': 'Galician',
+    'eu': 'Basque',
+    'el': 'Greek',
+    'da': 'Danish',
+    'ur': 'Urdu',
+    'ms': 'Malay'
+};
+
+export function isoAssoc(iso, prefix = '') {
+    if (!iso) {
+        throw new Error('ISO code is required');
+    }
+
+    const parts = iso.toLowerCase().split('-');
+    const lang = parts[0];
+    const country = parts.length > 1 ? parts[1] : null;
+
+    if (!langName[lang]) {
+        throw new Error(`Invalid language code: ${lang}`);
+    }
+
+    if (country && !countryName[country]) {
+        throw new Error(`Invalid country code: ${country}`);
+    }
+
+    return {
+        [prefix + 'ISO']: iso,
+        [prefix + 'LANG']: langName[lang],
+        [prefix + 'COUNTRY']: country ? countryName[country] : langName[lang],
+        [prefix + 'DENONYM']: country ? countryDenonym[country] : 'Universal',
+    };
+} 

package/package.json

@@ -0,0 +1,26 @@
+{
+  "name": "gptrans",
+  "type": "module",
+  "version": "1.0.0",
+  "description": "🚆 GPTrans - The smarter AI-powered way to translate.",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/clasen/GPTrans.git"
+  },
+  "main": "index.js",
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "author": "Martin Clasen",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/clasen/GPTrans/issues"
+  },
+  "homepage": "https://github.com/clasen/GPTrans#readme",
+  "dependencies": {
+    "deepbase": "^1.3.2",
+    "dotenv": "^16.4.7",
+    "modelmix": "^2.8.0",
+    "string-hash": "^1.1.3"
+  }
+}

package/prompt/translate.md

@@ -0,0 +1,19 @@
+# Goal
+Translation from FROM_ISO to TARGET_ISO (TARGET_DENONYM TARGET_LANG) with cultural adaptations.
+
+## Text to translate
+INPUT
+
+# Return Format
+   - Provide the final translation within a code block using ```.
+   - Do not include alternative translations, only provide the best translation.
+
+# Warnings
+   - **Context:** I will provide you with a text in FROM_DENONYM FROM_LANG. The goal is to translate it to TARGET_ISO (TARGET_DENONYM TARGET_LANG) while maintaining the essence, style, intention, and tone of the original.
+   - **Cultural references:** Adapt or explain references that are not familiar in TARGET_DENONYM culture, whenever necessary.
+   - **Wordplay and humor:** When it's impossible to directly translate wordplay, find a resource that recreates the playful effect.
+   - **Idioms:** Do not introduce new idioms or expressions that are not present in the original text.
+   - **Variables:** Do not translate content between curly braces {variable}. These are system variables and must remain exactly the same.
+
+# Context
+CONTEXT