gptrans 1.8.8 → 1.9.2

package/README.md

@@ -10,6 +10,8 @@ Whether you're building a multilingual website, a mobile app, or a localization
 
 - **AI-Powered Translations:** Harness advanced models like OpenAI's GPT and Anthropic's Sonnet for high-quality translations
 - **Smart Batching & Debouncing:** Translations are processed in batches, not only for efficiency but also to provide better context. By sending multiple related texts together, the AI model can better understand the overall context and produce more accurate and consistent translations across related terms and phrases.
+- **Reference Translations:** Use existing translations in other languages as context to improve accuracy and consistency across your multilingual content
+- **Flexible Base Language:** Translate from an intermediate language instead of the original, useful for avoiding gender-specific terms or leveraging more neutral language versions
 - **Caching with JSON:** Quickly retrieves cached translations to boost performance
 - **Parameter Substitution:** Dynamically replace placeholders in your translations
 - **Smart Context Handling:** Add contextual information to improve translation accuracy. Perfect for gender-aware translations, domain-specific content, or any scenario where additional context helps produce better results. The context is automatically cleared after each translation to prevent unintended effects.
@@ -117,12 +119,85 @@ If you're looking to streamline your translation workflow and bring your applica
 
 ## 🔄 Preloading Translations
 
-You can preload translations for specific languages using the `preload` method. This is particularly useful when you want to initialize translations based on dynamically generated keys:
+The `preload()` method allows you to pre-translate all texts in your database. It now supports advanced options for improved translation accuracy through reference translations and alternate base languages.
+
+### Basic Usage
+
+```javascript
+// Simple preload - translates all pending texts
+await gptrans.preload();
+```
+
+### Advanced Options
+
+#### Using Reference Translations
+
+Include translations from other languages as context to improve accuracy and consistency:
+
+```javascript
+// Use English and Portuguese translations as reference
+const gptrans = new GPTrans({ from: 'es', target: 'fr' });
+await gptrans.preload({ 
+  references: ['en', 'pt'] 
+});
+```
+
+The AI model will see existing translations in the reference languages, helping it produce more consistent and accurate translations.
+
+#### Using an Alternate Base Language
+
+Translate from an intermediate language instead of the original:
 
 ```javascript
-await gptrans.preload({ target:'ar' });
+// Original is Spanish, but translate FROM English TO Portuguese
+const gptrans = new GPTrans({ from: 'es', target: 'pt' });
+await gptrans.preload({ 
+  baseLanguage: 'en' 
+});
 ```
 
+This is useful when:
+- The intermediate language has characteristics that better suit the target (e.g., English "he/she" can be omitted in some languages)
+- You want to avoid gender-specific terms present in the source language
+- The intermediate translation is cleaner or more universal
+
+#### Combined Usage
+
+You can use both options together:
+
+```javascript
+// Translate from English to German, showing Spanish and Portuguese as reference
+const gptrans = new GPTrans({ from: 'es', target: 'de' });
+await gptrans.preload({ 
+  baseLanguage: 'en',
+  references: ['es', 'pt'] 
+});
+```
+
+### Real-World Example
+
+**Problem:** Gender-specific translations
+
+```javascript
+// Spanish: "El estudiante es muy bueno" (masculine)
+// English: "The student is very good" (neutral)
+
+// Solution: Translate to Portuguese using English as base
+const ptTranslator = new GPTrans({ from: 'es', target: 'pt' });
+await ptTranslator.preload({ 
+  baseLanguage: 'en',     // Use neutral English version
+  references: ['es']       // Show original Spanish for context
+});
+```
+
+### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `references` | `string[]` | Array of language codes (e.g., `['en', 'pt']`) to use as translation references |
+| `baseLanguage` | `string` | Language code to use as the base for translation instead of the original |
+
+
 ### Model Fallback System
 
 GPTrans supports a fallback mechanism for translation models. Instead of providing a single model, you can pass an array of models:

package/demo/case_references.js

@@ -20,7 +20,7 @@ async function testReferences() {
         target: 'en',
         model: 'sonnet45',
         name: 'ref_test',
-        debug: true
+        debug: 3
     });
     
     const ptTranslator = new GPTrans({ 
@@ -28,7 +28,7 @@ async function testReferences() {
         target: 'pt',
         model: 'sonnet45',
         name: 'ref_test',
-        debug: true
+        debug: 3
     });
     
     // Sample Spanish texts with gendered language
@@ -44,14 +44,16 @@ async function testReferences() {
         console.log(`   EN: ${enTranslator.t(text)}`);
     });
     
-    await new Promise(resolve => setTimeout(resolve, 5000));
+    // Wait for EN translations to complete if needed
+    await enTranslator.preload();
     
     console.log('\n📝 Creating Portuguese translations from Spanish...');
     spanishTexts.forEach(text => {
         console.log(`   PT: ${ptTranslator.t(text)}`);
     });
     
-    await new Promise(resolve => setTimeout(resolve, 5000));
+    // Wait for PT translations to complete if needed
+    await ptTranslator.preload();
     
     console.log('\n' + '='.repeat(70));
     console.log('\n📋 Test 2: Translation with English as reference\n');
@@ -62,7 +64,7 @@ async function testReferences() {
         target: 'fr',
         model: 'sonnet45',
         name: 'ref_test',
-        debug: true
+        debug: 3
     });
     
     console.log('🔄 Preloading French translations with English as reference...');
@@ -85,7 +87,7 @@ async function testReferences() {
         target: 'it',
         model: 'sonnet45',
         name: 'ref_test',
-        debug: true
+        debug: 3
     });
     
     console.log('🔄 Preloading Italian translations with English as base language...');
@@ -109,7 +111,7 @@ async function testReferences() {
         target: 'de',
         model: 'sonnet45',
         name: 'ref_test',
-        debug: true
+        debug: 3
     });
     
     console.log('🔄 Preloading German translations with multiple references...');

package/index.js

@@ -218,15 +218,45 @@ class GPTrans {
         const textsToTranslate = batch.map(([_, text]) => text).join(`\n${this.divider}\n`);
         try {
             const translations = await this._translate(textsToTranslate, batch, batchReferences, this.preloadBaseLanguage);
-            const translatedTexts = translations.split(`\n${this.divider}\n`);
+            
+            // Try different split strategies to be more robust
+            let translatedTexts = translations.split(`\n${this.divider}\n`);
+            
+            // If split doesn't match batch size, try alternative separators
+            if (translatedTexts.length !== batch.length) {
+                // Try without newlines around divider
+                translatedTexts = translations.split(this.divider);
+                
+                // If still doesn't match, try with just newline
+                if (translatedTexts.length !== batch.length) {
+                    translatedTexts = translations.split(/\n{2,}/); // Split by multiple newlines
+                }
+            }
 
             const contextHash = this._hash(context);
-            batch.forEach(([key], index) => {
-
-                if (!translatedTexts[index]) {
-                    console.log(translations);
-                    console.error(`No translation found for ${key}`);
+            
+            // Validate we have the right number of translations
+            if (translatedTexts.length !== batch.length) {
+                console.error(`❌ Translation count mismatch:`);
+                console.error(`   Expected: ${batch.length} translations`);
+                console.error(`   Received: ${translatedTexts.length} translations`);
+                console.error(`   Batch keys: ${batch.map(([key]) => key).join(', ')}`);
+                console.error(`\n📝 Full response from model:\n${translations}\n`);
+                
+                // Try to save what we can
+                const minLength = Math.min(translatedTexts.length, batch.length);
+                for (let i = 0; i < minLength; i++) {
+                    if (translatedTexts[i] && translatedTexts[i].trim()) {
+                        this.dbTarget.set(contextHash, batch[i][0], translatedTexts[i].trim());
+                    }
+                }
+                return;
+            }
 
+            batch.forEach(([key], index) => {
+                if (!translatedTexts[index] || !translatedTexts[index].trim()) {
+                    console.error(`❌ No translation found for ${key} at index ${index}`);
+                    console.error(`   Original text: ${batch[index][1]}`);
                     return;
                 }
 
@@ -234,7 +264,8 @@ class GPTrans {
             });
 
         } catch (e) {
-            console.error(e);
+            console.error('❌ Error in _processBatch:', e.message);
+            console.error(e.stack);
         }
     }
 
@@ -247,29 +278,50 @@ class GPTrans {
 
             model.setSystem("You are an expert translator specialized in literary translation between FROM_LANG and TARGET_DENONYM TARGET_LANG.");
 
-            model.addTextFromFile(this.promptFile);
+            // Read and process prompt file
+            let promptContent = fs.readFileSync(this.promptFile, 'utf-8');
 
             // Format references if available
             let referencesText = '';
             if (Object.keys(batchReferences).length > 0 && batch.length > 0) {
-                const textsArray = text.split(`\n${this.divider}\n`);
-
-                referencesText = textsArray.map((txt, index) => {
-                    const key = batch[index] ? batch[index][0] : null;
-                    if (key && batchReferences[key]) {
-                        const refs = batchReferences[key];
-                        const refLines = Object.entries(refs).map(([lang, translation]) => {
-                            try {
-                                const langInfo = isoAssoc(lang);
-                                return `${langInfo.DENONYM} ${langInfo.LANG} (${lang}): ${translation}`;
-                            } catch (e) {
-                                return `${lang}: ${translation}`;
+                // Group all references by language first
+                const refsByLang = {};
+                
+                batch.forEach(([key], index) => {
+                    if (batchReferences[key]) {
+                        Object.entries(batchReferences[key]).forEach(([lang, translation]) => {
+                            if (!refsByLang[lang]) {
+                                refsByLang[lang] = [];
                             }
-                        }).join(`\n${this.divider}\n`);
-                        return refLines;
+                            refsByLang[lang].push(translation);
+                        });
+                    }
+                });
+                
+                // Format: one language header, then all its translations with bullets
+                const refBlocks = Object.entries(refsByLang).map(([lang, translations]) => {
+                    try {
+                        const langInfo = isoAssoc(lang);
+                        const header = `### ${langInfo.DENONYM} ${langInfo.LANG} (${lang}):`;
+                        const content = translations.map(t => `- ${t}`).join('\n');
+                        return `${header}\n${content}`;
+                    } catch (e) {
+                        const header = `### ${lang}:`;
+                        const content = translations.map(t => `- ${t}`).join('\n');
+                        return `${header}\n${content}`;
                     }
-                    return '';
-                }).filter(r => r).join(`\n\n`);
+                });
+                
+                referencesText = refBlocks.join(`\n\n`);
+            }
+
+            // Remove reference section if no references
+            if (!referencesText) {
+                // Remove the entire "Reference Translations" section
+                promptContent = promptContent.replace(
+                    /## Reference Translations \(for context\)[\s\S]*?(?=\n#|$)/,
+                    ''
+                );
             }
 
             // Determine which FROM_ values to use
@@ -282,21 +334,31 @@ class GPTrans {
                 }
             }
 
-            model.replace({
-                INPUT: text,
-                CONTEXT: this.context,
-                REFERENCES: referencesText || 'None'
+            // Apply replacements to prompt
+            promptContent = promptContent
+                .replace(/INPUT/g, text)
+                .replace(/CONTEXT/g, this.context)
+                .replace(/REFERENCES/g, referencesText);
+
+            // Apply language-specific replacements
+            Object.entries(this.replaceTarget).forEach(([key, value]) => {
+                promptContent = promptContent.replace(new RegExp(key, 'g'), value);
             });
-            model.replace(this.replaceTarget);
-            model.replace(fromReplace);
+            Object.entries(fromReplace).forEach(([key, value]) => {
+                promptContent = promptContent.replace(new RegExp(key, 'g'), value);
+            });
+
+            model.addText(promptContent);
 
             const response = await model.message();
 
+            // Extract content from code block if present
             const codeBlockRegex = /```(?:\w*\n)?([\s\S]*?)```/;
             const match = response.match(codeBlockRegex);
-            const translatedText = match ? match[1].trim() : response;
+            const translatedText = match ? match[1].trim() : response.trim();
 
             return translatedText;
+
         } finally {
             // Always release the lock
             releaseLock();
@@ -347,8 +409,24 @@ class GPTrans {
             return this;
         }
 
+        // Filter out invalid references
+        const validReferences = references.filter(lang => {
+            const normalizedLang = this.normalizeBCP47(lang);
+            // Don't include target language as reference (we're translating TO it)
+            if (normalizedLang === this.replaceTarget.TARGET_ISO) {
+                console.warn(`Ignoring reference language '${lang}': cannot use target language as reference`);
+                return false;
+            }
+            // Don't include baseLanguage as reference (redundant)
+            if (baseLanguage && normalizedLang === this.normalizeBCP47(baseLanguage)) {
+                console.warn(`Ignoring reference language '${lang}': same as baseLanguage`);
+                return false;
+            }
+            return true;
+        });
+
         // Store preload options for use in translation
-        this.preloadReferences = references;
+        this.preloadReferences = validReferences;
         this.preloadBaseLanguage = baseLanguage;
 
         // Track which keys need translation
@@ -365,8 +443,9 @@ class GPTrans {
                 // Check if translation already exists
                 if (!this.dbTarget.get(contextHash, key)) {
                     keysNeedingTranslation.push({ context, contextHash, key });
+                    // Only call get() if translation doesn't exist
+                    this.get(key, text);
                 }
-                this.get(key, text);
             }
         }
 
@@ -378,10 +457,8 @@ class GPTrans {
         }
 
         // Wait for any pending translations to complete
-        const maxWaitTime = 120000; // 120 seconds timeout
-        const startTime = Date.now();
-        
-        await new Promise((resolve, reject) => {
+        // No global timeout - each translation request has its own timeout
+        await new Promise((resolve) => {
             const checkInterval = setInterval(() => {
                 // Check if there are still pending translations or batch being processed
                 const hasPending = this.pendingTranslations.size > 0 || this.isProcessingBatch;
@@ -399,13 +476,6 @@ class GPTrans {
                     clearInterval(checkInterval);
                     resolve();
                 }
-                
-                // Timeout check
-                if (Date.now() - startTime > maxWaitTime) {
-                    clearInterval(checkInterval);
-                    console.warn(`Preload timeout: ${keysNeedingTranslation.length} translations pending`);
-                    resolve(); // Resolve instead of reject to allow partial completion
-                }
             }, 100);
         });
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "gptrans",
   "type": "module",
-  "version": "1.8.8",
+  "version": "1.9.2",
   "description": "🚆 GPTrans - The smarter AI-powered way to translate.",
   "keywords": [
     "translate",

package/prompt/translate.md

@@ -8,13 +8,22 @@ INPUT
 
 ## Reference Translations (for context)
 These are existing translations in other languages that may help you provide a more accurate translation. Use them as reference but do not simply copy them:
-```
+
 REFERENCES
-```
 
 # Return Format
    - Provide the final translation within a code block using ```.
    - Do not include alternative translations, only provide the best translation.
+   - **IMPORTANT:** If the input contains multiple texts separated by `------`, you MUST return the translations in the SAME ORDER, separated by the EXACT SAME separator:
+     ```
+     Translation 1
+     ------
+     Translation 2
+     ------
+     Translation 3
+     ```
+   - Do not add extra newlines or modify the separator format.
+   - Each translation should correspond to each input text in the same order.
 
 # Warnings
    - **Context:** I will provide you with a text in FROM_DENONYM FROM_LANG. The goal is to translate it to TARGET_ISO (TARGET_DENONYM TARGET_LANG) while maintaining the essence, style, intention, and tone of the original.

package/demo/REFERENCES_USAGE.md

@@ -1,101 +0,0 @@
-# GPTrans - Referencias Múltiples y Idioma Base Alternativo
-
-## Nuevas Funcionalidades
-
-El método `preload()` ahora soporta dos parámetros opcionales que mejoran la precisión de las traducciones:
-
-### 1. `references` - Referencias Múltiples
-
-Permite incluir traducciones existentes en otros idiomas como contexto adicional para el modelo de IA.
-
-```javascript
-await gptrans.preload({ 
-  references: ['en', 'pt']  // Usar inglés y portugués como referencia
-});
-```
-
-**Caso de uso**: Cuando tienes traducciones en varios idiomas y quieres que la nueva traducción sea consistente con las existentes.
-
-### 2. `baseLanguage` - Idioma Base Alternativo
-
-Permite usar un idioma diferente al original como base para la traducción.
-
-```javascript
-await gptrans.preload({ 
-  baseLanguage: 'en'  // Traducir desde inglés en vez del idioma original
-});
-```
-
-**Caso de uso**: Cuando el texto original tiene características específicas del idioma (como he/she en inglés) que pueden omitirse en la traducción final.
-
-## Ejemplos de Uso
-
-### Ejemplo 1: Traducción Básica (sin cambios)
-
-```javascript
-const gptrans = new GPTrans({ from: 'en', target: 'es' });
-await gptrans.preload();  // Comportamiento original
-```
-
-### Ejemplo 2: Con Referencias
-
-```javascript
-// Original en español, traducir a francés usando inglés como referencia
-const gptrans = new GPTrans({ from: 'es', target: 'fr' });
-await gptrans.preload({ 
-  references: ['en']  // El modelo verá la traducción en inglés como contexto
-});
-```
-
-### Ejemplo 3: Con Idioma Base Alternativo
-
-```javascript
-// Original en español, pero traducir DE inglés A portugués
-const gptrans = new GPTrans({ from: 'es', target: 'pt' });
-await gptrans.preload({ 
-  baseLanguage: 'en'  // Usa la traducción en inglés como base
-});
-```
-
-### Ejemplo 4: Combinado
-
-```javascript
-// Original en español, traducir de inglés a alemán, mostrando español y portugués como referencia
-const gptrans = new GPTrans({ from: 'es', target: 'de' });
-await gptrans.preload({ 
-  baseLanguage: 'en',     // Traduce desde inglés
-  references: ['es', 'pt'] // Muestra español y portugués como contexto adicional
-});
-```
-
-## Caso de Uso Real: Evitar Problemas de Género
-
-### Problema
-En inglés: "The student is very good" (neutral)
-En español: "El estudiante es muy bueno" / "La estudiante es muy buena" (con género)
-En portugués: Se puede omitir el género en algunos contextos
-
-### Solución
-```javascript
-// Si el original está en español pero queremos la traducción al portugués
-// basada en la versión inglesa (que es neutral):
-
-const ptTranslator = new GPTrans({ from: 'es', target: 'pt' });
-await ptTranslator.preload({ 
-  baseLanguage: 'en',  // Usa la versión inglesa como base
-  references: ['es']    // Muestra el español original como referencia
-});
-```
-
-## Compatibilidad
-
-✅ **Totalmente retrocompatible**: Si no especificas ningún parámetro, `preload()` funciona exactamente igual que antes.
-
-```javascript
-// Esto sigue funcionando sin cambios:
-await gptrans.preload();
-```
-
-## Archivo de Prueba
-
-Ejecuta `demo/case_references.js` para ver ejemplos completos de todas las funcionalidades nuevas.