supervertaler 1.9.172__tar.gz → 1.9.173__tar.gz

{supervertaler-1.9.172 → supervertaler-1.9.173}/CHANGELOG.md

@@ -2,10 +2,22 @@
 
 All notable changes to Supervertaler are documented in this file.
 
-**Current Version:** v1.9.172 (January 28, 2026)
+**Current Version:** v1.9.173 (January 28, 2026)
 
 
 
+## v1.9.173 - January 28, 2026
+
+### ✨ Improvements
+
+- **Smarter TM Exact Matching**: Exact matches now use text normalization before hashing, so matches are found even when source text differs only in whitespace or Unicode normalization (e.g., non-breaking spaces vs regular spaces).
+
+- **Improved TM Pre-Translation Dialog**: The progress dialog now shows the current segment being processed, count of matches found so far, and elapsed time. A patience message appears after 10 seconds for large jobs.
+
+- **Higher Fuzzy Match Threshold**: Raised the minimum fuzzy match threshold from 70% to 75% for better quality matches across all TM operations.
+
+---
+
 ## v1.9.172 - January 28, 2026
 
 ### 🐛 Bug Fixes

{supervertaler-1.9.172 → supervertaler-1.9.173}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: supervertaler
-Version: 1.9.172
+Version: 1.9.173
 Summary: Professional AI-enhanced translation workbench with multi-LLM support, glossary system, TM, spellcheck, voice commands, and PyQt6 interface. Batteries included (core).
 Home-page: https://supervertaler.com
 Author: Michael Beijer
@@ -71,7 +71,7 @@ Dynamic: home-page
 Dynamic: license-file
 Dynamic: requires-python
 
-# 🚀 Supervertaler v1.9.172
+# 🚀 Supervertaler v1.9.173
 
 [![PyPI version](https://badge.fury.io/py/supervertaler.svg)](https://pypi.org/project/Supervertaler/)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
@@ -80,7 +80,13 @@ Dynamic: requires-python
 AI-enhanced CAT tool with multi-LLM support (GPT-4, Claude, Gemini, Ollama), innovative Superlookup concordance system offering access to multiple terminology sources (TMs, glossaries, web resources, etc.), and seamless CAT tool integration (memoQ, Trados, CafeTran, Phrase).
 
 
-**Current Version:** v1.9.172 (January 28, 2026)
+**Current Version:** v1.9.173 (January 28, 2026)
+
+### IMPROVED in v1.9.173 - 🎯 Smarter TM Pre-Translation
+
+- **Smarter TM Exact Matching**: Exact matches now use text normalization, so matches are found even with whitespace/Unicode differences.
+- **Improved Pre-Translation Dialog**: Shows current segment, match count, elapsed time, and patience message for large jobs.
+- **Higher Fuzzy Threshold**: Raised minimum fuzzy match threshold from 70% to 75% for better quality.
 
 ### FIXED in v1.9.172 - 🐛 Fresh Projects Start Clean
 

{supervertaler-1.9.172 → supervertaler-1.9.173}/README.md

@@ -1,4 +1,4 @@
-# 🚀 Supervertaler v1.9.172
+# 🚀 Supervertaler v1.9.173
 
 [![PyPI version](https://badge.fury.io/py/supervertaler.svg)](https://pypi.org/project/Supervertaler/)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
@@ -7,7 +7,13 @@
 AI-enhanced CAT tool with multi-LLM support (GPT-4, Claude, Gemini, Ollama), innovative Superlookup concordance system offering access to multiple terminology sources (TMs, glossaries, web resources, etc.), and seamless CAT tool integration (memoQ, Trados, CafeTran, Phrase).
 
 
-**Current Version:** v1.9.172 (January 28, 2026)
+**Current Version:** v1.9.173 (January 28, 2026)
+
+### IMPROVED in v1.9.173 - 🎯 Smarter TM Pre-Translation
+
+- **Smarter TM Exact Matching**: Exact matches now use text normalization, so matches are found even with whitespace/Unicode differences.
+- **Improved Pre-Translation Dialog**: Shows current segment, match count, elapsed time, and patience message for large jobs.
+- **Higher Fuzzy Threshold**: Raised minimum fuzzy match threshold from 70% to 75% for better quality.
 
 ### FIXED in v1.9.172 - 🐛 Fresh Projects Start Clean
 

{supervertaler-1.9.172 → supervertaler-1.9.173}/Supervertaler.py

@@ -5598,7 +5598,7 @@ class PreTranslationWorker(QThread):
                     match = matches[0]
                     match_pct = match.get('match_pct', 0)
                     print(f"🔍 TM PRE-TRANSLATE: Best match pct: {match_pct}")
-                    if match_pct >= 70:  # Accept matches 70% and above
+                    if match_pct >= 75:  # Accept matches 75% and above
                         return match.get('target', '')
                 return None
         except Exception as e:
@@ -41148,25 +41148,41 @@ OUTPUT ONLY THE SEGMENT MARKERS. DO NOT ADD EXPLANATIONS BEFORE OR AFTER."""
             self.log(f"📖 Pre-translate from TM: Using activated TMs: {tm_ids}")
             
             # Create progress dialog for TM pre-translation
+            import time
+            start_time = time.time()
+            total_segments = len(segments_needing_translation)
+
             progress = QProgressDialog(
-                f"Pre-translating {len(segments_needing_translation)} segments from TM...",
-                "Cancel", 0, len(segments_needing_translation), self
+                f"Pre-translating {total_segments} segments from TM...",
+                "Cancel", 0, total_segments, self
             )
-            progress.setWindowTitle("TM Pre-Translation")
+            progress.setWindowTitle("🔍 TM Pre-Translation")
             progress.setWindowModality(Qt.WindowModality.WindowModal)
             progress.setMinimumDuration(0)  # Show immediately
+            progress.setMinimumWidth(450)  # Wider dialog for more info
             progress.show()
             QApplication.processEvents()
-            
+
             success_count = 0
             no_match_count = 0
-            
+
             for idx, (row_index, segment) in enumerate(segments_needing_translation):
                 if progress.wasCanceled():
                     break
-                
+
                 progress.setValue(idx)
-                progress.setLabelText(f"Searching TM for segment {idx + 1}/{len(segments_needing_translation)}...")
+
+                # Build informative progress label
+                elapsed = time.time() - start_time
+                elapsed_str = f"{int(elapsed // 60)}:{int(elapsed % 60):02d}"
+                source_preview = segment.source[:50] + "..." if len(segment.source) > 50 else segment.source
+                label_text = (
+                    f"Searching TM for segment {idx + 1} of {total_segments}...\n\n"
+                    f"Current: \"{source_preview}\"\n"
+                    f"Matches found: {success_count}  |  Elapsed: {elapsed_str}\n\n"
+                    f"ℹ️ This may take a while for large documents."
+                )
+                progress.setLabelText(label_text)
                 QApplication.processEvents()
                 
                 try:
@@ -41184,7 +41200,7 @@ OUTPUT ONLY THE SEGMENT MARKERS. DO NOT ADD EXPLANATIONS BEFORE OR AFTER."""
                         else:
                             no_match_count += 1
                     else:
-                        # Fuzzy matching enabled - get best match ≥70%
+                        # Fuzzy matching enabled - get best match ≥75%
                         matches = self.tm_database.search_all(
                             segment.source,
                             tm_ids=tm_ids,
@@ -41194,7 +41210,7 @@ OUTPUT ONLY THE SEGMENT MARKERS. DO NOT ADD EXPLANATIONS BEFORE OR AFTER."""
                         if matches and len(matches) > 0:
                             best_match = matches[0]
                             match_pct = best_match.get('match_pct', 0)
-                            if match_pct >= 70:
+                            if match_pct >= 75:
                                 segment.target = best_match.get('target', '')
                                 segment.status = "Translated"
                                 success_count += 1
@@ -41562,7 +41578,7 @@ OUTPUT ONLY THE SEGMENT MARKERS. DO NOT ADD EXPLANATIONS BEFORE OR AFTER."""
                             match_pct = match.get('match_pct', 0)
                             tm_match = match.get('target', '')
                             
-                            if match_pct >= 70:  # Accept matches 70% and above
+                            if match_pct >= 75:  # Accept matches 75% and above
                                 segment.target = tm_match
                                 segment.status = "translated" if match_pct == 100 else "pre-translated"
                                 translated_count += 1

{supervertaler-1.9.172 → supervertaler-1.9.173}/docs/index.html

@@ -17,7 +17,7 @@
     <link rel="apple-touch-icon" href="apple-touch-icon.png">
     <link rel="manifest" href="site.webmanifest">
 
-    <link rel="stylesheet" href="styles.css?v=1.9.171">
+    <link rel="stylesheet" href="styles.css?v=1.9.173">
     <link rel="preconnect" href="https://fonts.googleapis.com">
     <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
     <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700;800&display=swap"

{supervertaler-1.9.172 → supervertaler-1.9.173}/modules/database_manager.py

@@ -17,12 +17,38 @@ import sqlite3
 import os
 import json
 import hashlib
+import unicodedata
+import re
 from datetime import datetime
 from typing import List, Dict, Optional, Tuple
 from pathlib import Path
 from difflib import SequenceMatcher
 
 
+def _normalize_for_matching(text: str) -> str:
+    """Normalize text for exact matching.
+
+    Handles invisible differences that would cause exact match to fail:
+    - Unicode normalization (NFC)
+    - Multiple whitespace -> single space
+    - Leading/trailing whitespace
+    - Non-breaking spaces -> regular spaces
+    """
+    if not text:
+        return ""
+    # Unicode normalize (NFC form)
+    text = unicodedata.normalize('NFC', text)
+    # Convert non-breaking spaces and other whitespace to regular space
+    text = text.replace('\u00a0', ' ')  # NBSP
+    text = text.replace('\u2007', ' ')  # Figure space
+    text = text.replace('\u202f', ' ')  # Narrow NBSP
+    # Collapse multiple whitespace to single space
+    text = re.sub(r'\s+', ' ', text)
+    # Strip leading/trailing whitespace
+    text = text.strip()
+    return text
+
+
 class DatabaseManager:
     """Manages SQLite database for translation resources"""
     
@@ -655,17 +681,19 @@ class DatabaseManager:
     # TRANSLATION MEMORY METHODS
     # ============================================
     
-    def add_translation_unit(self, source: str, target: str, source_lang: str, 
+    def add_translation_unit(self, source: str, target: str, source_lang: str,
                             target_lang: str, tm_id: str = 'project',
                             project_id: str = None, context_before: str = None,
                             context_after: str = None, notes: str = None) -> int:
         """
         Add translation unit to database
-        
+
         Returns: ID of inserted/updated entry
         """
-        # Generate hash for fast exact matching
-        source_hash = hashlib.md5(source.encode('utf-8')).hexdigest()
+        # Generate hash from NORMALIZED source for consistent exact matching
+        # This handles invisible differences like Unicode normalization, whitespace variations
+        normalized_source = _normalize_for_matching(source)
+        source_hash = hashlib.md5(normalized_source.encode('utf-8')).hexdigest()
         
         try:
             self.cursor.execute("""
@@ -687,33 +715,38 @@ class DatabaseManager:
             return None
     
     def get_exact_match(self, source: str, tm_ids: List[str] = None,
-                       source_lang: str = None, target_lang: str = None, 
+                       source_lang: str = None, target_lang: str = None,
                        bidirectional: bool = True) -> Optional[Dict]:
         """
         Get exact match from TM
-        
+
         Args:
             source: Source text to match
             tm_ids: List of TM IDs to search (None = all)
             source_lang: Filter by source language (base code matching: 'en' matches 'en-US', 'en-GB', etc.)
             target_lang: Filter by target language (base code matching)
             bidirectional: If True, search both directions (nl→en AND en→nl)
-        
+
         Returns: Dictionary with match data or None
         """
         from modules.tmx_generator import get_base_lang_code
-        
+
+        # Try both normalized and non-normalized hashes for backward compatibility
+        # This handles invisible differences like Unicode normalization, whitespace variations
         source_hash = hashlib.md5(source.encode('utf-8')).hexdigest()
-        
+        normalized_source = _normalize_for_matching(source)
+        normalized_hash = hashlib.md5(normalized_source.encode('utf-8')).hexdigest()
+
         # Get base language codes for comparison
         src_base = get_base_lang_code(source_lang) if source_lang else None
         tgt_base = get_base_lang_code(target_lang) if target_lang else None
-        
+
+        # Search using both original hash and normalized hash
         query = """
-            SELECT * FROM translation_units 
-            WHERE source_hash = ? AND source_text = ?
+            SELECT * FROM translation_units
+            WHERE (source_hash = ? OR source_hash = ?)
         """
-        params = [source_hash, source]
+        params = [source_hash, normalized_hash]
         
         if tm_ids:
             placeholders = ','.join('?' * len(tm_ids))

{supervertaler-1.9.172 → supervertaler-1.9.173}/modules/translation_memory.py

@@ -123,8 +123,8 @@ class TMDatabase:
         if source_lang and target_lang:
             self.set_tm_languages(source_lang, target_lang)
         
-        # Global fuzzy threshold (70% minimum similarity for fuzzy matches)
-        self.fuzzy_threshold = 0.7
+        # Global fuzzy threshold (75% minimum similarity for fuzzy matches)
+        self.fuzzy_threshold = 0.75
         
         # TM metadata cache (populated from database as needed)
         # Note: Legacy 'project' and 'big_mama' TMs are no longer used.

{supervertaler-1.9.172 → supervertaler-1.9.173}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "supervertaler"
-version = "1.9.172"
+version = "1.9.173"
 description = "Professional AI-enhanced translation workbench with multi-LLM support, glossary system, TM, spellcheck, voice commands, and PyQt6 interface. Batteries included (core)."
 readme = "README.md"
 requires-python = ">=3.10"

{supervertaler-1.9.172 → supervertaler-1.9.173}/supervertaler.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: supervertaler
-Version: 1.9.172
+Version: 1.9.173
 Summary: Professional AI-enhanced translation workbench with multi-LLM support, glossary system, TM, spellcheck, voice commands, and PyQt6 interface. Batteries included (core).
 Home-page: https://supervertaler.com
 Author: Michael Beijer
@@ -71,7 +71,7 @@ Dynamic: home-page
 Dynamic: license-file
 Dynamic: requires-python
 
-# 🚀 Supervertaler v1.9.172
+# 🚀 Supervertaler v1.9.173
 
 [![PyPI version](https://badge.fury.io/py/supervertaler.svg)](https://pypi.org/project/Supervertaler/)
 [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
@@ -80,7 +80,13 @@ Dynamic: requires-python
 AI-enhanced CAT tool with multi-LLM support (GPT-4, Claude, Gemini, Ollama), innovative Superlookup concordance system offering access to multiple terminology sources (TMs, glossaries, web resources, etc.), and seamless CAT tool integration (memoQ, Trados, CafeTran, Phrase).
 
 
-**Current Version:** v1.9.172 (January 28, 2026)
+**Current Version:** v1.9.173 (January 28, 2026)
+
+### IMPROVED in v1.9.173 - 🎯 Smarter TM Pre-Translation
+
+- **Smarter TM Exact Matching**: Exact matches now use text normalization, so matches are found even with whitespace/Unicode differences.
+- **Improved Pre-Translation Dialog**: Shows current segment, match count, elapsed time, and patience message for large jobs.
+- **Higher Fuzzy Threshold**: Raised minimum fuzzy match threshold from 70% to 75% for better quality.
 
 ### FIXED in v1.9.172 - 🐛 Fresh Projects Start Clean