text-rewrite 0.1.0__tar.gz

text_rewrite-0.1.0/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: text-rewrite
+Version: 0.1.0
+Summary: A high-performance text rewriting and fuzzy phonetic matching engine
+Author-email: Magic Jack <majic31@163.com>
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: flashtext>=2.7
+Requires-Dist: numba>=0.56.0
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: pypinyin>=0.49.0
+Requires-Dist: jieba>=0.42.1
+Provides-Extra: test
+Requires-Dist: pytest>=7.0.0; extra == "test"
+
+# text-rewrite 🚀
+
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**text-rewrite** 是一个专为工业级 ASR（语音识别）后处理设计的**高性能文本纠错与过滤引擎**。它将极致的计算效率与高精度的语义约束相结合，完美解决了传统 ASR 热词匹配中常见的“跨词误杀”、“谐音错认”以及“高并发性能瓶颈”问题。
+
+## ✨ 核心特性
+
+- **极致性能 ($O(N)$ 线性扩展)**：针对万级甚至十万级热词进行专项优化。处理近 2000 字的长文本只需不足 600 毫秒（常规短句 <10ms），绝不会拖垮流式并发服务器。
+- **Entity-Aware 实体约束引擎**：针对容易误杀的极短词（如人名“叶开” vs “也开心”），创新性引入基于 Jieba 词性标注的轻量级 NER 引擎。**内建极速音素预检机制与无 HMM 模式**，将千字长文本的 NER 解析时间从 400ms 暴砍至 5ms。
+- **双音节倒排索引 (Bigram Syllable Index)**：创新性地构建了基于“首位双音节”自适应哈希倒排池。将 10000 个热词在 2000 字长文本上的 DP 候选空间极致压缩了 98.8%，打破长文本下 $K=100\%$ 的魔咒。
+- **Numba Batch DP 加速**：底层基于 Numba JIT 编译的“批量化动态规划（DP）”核心引擎，彻底消除跨语言调度开销，支持纯发音级别的纠错（完美包容平翘舌、前后鼻音、形近音误差）。
+- **微秒级 FlashText 精确匹配**：对于全局安全大词表，底层自动退化为 Aho-Corasick 自动机，做到微秒级无感替换。
+- **Pipeline 乐高式组装**：提供高度可扩展的链式过滤器架构，正则清洗、精准替换、模糊纠错一气呵成。
+
+## 📦 安装依赖
+
+该项目核心依赖于高效计算与轻量级 NLP 组件：
+
+```bash
+pip install -r requirements.txt
+```
+> **要求**: `numba>=0.56.0`, `numpy>=1.21.0`, `pypinyin>=0.49.0`, `jieba>=0.42.1`
+
+## 🧩 核心过滤器 (Filters) 概览与最佳实践
+
+在真实的工业级落地场景中，最常用的“黄金三剑客”是以下三个过滤器的组合：
+
+1. **`RegexFilter` (打头阵：规则清洗)**
+   - **作用**：干脏活累活。负责前置格式规整。
+   - **场景**：将全角符号转半角、去除多余空格、清理语气词（“呃”、“啊”、“那个”），以及执行如 `四S -> 4S` 这种高度规律性的文本格式化。
+
+2. **`HotwordFilter` (中坚力量：业务绝对权威)**
+   - **作用**：基于 FlashText 的极速精确替换，零误杀，快、准、狠。
+   - **场景**：用于承载几万到几十万量级的“黑白名单 / 品牌库 / 敏感词库”（如“蔚来”、“极氪”）。用它拦截掉绝大部分必须 100% 准确的词，避免增加下游模糊匹配的开销和误杀率。
+
+3. **`EntityAwareFuzzyFilter` (最后兜底：长尾智能纠错)**
+   - **作用**：收拾残局。
+   - **场景**：经过前面精确词表的拦截，剩下的“长尾错别字”（如用户口音导致的“魏来”、“及克”）由它出马，通过发音的编辑距离计算把错别字捞回来。
+   - **语法**：`[标签]原词|替换词|阈值`
+     - 全局模糊拦截：`欧阳锋|欧阳锋|0.6` (长词，放心全局模糊替换)
+     - 短实体约束：`[nr]叶开|叶开|0.8` (`nr`为人名，只有 Jieba 认为是人名时才启动音素级检索)
+
+> **注**：除上述三者外，内部模块如 `FuzzyPhonemeFilter`（底层的 Numba DP 发音匹配引擎）和 `JiebaNERFilter`（底层 NER 引擎）主要作为组件被 `EntityAwareFuzzyFilter` 自动编排调用，在常规业务中通常无需直接操作。
+
+## 🚀 快速开始 (Quick Start)
+
+下面是一个将“正则清洗 -> 精确大词表拦截 -> 高危实体模糊纠错”串联起来的完整 Demo：
+
+```python
+from text_rewrite.pipeline import Pipeline
+from text_rewrite.filters.regex import RegexFilter
+from text_rewrite.filters.hotword import HotwordFilter
+from text_rewrite.filters.entity_fuzzy import EntityAwareFuzzyFilter
+
+# 1. 配置正则过滤器（清洗语气词）
+regex_rules = {
+    r"\b(嗯|啊|哦|那个)\b": ""
+}
+regex_filter = RegexFilter(rules=regex_rules)
+
+# 2. 配置精确热词过滤器（处理 10万级 安全大词表）
+exact_filter = HotwordFilter(hotwords={"确定性长词": "替换词"})
+
+# 3. 配置实体感知模糊过滤器 (EntityAwareFuzzyFilter)
+# 语法: [标签]目标词|阈值 (省略替换词会自动以目标词作为原词和替换词)
+fuzzy_rules = [
+    "张三|0.7",             # 无标签：全局极简配置（发音类似于张三的词，如展伞，都会被纠正为张三）
+    "[nr]叶开|0.8",         # 有标签：严格约束（必须是人名，且发音相似度>=0.8 才纠正）
+    "李四|0.7"              # 全局极简配置（里死 -> 李四）
+]
+entity_filter = EntityAwareFuzzyFilter(rules=fuzzy_rules)
+
+# 4. 组装 Pipeline 引擎（顺序即执行顺序）
+pipeline = Pipeline()
+pipeline.add_filter(regex_filter)
+pipeline.add_filter(exact_filter)
+pipeline.add_filter(entity_filter)
+
+# 5. 真实流式调用
+text = "那个，昨天通知了一下展伞和里死，但是他也开心了，最后通知了也开。"
+result = pipeline.process(text)
+
+print(result) 
+# 输出: "，昨天通知了一下张三和李四，但是他也开心了，最后通知了也开。"
+```
+
+> **注意**：由于初始化 Numba 引擎以及 JIT 预热需要少量时间，建议在服务启动时**单例初始化** `Pipeline` 实例，在后续流式请求中复用该实例调用 `.process(text)`。
+
+## 📊 性能压测 (Benchmark)
+
+您可以运行自带的压测脚本，体验在注入 **10000** 条工业级比例热词配置下的强悍性能：
+```bash
+python tests/bench_entity_fuzzy.py
+```
+*   **短句 (约 30 字)**: 端到端平均耗时 **~ 1 ms**
+*   **长文 (约 2000 字)**: 端到端平均耗时 **~ 150 ms** (呈现完美的 $O(N)$ 线性扩展，不受十万级词表拖累)
+
+---
+*Built with ❤️ for High-Performance NLP Engineering.*

text_rewrite-0.1.0/README.md

@@ -0,0 +1,101 @@
+# text-rewrite 🚀
+
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+**text-rewrite** 是一个专为工业级 ASR（语音识别）后处理设计的**高性能文本纠错与过滤引擎**。它将极致的计算效率与高精度的语义约束相结合，完美解决了传统 ASR 热词匹配中常见的“跨词误杀”、“谐音错认”以及“高并发性能瓶颈”问题。
+
+## ✨ 核心特性
+
+- **极致性能 ($O(N)$ 线性扩展)**：针对万级甚至十万级热词进行专项优化。处理近 2000 字的长文本只需不足 600 毫秒（常规短句 <10ms），绝不会拖垮流式并发服务器。
+- **Entity-Aware 实体约束引擎**：针对容易误杀的极短词（如人名“叶开” vs “也开心”），创新性引入基于 Jieba 词性标注的轻量级 NER 引擎。**内建极速音素预检机制与无 HMM 模式**，将千字长文本的 NER 解析时间从 400ms 暴砍至 5ms。
+- **双音节倒排索引 (Bigram Syllable Index)**：创新性地构建了基于“首位双音节”自适应哈希倒排池。将 10000 个热词在 2000 字长文本上的 DP 候选空间极致压缩了 98.8%，打破长文本下 $K=100\%$ 的魔咒。
+- **Numba Batch DP 加速**：底层基于 Numba JIT 编译的“批量化动态规划（DP）”核心引擎，彻底消除跨语言调度开销，支持纯发音级别的纠错（完美包容平翘舌、前后鼻音、形近音误差）。
+- **微秒级 FlashText 精确匹配**：对于全局安全大词表，底层自动退化为 Aho-Corasick 自动机，做到微秒级无感替换。
+- **Pipeline 乐高式组装**：提供高度可扩展的链式过滤器架构，正则清洗、精准替换、模糊纠错一气呵成。
+
+## 📦 安装依赖
+
+该项目核心依赖于高效计算与轻量级 NLP 组件：
+
+```bash
+pip install -r requirements.txt
+```
+> **要求**: `numba>=0.56.0`, `numpy>=1.21.0`, `pypinyin>=0.49.0`, `jieba>=0.42.1`
+
+## 🧩 核心过滤器 (Filters) 概览与最佳实践
+
+在真实的工业级落地场景中，最常用的“黄金三剑客”是以下三个过滤器的组合：
+
+1. **`RegexFilter` (打头阵：规则清洗)**
+   - **作用**：干脏活累活。负责前置格式规整。
+   - **场景**：将全角符号转半角、去除多余空格、清理语气词（“呃”、“啊”、“那个”），以及执行如 `四S -> 4S` 这种高度规律性的文本格式化。
+
+2. **`HotwordFilter` (中坚力量：业务绝对权威)**
+   - **作用**：基于 FlashText 的极速精确替换，零误杀，快、准、狠。
+   - **场景**：用于承载几万到几十万量级的“黑白名单 / 品牌库 / 敏感词库”（如“蔚来”、“极氪”）。用它拦截掉绝大部分必须 100% 准确的词，避免增加下游模糊匹配的开销和误杀率。
+
+3. **`EntityAwareFuzzyFilter` (最后兜底：长尾智能纠错)**
+   - **作用**：收拾残局。
+   - **场景**：经过前面精确词表的拦截，剩下的“长尾错别字”（如用户口音导致的“魏来”、“及克”）由它出马，通过发音的编辑距离计算把错别字捞回来。
+   - **语法**：`[标签]原词|替换词|阈值`
+     - 全局模糊拦截：`欧阳锋|欧阳锋|0.6` (长词，放心全局模糊替换)
+     - 短实体约束：`[nr]叶开|叶开|0.8` (`nr`为人名，只有 Jieba 认为是人名时才启动音素级检索)
+
+> **注**：除上述三者外，内部模块如 `FuzzyPhonemeFilter`（底层的 Numba DP 发音匹配引擎）和 `JiebaNERFilter`（底层 NER 引擎）主要作为组件被 `EntityAwareFuzzyFilter` 自动编排调用，在常规业务中通常无需直接操作。
+
+## 🚀 快速开始 (Quick Start)
+
+下面是一个将“正则清洗 -> 精确大词表拦截 -> 高危实体模糊纠错”串联起来的完整 Demo：
+
+```python
+from text_rewrite.pipeline import Pipeline
+from text_rewrite.filters.regex import RegexFilter
+from text_rewrite.filters.hotword import HotwordFilter
+from text_rewrite.filters.entity_fuzzy import EntityAwareFuzzyFilter
+
+# 1. 配置正则过滤器（清洗语气词）
+regex_rules = {
+    r"\b(嗯|啊|哦|那个)\b": ""
+}
+regex_filter = RegexFilter(rules=regex_rules)
+
+# 2. 配置精确热词过滤器（处理 10万级 安全大词表）
+exact_filter = HotwordFilter(hotwords={"确定性长词": "替换词"})
+
+# 3. 配置实体感知模糊过滤器 (EntityAwareFuzzyFilter)
+# 语法: [标签]目标词|阈值 (省略替换词会自动以目标词作为原词和替换词)
+fuzzy_rules = [
+    "张三|0.7",             # 无标签：全局极简配置（发音类似于张三的词，如展伞，都会被纠正为张三）
+    "[nr]叶开|0.8",         # 有标签：严格约束（必须是人名，且发音相似度>=0.8 才纠正）
+    "李四|0.7"              # 全局极简配置（里死 -> 李四）
+]
+entity_filter = EntityAwareFuzzyFilter(rules=fuzzy_rules)
+
+# 4. 组装 Pipeline 引擎（顺序即执行顺序）
+pipeline = Pipeline()
+pipeline.add_filter(regex_filter)
+pipeline.add_filter(exact_filter)
+pipeline.add_filter(entity_filter)
+
+# 5. 真实流式调用
+text = "那个，昨天通知了一下展伞和里死，但是他也开心了，最后通知了也开。"
+result = pipeline.process(text)
+
+print(result) 
+# 输出: "，昨天通知了一下张三和李四，但是他也开心了，最后通知了也开。"
+```
+
+> **注意**：由于初始化 Numba 引擎以及 JIT 预热需要少量时间，建议在服务启动时**单例初始化** `Pipeline` 实例，在后续流式请求中复用该实例调用 `.process(text)`。
+
+## 📊 性能压测 (Benchmark)
+
+您可以运行自带的压测脚本，体验在注入 **10000** 条工业级比例热词配置下的强悍性能：
+```bash
+python tests/bench_entity_fuzzy.py
+```
+*   **短句 (约 30 字)**: 端到端平均耗时 **~ 1 ms**
+*   **长文 (约 2000 字)**: 端到端平均耗时 **~ 150 ms** (呈现完美的 $O(N)$ 线性扩展，不受十万级词表拖累)
+
+---
+*Built with ❤️ for High-Performance NLP Engineering.*

text_rewrite-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "text-rewrite"
+version = "0.1.0"
+description = "A high-performance text rewriting and fuzzy phonetic matching engine"
+readme = "README.md"
+requires-python = ">=3.8"
+authors = [
+    { name = "Magic Jack", email = "majic31@163.com" }
+]
+dependencies = [
+    "flashtext>=2.7",
+    "numba>=0.56.0",
+    "numpy>=1.21.0",
+    "pypinyin>=0.49.0",
+    "jieba>=0.42.1"
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest>=7.0.0"
+]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["text_rewrite*"]

text_rewrite-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

text_rewrite-0.1.0/text_rewrite/__init__.py

@@ -0,0 +1,3 @@
+from .pipeline import Pipeline
+
+__all__ = ["Pipeline"]

text_rewrite-0.1.0/text_rewrite/filters/__init__.py

@@ -0,0 +1,9 @@
+from .base import BaseFilter
+from .hotword import HotwordFilter
+from .regex import RegexFilter
+from .ner import NERFilter
+from .jieba_ner import JiebaNERFilter
+from .fuzzy_phoneme import FuzzyPhonemeFilter
+from .entity_fuzzy import EntityAwareFuzzyFilter
+
+__all__ = ["BaseFilter", "HotwordFilter", "RegexFilter", "NERFilter", "JiebaNERFilter", "FuzzyPhonemeFilter", "EntityAwareFuzzyFilter"]

text_rewrite-0.1.0/text_rewrite/filters/base.py

@@ -0,0 +1,15 @@
+from abc import ABC, abstractmethod
+
+class BaseFilter(ABC):
+    """
+    Abstract base class for all text filters.
+    """
+    def __init__(self, name: str = None):
+        self.name = name or self.__class__.__name__
+
+    @abstractmethod
+    def process(self, text: str) -> str:
+        """
+        Process the input text and return the filtered text.
+        """
+        pass

text_rewrite-0.1.0/text_rewrite/filters/entity_fuzzy.py

@@ -0,0 +1,187 @@
+import logging
+import re
+from typing import List
+from collections import defaultdict
+from .base import BaseFilter
+from .fuzzy_phoneme import FuzzyPhonemeFilter
+logger = logging.getLogger(__name__)
+
+class EntityAwareFuzzyFilter(BaseFilter):
+    """
+    SOTA Entity-Aware Fuzzy Phoneme Filter.
+    Combines high-speed global phoneme matching with Jieba-based NER tagging
+    to correct ASR homophone errors accurately and efficiently.
+    """
+    def __init__(self, name: str = "EntityAwareFuzzyFilter", rules: List[str] = None):
+        """
+        :param rules: A list of rule strings. 
+                      Format: `[tag]word1:threshold1|word2:replacement2:threshold2`
+                      tag, replacement, and threshold are optional.
+                      Example: "[nr]叶开:0.8|张三:0.7", "李四:0.7", "欧阳锋"
+        """
+        super().__init__(name=name)
+        self.global_hotwords = {}
+        self.global_thresholds = {}
+        self.tagged_hotwords = defaultdict(dict)
+        self.tagged_thresholds = defaultdict(dict)
+        
+        self.global_filter = None
+        self.tagged_filters = {}
+        self.ner_filter = None
+        
+        if rules:
+            self.add_rules(rules)
+            
+    def add_rules(self, rules: List[str]):
+        tag_pattern = re.compile(r'^\[([^\]]+)\]')
+        for rule in rules:
+            self._parse_rule(rule, tag_pattern)
+                
+        self._rebuild_filters()
+        self._setup_ner_filter()
+
+    def _parse_rule(self, rule: str, tag_pattern: re.Pattern):
+        rule = rule.strip()
+        if not rule:
+            return
+            
+        tag_match = tag_pattern.match(rule)
+        tag = tag_match.group(1) if tag_match else None
+        
+        if tag_match:
+            rule = rule[tag_match.end():]
+            
+        parts = [p.strip() for p in rule.split('|')]
+        for part in parts:
+            if not part:
+                continue
+                
+            subparts = [sp.strip() for sp in part.split(':')]
+            word = subparts[0]
+            replacement = word
+            threshold = 0.6
+            
+            if len(subparts) == 2:
+                try:
+                    threshold = float(subparts[1])
+                except ValueError:
+                    replacement = subparts[1]
+            elif len(subparts) >= 3:
+                replacement = subparts[1]
+                try:
+                    threshold = float(subparts[2])
+                except ValueError:
+                    pass
+            
+            if tag:
+                self.tagged_hotwords[tag][word] = replacement
+                self.tagged_thresholds[tag][word] = threshold
+            else:
+                self.global_hotwords[word] = replacement
+                self.global_thresholds[word] = threshold
+
+    def _rebuild_filters(self):
+        # Rebuild global filter
+        if self.global_hotwords:
+            self.global_filter = FuzzyPhonemeFilter(hotwords=self.global_hotwords, threshold=0.6)
+            self.global_filter.custom_thresholds = self.global_thresholds
+            
+        # Rebuild tagged filters
+        for tag, hw_dict in self.tagged_hotwords.items():
+            f = FuzzyPhonemeFilter(hotwords=hw_dict, threshold=0.6)
+            f.custom_thresholds = self.tagged_thresholds[tag]
+            self.tagged_filters[tag] = f
+
+    def _setup_ner_filter(self):
+        # Init JiebaNERFilter if tagged rules exist
+        if not self.tagged_filters:
+            return
+
+        import functools
+        
+        # Cache the processing of individual words to avoid redundant phoneme DP searches
+        # Since tagged filters don't change state during processing, caching per tag is safe.
+        word_cache = {}
+        for tag in self.tagged_filters:
+            word_cache[tag] = functools.lru_cache(maxsize=4096)(self.tagged_filters[tag].process)
+            
+        def jieba_callback(terms, text):
+            new_text = text
+            
+            # Keep track of character offsets manually since jieba
+            # splits continuously.
+            offset = 0
+            entities = []
+            for term in terms:
+                word = term.word
+                tag_str = term.flag
+                
+                if tag_str in self.tagged_filters:
+                    entities.append({
+                        'word': word,
+                        'tag': tag_str,
+                        'start': offset,
+                        'end': offset + len(word)
+                    })
+                offset += len(word)
+            
+            # Process from back to front to avoid index shifting after string replacement
+            for ent in reversed(entities):
+                word = ent['word']
+                tag = ent['tag']
+                start = ent['start']
+                end = ent['end']
+                
+                # Fuzzy phoneme search on the isolated entity word (Cached)
+                corrected = word_cache[tag](word)
+                if corrected != word:
+                    new_text = new_text[:start] + corrected + new_text[end:]
+                    
+            return new_text
+
+        from .jieba_ner import JiebaNERFilter
+        self.ner_filter = JiebaNERFilter(replace_callback=jieba_callback)
+        self.ner_filter._load_model() # Pre-load it
+        
+        # Build prechecker
+        all_tagged_hw = {}
+        all_tagged_thresholds = {}
+        for tag, hw_dict in self.tagged_hotwords.items():
+            all_tagged_hw.update(hw_dict)
+            all_tagged_thresholds.update(self.tagged_thresholds[tag])
+            
+        if all_tagged_hw:
+            self.ner_prechecker = FuzzyPhonemeFilter(hotwords=all_tagged_hw, threshold=0.6)
+            self.ner_prechecker.custom_thresholds = all_tagged_thresholds
+        else:
+            self.ner_prechecker = None
+
+    def process(self, text: str) -> str:
+        """
+        Execute SOTA cascading replacement.
+        1. Global Numba match (fastest, for long words or safe thresholds).
+        2. Jieba NER constraint match (for tags like 'nr').
+        """
+        if not text:
+            return text
+
+            
+        # Step 1: Global substitution
+        if self.global_filter:
+            text = self.global_filter.process(text)
+            
+        # Step 2: Tag-constrained substitution
+        if self.ner_filter:
+            # 预检：如果不可能有带标签的热词出现，直接跳过 jieba 分词
+            skip_ner = False
+            if hasattr(self, 'ner_prechecker') and self.ner_prechecker:
+                from text_rewrite.utils.algo_phoneme import get_phoneme_info
+                inp = get_phoneme_info(text)
+                cands = self.ner_prechecker.rag.index.get_candidates(inp)
+                if not cands:
+                    skip_ner = True
+            
+            if not skip_ner:
+                text = self.ner_filter.process(text)
+            
+        return text

text_rewrite-0.1.0/text_rewrite/filters/fuzzy_phoneme.py

@@ -0,0 +1,130 @@
+import logging
+from typing import Dict
+from .base import BaseFilter
+from ..utils.algo_phoneme import get_phoneme_info, get_phoneme_seq
+from ..utils.rag_fast import FastRAG
+
+logger = logging.getLogger(__name__)
+
+class FuzzyPhonemeFilter(BaseFilter):
+    """
+    Fuzzy Phoneme Filter using Numba JIT DP acceleration.
+    Handles exact and fuzzy phoneme matches to correct ASR homophone errors.
+    """
+    def __init__(self, name: str = "FuzzyPhonemeFilter", hotwords: Dict[str, str] = None, threshold: float = 0.6):
+        """
+        :param threshold: The similarity threshold (0.0 to 1.0) for a phoneme sequence to match.
+                          0.6 is a good default for allowing some ASR errors.
+        """
+        super().__init__(name=name)
+        self.rag = FastRAG(threshold=threshold)
+        self.hotword_replacements = {}
+        # Cache hw_info to avoid recomputing
+        self._hw_info_cache = {}
+        
+        # Optional mapping of original word -> specific threshold
+        self.custom_thresholds = {}
+        
+        if hotwords:
+            self.add_hotwords(hotwords)
+            
+    def add_hotwords(self, hotwords: Dict[str, str], custom_thresholds: Dict[str, float] = None):
+        """
+        Add hotwords dynamically.
+        :param hotwords: Dictionary mapping the phonetic target (e.g., "张三") to the replacement string (e.g., "张三").
+        :param custom_thresholds: Optional dictionary mapping target word to its specific threshold.
+        """
+        if custom_thresholds:
+            self.custom_thresholds.update(custom_thresholds)
+            
+        hw_dict = {}
+        for original, replacement in hotwords.items():
+            phonemes = get_phoneme_seq(original)
+            hw_dict[original] = phonemes
+            self.hotword_replacements[original] = replacement
+            self._hw_info_cache[original] = [p.info for p in phonemes]
+        
+        self.rag.add_hotwords(hw_dict)
+        self.rag.numba_searcher.build_cache(self._hw_info_cache)
+
+    def process(self, text: str) -> str:
+        """
+        Process text by converting to phonemes, searching for fuzzy matches, 
+        mapping indices back, and replacing.
+        """
+        if not text:
+            return text
+            
+        # 1. Get input phonemes with char_start/char_end
+        input_phonemes = get_phoneme_info(text)
+        if not input_phonemes:
+            return text
+            
+        # 2. Get candidates from inverted index to avoid full scan
+        candidates = self.rag.index.get_candidates(input_phonemes)
+        if not candidates:
+            return text
+            
+        input_info = [p.info for p in input_phonemes]
+        
+        # Find the absolute minimum threshold to query Numba, post-filter later
+        min_thresh = self.rag.threshold
+        if self.custom_thresholds:
+            min_thresh = min(min_thresh, min(self.custom_thresholds.values()))
+            
+        # 3. Perform fine-grained Numba Substring DP search
+        replacements = []
+        
+        # Pre-encode input_info ONCE to avoid O(N_candidates) encoding overhead
+        inp_codes, inp_langs, _, inp_is_ws, inp_is_we = self.rag.numba_searcher.encode_input_vecs(input_info)
+        
+        candidate_keys = [hw_key for hw_key, _ in candidates]
+        
+        batch_res = self.rag.numba_searcher.search_batch_with_encoded_input(
+            candidate_keys, inp_codes, inp_langs, inp_is_ws, inp_is_we, threshold=min_thresh
+        )
+        
+        for hw_key, res_list in batch_res.items():
+            for score, start_idx, end_idx in res_list:
+                # Apply word-specific threshold
+                if score >= self.custom_thresholds.get(hw_key, self.rag.threshold):
+                    replacements.append({
+                        'score': score,
+                        'start_idx': start_idx,
+                        'end_idx': end_idx,
+                        'replacement': self.hotword_replacements[hw_key]
+                    })
+        
+        if not replacements:
+            return text
+            
+        # 4. Resolve overlaps (greedy: highest score, then longest span)
+        replacements.sort(key=lambda x: (-x['score'], -(x['end_idx'] - x['start_idx'])))
+        
+        final_reps = []
+        used_indices = set()
+        for r in replacements:
+            r_set = set(range(r['start_idx'], r['end_idx']))
+            if not r_set.intersection(used_indices):
+                final_reps.append(r)
+                used_indices.update(r_set)
+                
+        # 5. Execute replacements from back to front to avoid index shifting issues
+        final_reps.sort(key=lambda x: x['start_idx'], reverse=True)
+        
+        result_text = text
+        for r in final_reps:
+            # Map phoneme index to char index
+            # end_idx is exclusive in phoneme array
+            start_p = r['start_idx']
+            end_p = r['end_idx'] - 1
+            
+            if start_p >= len(input_phonemes) or end_p >= len(input_phonemes) or start_p > end_p:
+                continue
+                
+            char_start = input_phonemes[start_p].char_start
+            char_end = input_phonemes[end_p].char_end
+            
+            result_text = result_text[:char_start] + r['replacement'] + result_text[char_end:]
+            
+        return result_text

text_rewrite-0.1.0/text_rewrite/filters/hotword.py

@@ -0,0 +1,50 @@
+from typing import Dict, List
+from .base import BaseFilter
+
+try:
+    from flashtext import KeywordProcessor
+except ImportError:
+    KeywordProcessor = None
+
+
+class HotwordFilter(BaseFilter):
+    """
+    Hotword Filter for high-performance exact word replacements.
+    Uses Aho-Corasick algorithm via flashtext library.
+    Ideal for massive dictionaries (100k+ words).
+    """
+    def __init__(self, name: str = "HotwordFilter", hotwords: Dict[str, str] = None, case_sensitive: bool = False):
+        super().__init__(name=name)
+        if KeywordProcessor is None:
+            raise ImportError("flashtext library is required for HotwordFilter. Run `pip install flashtext`.")
+            
+        self.keyword_processor = KeywordProcessor(case_sensitive=case_sensitive)
+        if hotwords:
+            self.add_hotwords(hotwords)
+
+    def add_hotwords(self, hotwords: Dict[str, str]):
+        """
+        Add hotword replacements dynamically.
+        :param hotwords: Dictionary mapping original words to target replacement words.
+                         Note: flashtext format is { 'replacement': ['word1', 'word2'] } for add_keywords_from_dict.
+                         Since we get { 'word': 'replacement' }, we iterate or transform it.
+        """
+        for original_word, replacement_word in hotwords.items():
+            self.keyword_processor.add_keyword(original_word, replacement_word)
+
+    def remove_hotwords(self, words: List[str]):
+        """
+        Remove hotwords from the trie dynamically.
+        """
+        for word in words:
+            self.keyword_processor.remove_keyword(word)
+
+    def process(self, text: str) -> str:
+        """
+        Process text by efficiently finding and replacing all matching hotwords.
+        """
+        if not text:
+            return text
+            
+        # flashtext operates in O(N) where N is text length, extremely fast.
+        return self.keyword_processor.replace_keywords(text)