sembr 0.2.2__tar.gz → 0.2.4__tar.gz

{sembr-0.2.2 → sembr-0.2.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sembr
-Version: 0.2.2
+Version: 0.2.4
 Summary: A semantic linebreaker powered by transformers
 Author: admk
 License-Expression: MIT
@@ -14,17 +14,21 @@ Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: accelerate
-Requires-Dist: transformers
-Requires-Dist: torch
+Requires-Dist: fastmcp
+Requires-Dist: flask
+Requires-Dist: magika
+Requires-Dist: mcp[cli]
 Requires-Dist: numpy
-Requires-Dist: tqdm
+Requires-Dist: pydantic
 Requires-Dist: requests
-Requires-Dist: flask
-Requires-Dist: mcp[cli]>=1.2.0
-Requires-Dist: fastmcp>=2.10.6
+Requires-Dist: torch
+Requires-Dist: tqdm
+Requires-Dist: transformers
+Requires-Dist: tree-sitter>=0.25.0
+Requires-Dist: tree-sitter-markdown>=0.3.2
 Dynamic: license-file
 
-# Semantic Line Breaker (SemBr)
+# ⚡️ Semantic Line Breaker (SemBr)
 
 [![GitHub](https://img.shields.io/github/license/admk/sembr)](LICENSE)
 [![python](https://img.shields.io/badge/Python-3.10-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org)
@@ -45,6 +49,9 @@ SemBr is a command-line tool
 powered by [Transformer][transformers1] [models][transformers2]
 that performs [semantic linebreaks](#what-are-semantic-line-breaks)
 to breaks lines in a text file at semantic boundaries.
+It supports multiple file types
+including LaTeX, Markdown, and plain text,
+with automatic file type detection.
 
 ### Installation
 
@@ -66,6 +73,34 @@ sembr  # run
 uvx sembr  # install and run directly
 ```
 
+#### From GitHub (Latest Development Version)
+
+To install the latest development version directly from GitHub:
+
+```shell
+# Install from GitHub main branch
+uv tool install git+https://github.com/admk/sembr.git
+
+# Run directly without installing
+uvx --from git+https://github.com/admk/sembr.git sembr
+```
+
+Alternatively, clone and install in development mode:
+
+```shell
+# Clone the repository
+git clone https://github.com/admk/sembr.git
+cd sembr
+
+# Install in development mode
+pip install -e .
+
+# Or with uv
+uv pip install -e .
+```
+
+Note that the development version may include experimental features and could be less stable than the PyPI release.
+
 ### Supported Platforms
 
 SemBr is supported on Linux, Mac and Windows.
@@ -155,6 +190,9 @@ to customize the behavior of SemBr:
 * `--dtype <dtype>`:
   Data type for model weights (e.g. `float16`, `bfloat16`).
   Default is `float32`.
+* `--file-type <type>`:
+  File type (`plaintext`, `latex`, `markdown`, etc.).
+  Auto-detected using [Magika][magika] if not provided.
 * `--mcp`:
   Start MCP server mode instead of processing text.
 
@@ -357,7 +395,7 @@ to save best models.
   - Natural language support:
     - [ ] Support natural languages other than English.
   - Typesetting languages support:
-    - [ ] Markdown.
+    - [x] ~~Markdown.~~
     - [ ] Typst.
   - Usability:
     - [ ] Inference queue.
@@ -405,6 +443,7 @@ Semantic line breaking:
 [pypi]: https://pypi.org/project/sembr
 [uv]: https://github.com/astral-sh/uv
 [mcp]: https://modelcontextprotocol.io/overview
+[magika]: https://github.com/google/magika
 
 [sembr]: https://sembr.org
 [semlf]: https://rhodesmill.org/brandon/2012/one-sentence-per-line

sembr-0.2.2/sembr.egg-info/PKG-INFO → sembr-0.2.4/README.md

@@ -1,30 +1,4 @@
-Metadata-Version: 2.4
-Name: sembr
-Version: 0.2.2
-Summary: A semantic linebreaker powered by transformers
-Author: admk
-License-Expression: MIT
-Project-URL: Homepage, https://github.com/admk/sembr
-Project-URL: Issues, https://github.com/admk/sembr/issues
-Classifier: Programming Language :: Python :: 3
-Classifier: Operating System :: OS Independent
-Classifier: Topic :: Utilities
-Classifier: Environment :: Console
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: accelerate
-Requires-Dist: transformers
-Requires-Dist: torch
-Requires-Dist: numpy
-Requires-Dist: tqdm
-Requires-Dist: requests
-Requires-Dist: flask
-Requires-Dist: mcp[cli]>=1.2.0
-Requires-Dist: fastmcp>=2.10.6
-Dynamic: license-file
-
-# Semantic Line Breaker (SemBr)
+# ⚡️ Semantic Line Breaker (SemBr)
 
 [![GitHub](https://img.shields.io/github/license/admk/sembr)](LICENSE)
 [![python](https://img.shields.io/badge/Python-3.10-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org)
@@ -45,6 +19,9 @@ SemBr is a command-line tool
 powered by [Transformer][transformers1] [models][transformers2]
 that performs [semantic linebreaks](#what-are-semantic-line-breaks)
 to breaks lines in a text file at semantic boundaries.
+It supports multiple file types
+including LaTeX, Markdown, and plain text,
+with automatic file type detection.
 
 ### Installation
 
@@ -66,6 +43,34 @@ sembr  # run
 uvx sembr  # install and run directly
 ```
 
+#### From GitHub (Latest Development Version)
+
+To install the latest development version directly from GitHub:
+
+```shell
+# Install from GitHub main branch
+uv tool install git+https://github.com/admk/sembr.git
+
+# Run directly without installing
+uvx --from git+https://github.com/admk/sembr.git sembr
+```
+
+Alternatively, clone and install in development mode:
+
+```shell
+# Clone the repository
+git clone https://github.com/admk/sembr.git
+cd sembr
+
+# Install in development mode
+pip install -e .
+
+# Or with uv
+uv pip install -e .
+```
+
+Note that the development version may include experimental features and could be less stable than the PyPI release.
+
 ### Supported Platforms
 
 SemBr is supported on Linux, Mac and Windows.
@@ -155,6 +160,9 @@ to customize the behavior of SemBr:
 * `--dtype <dtype>`:
   Data type for model weights (e.g. `float16`, `bfloat16`).
   Default is `float32`.
+* `--file-type <type>`:
+  File type (`plaintext`, `latex`, `markdown`, etc.).
+  Auto-detected using [Magika][magika] if not provided.
 * `--mcp`:
   Start MCP server mode instead of processing text.
 
@@ -357,7 +365,7 @@ to save best models.
   - Natural language support:
     - [ ] Support natural languages other than English.
   - Typesetting languages support:
-    - [ ] Markdown.
+    - [x] ~~Markdown.~~
     - [ ] Typst.
   - Usability:
     - [ ] Inference queue.
@@ -405,6 +413,7 @@ Semantic line breaking:
 [pypi]: https://pypi.org/project/sembr
 [uv]: https://github.com/astral-sh/uv
 [mcp]: https://modelcontextprotocol.io/overview
+[magika]: https://github.com/google/magika
 
 [sembr]: https://sembr.org
 [semlf]: https://rhodesmill.org/brandon/2012/one-sentence-per-line

{sembr-0.2.2 → sembr-0.2.4}/pyproject.toml

@@ -15,14 +15,18 @@ license-files = ["LICEN[CS]E*"]
 requires-python = ">=3.10"
 dependencies = [
     "accelerate",
-    "transformers",
-    "torch",
+    "fastmcp",
+    "flask",
+    "magika",
+    "mcp[cli]",
     "numpy",
-    "tqdm",
+    "pydantic",
     "requests",
-    "flask",
-    "mcp[cli]>=1.2.0",
-    "fastmcp>=2.10.6",
+    "torch",
+    "tqdm",
+    "transformers",
+    "tree-sitter>=0.25.0",
+    "tree-sitter-markdown>=0.3.2",
 ]
 classifiers = [
     "Programming Language :: Python :: 3",
@@ -39,7 +43,7 @@ Homepage = "https://github.com/admk/sembr"
 Issues = "https://github.com/admk/sembr/issues"
 
 [tool.setuptools.packages.find]
-include = ["sembr"]
+include = ["sembr", "sembr.*"]
 exclude = ["data*", "tests*"]
 
 [tool.setuptools.dynamic]

{sembr-0.2.2 → sembr-0.2.4}/sembr/__init__.py

@@ -1,5 +1,5 @@
 __toolname__ = __name__
-__version__ = "0.2.2"
+__version__ = "0.2.4"
 __author__ = "admk"
 __license__ = "MIT"
 __url__ = f"https://github.com/admk/{__name__}"

{sembr-0.2.2 → sembr-0.2.4}/sembr/cli.py

@@ -18,7 +18,9 @@ def cli_parser():
     p = argparse.ArgumentParser(
         description='SemBr: Rewrap text with semantic breaks.')
     model_name = 'admko/sembr2023-bert-small'
-    p.add_argument('-v', '--version', action='version', version=__version__)
+    p.add_argument('-V', '--version', action='version', version=__version__)
+    p.add_argument(
+        '-v', '--verbose', action='store_true', help='Enable verbose output')
     p.add_argument('-m', '--model-name', type=str, default=model_name)
     p.add_argument('-i', '--input-file', type=str, default=None)
     p.add_argument('-o', '--output-file', type=str, default=None)
@@ -35,15 +37,25 @@ def cli_parser():
     p.add_argument('--dtype', type=str, default=None)
     p.add_argument('--debug', action='store_true')
     p.add_argument('--mcp', action='store_true', help='Start MCP server mode')
+    p.add_argument(
+        '--file-type', type=str, default=None,
+        help=(
+            'File type (plaintext, latex, markdown, etc.). '
+            'Auto-detect if not provided. '
+            'File type must be provided if using stdin.'))
     return p
 
 
-def init(model_name, bits=None, dtype=None):
+def init(model_name, bits=None, dtype=None, file_type=None, file_path=None, text=None, verbose=False):
     import torch
-    from transformers import (
-        AutoTokenizer, AutoModelForTokenClassification)
-    from .process import SemBrProcessor
-    tokenizer = AutoTokenizer.from_pretrained(model_name)
+    from transformers import AutoTokenizer, AutoModelForTokenClassification
+    from .processors import get_processor
+
+    try:
+        tokenizer = AutoTokenizer.from_pretrained(model_name)
+    except Exception:
+        tokenizer = AutoTokenizer.from_pretrained(model_name, local_files_only=True)
+
     dtype = getattr(torch, dtype) if dtype is not None else torch.float32
     kwargs = {}
     if torch.cuda.is_available():
@@ -59,20 +71,29 @@ def init(model_name, bits=None, dtype=None):
         if bits in [4, 8]:
             raise RuntimeError('MPS does not support quantization.')
         kwargs['device_map'] = 'mps'
-    model = AutoModelForTokenClassification.from_pretrained(
-        model_name, torch_dtype=dtype, **kwargs)
+
+    try:
+        model = AutoModelForTokenClassification.from_pretrained(
+            model_name, torch_dtype=dtype, **kwargs)
+    except Exception:
+        model = AutoModelForTokenClassification.from_pretrained(
+            model_name, torch_dtype=dtype, local_files_only=True, **kwargs)
+
     model.eval()
-    processor = SemBrProcessor()
+    processor = get_processor(
+        file_type=file_type, file_path=file_path, text=text, verbose=verbose)
     return tokenizer, model, processor
 
 
-def start_server(port, tokenizer, model, processor, wrap_kwargs=None):
+def start_server(
+    port, tokenizer, model, default_file_type=None, wrap_kwargs=None
+):
     from flask import Flask, request
+    from .processors import get_processor
     app = Flask(__name__)
     base_rv = {
         'model': model.__class__.__name__,
         'tokenizer': tokenizer.__class__.__name__,
-        'processor': processor.__class__.__name__,
     }
 
     @app.route('/check')
@@ -88,8 +109,17 @@ def start_server(port, tokenizer, model, processor, wrap_kwargs=None):
         form = request.form
         text = form['text']
         kwargs = dict(wrap_kwargs or {})
+
+        # Get file_type from form data or use default
+        file_type = form.get('file_type', default_file_type)
+
+        # Create processor dynamically based on file type or text content
+        processor = get_processor(
+            file_type=file_type, text=text if not file_type else None)
+
+        # Process other form parameters
         for k, v in form.items():
-            if k == 'text':
+            if k in ['text', 'file_type']:
                 continue
             if k in ['batch_size', 'tokens_per_line', 'overlap_divisor']:
                 v = int(v)
@@ -99,6 +129,8 @@ def start_server(port, tokenizer, model, processor, wrap_kwargs=None):
             return {
                 'status': 'success',
                 **base_rv,
+                'processor': processor.__class__.__name__,
+                'file_type': file_type,
                 **kwargs,
                 'text': results,
             }
@@ -106,6 +138,8 @@ def start_server(port, tokenizer, model, processor, wrap_kwargs=None):
             return {
                 'status': 'error',
                 **base_rv,
+                'processor': processor.__class__.__name__,
+                'file_type': file_type,
                 **kwargs,
                 'error': str(e),
                 'traceback': traceback.format_exc(),
@@ -180,9 +214,9 @@ def main() -> int:
         return 0
     kwargs = wrap_kwargs(args)
     if args.listen:
-        tokenizer, model, processor = init(
-            args.model_name, args.bits, args.dtype)
-        start_server(args.port, tokenizer, model, processor, kwargs)
+        tokenizer, model, _ = init(
+            args.model_name, args.bits, args.dtype, args.file_type, None, None, args.verbose)
+        start_server(args.port, tokenizer, model, args.file_type, kwargs)
         return 0
     if args.input_file is not None:
         with open(args.input_file, 'r', encoding='utf-8') as f:
@@ -198,7 +232,8 @@ def main() -> int:
     else:
         from .inference import sembr
         tokenizer, model, processor = init(
-            args.model_name, args.bits, args.dtype)
+            args.model_name, args.bits, args.dtype,
+            args.file_type, args.input_file, text, args.verbose)
         result = sembr(text, tokenizer, model, processor, **kwargs)
     if args.output_file is None:
         print(result)

{sembr-0.2.2 → sembr-0.2.4}/sembr/inference.py

@@ -108,7 +108,7 @@ def inference(
     if text.strip() == '':
         return []
     collator = DataCollatorForTokenClassification(tokenizer, padding='longest')
-    results = processor(text, split=isinstance(text, str))
+    results = processor.parse_text(text, split=isinstance(text, str))
     results = processor.tokenize_with_modes(tokenizer, results)
     logits, counts = _tiled_inference(
         model, collator, results, batch_size, overlap_divisor)

{sembr-0.2.2 → sembr-0.2.4}/sembr/mcp.py

@@ -9,16 +9,25 @@ from .cli import init, cli_parser, wrap_kwargs
 
 
 class SembrModel:
-    def __init__(self, tokenizer, model, processor, kwargs):
+    def __init__(self, tokenizer, model, default_file_type=None, kwargs=None):
         self.tokenizer = tokenizer
         self.model = model
-        self.processor = processor
-        self.kwargs = kwargs
+        self.default_file_type = default_file_type
+        self.kwargs = kwargs or {}
 
-    def process_text(self, text: str) -> str:
+    def process_text(self, text: str, file_type: Optional[str] = None) -> str:
         from .inference import sembr
+        from .processors import get_processor
+
+        # Use provided file_type, default, or auto-detect from text
+        effective_file_type = file_type or self.default_file_type
+        processor = get_processor(
+            file_type=effective_file_type,
+            text=text if not effective_file_type else None
+        )
+
         return sembr(
-            text, self.tokenizer, self.model, self.processor, **self.kwargs)
+            text, self.tokenizer, self.model, processor, **self.kwargs)
 
 
 _sembr_model: Optional[SembrModel] = None
@@ -31,9 +40,10 @@ def get_sembr_model() -> SembrModel:
         return _sembr_model
     parser = cli_parser()
     args, _ = parser.parse_known_args()
-    tokenizer, model, processor = init(args.model_name, args.bits, args.dtype)
+    tokenizer, model, _ = init(
+        args.model_name, args.bits, args.dtype, args.file_type)
     kwargs = wrap_kwargs(args)
-    _sembr_model = SembrModel(tokenizer, model, processor, kwargs)
+    _sembr_model = SembrModel(tokenizer, model, args.file_type, kwargs)
     return _sembr_model
 
 
@@ -46,18 +56,26 @@ mcp = FastMCP("SemBr")
 )
 def wrap_text(
     text: Annotated[str, Field(description="Text to wrap")],
+    file_type: Annotated[Optional[str], Field(
+        description=(
+            "File type (latex, markdown, plaintext, etc.). "
+            "Auto-detect if not provided."),
+        default=None
+    )] = None,
 ) -> ToolResult:
     try:
-        wrapped_text = get_sembr_model().process_text(text)
+        wrapped_text = get_sembr_model().process_text(text, file_type)
     except Exception as e:
         return ToolResult(
             content=[TextContent(type="text", text=f"Error processing text: {str(e)}")],
             structured_content={"success": False, "error": str(e)})
     num_lines = len(wrapped_text.splitlines())
     readable = f"Performed semantic line breaks to {num_lines} lines."
+    if file_type:
+        readable += f" File type: {file_type}."
     return ToolResult(
         content=[TextContent(type="text", text=readable)],
-        structured_content={"success": True, "output": wrapped_text})
+        structured_content={"success": True, "output": wrapped_text, "file_type": file_type})
 
 
 if __name__ == "__main__":

sembr-0.2.4/sembr/processors/__init__.py

@@ -0,0 +1,15 @@
+"""
+Grammar-based text processors for different file types.
+"""
+
+from .base import BaseProcessor
+from .latex import LaTeXProcessor
+from .markdown import MarkdownProcessor
+from .plaintext import PlainTextProcessor
+from .utils import get_processor, detect_file_type_from_text
+
+
+__all__ = [
+    'BaseProcessor',
+    'LaTeXProcessor', 'MarkdownProcessor', 'PlainTextProcessor',
+    'get_processor', 'detect_file_type_from_text']