flowmark 0.3.2__tar.gz → 0.3.3__tar.gz

{flowmark-0.3.2 → flowmark-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flowmark
-Version: 0.3.2
+Version: 0.3.3
 Summary: Better line wrapping and formatting for plaintext and Markdown
 Project-URL: Repository, https://github.com/jlevy/flowmark
 Author-email: Joshua Levy <joshua@cal.berkeley.edu>
@@ -18,10 +18,9 @@ Description-Content-Type: text/markdown
 Flowmark is a new Python implementation of **text and Markdown line wrapping and
 filling**, with an emphasis on making **git diffs** and **LLM edits** to text documents
 easier to diff and review.
-after updating
 
-In addition, it adds **support for Markdown** and offers **Markdown auto-formatting and
-normalization** as a library or from the command line.
+In addition, it offers **Markdown auto-formatting and normalization** as a library or
+from the command line.
 This is much like [markdownfmt](https://github.com/shurcooL/markdownfmt) or
 [prettier's Markdown support](https://prettier.io/blog/2017/11/07/1.8.0) but is pure
 Python and has (in my humble opinion) better options and defaults.
@@ -73,8 +72,7 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
 
 ## Use Cases
 
-- As a **command line formatter** to format text or Markdown files using the `flowmark`
-  command.
+The main ways to use Flowmark are:
 
 - To **autoformat Markdown on save in VSCode/Cursor** or any other editor that supports
   running a command on save.
@@ -83,6 +81,9 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
   formatting styles). This can be especially useful for documentation and editing
   workflows where clean diffs and minimal merge conflicts on GitHub are important.
 
+- As a **command line formatter** to format text or Markdown files using the `flowmark`
+  command.
+
 - As a **library to autoformat Markdown**. For example, it is great to normalize the
   outputs from LLMs to be consistent, or to run on the inputs and outputs of LLM
   transformations that edit text, so that the resulting diffs are clean.
@@ -95,6 +96,8 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
   subsequent indentation** and **when to split words and lines**, e.g. using a word
   splitter that won't break lines within HTML tags.
 
+Other features:
+
 - Flowmark has the option to to use **semantic line breaks** (using a heuristic to break
   lines on sentences sentences when that is reasonable), which is an underrated feature
   that can **make diffs on GitHub much more readable**. The the change may seem subtle
@@ -104,8 +107,11 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
   [Markdown source](https://github.com/jlevy/flowmark/blob/main/README.md?plain=1) of
   this readme file.)
 
-- Very, very simple and fast **regex-based sentence splitting**. This should work fine
-  for English and many other latin/Cyrillic languages but it hasn't been tested on CJK.
+- Very simple and fast **regex-based sentence splitting**. It's just based on letters
+  and punctuation so isn't perfect but works well for these purposes (and is much faster
+  and simpler than a proper sentence parser like SpaCy).
+  It should work fine for English and many other latin/Cyrillic languages but hasn't
+  been tested on CJK.
 
 It aims to be small and simple and have only a few dependencies, currently only
 [`marko`](https://github.com/frostming/marko),

{flowmark-0.3.2 → flowmark-0.3.3}/README.md

@@ -3,10 +3,9 @@
 Flowmark is a new Python implementation of **text and Markdown line wrapping and
 filling**, with an emphasis on making **git diffs** and **LLM edits** to text documents
 easier to diff and review.
-after updating
 
-In addition, it adds **support for Markdown** and offers **Markdown auto-formatting and
-normalization** as a library or from the command line.
+In addition, it offers **Markdown auto-formatting and normalization** as a library or
+from the command line.
 This is much like [markdownfmt](https://github.com/shurcooL/markdownfmt) or
 [prettier's Markdown support](https://prettier.io/blog/2017/11/07/1.8.0) but is pure
 Python and has (in my humble opinion) better options and defaults.
@@ -58,8 +57,7 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
 
 ## Use Cases
 
-- As a **command line formatter** to format text or Markdown files using the `flowmark`
-  command.
+The main ways to use Flowmark are:
 
 - To **autoformat Markdown on save in VSCode/Cursor** or any other editor that supports
   running a command on save.
@@ -68,6 +66,9 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
   formatting styles). This can be especially useful for documentation and editing
   workflows where clean diffs and minimal merge conflicts on GitHub are important.
 
+- As a **command line formatter** to format text or Markdown files using the `flowmark`
+  command.
+
 - As a **library to autoformat Markdown**. For example, it is great to normalize the
   outputs from LLMs to be consistent, or to run on the inputs and outputs of LLM
   transformations that edit text, so that the resulting diffs are clean.
@@ -80,6 +81,8 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
   subsequent indentation** and **when to split words and lines**, e.g. using a word
   splitter that won't break lines within HTML tags.
 
+Other features:
+
 - Flowmark has the option to to use **semantic line breaks** (using a heuristic to break
   lines on sentences sentences when that is reasonable), which is an underrated feature
   that can **make diffs on GitHub much more readable**. The the change may seem subtle
@@ -89,8 +92,11 @@ The `--auto` option is just the same as `--inplace --nobackup --semantic`.
   [Markdown source](https://github.com/jlevy/flowmark/blob/main/README.md?plain=1) of
   this readme file.)
 
-- Very, very simple and fast **regex-based sentence splitting**. This should work fine
-  for English and many other latin/Cyrillic languages but it hasn't been tested on CJK.
+- Very simple and fast **regex-based sentence splitting**. It's just based on letters
+  and punctuation so isn't perfect but works well for these purposes (and is much faster
+  and simpler than a proper sentence parser like SpaCy).
+  It should work fine for English and many other latin/Cyrillic languages but hasn't
+  been tested on CJK.
 
 It aims to be small and simple and have only a few dependencies, currently only
 [`marko`](https://github.com/frostming/marko),

{flowmark-0.3.2 → flowmark-0.3.3}/src/flowmark/__init__.py

@@ -1,9 +1,12 @@
 __all__ = (
     "fill_text",
     "fill_markdown",
+    "first_sentence",
+    "first_sentences",
     "html_md_word_splitter",
     "line_wrap_by_sentence",
     "line_wrap_to_width",
+    "split_sentences_regex",
     "wrap_paragraph",
     "wrap_paragraph_lines",
     "Wrap",
@@ -11,5 +14,6 @@ __all__ = (
 
 from .line_wrappers import line_wrap_by_sentence, line_wrap_to_width
 from .markdown_filling import fill_markdown
+from .sentence_split_regex import first_sentence, first_sentences, split_sentences_regex
 from .text_filling import Wrap, fill_text
 from .text_wrapping import html_md_word_splitter, wrap_paragraph, wrap_paragraph_lines

{flowmark-0.3.2 → flowmark-0.3.3}/src/flowmark/sentence_split_regex.py

@@ -2,43 +2,43 @@ from collections.abc import Callable
 
 import regex
 
-# These heuristics are from Flowmark:
-# https://github.com/jlevy/atom-flowmark/blob/master/lib/remark-smart-word-wrap.js#L17-L33
-
-# They work pretty well when used for formatting and editing documents in English.
-# Note this is smarter than Python textwrap's simple heuristic:
-# https://github.com/python/cpython/blob/main/Lib/textwrap.py#L105-L110
-
-# Heuristic: End of sentence must be two letters or more, with the last letter lowercase,
-# followed by a period, exclamation point, question mark. A final or preceding parenthesis
-# or quote is allowed.
-#
-# Does not break on colon or semicolon currently as that seems to have false positives too
-# often with code or other syntax.
-#
 # XXX: Could also handle rare cases with both quotes and parentheses at sentence end
 # but may not be worth it. Also does not detect sentences ending in numerals, which
 # tends to cause too many false positives. Should be OK for most Latin languages but
 # may need to rethink the 2-letter restriction for some languages.
-SENTENCE_RE = regex.compile(r"(\b\p{L}+[\p{Ll}])([.?!]['\"’”)]?|['\"’”)][.?!]) *$")
+# See also:
+# https://github.com/jlevy/atom-flowmark/blob/master/lib/remark-smart-word-wrap.js#L17-L33
+SENTENCE_END_RE = regex.compile(r"(\b\p{L}+[\p{Ll}])([.?!]['\"’”)]?|['\"’”)][.?!]) *$")
 
 # Second heuristic: Very short sentences often not so useful.
 SENTENCE_MIN_LENGTH = 15
 
 
 def heuristic_end_of_sentence(word: str) -> bool:
-    return bool(SENTENCE_RE.search(word))
+    return bool(SENTENCE_END_RE.search(word))
 
 
 def split_sentences_regex(
     text: str,
-    heuristic: Callable[[str], bool] = heuristic_end_of_sentence,
     min_length: int = SENTENCE_MIN_LENGTH,
+    heuristic: Callable[[str], bool] = heuristic_end_of_sentence,
 ) -> list[str]:
     """
-    Split text into sentences using an approximate, fast regex heuristic. (English.)
+    Split text into sentences using an approximate, fast regex heuristic.
+
     Goal is to be conservative, not perfect, avoiding excessive breaks.
 
+    The default heuristic: End of sentence must be two letters or more,
+    with the last letter lowercase, followed by a period, exclamation point,
+    question mark. A final or preceding parenthesis or quote is allowed.
+    Does not break on colon or semicolon as that seems to have false
+    positives too often with code or other syntax.
+
+    They work pretty well when used for formatting and editing documents
+    in English. It should be reasonable for most Latin languages.
+    Note this is smarter than Python textwrap's simpler heuristic:
+    https://github.com/python/cpython/blob/main/Lib/textwrap.py#L105-L110
+
     :param text: The text to split into sentences.
     :param heuristic: A callable that returns True if text ends at the end of a sentence.
     :param min_length: The minimum length of a sentence in characters.
@@ -59,3 +59,28 @@ def split_sentences_regex(
     if sentence:
         sentences.append(" ".join(sentence))
     return sentences
+
+
+def first_sentences(
+    text: str,
+    n: int,
+    min_length: int = SENTENCE_MIN_LENGTH,
+    heuristic: Callable[[str], bool] = heuristic_end_of_sentence,
+) -> list[str]:
+    """
+    Return the first n sentences from the text.
+    """
+    return split_sentences_regex(text, min_length=min_length, heuristic=heuristic)[:n]
+
+
+def first_sentence(
+    text: str,
+    min_length: int = SENTENCE_MIN_LENGTH,
+    heuristic: Callable[[str], bool] = heuristic_end_of_sentence,
+) -> str:
+    """
+    Return the first sentence from the text. Returns input text unchanged if no
+    sentences are found.
+    """
+    sentences = split_sentences_regex(text, min_length=min_length, heuristic=heuristic)
+    return sentences[0] if sentences else text

flowmark-0.3.3/tests/test_sentences.py

@@ -0,0 +1,30 @@
+from flowmark import first_sentence, split_sentences_regex
+
+LONG_TEXT = """
+End of sentence must be two letters or more,
+with the last letter lowercase, followed by a period, exclamation point,
+question mark. A final or preceding parenthesis or quote is allowed.
+Does not break on colon or semicolon as that seems to have false
+positives too often with code or other syntax.
+"""
+
+FIRST_SENTENCE = "End of sentence must be two letters or more, with the last letter lowercase, followed by a period, exclamation point, question mark."
+
+
+def test_split_sentences():
+    assert split_sentences_regex("test!") == ["test!"]
+    assert split_sentences_regex("test! random words") == ["test! random words"]
+
+    split_sentences = split_sentences_regex(LONG_TEXT)
+    print(split_sentences)
+    assert len(split_sentences) == 3
+    assert split_sentences[0] == FIRST_SENTENCE
+
+
+def test_first_sentence():
+    assert first_sentence(LONG_TEXT) == FIRST_SENTENCE
+
+    assert first_sentence("") == ""
+    assert first_sentence(" ") == " "
+    assert first_sentence("hello") == "hello"
+    assert first_sentence(" hello\n") == "hello"