flowmark 0.4.7__tar.gz → 0.4.8__tar.gz

{flowmark-0.4.7 → flowmark-0.4.8}/.cursor/rules/flowmark.mdc

@@ -10,7 +10,7 @@ Most comprehensive testing is done via the test files in the tests/testdocs dire
 
 The input is tests/testdocs/testdoc.orig.md.
 
-The expected output is in tests/testdocs/testdoc.out.*.md.
+The expected output is in tests/testdocs/testdoc.expected.*.md.
 
 When you run `make test`, the actual output from the code is written to tests/testdocs/testdoc.actual.*.md.
 
@@ -18,8 +18,9 @@ The test passes if the actual output matches the expected output.
 
 If they do NOT match, it is a bug and the test fails.
 After a failure, the testdoc.actual.*.md files are still updated and can be
-compared to the expected testdoc.out.*.md files to understand what went wrong.
+compared to the expected testdoc.expected.*.md files to understand what went wrong.
 
-DO NOT change the testdoc.out.*.md files since they should be the expected output.
+DO NOT change the testdoc.expected.*.md files since they should be the expected output,
+unless the user confirms that the testdoc.expected.*.md files are incorrect.
 
-Change the code to fix bugs until the testdoc.actual.*.md files match the expected testdoc.out.*.md files.
+Change the code to fix bugs until the testdoc.actual.*.md files match the expected testdoc.expected.*.md files.

{flowmark-0.4.7 → flowmark-0.4.8}/Makefile

@@ -4,9 +4,9 @@
 
 .DEFAULT_GOAL := default
 
-.PHONY: default install lint test upgrade build clean agent-rules
+.PHONY: default install lint test upgrade build clean agent-rules format-docs
 
-default: agent-rules install lint test 
+default: agent-rules format-docs install lint test 
 
 install:
 	uv sync --all-extras
@@ -42,6 +42,5 @@ clean:
 	-rm -rf CLAUDE.md AGENTS.md
 	-find . -type d -name "__pycache__" -exec rm -rf {} +
 
-gen_test_docs:
-	poetry run flowmark tests/testdocs/testdoc.orig.md -o tests/testdocs/testdoc.out.plain.md
-	poetry run flowmark --semantic tests/testdocs/testdoc.orig.md -o tests/testdocs/testdoc.out.semantic.md
+format-docs:
+	uv run flowmark --auto README.md

{flowmark-0.4.7 → flowmark-0.4.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flowmark
-Version: 0.4.7
+Version: 0.4.8
 Summary: Better line wrapping and formatting for plaintext and Markdown
 Project-URL: Repository, https://github.com/jlevy/flowmark
 Author-email: Joshua Levy <joshua@cal.berkeley.edu>
@@ -24,15 +24,19 @@ Description-Content-Type: text/markdown
 
 # flowmark
 
-Flowmark is a new Python implementation of **text and Markdown line wrapping and
-filling**, with an emphasis on making **git diffs** and **LLM edits** to text documents
-easier to diff and review.
+Flowmark is a pure Python implementation of **Markdown auto-formatting and
+normalization** and **text and Markdown line wrapping and filling**. It can be used from
+the command line or a library.
 
-In addition, it offers **Markdown auto-formatting and normalization** as a library or
-from the command line.
 This is much like [markdownfmt](https://github.com/shurcooL/markdownfmt) or
-[prettier's Markdown support](https://prettier.io/blog/2017/11/07/1.8.0) but is pure
-Python and has (in my humble opinion) better options and defaults.
+[prettier’s Markdown support](https://prettier.io/blog/2017/11/07/1.8.0) but is pure
+Python and has more options and (in my humble opinion) better defaults.
+In particular, it was written to make **git diffs** and **LLM edits** of Markdown text
+documents much easier to review.
+
+It also offers optional **automatic smart quotes** to convert \"non-oriented quotes\" to
+“oriented quotes” and apostrophes intelligently and conservatively (in particular,
+avoiding code blocks).
 
 It aims to be small and simple and have only a few dependencies, currently only
 [`marko`](https://github.com/frostming/marko),
@@ -73,7 +77,7 @@ To use as a library, use uv/poetry/pip to install
 ## Use in VSCode/Cursor
 
 You can use Flowmark to auto-format Markdown on save in VSCode or Cursor.
-Install the "Run on Save" (`emeraldwalk.runonsave`) extension.
+Install the “Run on Save” (`emeraldwalk.runonsave`) extension.
 Then add to your `settings.json`:
 
 ```json
@@ -87,7 +91,8 @@ Then add to your `settings.json`:
   }
 ```
 
-The `--auto` option is just the same as `--inplace --nobackup --semantic`.
+The `--auto` option is just the same as `--inplace --nobackup --semantic --cleanups
+--smartquotes`.
 
 ## Use Cases
 
@@ -109,11 +114,11 @@ The main ways to use Flowmark are:
   Having this as a simple Python library makes this easy in AI-related document
   pipelines.
 
-- As a **drop-in replacement library for Python's default
+- As a **drop-in replacement library for Python’s default
   [`textwrap`](https://docs.python.org/3/library/textwrap.html)** but with more options.
   It simplifies and generalizes that library, offering better control over **initial and
   subsequent indentation** and **when to split words and lines**, e.g. using a word
-  splitter that won't break lines within HTML tags.
+  splitter that won’t break lines within HTML tags.
 
 Other features:
 
@@ -126,10 +131,10 @@ Other features:
   [Markdown source](https://github.com/jlevy/flowmark/blob/main/README.md?plain=1) of
   this readme file.)
 
-- Very simple and fast **regex-based sentence splitting**. It's just based on letters
-  and punctuation so isn't perfect but works well for these purposes (and is much faster
+- Very simple and fast **regex-based sentence splitting**. It’s just based on letters
+  and punctuation so isn’t perfect but works well for these purposes (and is much faster
   and simpler than a proper sentence parser like SpaCy).
-  It should work fine for English and many other latin/Cyrillic languages but hasn't
+  It should work fine for English and many other latin/Cyrillic languages but hasn’t
   been tested on CJK.
 
 Because **YAML frontmatter** is common on Markdown files, the Markdown autoformat
@@ -137,7 +142,7 @@ preserves all frontmatter (content between `---` delimiters at the front of a fi
 
 ## Why a New Markdown Formatter?
 
-Previously I'd implemented something very similar with
+Previously I’d implemented something very similar with
 [for Atom](https://github.com/jlevy/atom-flowmark).
 I found the Markdown formatting conventions enforced by the that plugin worked really
 well for editing and publishing large or collaboratively edited documents.
@@ -178,7 +183,7 @@ options:
                        boldfaced section headers (only applies to Markdown mode)
   -i, --inplace        Edit the file in place (ignores --output)
   --nobackup           Do not make a backup of the original file when using --inplace
-  --auto               Same as `--inplace --nobackup --semantic --cleanups`, as a
+  --auto               Same as `--inplace --nobackup --semantic --cleanups --smartquotes`, as a
                        convenience for fully auto-formatting files
   --version            Show version information and exit
 
@@ -221,13 +226,13 @@ For more details, see: https://github.com/jlevy/flowmark
 - This enables
   [GitHub-flavored Markdown support](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)
   using
-  [Marko's extension](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
+  [Marko’s extension](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
 
 - GFM-style tables are supported and also auto-formatted.
 
 - GFM-style footnotes are supported.
-  But note these aren't actually in the GFM spec, but we follow
-  [micromark's conventions](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
+  But note these aren’t actually in the GFM spec, but we follow
+  [micromark’s conventions](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
 
 ## Project Docs
 

{flowmark-0.4.7 → flowmark-0.4.8}/README.md

@@ -1,14 +1,18 @@
 # flowmark
 
-Flowmark is a new Python implementation of **text and Markdown line wrapping and
-filling**, with an emphasis on making **git diffs** and **LLM edits** to text documents
-easier to diff and review.
+Flowmark is a pure Python implementation of **Markdown auto-formatting and
+normalization** and **text and Markdown line wrapping and filling**. It can be used from
+the command line or a library.
 
-In addition, it offers **Markdown auto-formatting and normalization** as a library or
-from the command line.
 This is much like [markdownfmt](https://github.com/shurcooL/markdownfmt) or
-[prettier's Markdown support](https://prettier.io/blog/2017/11/07/1.8.0) but is pure
-Python and has (in my humble opinion) better options and defaults.
+[prettier’s Markdown support](https://prettier.io/blog/2017/11/07/1.8.0) but is pure
+Python and has more options and (in my humble opinion) better defaults.
+In particular, it was written to make **git diffs** and **LLM edits** of Markdown text
+documents much easier to review.
+
+It also offers optional **automatic smart quotes** to convert \"non-oriented quotes\" to
+“oriented quotes” and apostrophes intelligently and conservatively (in particular,
+avoiding code blocks).
 
 It aims to be small and simple and have only a few dependencies, currently only
 [`marko`](https://github.com/frostming/marko),
@@ -49,7 +53,7 @@ To use as a library, use uv/poetry/pip to install
 ## Use in VSCode/Cursor
 
 You can use Flowmark to auto-format Markdown on save in VSCode or Cursor.
-Install the "Run on Save" (`emeraldwalk.runonsave`) extension.
+Install the “Run on Save” (`emeraldwalk.runonsave`) extension.
 Then add to your `settings.json`:
 
 ```json
@@ -63,7 +67,8 @@ Then add to your `settings.json`:
   }
 ```
 
-The `--auto` option is just the same as `--inplace --nobackup --semantic`.
+The `--auto` option is just the same as `--inplace --nobackup --semantic --cleanups
+--smartquotes`.
 
 ## Use Cases
 
@@ -85,11 +90,11 @@ The main ways to use Flowmark are:
   Having this as a simple Python library makes this easy in AI-related document
   pipelines.
 
-- As a **drop-in replacement library for Python's default
+- As a **drop-in replacement library for Python’s default
   [`textwrap`](https://docs.python.org/3/library/textwrap.html)** but with more options.
   It simplifies and generalizes that library, offering better control over **initial and
   subsequent indentation** and **when to split words and lines**, e.g. using a word
-  splitter that won't break lines within HTML tags.
+  splitter that won’t break lines within HTML tags.
 
 Other features:
 
@@ -102,10 +107,10 @@ Other features:
   [Markdown source](https://github.com/jlevy/flowmark/blob/main/README.md?plain=1) of
   this readme file.)
 
-- Very simple and fast **regex-based sentence splitting**. It's just based on letters
-  and punctuation so isn't perfect but works well for these purposes (and is much faster
+- Very simple and fast **regex-based sentence splitting**. It’s just based on letters
+  and punctuation so isn’t perfect but works well for these purposes (and is much faster
   and simpler than a proper sentence parser like SpaCy).
-  It should work fine for English and many other latin/Cyrillic languages but hasn't
+  It should work fine for English and many other latin/Cyrillic languages but hasn’t
   been tested on CJK.
 
 Because **YAML frontmatter** is common on Markdown files, the Markdown autoformat
@@ -113,7 +118,7 @@ preserves all frontmatter (content between `---` delimiters at the front of a fi
 
 ## Why a New Markdown Formatter?
 
-Previously I'd implemented something very similar with
+Previously I’d implemented something very similar with
 [for Atom](https://github.com/jlevy/atom-flowmark).
 I found the Markdown formatting conventions enforced by the that plugin worked really
 well for editing and publishing large or collaboratively edited documents.
@@ -154,7 +159,7 @@ options:
                        boldfaced section headers (only applies to Markdown mode)
   -i, --inplace        Edit the file in place (ignores --output)
   --nobackup           Do not make a backup of the original file when using --inplace
-  --auto               Same as `--inplace --nobackup --semantic --cleanups`, as a
+  --auto               Same as `--inplace --nobackup --semantic --cleanups --smartquotes`, as a
                        convenience for fully auto-formatting files
   --version            Show version information and exit
 
@@ -197,13 +202,13 @@ For more details, see: https://github.com/jlevy/flowmark
 - This enables
   [GitHub-flavored Markdown support](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax)
   using
-  [Marko's extension](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
+  [Marko’s extension](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
 
 - GFM-style tables are supported and also auto-formatted.
 
 - GFM-style footnotes are supported.
-  But note these aren't actually in the GFM spec, but we follow
-  [micromark's conventions](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
+  But note these aren’t actually in the GFM spec, but we follow
+  [micromark’s conventions](https://github.com/frostming/marko/blob/master/marko/ext/footnote.py).
 
 ## Project Docs
 

{flowmark-0.4.7 → flowmark-0.4.8}/src/flowmark/cli.py

@@ -54,6 +54,7 @@ class Options:
     plaintext: bool
     semantic: bool
     cleanups: bool
+    smartquotes: bool
     inplace: bool
     nobackup: bool
     version: bool
@@ -105,6 +106,13 @@ def _parse_args(args: list[str] | None = None) -> Options:
         help="Enable (safe) cleanups for common issues like accidentally boldfaced section "
         "headers (only applies to Markdown mode)",
     )
+    parser.add_argument(
+        "--smartquotes",
+        action="store_true",
+        default=False,
+        help="Convert straight quotes to typographic (curly) quotes and apostrophes "
+        "(only applies to Markdown mode)",
+    )
     parser.add_argument(
         "-i", "--inplace", action="store_true", help="Edit the file in place (ignores --output)"
     )
@@ -116,7 +124,7 @@ def _parse_args(args: list[str] | None = None) -> Options:
     parser.add_argument(
         "--auto",
         action="store_true",
-        help="Same as `--inplace --nobackup --semantic --cleanups`, as a convenience for "
+        help="Same as `--inplace --nobackup --semantic --cleanups --smartquotes`, as a convenience for "
         "fully auto-formatting files",
     )
     parser.add_argument(
@@ -131,6 +139,7 @@ def _parse_args(args: list[str] | None = None) -> Options:
         opts.nobackup = True
         opts.semantic = True
         opts.cleanups = True
+        opts.smartquotes = True
 
     return Options(
         file=opts.file,
@@ -139,6 +148,7 @@ def _parse_args(args: list[str] | None = None) -> Options:
         plaintext=opts.plaintext,
         semantic=opts.semantic,
         cleanups=opts.cleanups,
+        smartquotes=opts.smartquotes,
         inplace=opts.inplace,
         nobackup=opts.nobackup,
         version=opts.version,
@@ -176,6 +186,7 @@ def main(args: list[str] | None = None) -> int:
             plaintext=options.plaintext,
             semantic=options.semantic,
             cleanups=options.cleanups,
+            smartquotes=options.smartquotes,
             make_parents=True,
         )
     except ValueError as e:

{flowmark-0.4.7 → flowmark-0.4.8}/src/flowmark/custom_marko.py

@@ -81,6 +81,10 @@ class MarkdownNormalizer(Renderer):
         # Suppress item breaks on list items following a top-level paragraph.
         if not self._prefix:
             self._suppress_item_break = True
+        else:
+            # For paragraphs within list items, ensure proper spacing after multi-paragraph items
+            # This handles the case where a paragraph follows a BlankLine within a list item
+            self._suppress_item_break = False
 
         children: Any = self.render_children(element)
 
@@ -97,14 +101,6 @@ class MarkdownNormalizer(Renderer):
         self._prefix = self._second_prefix
         return wrapped_text + "\n"
 
-    def _has_multiple_paragraphs(self, item: object) -> bool:
-        """
-        Check if a list item contains multiple paragraphs.
-        """
-        list_item = cast(block.ListItem, item)
-        paragraphs = [c for c in list_item.children if isinstance(c, block.Paragraph)]
-        return len(paragraphs) > 1
-
     def render_list(self, element: block.List) -> str:
         result: list[str] = []
 
@@ -119,11 +115,8 @@ class MarkdownNormalizer(Renderer):
                 subsequent_indent = "  "
 
             with self.container(prefix, subsequent_indent):
-                # Add an extra newline before multi-paragraph list items (except the first)
-                if i > 0 and self._has_multiple_paragraphs(child):
-                    result.append(self._second_prefix.strip() + "\n")
-
-                result.append(self.render(child))
+                rendered_item = self.render(child)
+                result.append(rendered_item)
 
         self._prefix = self._second_prefix
         return "".join(result)
@@ -139,12 +132,16 @@ class MarkdownNormalizer(Renderer):
             result += self._second_prefix.strip() + "\n"
 
         result += self.render_children(element)
+
         return result
 
     def render_quote(self, element: block.Quote) -> str:
         with self.container("> ", "> "):
             result = self.render_children(element).rstrip("\n")
         self._prefix = self._second_prefix
+        # After rendering a quote block, don't suppress the next item break
+        # This ensures proper spacing after list items with quote blocks
+        self._suppress_item_break = False
         return f"{result}\n"
 
     def _render_code(self, element: block.CodeBlock | block.FencedCode) -> str:
@@ -159,6 +156,9 @@ class MarkdownNormalizer(Renderer):
         lines.extend(f"{self._second_prefix}{line}" for line in code_content.splitlines())
         lines.append(f"{self._second_prefix}```")
         self._prefix = self._second_prefix
+        # After rendering a code block, don't suppress the next item break
+        # This ensures proper spacing after list items with code blocks
+        self._suppress_item_break = False
         return "\n".join(lines) + "\n"
 
     def render_fenced_code(self, element: block.FencedCode) -> str:

{flowmark-0.4.7 → flowmark-0.4.8}/src/flowmark/markdown_filling.py

@@ -15,11 +15,12 @@ import re
 from collections.abc import Callable
 from textwrap import dedent
 
-from flowmark.cleanups import doc_cleanups
+from flowmark.cleanups import doc_cleanups, rewrite_text_content
 from flowmark.custom_marko import custom_marko
 from flowmark.frontmatter import split_frontmatter
 from flowmark.line_wrappers import LineWrapper, line_wrap_by_sentence, line_wrap_to_width
 from flowmark.sentence_split_regex import split_sentences_regex
+from flowmark.smartquotes import smart_quotes
 from flowmark.text_filling import DEFAULT_WRAP_WIDTH
 
 
@@ -79,6 +80,7 @@ def fill_markdown(
     width: int = DEFAULT_WRAP_WIDTH,
     semantic: bool = False,
     cleanups: bool = False,
+    smartquotes: bool = False,
     line_wrapper: LineWrapper | None = None,
 ) -> str:
     """
@@ -125,6 +127,8 @@ def fill_markdown(
     document = marko.parse(markdown_text)
     if cleanups:
         doc_cleanups(document)
+    if smartquotes:
+        rewrite_text_content(document, smart_quotes)
     result = marko.render(document)
 
     # Reattach frontmatter if it was present

{flowmark-0.4.7 → flowmark-0.4.8}/src/flowmark/reformat_api.py

@@ -14,6 +14,7 @@ def reformat_text(
     plaintext: bool = False,
     semantic: bool = True,
     cleanups: bool = True,
+    smartquotes: bool = False,
 ) -> str:
     """
     Reformat text or markdown and wrap lines. Simply a convenient wrapper
@@ -29,7 +30,9 @@ def reformat_text(
         )
     else:
         # Markdown mode
-        result = fill_markdown(text, width=width, semantic=semantic, cleanups=cleanups)
+        result = fill_markdown(
+            text, width=width, semantic=semantic, cleanups=cleanups, smartquotes=smartquotes
+        )
 
     return result
 
@@ -43,6 +46,7 @@ def reformat_file(
     plaintext: bool = False,
     semantic: bool = False,
     cleanups: bool = True,
+    smartquotes: bool = False,
     make_parents: bool = True,
 ) -> None:
     """
@@ -60,6 +64,8 @@ def reformat_file(
         semantic: Use semantic line breaks (based on sentences) heuristic.
         cleanups: Enable (safe) cleanups for common issues like accidentally boldfaced section
             headers (only applies to Markdown mode).
+        smartquotes: Convert straight quotes to typographic (curly) quotes and apostrophes
+            (only applies to Markdown mode).
         make_parents: Whether to make parent directories if they don't exist.
     """
     read_stdin = path == "-"
@@ -73,7 +79,7 @@ def reformat_file(
     else:
         text = Path(path).read_text()
 
-    result = reformat_text(text, width, plaintext, semantic, cleanups)
+    result = reformat_text(text, width, plaintext, semantic, cleanups, smartquotes)
 
     if inplace:
         backup_suffix = ".orig" if not nobackup else ""

flowmark-0.4.8/src/flowmark/smartquotes.py

@@ -0,0 +1,98 @@
+import re
+from re import Match
+
+
+def smart_quotes(text: str) -> str:
+    r"""
+    Replace straight ASCII quotes and apostrophes with typographic quotes and apostrophes
+    when this can be done safely. Aims to be conservative so it doesn't break code or
+    things that aren't language.
+
+    Text that is wrapped in single or double quotes is replaced with typographic quotes
+    if it has whitespace or a newline at the front and is followed by whitespace or
+    a [.,?!]. The content inside quotes must not contain any of the same type (single
+    or double).
+
+    Straight quotes are converted to apostrophes if they are the only straight quote
+    in the word, and have word characters on both sides:
+
+    I'm there with "George" -> I’m there with “George”
+    "Hello," he said. -> “Hello,” he said.
+    "I know!" -> “I know!”
+
+    Words in 'single quotes' work too -> Words in 'single quotes' work too
+
+    I'm there -> I’m there
+    I'll be there, don't worry -> I’ll be there, don’t worry
+    X is 'foo' -> X is ‘foo’
+
+    A few special rules to better help with English:
+
+    Jill's -> Jill’s
+    James' -> James’
+
+    Other patterns are unchanged:
+
+    x="foo" -> x="foo"
+    x='foo' -> x='foo'
+    Blah'blah'blah -> Blah'blah'blah
+    ""quotes"s -> ""quotes"s
+    \"escaped\" -> \"escaped\"
+    'apos -> 'apos
+    'apos'trophes -> 'apos'trophes
+    $James' -> $James'
+
+    """
+
+    # First handle quoted text - both single and double quotes
+    # Pattern excludes content that contains the same type of quote characters
+    # Double quotes exclude double quotes, single quotes exclude single quotes
+    quote_pattern = r'(^|\s)(?:"([^"\u201c\u201d]*)"|\'([^\'\u2018\u2019]*)\')(\s|$|\.|,|;|:|\?|!)'
+
+    def replace_quotes(match: Match[str]) -> str:
+        prefix = match.group(1)
+        double_content = match.group(2)  # Content of double quotes
+        single_content = match.group(3)  # Content of single quotes
+        suffix = match.group(4)
+
+        if double_content is not None:
+            # Replace double quotes with typographic quotes
+            return prefix + "\u201c" + double_content + "\u201d" + suffix
+        else:
+            # Replace single quotes with typographic quotes
+            return prefix + "\u2018" + single_content + "\u2019" + suffix
+
+    result = re.sub(quote_pattern, replace_quotes, text)
+
+    # Now handle apostrophes/contractions
+    # Only convert single quotes that are:
+    # 1. The only quote in the word
+    # 2. Have word characters on both sides OR are possessives at end of words ending in s/S
+
+    # Pattern for apostrophes: word char + ' + word char, where ' is the only quote in the word
+    # We need to be careful not to match words that have multiple quotes
+
+    # Split by whitespace to process words individually
+    words = re.split(r"(\s+)", result)
+
+    for i, word in enumerate(words):
+        # Skip whitespace
+        if word.isspace():
+            continue
+
+        # Count straight quotes in the word
+        quote_count = word.count("'")
+
+        # Only process if there's exactly one straight quote
+        if quote_count == 1:
+            # Check if it's surrounded by word characters (contractions)
+            apostrophe_pattern = r"(\w)\'(\w)"
+            if re.search(apostrophe_pattern, word):
+                # Replace the single quote with apostrophe
+                words[i] = re.sub(r"\'", "\u2019", word)
+            # Check if it's a possessive at the end of a word ending in s/S
+            elif re.match(r"\w*[sS]\'$", word):
+                # Replace the single quote with apostrophe
+                words[i] = re.sub(r"\'", "\u2019", word)
+
+    return "".join(words)