pysbd-plus 0.3.5__tar.gz

pysbd_plus-0.3.5/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 Nipun Sadvilkar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pysbd_plus-0.3.5/MANIFEST.in

@@ -0,0 +1 @@
+include README.md LICENSE

pysbd_plus-0.3.5/PKG-INFO

@@ -0,0 +1,139 @@
+Metadata-Version: 2.4
+Name: pysbd-plus
+Version: 0.3.5
+Summary: pysbd (Python Sentence Boundary Disambiguation) is a rule-based sentence boundary detection that works out-of-the-box across many languages.
+Home-page: http://nipunsadvilkar.github.io/
+Author: Nipun Sadvilkar
+Author-email: nipunsadvilkar@gmail.com
+License: MIT
+Keywords: natural-language-processing nlp
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Topic :: Software Development
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: License :: OSI Approved :: MIT License
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-python
+Dynamic: summary
+
+
+![PySBD logo](artifacts/pysbd_logo.png?raw=true "pysbd logo")
+# pySBD: Python Sentence Boundary Disambiguation (SBD)
+
+![Python package](https://github.com/nipunsadvilkar/pySBD/workflows/Python%20package/badge.svg) [![codecov](https://codecov.io/gh/nipunsadvilkar/pySBD/branch/master/graph/badge.svg)](https://codecov.io/gh/nipunsadvilkar/pySBD) [![License](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat)](https://github.com/nipunsadvilkar/pySBD/blob/master/LICENSE) [![PyPi](https://img.shields.io/pypi/v/pysbd?color=blue&logo=pypi&logoColor=white)](https://pypi.python.org/pypi/pysbd) [![GitHub](https://img.shields.io/github/v/release/nipunsadvilkar/pySBD.svg?include_prereleases&logo=github&style=flat)](https://github.com/nipunsadvilkar/pySBD)
+
+pySBD - python Sentence Boundary Disambiguation (SBD) - is a rule-based sentence boundary detection module that works out-of-the-box.
+
+This project is a direct port of ruby gem - [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter) which provides rule-based sentence boundary detection.
+
+![pysbd_code](artifacts/pysbd_code.png?raw=true "pysbd_code")
+
+## Highlights
+**'PySBD: Pragmatic Sentence Boundary Disambiguation'** a short research paper got accepted into 2nd Workshop for Natural Language Processing Open Source Software (NLP-OSS) at EMNLP 2020. </br>
+
+**Research Paper:**</br>
+
+https://arxiv.org/abs/2010.09657</br>
+
+**[Recorded Talk:](https://slideslive.com/38939754)**</br>
+
+[![pysbd_talk](artifacts/pysbd_talk.png)](https://slideslive.com/38939754)</br>
+
+**Poster:**</br>
+
+[![name](artifacts/pysbd_poster.png)](artifacts/pysbd_poster.png)
+
+## Install
+
+**Python**
+
+    pip install pysbd
+
+## Usage
+
+-   Currently pySBD supports 22 languages.
+
+```python
+import pysbd
+text = "My name is Jonas E. Smith. Please turn to p. 55."
+seg = pysbd.Segmenter(language="en", clean=False)
+print(seg.segment(text))
+# ['My name is Jonas E. Smith.', 'Please turn to p. 55.']
+```
+
+-   Use `pysbd` as a [spaCy](https://spacy.io/usage/processing-pipelines) pipeline component. (recommended)</br>Please refer to example [pysbd\_as\_spacy\_component.py](https://github.com/nipunsadvilkar/pySBD/blob/master/examples/pysbd_as_spacy_component.py)
+- Use pysbd through [entrypoints](https://spacy.io/usage/saving-loading#entry-points-components)
+
+```python
+import spacy
+from pysbd.utils import PySBDFactory
+
+nlp = spacy.blank('en')
+
+# explicitly adding component to pipeline
+# (recommended - makes it more readable to tell what's going on)
+nlp.add_pipe(PySBDFactory(nlp))
+
+# or you can use it implicitly with keyword
+# pysbd = nlp.create_pipe('pysbd')
+# nlp.add_pipe(pysbd)
+
+doc = nlp('My name is Jonas E. Smith. Please turn to p. 55.')
+print(list(doc.sents))
+# [My name is Jonas E. Smith., Please turn to p. 55.]
+
+```
+
+## Contributing
+
+If you want to contribute new feature/language support or found a text that is incorrectly segmented using pySBD, then please head to [CONTRIBUTING.md](https://github.com/nipunsadvilkar/pySBD/blob/master/CONTRIBUTING.md) to know more and follow these steps.
+
+1.  Fork it ( https://github.com/nipunsadvilkar/pySBD/fork )
+2.  Create your feature branch (`git checkout -b my-new-feature`)
+3.  Commit your changes (`git commit -am 'Add some feature'`)
+4.  Push to the branch (`git push origin my-new-feature`)
+5.  Create a new Pull Request
+
+## Citation
+If you use `pysbd` package in your projects or research, please cite [PySBD: Pragmatic Sentence Boundary Disambiguation](https://www.aclweb.org/anthology/2020.nlposs-1.15).
+```
+@inproceedings{sadvilkar-neumann-2020-pysbd,
+    title = "{P}y{SBD}: Pragmatic Sentence Boundary Disambiguation",
+    author = "Sadvilkar, Nipun  and
+      Neumann, Mark",
+    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
+    month = nov,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/2020.nlposs-1.15",
+    pages = "110--114",
+    abstract = "We present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter which we ported to Python with additional improvements and functionality. PySBD passes 97.92{\%} of the Golden Rule Set examplars for English, an improvement of 25{\%} over the next best open source Python tool.",
+}
+```
+
+## Credit
+
+This project wouldn't be possible without the great work done by [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter) team.

pysbd_plus-0.3.5/README.md

@@ -0,0 +1,98 @@
+![PySBD logo](artifacts/pysbd_logo.png?raw=true "pysbd logo")
+# pySBD: Python Sentence Boundary Disambiguation (SBD)
+
+![Python package](https://github.com/nipunsadvilkar/pySBD/workflows/Python%20package/badge.svg) [![codecov](https://codecov.io/gh/nipunsadvilkar/pySBD/branch/master/graph/badge.svg)](https://codecov.io/gh/nipunsadvilkar/pySBD) [![License](https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat)](https://github.com/nipunsadvilkar/pySBD/blob/master/LICENSE) [![PyPi](https://img.shields.io/pypi/v/pysbd?color=blue&logo=pypi&logoColor=white)](https://pypi.python.org/pypi/pysbd) [![GitHub](https://img.shields.io/github/v/release/nipunsadvilkar/pySBD.svg?include_prereleases&logo=github&style=flat)](https://github.com/nipunsadvilkar/pySBD)
+
+pySBD - python Sentence Boundary Disambiguation (SBD) - is a rule-based sentence boundary detection module that works out-of-the-box.
+
+This project is a direct port of ruby gem - [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter) which provides rule-based sentence boundary detection.
+
+![pysbd_code](artifacts/pysbd_code.png?raw=true "pysbd_code")
+
+## Highlights
+**'PySBD: Pragmatic Sentence Boundary Disambiguation'** a short research paper got accepted into 2nd Workshop for Natural Language Processing Open Source Software (NLP-OSS) at EMNLP 2020. </br>
+
+**Research Paper:**</br>
+
+https://arxiv.org/abs/2010.09657</br>
+
+**[Recorded Talk:](https://slideslive.com/38939754)**</br>
+
+[![pysbd_talk](artifacts/pysbd_talk.png)](https://slideslive.com/38939754)</br>
+
+**Poster:**</br>
+
+[![name](artifacts/pysbd_poster.png)](artifacts/pysbd_poster.png)
+
+## Install
+
+**Python**
+
+    pip install pysbd
+
+## Usage
+
+-   Currently pySBD supports 22 languages.
+
+```python
+import pysbd
+text = "My name is Jonas E. Smith. Please turn to p. 55."
+seg = pysbd.Segmenter(language="en", clean=False)
+print(seg.segment(text))
+# ['My name is Jonas E. Smith.', 'Please turn to p. 55.']
+```
+
+-   Use `pysbd` as a [spaCy](https://spacy.io/usage/processing-pipelines) pipeline component. (recommended)</br>Please refer to example [pysbd\_as\_spacy\_component.py](https://github.com/nipunsadvilkar/pySBD/blob/master/examples/pysbd_as_spacy_component.py)
+- Use pysbd through [entrypoints](https://spacy.io/usage/saving-loading#entry-points-components)
+
+```python
+import spacy
+from pysbd.utils import PySBDFactory
+
+nlp = spacy.blank('en')
+
+# explicitly adding component to pipeline
+# (recommended - makes it more readable to tell what's going on)
+nlp.add_pipe(PySBDFactory(nlp))
+
+# or you can use it implicitly with keyword
+# pysbd = nlp.create_pipe('pysbd')
+# nlp.add_pipe(pysbd)
+
+doc = nlp('My name is Jonas E. Smith. Please turn to p. 55.')
+print(list(doc.sents))
+# [My name is Jonas E. Smith., Please turn to p. 55.]
+
+```
+
+## Contributing
+
+If you want to contribute new feature/language support or found a text that is incorrectly segmented using pySBD, then please head to [CONTRIBUTING.md](https://github.com/nipunsadvilkar/pySBD/blob/master/CONTRIBUTING.md) to know more and follow these steps.
+
+1.  Fork it ( https://github.com/nipunsadvilkar/pySBD/fork )
+2.  Create your feature branch (`git checkout -b my-new-feature`)
+3.  Commit your changes (`git commit -am 'Add some feature'`)
+4.  Push to the branch (`git push origin my-new-feature`)
+5.  Create a new Pull Request
+
+## Citation
+If you use `pysbd` package in your projects or research, please cite [PySBD: Pragmatic Sentence Boundary Disambiguation](https://www.aclweb.org/anthology/2020.nlposs-1.15).
+```
+@inproceedings{sadvilkar-neumann-2020-pysbd,
+    title = "{P}y{SBD}: Pragmatic Sentence Boundary Disambiguation",
+    author = "Sadvilkar, Nipun  and
+      Neumann, Mark",
+    booktitle = "Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)",
+    month = nov,
+    year = "2020",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/2020.nlposs-1.15",
+    pages = "110--114",
+    abstract = "We present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter which we ported to Python with additional improvements and functionality. PySBD passes 97.92{\%} of the Golden Rule Set examplars for English, an improvement of 25{\%} over the next best open source Python tool.",
+}
+```
+
+## Credit
+
+This project wouldn't be possible without the great work done by [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter) team.

pysbd_plus-0.3.5/benchmarks/benchmark_sbd_tools.py

@@ -0,0 +1,84 @@
+import blingfire
+import nltk
+import pysbd
+import spacy
+import stanza
+
+from syntok.tokenizer import Tokenizer
+import syntok.segmenter as syntok_segmenter
+
+from english_golden_rules import GOLDEN_EN_RULES
+
+pysbd_segmenter = pysbd.Segmenter(language="en", clean=False, char_span=False)
+
+nlp = spacy.blank('en')
+nlp.add_pipe(nlp.create_pipe("sentencizer"))
+nlp_dep = spacy.load('en_core_web_sm', disable=["ner"])
+#stanza.download('en')
+stanza_nlp = stanza.Pipeline(lang='en', processors='tokenize')
+
+syntok_tokenizer = Tokenizer()
+
+def blingfire_tokenize(text):
+    return blingfire.text_to_sentences(text).split('\n')
+
+def nltk_tokenize(text):
+    return nltk.sent_tokenize(text)
+
+def pysbd_tokenize(text):
+    segments = pysbd_segmenter.segment(text)
+    return [s.strip() for s in segments]
+
+def spacy_tokenize(text):
+    return [sent.text for sent in nlp(text).sents]
+
+def spacy_dep_tokenize(text):
+    return [sent.text for sent in nlp_dep(text).sents]
+
+def stanza_tokenize(text):
+    return [e.text for e in stanza_nlp(text).sentences]
+
+def make_sentences(segmented_tokens):
+    for sentence in segmented_tokens:
+        yield "".join(str(token) for token in sentence).strip()
+
+def syntok_tokenize(text):
+    tokens = syntok_tokenizer.split(text)
+    result = syntok_segmenter.split(iter(tokens))
+    segments = [sent for sent in make_sentences(result)]
+    return segments
+
+
+total_rules = len(GOLDEN_EN_RULES)
+
+def benchmark(golden_rules, tokenize_func):
+    score = 0
+    for rule in golden_rules:
+        text, expected = rule
+        segments = tokenize_func(text)
+        if segments == expected:
+            score += 1
+    percent_score = (score / total_rules) * 100.0
+
+    return percent_score
+
+if __name__ == "__main__":
+    import time
+    libraries = (
+        blingfire_tokenize,
+        nltk_tokenize,
+        pysbd_tokenize,
+        spacy_tokenize,
+        spacy_dep_tokenize,
+        stanza_tokenize,
+        syntok_tokenize)
+    for tokenize_func in libraries:
+        t = time.time()
+        for i in range(100):
+            percent_score = benchmark(GOLDEN_EN_RULES, tokenize_func)
+
+        time_taken = time.time() - t
+        print()
+        print(tokenize_func.__name__)
+        print('GRS score: {:0.2f}%'.format(percent_score))
+        print('Speed(Avg over 100 runs): {:>10.2f} ms'.format(time_taken*1000/100))

pysbd_plus-0.3.5/benchmarks/bigtext_speed_benchmark.py

@@ -0,0 +1,75 @@
+import blingfire
+import nltk
+import pysbd
+import spacy
+import stanza
+
+from syntok.tokenizer import Tokenizer
+import syntok.segmenter as syntok_segmenter
+
+pysbd_segmenter = pysbd.Segmenter(language="en", clean=False, char_span=False)
+
+nlp = spacy.blank('en')
+nlp.add_pipe(nlp.create_pipe("sentencizer"))
+nlp_dep = spacy.load('en_core_web_sm', disable=["ner"])
+#stanza.download('en')
+stanza_nlp = stanza.Pipeline(lang='en', processors='tokenize')
+
+syntok_tokenizer = Tokenizer()
+
+def blingfire_tokenize(text):
+    return blingfire.text_to_sentences(text).split('\n')
+
+def nltk_tokenize(text):
+    return nltk.sent_tokenize(text)
+
+def pysbd_tokenize(text):
+    segments = pysbd_segmenter.segment(text)
+    segments = [s.strip() for s in segments]
+    return segments
+
+def spacy_tokenize(text):
+    return [sent.text.strip("\n") for sent in nlp(text).sents]
+
+def spacy_dep_tokenize(text):
+    return [sent.text.strip("\n") for sent in nlp_dep(text).sents]
+
+def stanza_tokenize(text):
+    return [e.text for e in stanza_nlp(text).sentences]
+
+def make_sentences(segmented_tokens):
+    for sentence in segmented_tokens:
+        yield "".join(str(token) for token in sentence).strip()
+
+def syntok_tokenize(text):
+    tokens = syntok_tokenizer.split(text)
+    result = syntok_segmenter.split(iter(tokens))
+    segments = [sent for sent in make_sentences(result)]
+    return segments
+
+def speed_benchmark(big_text, tokenize_func):
+    segments = tokenize_func(big_text)
+    return segments
+
+if __name__ == "__main__":
+    import time
+    libraries = (
+        blingfire_tokenize,
+        nltk_tokenize,
+        pysbd_tokenize,
+        spacy_tokenize,
+        spacy_dep_tokenize,
+        stanza_tokenize,
+        syntok_tokenize)
+
+    for tokenize_func in libraries:
+        t = time.time()
+        # wget http://www.gutenberg.org/files/1661/1661-0.txt -P benchmarks/
+        with open('benchmarks/1661-0.txt') as bigfile:
+            big_text = bigfile.read()
+        sentences = speed_benchmark(big_text, tokenize_func)
+
+        time_taken = time.time() - t
+        print()
+        print(tokenize_func.__name__)
+        print('Speed : {:>20.2f} ms'.format(time_taken * 1000))

pysbd_plus-0.3.5/benchmarks/english_golden_rules.py

@@ -0,0 +1,210 @@
+# -*- coding: utf-8 -*-
+
+GOLDEN_EN_RULES = [
+    # 1) Simple period to end sentence
+    ("Hello World. My name is Jonas.", ["Hello World.", "My name is Jonas."]),
+    # 2) Question mark to end sentence
+    ("What is your name? My name is Jonas.", ["What is your name?", "My name is Jonas."]),
+    # 3) Exclamation point to end sentence
+    ("There it is! I found it.", ["There it is!", "I found it."]),
+    # 4) One letter upper case abbreviations
+    ("My name is Jonas E. Smith.", ["My name is Jonas E. Smith."]),
+    # 5) One letter lower case abbreviations
+    ("Please turn to p. 55.", ["Please turn to p. 55."]),
+    # 6) Two letter lower case abbreviations in the middle of a sentence
+    ("Were Jane and co. at the party?", ["Were Jane and co. at the party?"]),
+    # 7) Two letter upper case abbreviations in the middle of a sentence
+    ("They closed the deal with Pitt, Briggs & Co. at noon.",
+        ["They closed the deal with Pitt, Briggs & Co. at noon."]),
+    # 8) Two letter lower case abbreviations at the end of a sentence
+    (
+        "Let's ask Jane and co. They should know.",
+        ["Let's ask Jane and co.", "They should know."]),
+    # 9) Two letter upper case abbreviations at the end of a sentence
+    (
+        "They closed the deal with Pitt, Briggs & Co. It closed yesterday.", [
+            "They closed the deal with Pitt, Briggs & Co.",
+            "It closed yesterday."
+        ],
+    ),
+    # 10) Two letter (prepositive) abbreviations
+    ("I can see Mt. Fuji from here.", ["I can see Mt. Fuji from here."]),
+    # 11) Two letter (prepositive & postpositive) abbreviations
+    (
+        "St. Michael's Church is on 5th st. near the light.",
+        ["St. Michael's Church is on 5th st. near the light."],
+    ),
+    # 12) Possesive two letter abbreviations
+    ("That is JFK Jr.'s book.", ["That is JFK Jr.'s book."]),
+    # 13) Multi-period abbreviations in the middle of a sentence
+    ("I visited the U.S.A. last year.", ["I visited the U.S.A. last year."]),
+    # 14) Multi-period abbreviations at the end of a sentence
+    (
+        "I live in the E.U. How about you?",
+        ["I live in the E.U.", "How about you?"],
+    ),
+    # 15) U.S. as sentence boundary
+    (
+        "I live in the U.S. How about you?",
+        ["I live in the U.S.", "How about you?"],
+    ),
+    # 16) U.S. as non sentence boundary with next word capitalized
+    ("I work for the U.S. Government in Virginia.",
+        ["I work for the U.S. Government in Virginia."]),
+    # 17) U.S. as non sentence boundary
+    ("I have lived in the U.S. for 20 years.",
+        ["I have lived in the U.S. for 20 years."]),
+    # Most difficult sentence to crack
+    # 18) A.M. / P.M. as non sentence boundary and sentence boundary
+    (
+         "At 5 a.m. Mr. Smith went to the bank. He left the bank at 6 P.M. Mr. Smith then went to the store.",
+         [
+             "At 5 a.m. Mr. Smith went to the bank.",
+             "He left the bank at 6 P.M.", "Mr. Smith then went to the store."
+         ]
+    ),
+    # 19) Number as non sentence boundary
+    ("She has $100.00 in her bag.", ["She has $100.00 in her bag."]),
+    # 20) Number as sentence boundary
+    ("She has $100.00. It is in her bag.", ["She has $100.00.", "It is in her bag."]),
+    # 21) Parenthetical inside sentence
+    ("He teaches science (He previously worked for 5 years as an engineer.) at the local University.",
+        ["He teaches science (He previously worked for 5 years as an engineer.) at the local University."]),
+    # 22) Email addresses
+    ("Her email is Jane.Doe@example.com. I sent her an email.",
+        ["Her email is Jane.Doe@example.com.", "I sent her an email."]),
+    # 23) Web addresses
+    ("The site is: https://www.example.50.com/new-site/awesome_content.html. Please check it out.",
+        ["The site is: https://www.example.50.com/new-site/awesome_content.html.",
+            "Please check it out."]),
+    # 24) Single quotations inside sentence
+    (
+        "She turned to him, 'This is great.' she said.",
+        ["She turned to him, 'This is great.' she said."],
+    ),
+    # 25) Double quotations inside sentence
+    (
+        'She turned to him, "This is great." she said.',
+        ['She turned to him, "This is great." she said.'],
+    ),
+    # 26) Double quotations at the end of a sentence
+    (
+        'She turned to him, "This is great." She held the book out to show him.',
+        [
+            'She turned to him, "This is great."',
+            "She held the book out to show him."
+        ],
+    ),
+    # 27) Double punctuation (exclamation point)
+    ("Hello!! Long time no see.", ["Hello!!", "Long time no see."]),
+    # 28) Double punctuation (question mark)
+    ("Hello?? Who is there?", ["Hello??", "Who is there?"]),
+    # 29) Double punctuation (exclamation point / question mark)
+    ("Hello!? Is that you?", ["Hello!?", "Is that you?"]),
+    # 30) Double punctuation (question mark / exclamation point)
+    ("Hello?! Is that you?", ["Hello?!", "Is that you?"]),
+    # 31) List (period followed by parens and no period to end item)
+    (
+        "1.) The first item 2.) The second item",
+        ["1.) The first item", "2.) The second item"],
+    ),
+    # 32) List (period followed by parens and period to end item)
+    (
+        "1.) The first item. 2.) The second item.",
+        ["1.) The first item.", "2.) The second item."],
+    ),
+    # 33) List (parens and no period to end item)
+    (
+        "1) The first item 2) The second item",
+        ["1) The first item", "2) The second item"],
+    ),
+    # 34) List (parens and period to end item)
+    ("1) The first item. 2) The second item.",
+        ["1) The first item.", "2) The second item."]),
+    # 35) List (period to mark list and no period to end item)
+    (
+        "1. The first item 2. The second item",
+        ["1. The first item", "2. The second item"],
+    ),
+    # 36) List (period to mark list and period to end item)
+    (
+        "1. The first item. 2. The second item.",
+        ["1. The first item.", "2. The second item."],
+    ),
+    # 37) List with bullet
+    (
+        "• 9. The first item • 10. The second item",
+        ["• 9. The first item", "• 10. The second item"],
+    ),
+    # 38) List with hypthen
+    (
+        "⁃9. The first item ⁃10. The second item",
+        ["⁃9. The first item", "⁃10. The second item"],
+    ),
+    # 39) Alphabetical list
+    (
+        "a. The first item b. The second item c. The third list item",
+        ["a. The first item", "b. The second item", "c. The third list item"],
+    ),
+    # 40) Geo Coordinates
+    (
+        "You can find it at N°. 1026.253.553. That is where the treasure is.",
+        [
+            "You can find it at N°. 1026.253.553.",
+            "That is where the treasure is."
+        ],
+    ),
+    # 41) Named entities with an exclamation point
+    (
+        "She works at Yahoo! in the accounting department.",
+        ["She works at Yahoo! in the accounting department."],
+    ),
+    # 42) I as a sentence boundary and I as an abbreviation
+    (
+        "We make a good team, you and I. Did you see Albert I. Jones yesterday?",
+        [
+            "We make a good team, you and I.",
+            "Did you see Albert I. Jones yesterday?"
+        ],
+    ),
+    # 43) Ellipsis at end of quotation
+    (
+        "Thoreau argues that by simplifying one’s life, “the laws of the universe will appear less complex. . . .”",
+        [
+            "Thoreau argues that by simplifying one’s life, “the laws of the universe will appear less complex. . . .”"
+        ],
+    ),
+    # 44) Ellipsis with square brackets
+    (
+        """"Bohr [...] used the analogy of parallel stairways [...]" (Smith 55).""",
+        [
+            '"Bohr [...] used the analogy of parallel stairways [...]" (Smith 55).'
+        ],
+    ),
+    # 45) Ellipsis as sentence boundary (standard ellipsis rules)
+    ("If words are left off at the end of a sentence, and that is all that is omitted, indicate the omission with ellipsis marks (preceded and followed by a space) and then indicate the end of the sentence with a period . . . . Next sentence.",
+        [
+            "If words are left off at the end of a sentence, and that is all that is omitted, indicate the omission with ellipsis marks (preceded and followed by a space) and then indicate the end of the sentence with a period . . . .",
+            "Next sentence."
+        ]),
+    # 46) Ellipsis as sentence boundary (non-standard ellipsis rules)
+    (
+        "I never meant that.... She left the store.",
+        ["I never meant that....", "She left the store."],
+    ),
+    # 47) Ellipsis as non sentence boundary
+    (
+        "I wasn’t really ... well, what I mean...see . . . what I'm saying, the thing is . . . I didn’t mean it.",
+        [
+            "I wasn’t really ... well, what I mean...see . . . what I'm saying, the thing is . . . I didn’t mean it."
+        ],
+    ),
+    # 48) 4-dot ellipsis
+    (
+        "One further habit which was somewhat weakened . . . was that of combining words into self-interpreting compounds. . . . The practice was not abandoned. . . .",
+        [
+            "One further habit which was somewhat weakened . . . was that of combining words into self-interpreting compounds.",
+            ". . . The practice was not abandoned. . . ."
+        ],
+    )
+]