scripturelookup 0.0.7__tar.gz → 0.1.0__tar.gz

{scripturelookup-0.0.7/src/scripturelookup.egg-info → scripturelookup-0.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scripturelookup
-Version: 0.0.7
+Version: 0.1.0
 Summary: Python and command-line utility for converting scripture references between formats.
 Author-email: Samuel Bradshaw <samuel.h.bradshaw@gmail.com>
 License: MIT License
@@ -40,7 +40,7 @@ Dynamic: license-file
 
 # Scripture Lookup
 
-Scripture Lookup is a Python library and command-line tool for looking up scripture verses or chapters. It can also convert scripture references between formats and languages. The tool is built around data from [Python Scripture Scraper](https://github.com/samuelbradshaw/python-scripture-scraper).
+Scripture Lookup is a Python library and command-line tool for looking up scripture verses or chapters. It can also detect scripture references in a string, and convert them between formats and languages. The tool uses scripture and language metadata from [Python Scripture Scraper](https://github.com/samuelbradshaw/python-scripture-scraper).
 
 
 ## Installation
@@ -79,6 +79,12 @@ https://www.churchofjesuschrist.org/study/scriptures/bofm/hel/5?id=p12&lang=eng#
 
 % scripturelookup get_church_url "/scriptures/ot" --lang "es"
 https://www.churchofjesuschrist.org/study/scriptures/ot?lang=spa
+
+% scripturelookup detect_references "As Nephi taught in 1 Ne. 3:7, and again in 2 Nephi 2:25."
+[['1 Ne. 3:7', 19, 28], ['2 Nephi 2:25', 43, 55]]
+
+% scripturelookup refresh_metadata
+Updated metadata: metadata-languages.min.json, metadata-scriptures.min.json
 ```
 
 
@@ -102,6 +108,12 @@ lookup.get_label('/scriptures/nt/1-jn/3.2-3', lang = 'cmn-Hant')
 
 lookup.get_church_url('/scriptures/ot', lang = 'es')
 # https://www.churchofjesuschrist.org/study/scriptures/ot?lang=spa
+
+lookup.detect_references('As Nephi taught in 1 Ne. 3:7, and again in 2 Nephi 2:25.')
+# [['1 Ne. 3:7', 19, 28], ['2 Nephi 2:25', 43, 55]]
+
+lookup.refresh_metadata()
+# Updated metadata: metadata-languages.min.json, metadata-scriptures.min.json
 ```
 
 
@@ -117,6 +129,8 @@ lookup.get_church_url('/scriptures/ot', lang = 'es')
 - **get_reference_objects** – Get a list of references as objects.
 - **get_reference_attributes** – Get a list of references as dictionaries.
 - **sort_references** – Sort a list of references by label or in traditional book order.
+- **detect_references** – Find scripture references embedded in a string of text.
+- **refresh_metadata** – Download the latest scripture and language metadata. Takes no input.
 
 ### Inputs
 
@@ -145,6 +159,8 @@ Any of the following input types are supported. You can also provide several inp
   - http://lds.org/scriptures/bofm/1-ne/3.7?lang=eng
   - [gospellibrary://content/scriptures/nt/john/3.16?lang=eng#16](https://www.churchofjesuschrist.org/study/scriptures/nt/john/3?id=p16&lang=eng#p16)
 
+`detect_references` takes an arbitrary text string and returns a list of `[label, start_offset, end_offset]`. The label is the exact detected text (not normalized). Start and end offset indicate the exclusive range of characters in the input string where the reference was found.
+
 ### Options
 
 Several options are available. Some are only applicable to certain commands.

{scripturelookup-0.0.7 → scripturelookup-0.1.0}/README.md

@@ -1,6 +1,6 @@
 # Scripture Lookup
 
-Scripture Lookup is a Python library and command-line tool for looking up scripture verses or chapters. It can also convert scripture references between formats and languages. The tool is built around data from [Python Scripture Scraper](https://github.com/samuelbradshaw/python-scripture-scraper).
+Scripture Lookup is a Python library and command-line tool for looking up scripture verses or chapters. It can also detect scripture references in a string, and convert them between formats and languages. The tool uses scripture and language metadata from [Python Scripture Scraper](https://github.com/samuelbradshaw/python-scripture-scraper).
 
 
 ## Installation
@@ -39,6 +39,12 @@ https://www.churchofjesuschrist.org/study/scriptures/bofm/hel/5?id=p12&lang=eng#
 
 % scripturelookup get_church_url "/scriptures/ot" --lang "es"
 https://www.churchofjesuschrist.org/study/scriptures/ot?lang=spa
+
+% scripturelookup detect_references "As Nephi taught in 1 Ne. 3:7, and again in 2 Nephi 2:25."
+[['1 Ne. 3:7', 19, 28], ['2 Nephi 2:25', 43, 55]]
+
+% scripturelookup refresh_metadata
+Updated metadata: metadata-languages.min.json, metadata-scriptures.min.json
 ```
 
 
@@ -62,6 +68,12 @@ lookup.get_label('/scriptures/nt/1-jn/3.2-3', lang = 'cmn-Hant')
 
 lookup.get_church_url('/scriptures/ot', lang = 'es')
 # https://www.churchofjesuschrist.org/study/scriptures/ot?lang=spa
+
+lookup.detect_references('As Nephi taught in 1 Ne. 3:7, and again in 2 Nephi 2:25.')
+# [['1 Ne. 3:7', 19, 28], ['2 Nephi 2:25', 43, 55]]
+
+lookup.refresh_metadata()
+# Updated metadata: metadata-languages.min.json, metadata-scriptures.min.json
 ```
 
 
@@ -77,6 +89,8 @@ lookup.get_church_url('/scriptures/ot', lang = 'es')
 - **get_reference_objects** – Get a list of references as objects.
 - **get_reference_attributes** – Get a list of references as dictionaries.
 - **sort_references** – Sort a list of references by label or in traditional book order.
+- **detect_references** – Find scripture references embedded in a string of text.
+- **refresh_metadata** – Download the latest scripture and language metadata. Takes no input.
 
 ### Inputs
 
@@ -105,6 +119,8 @@ Any of the following input types are supported. You can also provide several inp
   - http://lds.org/scriptures/bofm/1-ne/3.7?lang=eng
   - [gospellibrary://content/scriptures/nt/john/3.16?lang=eng#16](https://www.churchofjesuschrist.org/study/scriptures/nt/john/3?id=p16&lang=eng#p16)
 
+`detect_references` takes an arbitrary text string and returns a list of `[label, start_offset, end_offset]`. The label is the exact detected text (not normalized). Start and end offset indicate the exclusive range of characters in the input string where the reference was found.
+
 ### Options
 
 Several options are available. Some are only applicable to certain commands.

{scripturelookup-0.0.7 → scripturelookup-0.1.0}/src/scripturelookup/command_line.py

@@ -1,13 +1,14 @@
 # Python standard libraries
 import argparse
+import inspect
 
 # Internal imports
-from . import data, numbers, lookup
+from . import lookup
 
 def main_cli():
   parser = argparse.ArgumentParser(description='Scripture lookup')
   parser.add_argument('command', help='Command to run. Required.')
-  parser.add_argument('input', help='Input text to parse (one or more references).')
+  parser.add_argument('input', nargs='?', default='', help='Input text to parse (one or more references). Not needed for commands that don’t take input, such as refresh_metadata.')
   parser.add_argument('--lang', help='Output language. Default: "en".')
   parser.add_argument('--separator', help='Separator when there are multiple results. Default: "\n".')
   parser.add_argument('--sort-by', help='Sort the returned references ("none", "traditional", or "label"). Default: "none".')
@@ -24,9 +25,16 @@ def main_cli():
   
   args = parser.parse_args()
   
-  command = getattr(lookup, args.command)
-  result = command(
-    args.input,
+  command = getattr(lookup, args.command, None)
+  if not callable(command) or args.command.startswith('_'):
+    parser.error(f'Unknown command: “{args.command}”. See README.md for the list of commands.')
+
+  # Some commands (refresh_metadata, get_langs, get_punctuation, get_numerals) don't take input text
+  command_takes_input = 'input_string' in inspect.signature(command).parameters
+  if command_takes_input and not args.input:
+    parser.error(f'The “{args.command}” command needs input text.')
+
+  options = dict(
     lang = args.lang or 'en',
     separator = args.separator or '\n',
     sort_by = args.sort_by,
@@ -41,5 +49,6 @@ def main_cli():
     skip_cleanup = args.skip_cleanup,
     range_split_limit = args.range_split_limit or 1,
   )
-  
+  result = command(args.input, **options) if command_takes_input else command(**options)
+
   print(result)

{scripturelookup-0.0.7 → scripturelookup-0.1.0}/src/scripturelookup/data.py

@@ -6,9 +6,8 @@ import time
 import re
 import unicodedata
 
-# Third-party libraries
-import requests
-from bs4 import BeautifulSoup
+# Third-party libraries are imported where they're used, rather than here, so that commands that
+# never reach the network don't pay for loading them. Together they add about 0.12 seconds to startup.
 
 
 data_directory = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'data')
@@ -16,6 +15,8 @@ os.makedirs(data_directory, exist_ok = True)
 
 # Download JSON data
 def download_data(filename, filepath):
+  import requests
+
   request_url = f'https://cdn.jsdelivr.net/gh/samuelbradshaw/python-scripture-scraper@main/sample/{filename}'
   r = requests.get(request_url)
   r.encoding = 'utf-8'
@@ -36,10 +37,12 @@ def load_data(filename):
     download_data(filename, filepath)
     return load_data(filename)
 
-# Update JSON data
+# Update JSON data. Returns the names of the files that were downloaded.
 def update_data():
-  for filename in ('metadata-languages.min.json', 'metadata-scriptures.min.json',):
+  filenames = ('metadata-languages.min.json', 'metadata-scriptures.min.json',)
+  for filename in filenames:
     download_data(filename, os.path.join(data_directory, filename))
+  return filenames
 
 # Normalize text by removing anything that's not a letter or number, and converting to lowercase. This allows for a fuzzy comparison between input text and a known list of values.
 def normalize_for_compare(text):
@@ -47,20 +50,79 @@ def normalize_for_compare(text):
   normalized_text = ''.join([c for c in decomposed_text if unicodedata.category(c)[0] in ['L', 'N']]).lower()
   return normalized_text
 
+# Words that can appear in a scripture reference, by language. Longer words should come before shorter words that they start with, so that (for example) "chapters" is matched before "chapter".
+reference_words = {
+  'en': {
+    'list_conjunctions': ['and', '&'],
+    'range_conjunctions': ['through', 'thru', 'to'],
+    'verse_words': ['verses', 'verse', 'vv.', 'v.'],
+    'chapter_words': ['chapters', 'chapter', 'chs.', 'ch.'],
+  },
+  'es': {
+    'list_conjunctions': ['y', 'e'],
+    'range_conjunctions': ['a', 'al'],
+    'verse_words': ['versículos', 'versículo'],
+    'chapter_words': ['capítulos', 'capítulo'],
+  },
+  'fr': {
+    'list_conjunctions': ['et'],
+    'range_conjunctions': ['à'],
+    'verse_words': ['versets', 'verset'],
+    'chapter_words': ['chapitres', 'chapitre'],
+  },
+  'pt': {
+    'list_conjunctions': ['e'],
+    'range_conjunctions': ['a'],
+    'verse_words': ['versículos', 'versículo'],
+    'chapter_words': ['capítulos', 'capítulo'],
+  },
+}
+reference_word_keys = ('list_conjunctions', 'range_conjunctions', 'verse_words', 'chapter_words',)
+
+# List conjunctions from every language, split into the ones written as a word ("and", "y", "et") and the ones written as a symbol ("&")
+all_list_conjunctions = sorted({c for words in reference_words.values() for c in words['list_conjunctions']})
+list_conjunction_words = [c for c in all_list_conjunctions if any(char.isalpha() for char in c)]
+list_conjunction_symbols = [c for c in all_list_conjunctions if not any(char.isalpha() for char in c)]
+
+# Get alternate spellings of a name where a spelled-out list conjunction is swapped for a symbol, or the other way around. Example: "Doctrine and Covenants" –> "Doctrine & Covenants"
+def get_conjunction_aliases(name):
+  aliases = set()
+  for word in list_conjunction_words:
+    for symbol in list_conjunction_symbols:
+      if f' {word} ' in name:
+        aliases.add(name.replace(f' {word} ', f' {symbol} '))
+      if f' {symbol} ' in name:
+        aliases.add(name.replace(f' {symbol} ', f' {word} '))
+  return aliases
+
+
 languages = load_data('metadata-languages.min.json')
 scriptures = load_data('metadata-scriptures.min.json')
 
-scriptures['mapToSlugNormalized'] = {}
-for key, value in scriptures['mapToSlug'].items():
-  normalized_key = normalize_for_compare(key)
-  scriptures['mapToSlugNormalized'][normalized_key] = value
+# Accept a symbol conjunction ("&") wherever a name spells the conjunction out, and vice versa. The words come from the language data below rather than being hard-coded, so this covers "Doctrine & Covenants" for "Doctrine and Covenants" and "Doctrina & Convenios" for "Doctrina y Convenios".
+for key, value in list(scriptures['mapToSlug'].items()):
+  for alias in get_conjunction_aliases(key):
+    scriptures['mapToSlug'].setdefault(alias, value)
+
+# Get a map of normalized names to book slugs, for comparing input that doesn't match a known name exactly. It's built the first time it's needed, rather than at import, since normalizing all ~10,000 names takes a moment and input that's already well-formed never needs it.
+map_to_slug_normalized = None
+def get_map_to_slug_normalized():
+  global map_to_slug_normalized
+  if map_to_slug_normalized is None:
+    map_to_slug_normalized = {normalize_for_compare(key): value for key, value in scriptures['mapToSlug'].items()}
+  return map_to_slug_normalized
 
-reference_separators_pattern = r'|'.join([re.escape(s.strip()) for s in scriptures['summary']['punctuation']['referenceSeparator']] + [re.escape(';'), re.escape('|'), re.escape('•'), re.escape('\n')])
-chapter_verse_separators_pattern = r'|'.join([re.escape(s.strip()) for s in scriptures['summary']['punctuation']['chapterVerseSeparator']] + [re.escape(':')])
-verse_group_separators_pattern = r'|'.join([re.escape(s.strip()) for s in scriptures['summary']['punctuation']['verseGroupSeparator']] + [re.escape(',')])
-verse_range_separators_pattern = r'|'.join([re.escape(s.strip()) for s in scriptures['summary']['punctuation']['verseRangeSeparator']] + [re.escape('-'), re.escape('–'), re.escape('〜')])
-opening_parenthesis_pattern = r'|'.join([re.escape(s.strip()) for s in scriptures['summary']['punctuation']['openingParenthesis']] + [re.escape('(')])
-closing_parenthesis_pattern = r'|'.join([re.escape(s.strip()) for s in scriptures['summary']['punctuation']['closingParenthesis']] + [re.escape(')')])
+
+# Get regex patterns for the words that can appear in a scripture reference in a given language. Languages that aren't listed above return empty patterns.
+reference_words_cache = {}
+def get_reference_words(lang):
+  if lang not in reference_words_cache:
+    words_for_lang = reference_words.get(lang, {})
+    reference_words_cache[lang] = {
+      key: r'|'.join([re.escape(w) for w in words_for_lang.get(key, [])])
+      for key in reference_word_keys
+    }
+  return reference_words_cache[lang]
 
 
 # Get the BCP 47 language tag for a given language code
@@ -81,6 +143,9 @@ def get_bcp47(lang):
 
 # Get the content for a given chapter verse from python-scripture-scraper or ChurchofJesusChrist.org
 def request_content(publication_slug, book_slug, chapter, verse_groups, church_url, lang = 'en', source = 'python-scripture-scraper'):
+  import requests
+  from bs4 import BeautifulSoup
+
   text_content = ''
   
   if not publication_slug and book_slug and chapter:

{scripturelookup-0.0.7 → scripturelookup-0.1.0}/src/scripturelookup/lookup.py

@@ -6,10 +6,118 @@ import re
 import icu
 
 # Internal imports
-from . import data, numbers
-
+from . import data, numbers, patterns
 
+# Caching dicts
 natural_sort_collators = {}
+book_name_cache = {}
+detection_pattern_cache = {}
+
+
+# Get scripture book names and abbreviations for a language, sorted longest first so that (for example) "1 John" is recognized before "John"
+def get_book_names(lang):
+  if lang not in book_name_cache:
+    scripture_book_names = set()
+    for volume_data in data.scriptures['structure'].values():
+      for book_slug in volume_data['books'].keys():
+        book_info = data.scriptures['languages'][lang]['translatedNames'].get(book_slug)
+        if book_info:
+          book_name = patterns.whitespace.sub(' ', book_info.get('name') or '')
+          if book_name:
+            scripture_book_names.add(book_name)
+          book_abbrev = patterns.whitespace.sub(' ', book_info.get('abbrev') or '')
+          if book_abbrev:
+            scripture_book_names.add(book_abbrev)
+    scripture_book_names = sorted(scripture_book_names, key=lambda x: (-len(x), x))
+
+    # Book names with commas removed, so further normalization doesn't try to split one book name into two references. Example: "JST, Genesis 1" –> "JST Genesis 1"
+    scripture_book_names_without_commas = []
+    scripture_book_names_with_commas = []
+    for scripture_book_name in scripture_book_names:
+      scripture_book_name_without_comma = patterns.verse_group_separators.sub('', scripture_book_name)
+      scripture_book_names_without_commas.append(scripture_book_name_without_comma)
+      if patterns.verse_group_separators.search(scripture_book_name):
+        scripture_book_names_with_commas.append((scripture_book_name, scripture_book_name_without_comma,))
+
+    names_pattern = patterns.build_trie_pattern(scripture_book_names_without_commas)
+    book_name_cache[lang] = {
+      'names_with_commas': scripture_book_names_with_commas,
+      'separate_references_pattern': re.compile(rf'(?:^|[^\-])\b({names_pattern})', flags=re.IGNORECASE),
+    }
+  return book_name_cache[lang]
+
+
+# Study helps aren't scripture references, and their abbreviations ("TG", "BD", "IT", "GS") are too short to tell apart from ordinary text
+slugs_to_skip_when_detecting = ('topical-guide', 'bible-dictionary', 'index-to-the-triple-combination', 'guide-to-the-scriptures', 'joseph-smith-translation',)
+
+# Build a regex alternation that matches anything that can start a scripture reference, allowing flexible whitespace and optional commas. Example: "JST, Genesis" –> "JST(?:,?)\sGenesis"
+def build_book_names_pattern(lang):
+  # Every translated name is included, not just the books in "structure". That picks up publication names ("Doctrine and Covenants"), alternate forms ("Psalm" alongside "Psalms"), and things like "Official Declaration" and "Facsimile", which can all start a reference.
+  names = set()
+  for slug, translated_name in data.scriptures['languages'][lang]['translatedNames'].items():
+    if slug in slugs_to_skip_when_detecting:
+      continue
+    for key in ('name', 'abbrev',):
+      # Whitespace inside the name doesn't need to be normalized here – it's all converted to "\s" below
+      name = translated_name.get(key) or ''
+      if name:
+        names.add(name)
+        names.update(data.get_conjunction_aliases(name))
+
+  # A comma in a name is optional, so both spellings are listed. They have to be separate entries
+  # rather than an optional character, so that every path through the trie consumes the same
+  # characters – otherwise "TJS Génesis" and "TJS, Génesis 1–8" end up on branches that can both
+  # match the same text, and the shorter one wins.
+  for name in list(names):
+    name_without_comma = patterns.verse_group_separators.sub('', name)
+    if name_without_comma != name:
+      names.add(name_without_comma)
+
+  # Whitespace inside a name is matched loosely, so a non-breaking space in the data still matches a
+  # regular space in the text. That mapping is one character for one character, so the trie is safe.
+  normalized_names = [patterns.whitespace.sub(' ', name) for name in names]
+  return patterns.build_trie_pattern(normalized_names, character_patterns = {' ': r'\s'})
+
+
+# Get a compiled regex for detecting scripture references embedded in text
+def get_detection_pattern(lang):
+  if lang not in detection_pattern_cache:
+    words = data.get_reference_words(lang)
+
+    # Words that can be spelled out between two numbers, or between a book name and a number. Examples: "1, 2, and 3"; "Alma chapter 32 verse 21"
+    inline_words = r'|'.join([w for w in (words['list_conjunctions'], words['range_conjunctions'], words['verse_words'], words['chapter_words'],) if w])
+    leading_words = r'|'.join([w for w in (words['verse_words'], words['chapter_words'],) if w])
+
+    books = build_book_names_pattern(lang)
+    anchor = rf'(?P<book>{books})'
+    if words['chapter_words']:
+      anchor += rf'|(?P<chapter_word>{words["chapter_words"]})'
+    anchor_continued = rf'{books}|{words["chapter_words"]}' if words['chapter_words'] else books
+
+    # Separator between two numbers in a reference. A period is one of the possible chapter/verse separators, and it can't be followed by whitespace – without that restriction, "Alma 32. 5 people" would be read as "Alma 32:5". The other chapter/verse separators are unambiguous, so "Mosiah 28: 13" is fine.
+    chapter_verse_separators_allowing_space = r'|'.join([s for s in patterns.chapter_verse_separators_pattern.split(r'|') if s != re.escape('.')])
+    number_separator = rf'(?:{chapter_verse_separators_allowing_space})\s*|(?:{patterns.chapter_verse_separators_pattern})|\s*(?:{patterns.verse_group_separators_pattern}|{patterns.verse_range_separators_pattern})\s*'
+    if inline_words:
+      number_separator = rf'(?:{number_separator})(?:(?:{inline_words})\s+)?|\s+(?:{inline_words})\s+'
+
+    # Chapter and verses, ending on a digit or a closing parenthesis so that trailing whitespace and punctuation are never included in the match
+    leading_word = rf'(?:(?:{leading_words})\s+)?' if leading_words else ''
+    # A number that starts a book name belongs to the next reference, not to this one's verse list. Without this, "Alma 5 and 2 Ne. 2:25" would run together as "Alma 5 and 2".
+    not_the_next_book = rf'(?!(?:{books})\s*\d)'
+
+    # A chapter or verse is never more than three digits, so a longer run of digits – a year, a page number – isn't part of a reference. The lookahead rejects the whole number, rather than matching just its first three digits.
+    number = r'\d{1,3}(?!\d)'
+
+    context = rf'(?:\s*(?:{patterns.opening_parenthesis_pattern})\s*{number}(?:(?:{number_separator}){number})*\s*(?:{patterns.closing_parenthesis_pattern}))?'
+    tail = rf'{leading_word}{number}(?:(?:{number_separator}){not_the_next_book}{number})*{context}'
+
+    # Additional references that continue the same run. A conjunction can follow the separator, as in "Genesis 11:29; 22:23; and 24:15".
+    conjunction_after_separator = rf'(?:(?:{inline_words})\s+)?' if inline_words else ''
+    continued = rf'(?:\s*(?:{patterns.reference_separators_pattern})\s*{conjunction_after_separator}(?:(?:{anchor_continued})\s*)?{tail})*'
+
+    detection_pattern_cache[lang] = re.compile(rf'(?<![-\w])(?:{anchor})\s*{tail}{continued}', flags=re.IGNORECASE)
+  return detection_pattern_cache[lang]
+
 
 class Reference:
   def __init__(self, lang = 'en', publication_slug = None, book_slug = None, chapter = None, verse_groups = [], context_verse_groups = []):
@@ -61,13 +169,13 @@ class Reference:
       
       # Get localized chapter name
       def format_chapter_range(chapter_string):
-        groups = re.split(data.verse_group_separators_pattern, chapter_string)
+        groups = patterns.verse_group_separators.split(chapter_string)
         new_groups = []
         for group in groups:
-          range_parts = re.split(data.verse_range_separators_pattern, group)
+          range_parts = patterns.verse_range_separators.split(group)
           new_range_parts = []
           for range_part in range_parts:
-            chapter_verse_parts = re.split(data.chapter_verse_separators_pattern, range_part)
+            chapter_verse_parts = patterns.chapter_verse_separators.split(range_part)
             new_chapter_verse_parts = []
             for num in chapter_verse_parts:
               new_chapter_verse_parts.append(numbers.get_formatted_number(num, target_lang = self.lang, target_custom_numerals = numerals))
@@ -109,9 +217,9 @@ class Reference:
     
     if uri and self.chapter:
       chapter = str(self.chapter)
-      if re.match(rf'\d+{data.verse_range_separators_pattern}\d+', chapter):
+      if patterns.chapter_range.match(chapter):
         # Chapter range – only use the first chapter
-        chapter = re.split(data.verse_range_separators_pattern, chapter)[0]
+        chapter = patterns.verse_range_separators.split(chapter)[0]
       uri += '/' + str(chapter)
       if self.verse_groups:
         if use_query_parameters:
@@ -174,8 +282,8 @@ def parse_verses_string(verses_string, lang = 'en', range_split_limit = 1):
   
   unique_verses = set()
   all_verses_are_integers = True
-  for verse_group_string in re.split(rf'(?:{data.verse_group_separators_pattern})+', verses_string):
-    verse_strings = re.split(data.verse_range_separators_pattern, verse_group_string)
+  for verse_group_string in patterns.verse_group_separators_repeated.split(verses_string):
+    verse_strings = patterns.verse_range_separators.split(verse_group_string)
     lower_int = numbers.convert_number_to_int(verse_strings[0])
     upper_int = numbers.convert_number_to_int(verse_strings[-1])
     
@@ -257,66 +365,61 @@ def parse_references_string(input_string, lang = 'en', sort_by = None, skip_clea
     input_string = input_string.strip().strip(punctuation_to_strip).rstrip(':').strip()
     
     # If language is English, replace roman numerals with numbers. Example: 'II Corinthians" –> "2 Corinthians"
-    if lang == 'en':
-      input_string = re.sub(r'\bi\s', '1', input_string, flags=re.IGNORECASE)
-      input_string = re.sub(r'\bii\s', '2', input_string, flags=re.IGNORECASE)
-      input_string = re.sub(r'\biii\s', '3', input_string, flags=re.IGNORECASE)
-      input_string = re.sub(r'\biv\s', '4', input_string, flags=re.IGNORECASE)
+    # The substitutions have to run in order, since replacing "i " with "1" can join it to the text that follows, so they're guarded by a single search rather than combined into one pass.
+    if lang == 'en' and patterns.any_roman_numeral.search(input_string):
+      for roman_numeral_pattern, arabic_numeral in patterns.roman_numerals:
+        input_string = roman_numeral_pattern.sub(arabic_numeral, input_string)
     
+    # Normalize whitespace so it matches the spaces in book names, but keep line breaks, since they separate one reference from the next
+    input_string = patterns.whitespace_except_line_breaks.sub(' ', input_string)
+
     # Remove commas from book names so further normalization doesn't try to split it into two references. Example: "JST, Genesis 1" –> "JST Genesis 1"
-    input_string = input_string.replace('\xa0', ' ')
-    scripture_book_names = set()
-    scripture_book_names_without_commas = []
-    for volume_data in data.scriptures['structure'].values():
-      for book_slug in volume_data['books'].keys():
-        book_info = data.scriptures['languages'][lang]['translatedNames'].get(book_slug)
-        if book_info:
-          book_name = (book_info.get('name') or '').replace('\xa0', ' ')
-          if book_name:
-            scripture_book_names.add(book_name)
-          book_abbrev = (book_info.get('abbrev') or '').replace('\xa0', ' ')
-          if book_abbrev:
-            scripture_book_names.add(book_abbrev)
-    scripture_book_names = sorted(scripture_book_names, key=lambda x: (-len(x), x))
-    for scripture_book_name in scripture_book_names:
-      scripture_book_name_without_comma = re.sub(data.verse_group_separators_pattern, '', scripture_book_name)
-      scripture_book_names_without_commas.append(scripture_book_name_without_comma)
-      if scripture_book_name in input_string and re.search(data.verse_group_separators_pattern, scripture_book_name):
+    book_names = get_book_names(lang)
+    for scripture_book_name, scripture_book_name_without_comma in book_names['names_with_commas']:
+      if scripture_book_name in input_string:
         input_string = input_string.replace(scripture_book_name, scripture_book_name_without_comma)
-    
+
     # Normalize whitespace-separated references. Example: "Genesis 1:2 1 Nephi 3:7" –> "; Genesis 1:2 ; 1 Nephi 3:7"
-    scripture_book_names_pattern = '|'.join([re.escape(sbn) for sbn in scripture_book_names_without_commas])
-    input_string = re.sub(rf'(?:^|[^\-])\b({scripture_book_names_pattern})', r'; \1', input_string, flags=re.IGNORECASE)
-    
+    input_string = book_names['separate_references_pattern'].sub(r'; \1', input_string)
+
+    normalization_patterns = patterns.get_normalization_patterns(lang)
+
     # Normalize lists and ranges. Example: "Genesis 12:1, 2, and 3; verses 1 and 4; John 2 through 7" –> "Genesis 12:1, 2,,3; verses 1,4; John 2–7"
-    input_string = re.sub(r'\s+(?:and|y|e|et|&)\s+(\d+)', r',\1', input_string)
-    input_string = re.sub(r'\s+(?:through|thru|to|al|a|à)\s+(\d+)', r'–\1', input_string)
-    
+    if normalization_patterns['list_conjunctions']:
+      input_string = normalization_patterns['list_conjunctions'].sub(r',\1', input_string)
+    if normalization_patterns['range_conjunctions']:
+      input_string = normalization_patterns['range_conjunctions'].sub(r'–\1', input_string)
+
     # Normalize verse sets. Example: "chapter 3 verse 7; vv. 3, 6" –> "chapter 3:7; :3, 6"
-    input_string = re.sub(r'(?:^|\s)(?:verses|verse|vv\.|v\.|versículos|versículo|versets|verset)\s(\d+)', r':\1', input_string).replace('::', ':')
-    
+    if normalization_patterns['verse_words']:
+      input_string = normalization_patterns['verse_words'].sub(r':\1', input_string).replace('::', ':')
+
+    # Normalize chapter words. Example: "Alma chapter 32 verse 21" –> "Alma 32:21"
+    if normalization_patterns['chapter_words']:
+      input_string = normalization_patterns['chapter_words'].sub(r' \1', input_string)
+
     # Normalize chapter sets. Example: "Genesis 1, 2, 4–5, Exodus 10; Alma 32" –> "Genesis 1; 2; 4–5; Exodus 10; Alma 32"
-    if re.search(data.verse_group_separators_pattern, input_string) and not re.search(data.chapter_verse_separators_pattern, input_string):
-      input_string = re.sub(rf'(?:{data.verse_group_separators_pattern})+', ';', input_string)
+    if patterns.verse_group_separators.search(input_string) and not patterns.chapter_verse_separators.search(input_string):
+      input_string = patterns.verse_group_separators_repeated.sub(';', input_string)
     
     # Normalize chapter:verse sets. Example: "Genesis 6:7a, 6:13a, 15; 1 Nephi 3:7 (twice), 8:21" –> "Genesis 6:7a; 6:13a, 15; 1 Nephi 3:7 (twice); 8:21"
-    if re.search(data.chapter_verse_separators_pattern, input_string):
-      references_list = re.split(data.reference_separators_pattern, input_string)
+    if patterns.chapter_verse_separators.search(input_string):
+      references_list = patterns.reference_separators.split(input_string)
       new_references_list = []
       for reference in references_list:
-        reference_parts = re.split(data.verse_group_separators_pattern, reference)
+        reference_parts = patterns.verse_group_separators.split(reference)
         reference_input_string = ''
         for part in reference_parts:
           if reference_input_string == '':
             reference_input_string += part
-          elif re.search(data.chapter_verse_separators_pattern, part):
+          elif patterns.chapter_verse_separators.search(part):
             reference_input_string += ';' + part
           else:
             reference_input_string += ',' + part
         new_references_list.append(reference_input_string)
       input_string = ';'.join(new_references_list)
   
-  input_list = re.split(data.reference_separators_pattern, input_string)
+  input_list = patterns.reference_separators.split(input_string)
   
   references = []
   previous_book_slug = None
@@ -329,7 +432,7 @@ def parse_references_string(input_string, lang = 'en', sort_by = None, skip_clea
     
     if not skip_cleanup:
       # Remove trailing text. Example: "1 John 3:2 2" –> "1 John 3:2"
-      trailing_text_match = re.match(rf'^.*?\d((?:\:|{data.closing_parenthesis_pattern})?\s+[^{data.opening_parenthesis_pattern}|\s]+)$', input_string)
+      trailing_text_match = patterns.trailing_text.match(input_string)
       if trailing_text_match:
         trailing_text_string = trailing_text_match.group(1)
         input_string = input_string.removesuffix(trailing_text_string)
@@ -376,8 +479,14 @@ def parse_references_string(input_string, lang = 'en', sort_by = None, skip_clea
       # Scripture reference or slug
       # Examples: Old Testament; 1 Nephi; Matthew 1; Helaman 5:12; words-of-mormon
       
-      # Get verses string
-      parts = re.split(data.chapter_verse_separators_pattern, input_string)
+      # Get context verses from a trailing parenthetical, so it doesn't interfere with parsing the rest of the reference. Example: "Gen. 1:3 (3–4)"
+      context_match = patterns.trailing_parenthetical.match(input_string)
+      if context_match and patterns.numbers_and_separators.match(context_match.group(2)):
+        input_string = context_match.group(1)
+        context_verses_string = context_match.group(2)
+
+      # Get verses string. The separator has to sit between two digits, so that the period in an abbreviation like "Gen." isn't mistaken for a chapter/verse separator.
+      parts = patterns.chapter_verse_separator_between_digits.split(input_string)
       if len(parts) == 2:
         # Regular chapter and verse found
         unparsed, verses_string = parts
@@ -385,11 +494,13 @@ def parse_references_string(input_string, lang = 'en', sort_by = None, skip_clea
         # Chapter only, or special case like 'Genesis 7:17–8:9' or 'Genesis 1–5'
         unparsed = input_string
         verses_string = ''
-      verses_string, context_verses_string = (re.split(data.opening_parenthesis_pattern, re.sub(data.closing_parenthesis_pattern, '', verses_string)) + [''])[:2]
-      
+      verses_string, remaining_context_verses_string = (patterns.opening_parenthesis.split(patterns.closing_parenthesis.sub('', verses_string)) + [''])[:2]
+      if context_verses_string is None:
+        context_verses_string = remaining_context_verses_string
+
       # Get chapter string and book string
       book_string = unparsed
-      chapter_match = re.match(rf'^.*?(\d(?:\d|\s|{data.chapter_verse_separators_pattern}|{data.verse_range_separators_pattern}|{data.verse_group_separators_pattern})*)$', book_string)
+      chapter_match = patterns.trailing_chapter.match(book_string)
       if chapter_match:
         chapter_string = chapter_match.group(1)
         book_string = book_string.removesuffix(chapter_string).strip()
@@ -400,7 +511,7 @@ def parse_references_string(input_string, lang = 'en', sort_by = None, skip_clea
     book_slug = None
     skip_book_name = False
     if book_string:
-      book_slug = data.scriptures['mapToSlug'].get(book_string, None) or data.scriptures['mapToSlugNormalized'].get(data.normalize_for_compare(book_string), None)
+      book_slug = data.scriptures['mapToSlug'].get(book_string, None) or data.get_map_to_slug_normalized().get(data.normalize_for_compare(book_string), None)
       # Special handling for Abraham facsimiles
       if book_slug == 'facsimiles' or (not book_slug and 'fac' in book_string.lower()):
         if previous_book_slug == 'abraham' and not previous_chapter:
@@ -456,7 +567,7 @@ def get_label(input_string, lang = 'en', separator = '\n', sort_by = None, skip_
   references = parse_references_string(input_string, lang = lang, sort_by = sort_by, skip_cleanup = skip_cleanup, range_split_limit = range_split_limit)
   return separator.join([ref.label(skip_book_name = skip_book_name, abbreviated = abbreviated) for ref in references])
 
-def get_church_uri(input_string, separator = '\n', sort_by = None, use_query_parameters = False, skip_cleanup = False, range_split_limit = 1, **kwargs):
+def get_church_uri(input_string, lang = 'en', separator = '\n', sort_by = None, use_query_parameters = False, skip_cleanup = False, range_split_limit = 1, **kwargs):
   references = parse_references_string(input_string, lang = lang, sort_by = sort_by, skip_cleanup = skip_cleanup, range_split_limit = range_split_limit)
   return separator.join([ref.church_uri(use_query_parameters = use_query_parameters) for ref in references])
 
@@ -468,6 +579,28 @@ def get_church_link(input_string, lang = 'en', separator = '\n', sort_by = None,
   references = parse_references_string(input_string, lang = lang, sort_by = sort_by, skip_cleanup = skip_cleanup, range_split_limit = range_split_limit)
   return separator.join([ref.church_link(link_class = link_class, link_target = link_target, skip_book_name = skip_book_name, abbreviated = abbreviated, skip_lang = skip_lang, skip_fragment = skip_fragment) for ref in references])
   
+def detect_references(input_string, lang = 'en', range_split_limit = 1, **kwargs):
+  # Every reference includes a chapter number, so text with no digits in it can be skipped without
+  # building the language's detection pattern or scanning for book names.
+  if not input_string or not patterns.any_digit.search(input_string):
+    return []
+
+  lang = data.get_bcp47(lang)
+
+  # Replace newlines and other whitespace with regular spaces, one character for one character, so offsets stay valid in the original string
+  working_string = patterns.whitespace.sub(' ', input_string)
+
+  detections = []
+  for match in get_detection_pattern(lang).finditer(working_string):
+    references = parse_references_string(match.group(0), lang = lang, range_split_limit = range_split_limit)
+    if match.group('book') and not any(ref.book_slug or ref.publication_slug for ref in references):
+      # Looked like a book name, but it didn't resolve to a known book
+      continue
+    if not any(ref.chapter for ref in references):
+      continue
+    detections.append([input_string[match.start():match.end()], match.start(), match.end()])
+  return detections
+
 def get_reference_objects(input_string, lang = 'en', sort_by = None, skip_cleanup = False, range_split_limit = 1, **kwargs):
   return parse_references_string(input_string, lang = lang, sort_by = sort_by, skip_cleanup = skip_cleanup, range_split_limit = range_split_limit)
 
@@ -475,6 +608,11 @@ def get_reference_attributes(input_string, lang = 'en', sort_by = None, skip_cle
   references = parse_references_string(input_string, lang = lang, sort_by = sort_by, skip_cleanup = skip_cleanup, range_split_limit = range_split_limit)
   return [ref.attributes() for ref in references]
 
+# Download the latest scripture and language metadata. Metadata that's already been loaded stays in memory, so the new data is used from the next run onward.
+def refresh_metadata(**kwargs):
+  filenames = data.update_data()
+  return 'Updated metadata: ' + ', '.join(filenames)
+
 def get_langs(**kwargs):
   return data.scriptures['languages'].keys()
 

scripturelookup-0.1.0/src/scripturelookup/patterns.py

@@ -0,0 +1,132 @@
+# Regex patterns used for parsing and detecting scripture references.
+#
+# Patterns ending in "_pattern" are strings, since they get interpolated into larger patterns.
+# Everything else is compiled once here, rather than being rebuilt every time a reference is parsed.
+
+# Python standard libraries
+import re
+
+# Internal imports
+from . import data
+
+
+# Separators and parentheses, gathered from the punctuation used across all languages
+reference_separators_pattern = r'|'.join([re.escape(s.strip()) for s in data.scriptures['summary']['punctuation']['referenceSeparator']] + [re.escape(';'), re.escape('|'), re.escape('•'), re.escape('\n')])
+chapter_verse_separators_pattern = r'|'.join([re.escape(s.strip()) for s in data.scriptures['summary']['punctuation']['chapterVerseSeparator']] + [re.escape(':'), re.escape('.')])
+verse_group_separators_pattern = r'|'.join([re.escape(s.strip()) for s in data.scriptures['summary']['punctuation']['verseGroupSeparator']] + [re.escape(',')])
+verse_range_separators_pattern = r'|'.join([re.escape(s.strip()) for s in data.scriptures['summary']['punctuation']['verseRangeSeparator']] + [re.escape('-'), re.escape('–'), re.escape('〜'), re.escape('~')])
+opening_parenthesis_pattern = r'|'.join([re.escape(s.strip()) for s in data.scriptures['summary']['punctuation']['openingParenthesis']] + [re.escape('(')])
+closing_parenthesis_pattern = r'|'.join([re.escape(s.strip()) for s in data.scriptures['summary']['punctuation']['closingParenthesis']] + [re.escape(')')])
+
+# Any Unicode whitespace character – newlines, tabs, non-breaking spaces, ideographic spaces, and so on
+whitespace = re.compile(r'\s')
+# The same, except line breaks, which separate one reference from the next
+whitespace_except_line_breaks = re.compile(r'[^\S\n]')
+
+# Separators between references, chapters, verses, and verse groups
+reference_separators = re.compile(reference_separators_pattern)
+chapter_verse_separators = re.compile(chapter_verse_separators_pattern)
+verse_group_separators = re.compile(verse_group_separators_pattern)
+verse_range_separators = re.compile(verse_range_separators_pattern)
+verse_group_separators_repeated = re.compile(rf'(?:{verse_group_separators_pattern})+')
+opening_parenthesis = re.compile(opening_parenthesis_pattern)
+closing_parenthesis = re.compile(closing_parenthesis_pattern)
+
+# A chapter/verse separator that sits between two digits, so the period in an abbreviation like "Gen." isn't mistaken for one
+chapter_verse_separator_between_digits = re.compile(rf'(?<=\d)(?:{chapter_verse_separators_pattern})(?=\d)')
+# A range of chapters. Example: "1–5"
+chapter_range = re.compile(rf'\d+(?:{verse_range_separators_pattern})\d+')
+
+# Anything that can follow the first digit of a chapter or verse: another digit, whitespace, or any separator
+number_continuation_pattern = rf'\d|\s|{chapter_verse_separators_pattern}|{verse_range_separators_pattern}|{verse_group_separators_pattern}'
+# A string made up entirely of numbers and separators. Example: "3, 5–7"
+numbers_and_separators = re.compile(rf'^\d(?:{number_continuation_pattern})*$')
+# The chapter and verses at the end of a reference, so the book name can be split off the front
+trailing_chapter = re.compile(rf'^.*?(\d(?:{number_continuation_pattern})*)$')
+# Text after the end of a reference. Example: "1 John 3:2 2" –> " 2"
+trailing_text = re.compile(rf'^.*?\d((?:\:|{closing_parenthesis_pattern})?\s+[^{opening_parenthesis_pattern}|\s]+)$')
+# A parenthetical at the end of a reference, which holds context verses. Example: "Gen. 1:3 (3–4)"
+trailing_parenthetical = re.compile(rf'^(.*?)\s*(?:{opening_parenthesis_pattern})([^)）]*)(?:{closing_parenthesis_pattern})$')
+
+# English roman numeral prefixes, paired with the number they stand for. Example: "II Corinthians" –> "2 Corinthians"
+roman_numerals = [
+  (re.compile(r'\bi\s', flags=re.IGNORECASE), '1'),
+  (re.compile(r'\bii\s', flags=re.IGNORECASE), '2'),
+  (re.compile(r'\biii\s', flags=re.IGNORECASE), '3'),
+  (re.compile(r'\biv\s', flags=re.IGNORECASE), '4'),
+]
+# Matches any of the prefixes above, so a string with no roman numeral at all – which is most of them – can skip the substitutions entirely
+any_roman_numeral = re.compile(r'\b(?:i|ii|iii|iv)\s', flags=re.IGNORECASE)
+
+# A single digit. Every reference needs a chapter number, so text with no digits anywhere can't contain one.
+any_digit = re.compile(r'\d')
+
+
+# Build a regex alternation that matches any of the given words, sharing common prefixes so the regex
+# engine doesn't retry every word at every position. Example: ["Genesis", "Genes"] –> "Gene(?:sis|s)".
+# Shared prefixes make this several times faster than a flat "Genesis|Genes" alternation, which the
+# engine has to walk one branch at a time. Longer words still win over shorter ones that they start
+# with, because the trailing group is greedy.
+#
+# Each character can be given its own sub-pattern, for names where a character shouldn't be matched
+# literally – see character_patterns in build_book_names_pattern.
+def build_trie_pattern(words, character_patterns = None, case_insensitive = True):
+  character_patterns = character_patterns or {}
+
+  # Characters that differ only by case have to share a trie node, since the pattern is meant to be
+  # compiled with re.IGNORECASE. Without this, "OP" and "Opisyal" would sit in sibling branches, and
+  # "OP" would win on the input "Opisyal" – a flat alternation avoids that by sorting longest first.
+  def trie_key(character):
+    lowercased = character.lower()
+    return lowercased if case_insensitive and len(lowercased) == 1 else character
+
+  trie = {}
+  for word in words:
+    node = trie
+    for character in word:
+      node = node.setdefault(trie_key(character), {})
+    node[''] = {}
+
+  def build_branch(node):
+    # A node with nothing but an end marker is the end of a word
+    if '' in node and len(node) == 1:
+      return None
+
+    branches = []
+    word_ends_here = False
+    for character, child_node in sorted(node.items()):
+      if character == '':
+        word_ends_here = True
+        continue
+      remainder = build_branch(child_node)
+      character_pattern = character_patterns.get(character) or re.escape(character)
+      branches.append(character_pattern + (f'(?:{remainder})' if remainder and len(remainder) > 1 else (remainder or '')))
+
+    branch_pattern = r'|'.join(branches)
+    return f'(?:{branch_pattern})?' if word_ends_here else branch_pattern
+
+  return build_branch(trie)
+
+
+# Get compiled regexes for normalizing the words that can appear in a reference in a given language
+normalization_pattern_cache = {}
+def get_normalization_patterns(lang):
+  if lang not in normalization_pattern_cache:
+    words = data.get_reference_words(lang)
+
+    # A conjunction sits between two numbers, so it needs whitespace on both sides. A verse or chapter word can also start the string.
+    conjunction_template = r'\s+(?:{words})\s+(\d+)'
+    reference_word_template = r'(?:^|\s)(?:{words})\s+(\d+)'
+    templates = {
+      'list_conjunctions': conjunction_template,
+      'range_conjunctions': conjunction_template,
+      'verse_words': reference_word_template,
+      'chapter_words': reference_word_template,
+    }
+
+    normalization_patterns = {}
+    for key in data.reference_word_keys:
+      template = templates[key]
+      normalization_patterns[key] = re.compile(template.format(words = words[key]), flags=re.IGNORECASE) if words[key] else None
+    normalization_pattern_cache[lang] = normalization_patterns
+  return normalization_pattern_cache[lang]

{scripturelookup-0.0.7 → scripturelookup-0.1.0/src/scripturelookup.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scripturelookup
-Version: 0.0.7
+Version: 0.1.0
 Summary: Python and command-line utility for converting scripture references between formats.
 Author-email: Samuel Bradshaw <samuel.h.bradshaw@gmail.com>
 License: MIT License
@@ -40,7 +40,7 @@ Dynamic: license-file
 
 # Scripture Lookup
 
-Scripture Lookup is a Python library and command-line tool for looking up scripture verses or chapters. It can also convert scripture references between formats and languages. The tool is built around data from [Python Scripture Scraper](https://github.com/samuelbradshaw/python-scripture-scraper).
+Scripture Lookup is a Python library and command-line tool for looking up scripture verses or chapters. It can also detect scripture references in a string, and convert them between formats and languages. The tool uses scripture and language metadata from [Python Scripture Scraper](https://github.com/samuelbradshaw/python-scripture-scraper).
 
 
 ## Installation
@@ -79,6 +79,12 @@ https://www.churchofjesuschrist.org/study/scriptures/bofm/hel/5?id=p12&lang=eng#
 
 % scripturelookup get_church_url "/scriptures/ot" --lang "es"
 https://www.churchofjesuschrist.org/study/scriptures/ot?lang=spa
+
+% scripturelookup detect_references "As Nephi taught in 1 Ne. 3:7, and again in 2 Nephi 2:25."
+[['1 Ne. 3:7', 19, 28], ['2 Nephi 2:25', 43, 55]]
+
+% scripturelookup refresh_metadata
+Updated metadata: metadata-languages.min.json, metadata-scriptures.min.json
 ```
 
 
@@ -102,6 +108,12 @@ lookup.get_label('/scriptures/nt/1-jn/3.2-3', lang = 'cmn-Hant')
 
 lookup.get_church_url('/scriptures/ot', lang = 'es')
 # https://www.churchofjesuschrist.org/study/scriptures/ot?lang=spa
+
+lookup.detect_references('As Nephi taught in 1 Ne. 3:7, and again in 2 Nephi 2:25.')
+# [['1 Ne. 3:7', 19, 28], ['2 Nephi 2:25', 43, 55]]
+
+lookup.refresh_metadata()
+# Updated metadata: metadata-languages.min.json, metadata-scriptures.min.json
 ```
 
 
@@ -117,6 +129,8 @@ lookup.get_church_url('/scriptures/ot', lang = 'es')
 - **get_reference_objects** – Get a list of references as objects.
 - **get_reference_attributes** – Get a list of references as dictionaries.
 - **sort_references** – Sort a list of references by label or in traditional book order.
+- **detect_references** – Find scripture references embedded in a string of text.
+- **refresh_metadata** – Download the latest scripture and language metadata. Takes no input.
 
 ### Inputs
 
@@ -145,6 +159,8 @@ Any of the following input types are supported. You can also provide several inp
   - http://lds.org/scriptures/bofm/1-ne/3.7?lang=eng
   - [gospellibrary://content/scriptures/nt/john/3.16?lang=eng#16](https://www.churchofjesuschrist.org/study/scriptures/nt/john/3?id=p16&lang=eng#p16)
 
+`detect_references` takes an arbitrary text string and returns a list of `[label, start_offset, end_offset]`. The label is the exact detected text (not normalized). Start and end offset indicate the exclusive range of characters in the input string where the reference was found.
+
 ### Options
 
 Several options are available. Some are only applicable to certain commands.

{scripturelookup-0.0.7 → scripturelookup-0.1.0}/src/scripturelookup.egg-info/SOURCES.txt

@@ -16,11 +16,15 @@ src/scripturelookup/command_line.py
 src/scripturelookup/data.py
 src/scripturelookup/lookup.py
 src/scripturelookup/numbers.py
+src/scripturelookup/patterns.py
 src/scripturelookup.egg-info/PKG-INFO
 src/scripturelookup.egg-info/SOURCES.txt
 src/scripturelookup.egg-info/dependency_links.txt
 src/scripturelookup.egg-info/entry_points.txt
 src/scripturelookup.egg-info/requires.txt
+src/scripturelookup.egg-info/scm_file_list.json
+src/scripturelookup.egg-info/scm_version.json
 src/scripturelookup.egg-info/top_level.txt
 src/scripturelookup/data/metadata-languages.min.json
-src/scripturelookup/data/metadata-scriptures.min.json
+src/scripturelookup/data/metadata-scriptures.min.json
+tests/test_detect.py

scripturelookup-0.1.0/src/scripturelookup.egg-info/scm_file_list.json

@@ -0,0 +1,26 @@
+{
+  "files": [
+    ".github/workflows/publish.yml",
+    ".gitignore",
+    "LICENSE",
+    "README.md",
+    "pyproject.toml",
+    "requirements.txt",
+    "src/geezify-python-main/LICENSE",
+    "src/geezify-python-main/README.md",
+    "src/geezify-python-main/arabify.py",
+    "src/geezify-python-main/arabify_test.py",
+    "src/geezify-python-main/geezify.py",
+    "src/geezify-python-main/geezify_test.py",
+    "src/geezify-python-main/test_data.py",
+    "src/scripturelookup/__init__.py",
+    "src/scripturelookup/command_line.py",
+    "src/scripturelookup/data.py",
+    "src/scripturelookup/data/metadata-languages.min.json",
+    "src/scripturelookup/data/metadata-scriptures.min.json",
+    "src/scripturelookup/lookup.py",
+    "src/scripturelookup/numbers.py",
+    "src/scripturelookup/patterns.py",
+    "tests/test_detect.py"
+  ]
+}

scripturelookup-0.1.0/src/scripturelookup.egg-info/scm_version.json

@@ -0,0 +1,8 @@
+{
+  "tag": "0.1.0",
+  "distance": 0,
+  "node": "ga8b30f8a91100afe8fd53885f129ff7f22d83e70",
+  "dirty": false,
+  "branch": "HEAD",
+  "node_date": "2026-08-28"
+}

scripturelookup-0.1.0/tests/test_detect.py

@@ -0,0 +1,137 @@
+# Tests for lookup.detect_references
+#
+# Run with:
+#   python3 -m pytest tests
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.join(os.path.abspath(os.path.dirname(__file__)), '..', 'src'))
+from scripturelookup import lookup
+
+
+# (input text, language, expected labels in order)
+detection_cases = [
+  # Nothing to find
+  ('', 'en', []),
+  ('Nothing here at all.', 'en', []),
+
+  # A run of related references stays a single detection
+  ('See 1 Nephi 3:7, 5:2; Alma 5 for details.', 'en', ['1 Nephi 3:7, 5:2; Alma 5']),
+  ('Doctrine and Covenants 45:56-59 | 1 Nephi 22:24-28', 'en', ['Doctrine and Covenants 45:56-59 | 1 Nephi 22:24-28']),
+  ('Doctrine & Covenants 45:56 • 1 Nephi 22:24', 'en', ['Doctrine & Covenants 45:56 • 1 Nephi 22:24']),
+
+  # "&" stands in for a spelled-out list conjunction in any language, not just English
+  ('Vea Doctrina & Convenios 45:56 aquí.', 'es', ['Doctrina & Convenios 45:56']),
+  ('Vea Doctrina y Convenios 45:56 aquí.', 'es', ['Doctrina y Convenios 45:56']),
+
+  # Abbreviations
+  ('In 2 Ne. 2:25 we read.', 'en', ['2 Ne. 2:25']),
+  ('Compare Matt. 5:3 today.', 'en', ['Matt. 5:3']),
+
+  # A number is required, so ordinary words that are also book names are skipped
+  ('He found a job in Job 5.', 'en', ['Job 5']),
+  ('Read the Book of Mormon.', 'en', []),
+
+  # Trailing whitespace and unbalanced parentheses are never part of a match
+  ('Read Genesis 1-3 (see also Moses 2).', 'en', ['Genesis 1-3', 'Moses 2']),
+  ('Gen. 1:3 (3-4) is nice.', 'en', ['Gen. 1:3 (3-4)']),
+
+  # A period is a chapter/verse separator in some languages, but only between digits
+  ('Alma 32. 5 people came.', 'en', ['Alma 32']),
+
+  # A chapter or verse is never more than three digits, so years and page numbers aren't references.
+  # The whole number is rejected rather than being truncated to its first three digits.
+  ('Psalm 119:176 is the last verse.', 'en', ['Psalm 119:176']),
+  ('D&C 124:123-45 was given.', 'en', ['D&C 124:123-45']),
+  ('Alma 1978 was a year.', 'en', []),
+  ('Alma 12345 xyz', 'en', []),
+  ('Alma 32:21 was quoted in 1978.', 'en', ['Alma 32:21']),
+
+  # Chapter words can stand in for a book name; verse words cannot
+  ('See chapter 3 and Alma chapter 32 verse 21.', 'en', ['chapter 3', 'Alma chapter 32 verse 21']),
+  ('See verses 3-5 below.', 'en', []),
+
+  # Bare chapter:verse with no preceding anchor is not a reference
+  ('As in Alma 32. See also 3:7.', 'en', ['Alma 32']),
+  ('Meet at 3:15 tomorrow.', 'en', []),
+
+  # Spelled-out lists and ranges
+  ('Genesis 12:1, 2, and 3 plus John 2 through 7.', 'en', ['Genesis 12:1, 2, and 3', 'John 2 through 7']),
+
+  # A number that starts the next book name isn't pulled into this reference's verse list
+  ('See 1 Nephi 3:7, 5:2; Alma 5 and 2 Ne. 2:25 in chapter 9.', 'en', ['1 Nephi 3:7, 5:2; Alma 5', '2 Ne. 2:25', 'chapter 9']),
+  ('Alma 5, 2 Ne. 2:25 today.', 'en', ['Alma 5', '2 Ne. 2:25']),
+  ('Read 1 Nephi 3 and 3 John 1:4.', 'en', ['1 Nephi 3', '3 John 1:4']),
+
+  # A conjunction can follow a reference separator
+  ('mentioned in Genesis 11:29; 22:23; and 24:15 as well as in Abraham 2:2.', 'en', ['Genesis 11:29; 22:23; and 24:15', 'Abraham 2:2']),
+
+  # A colon may be followed by a space, but a period may not (a period is also a chapter/verse separator)
+  ('Mosiah 28: 13-15 is next.', 'en', ['Mosiah 28: 13-15']),
+  ('Genesis 1.3 and Abr. 3.22-23 here.', 'en', ['Genesis 1.3', 'Abr. 3.22-23']),
+  ('It cost 1.50 today.', 'en', []),
+
+  # Names that aren't in the "structure" book list
+  ('See Psalm 23 and Official Declaration 2 and Facsimile 2.', 'en', ['Psalm 23', 'Official Declaration 2', 'Facsimile 2']),
+
+  # Line-wrapped references are found, and the label keeps the original whitespace
+  ('Alma\n32:21 was quoted.', 'en', ['Alma\n32:21']),
+  ('Alma\r\n32:21 was quoted.', 'en', ['Alma\r\n32:21']),
+  ('See Alma\n  32:21, 22 here.', 'en', ['Alma\n  32:21, 22']),
+
+  # Any kind of Unicode whitespace works, not just a regular space
+  ('See 1 Samuel 3:1 now.', 'en', ['1 Samuel 3:1']),
+  ('See 1\xa0Samuel 3:1 now.', 'en', ['1\xa0Samuel 3:1']),
+  ('See Alma　32:21 now.', 'en', ['Alma　32:21']),
+
+  # Reference words are language-specific. German has no seeded word list, so "capítulo 3" isn't a
+  # reference there, but "Alma" is spelled the same in German and Spanish and is still detected.
+  ('Consulte el capítulo 3 y Alma 32:21 aquí.', 'es', ['capítulo 3', 'Alma 32:21']),
+  ('Consulte el capítulo 3 y Alma 32:21 aquí.', 'de', ['Alma 32:21']),
+
+  # Book names are language-specific too
+  ('Siehe Alma 32:21 hier.', 'de', ['Alma 32:21']),
+  ('Siehe Alma 32:21 hier.', 'ko', []),
+]
+
+
+@pytest.mark.parametrize('text, lang, expected_labels', detection_cases)
+def test_detect_references(text, lang, expected_labels):
+  detections = lookup.detect_references(text, lang = lang)
+  assert [label for label, start, end in detections] == expected_labels
+
+
+@pytest.mark.parametrize('text, lang, expected_labels', detection_cases)
+def test_offsets_match_the_original_string(text, lang, expected_labels):
+  for label, start, end in lookup.detect_references(text, lang = lang):
+    assert text[start:end] == label
+
+
+@pytest.mark.parametrize('text, lang, expected_labels', detection_cases)
+def test_detections_do_not_overlap(text, lang, expected_labels):
+  previous_end = 0
+  for label, start, end in lookup.detect_references(text, lang = lang):
+    assert start >= previous_end
+    assert start < end
+    previous_end = end
+
+
+def test_detected_labels_can_be_parsed_back():
+  text = 'See 1 Nephi 3:7 and Mosiah 2:17 for details.'
+  labels = [lookup.get_label(label) for label, start, end in lookup.detect_references(text)]
+  assert labels == ['1\xa0Nephi\xa03:7', 'Mosiah\xa02:17']
+
+
+# Parsing (not detection) treats a line break as a separator between references, so whitespace
+# normalization there has to leave line breaks alone.
+def test_line_breaks_still_separate_references_when_parsing():
+  assert lookup.get_label('Alma 5\nMosiah 2:17') == 'Alma\xa05\nMosiah\xa02:17'
+  assert lookup.get_label('Alma 5\r\nMosiah 2:17') == 'Alma\xa05\nMosiah\xa02:17'
+
+
+@pytest.mark.parametrize('space', [' ', '\xa0', ' ', ' '])
+def test_book_names_match_any_kind_of_space(space):
+  assert lookup.get_label(f'1{space}Samuel 3:1') == '1\xa0Samuel\xa03:1'