usfm-grammar-web 3.2.0 → 3.2.2

package/README.md

@@ -3,6 +3,8 @@
 ## Description
 This is the web alternative to the (USFM-Grammar 3.x)[https://www.npmjs.com/package/usfm-grammar] library to be used from HTML, react etc. USFM Grammar is a JavaScript library for parsing and converting USFM (Unified Standard Format Markers) to/from USJ (Unified Standard JSON) format. This library provides functionalities to parse USFM strings into a syntax tree and convert them into a JSON-like structure (USJ), and vice versa.
 
+> :information_source: For Node.js, use the [usfm-grammar](https://www.npmjs.com/package/usfm-grammar) package; `usfm-grammar-web` is for browsers and bundlers.
+
 ## Installation
 You can install USFM Grammar via npm:
 
@@ -23,10 +25,10 @@ function App() {
 
   useEffect(() => {
     const initParser = async () => {
-      await USFMParser.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter-usfm.wasm",
-                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter.wasm");
-      await Validator.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter-usfm.wasm",
-                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter.wasm");
+      await USFMParser.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/tree-sitter-usfm.wasm",
+                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/web-tree-sitter.wasm");
+      await Validator.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/tree-sitter-usfm.wasm",
+                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/web-tree-sitter.wasm");
 
     };
     initParser();
@@ -60,13 +62,13 @@ It can be used directly in the HTML script tag too. Please ensure its dependenci
 
 ```html
 <script type="module">
-  import { USFMParser, Filter, Validator } from 'https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/dist/bundle.mjs';
+  import { USFMParser, Filter, Validator } from 'https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/dist/bundle.mjs';
   console.log('Hello world');
   (async () => {
-  await USFMParser.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter-usfm.wasm",
-                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter.wasm");
-  await Validator.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter-usfm.wasm",
-                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.0/tree-sitter.wasm");
+  await USFMParser.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/tree-sitter-usfm.wasm",
+                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/web-tree-sitter.wasm");
+  await Validator.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/tree-sitter-usfm.wasm",
+                            "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/web-tree-sitter.wasm");
   const usfmParser = new USFMParser('\\id GEN\n\\c 1\n\\p\n\\v 1 In the begining..\\v 2 more text')
   const output = usfmParser.toUSJ()
   console.log({ output })
@@ -109,7 +111,7 @@ Working with USJ, also gives options like filtering selected markers, to edit th
 
 To work with the XML format, USX:
 ```javascript
-import { DOMImplementation, XMLSerializer } from 'xmldom';
+import { DOMParser, XMLSerializer } from '@xmldom/xmldom';
 
 const USFM = '\\id GEN\n\\c 1\n\\p\n\\v 1 In the begining..\\v 2 some more text'
 const usfmParser = new USFMParser(USFM);
@@ -123,6 +125,11 @@ console.log(usx);
 const usfmParser2 = new USFMParser(null, null, usxElem) // USX to USFM
 const usfmGen = usfmParser2.usfm;
 console.log(usfmGen);
+
+// USX from a string e.g. uploaded file contents
+const usxText = '<usx version="3.1"><book code="GEN" style="id"/><chapter number="1" style="c"/><para style="p"><verse number="1" style="v"/>In the beginning...</para></usx>'; 
+const usxDoc = new DOMParser().parseFromString(usxText.replace(/^/, ''), 'text/xml'); // strip BOM, if any
+const usfmParser3 = new USFMParser(null, null, usxDoc.documentElement);
 ```
 ### BibleNLP format
 Bible NLP format consists of two `txt` files: the first, with verse texts, one per line and the second, with corresponding references. The API generates a JSON with two fields, `text` and `vref`, each containing an array of strings.
@@ -197,9 +204,9 @@ const simpleUSJ = {
     { type: 'book', marker: 'id', code: 'GEN', content: [] },
     { type: 'chapter', marker: 'c', number: '1', sid: 'GEN 1' },
     { type: 'para', marker: 'p', content: [
-        {type: 'verse', marker: 'v', number: 1 },
+        {type: 'verse', marker: 'v', number: '1' },
         "In the begining..",
-        {type: 'verse', marker: 'v', number: 2 }
+        {type: 'verse', marker: 'v', number: '2' }
       ] }
   ]
 }
@@ -222,7 +229,7 @@ The filtering on USJ, the JSON output, is a feature incorporated to allow data e
 
 - *combineTexts*
 
-   Fourth input parameter to `toUSJ()` and `toList` of `USFMParser` class. Defaults to `true`. After filtering out makers like paragraphs and characters, we are left with texts from within them, if 'text-in-excluded-parent' is also not excluded. These text snippets may come as separate components in the contents list. When this option is `True`, the consequetive text snippets will be concatinated together. The text concatination is done in a puctuation and space aware manner. If users need more control over the space handling or for any other reason, would prefer the texts snippets as different components in the output, this can be set to `False`.
+   Fourth input parameter to `toUSJ()` and `toList` of `USFMParser` class. Defaults to `true`. After filtering out makers like paragraphs and characters, we are left with texts from within them, if 'text-in-excluded-parent' is also not excluded. These text snippets may come as separate components in the contents list. When this option is `True`, the consequetive text snippets will be concatenated together. The text concatenation is done in a puctuation and space aware manner. If users need more control over the space handling or for any other reason, would prefer the texts snippets as different components in the output, this can be set to `False`.
 
 - *usfm_grammar.Filter*
 
@@ -234,11 +241,13 @@ The filtering on USJ, the JSON output, is a feature incorporated to allow data e
     PARAGRAPHS : paragraph markers like \p, poetry markers, list table markers
     CHARACTERS : all character level markups like \em, \w, \wj etc and their nested versions with +
     NOTES : foot note, cross-reference and their content markers
+    LISTS : list markers like \li, \lh, \lim and the \list-s/\list-e milestones
     STUDY_BIBLE : \esb and \cat
     BCV : \id, \c and \v
-    TEXT : 'text-in-excluded-parent'
+    TEXT : 'text-in-excluded-parent' and 'text'
+    ZNAMESPACES : all user-extended \z markers, as one group
     ```
-    To inspect which are the markers in each of these options, it could be just printed out, `print(Filter.TITLES)`. These could be used individually or concatinated to get the desired filtering of markers and data:
+    To inspect which are the markers in each of these options, it could be just printed out, `console.log(Filter.TITLES)`. These could be used individually or concatenated to get the desired filtering of markers and data:
     ```javascript
     output = usfmParser.toUSJ(null, Filter.BCV) //to include
     output = usfmParser.toUSJ(null, [...Filter.BCV, ...Filter.TEXT]) //to include
@@ -254,6 +263,146 @@ The filtering on USJ, the JSON output, is a feature incorporated to allow data e
 
 
 
+### User-extended markers (\z)
+
+The USFM spec reserves markers beginning with `\z` for [user-defined extensions](https://docs.usfm.bible/usfm/3.1.2/extensions.html). usfm-grammar parses these, and represents them in USJ and USX using the type the extension belongs to. A `markers.ext` file can be supplied to declare which markers exist and what each one is, so that the output is what you expect. Without it, the type is guessed from how the marker is written.
+
+- *Supplying a markers.ext*
+
+  `markersExt`, the sixth argument to the `USFMParser` constructor, takes the **contents** of the file, not its path. Supply it when parsing USFM; it is not needed, and should not be passed, when converting from USJ or USX.
+  ```javascript
+  import {USFMParser, Filter} from 'usfm-grammar-web';
+
+  await USFMParser.init("https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/tree-sitter-usfm.wasm",
+                        "https://cdn.jsdelivr.net/npm/usfm-grammar-web@3.2.2/web-tree-sitter.wasm");
+
+  const myParser = new USFMParser(inputUsfm, null, null, null, null, markersExt);
+  const output = myParser.toUSJ();
+  ```
+
+- *The markers.ext file*
+
+  A plain text file, in a USFM-like syntax, with one block per extension marker. Blank lines separate the blocks.
+  ```
+  \marker zheading
+  \category title
+  \description A custom section heading.
+
+  \marker zbadge
+  \category char
+  \description A custom character marker.
+
+  \marker zalign-s
+  \category milestone
+  \description A custom alignment milestone.
+
+  \marker znote
+  \category footnote
+  \description A custom note marker.
+  ```
+  `\marker` and `\category` are the two fields that matter. `\description`, and any other field, is read but does not affect parsing. Only markers starting with `z` are accepted; a block declaring anything else is ignored.
+
+- *Categories and the four major types*
+
+  `\category` must be one of the values below. Each maps onto one of four major types, and that major type is what decides how the marker is parsed and what it becomes in the output.
+
+  | Major type | `\category` values | USJ `type` | USX element |
+  | --- | --- | --- | --- |
+  | para | `para`, `header`, `title`, `introduction`, `section`, `versepara`, `list`, `otherpara` | `para` | `<para>` |
+  | char | `char`, `introchar`, `listchar`, `footnotechar`, `crossreferencechar` | `char` | `<char>` |
+  | note | `note`, `footnote`, `crossreference` | `note` | `<note>` |
+  | milestone | `milestone` | `ms` | `<ms>` |
+
+  A `\category` outside this list, or a block with no `\category` at all, raises an error.
+
+  :warning: Only the major type is used. The minor type, and the rules the spec attaches to it about where such a marker may occur, are not enforced — a declared extension marker is accepted anywhere in the document. So declaring `\category title` rather than `\category para` changes nothing about the output; both produce a `para`. In particular, a `versepara` extension is not treated as verse text: its content is not attributed to the preceding verse (e.g. for `vid` on continuation paragraphs or the BibleNLP/list outputs and filtered USJs).
+
+- *Expected syntax per major type*
+
+  Each major type has to be written the way the corresponding standard markers are written, otherwise parsing fails.
+  ```
+  para        \zheading A line of its own
+  char        \zbadge inline content\zbadge*          (closing marker required)
+  milestone   \zalign-s\*   or   \zalign-s|x-strong="G25320"\*
+  note        \znote + \zbadge note body\zbadge*\znote*
+  ```
+  A note takes a caller as its first token, exactly like `\f` and `\x` do. Its body must be made of extension markers; plain text directly inside a note is not accepted.
+
+- *What may appear inside an extension marker*
+
+  Within a user-extended marker, only other user-extended markers (`\z...`, or `\+z...` when nested) and custom attributes (`x-...`) are allowed. Markers from the common pool are not. So this parses:
+  ```
+  \zbadge outer \+zbadge nested\+zbadge* text\zbadge*
+  \zbadge content|x-key="value"\zbadge*
+  ```
+  and these are parsing errors:
+  ```
+  \zbadge outer \nd Lord\nd* text\zbadge*
+  \zbadge content|lemma="value"\zbadge*
+  \c 1
+  \zheading A line \v 1 verse text
+  ```
+  A verse cannot be part of a custom para either. A `\v` ends the custom para. When the custom para sits within a verse para such as `\p`, the verse continues in that enclosing para; otherwise, as in the last example, it is a parsing error.
+
+- *When no markers.ext is given*
+
+  The markers are still parsed, and the type is inferred from the syntax alone. A warning is recorded for each one, listing the node and suggesting that a markers.ext be supplied.
+  ```
+  \zheading A line of its own       ->  para   (no closing marker)
+  \zbadge inline\zbadge*            ->  char   (closed with its own marker)
+  \zalign-s\*                       ->  ms     (closed with \*)
+  ```
+  :warning: Notes cannot be told apart from character markers this way, since both are written as an opening marker closed by its own marker. An undeclared note-like marker is reported as `char`. Declare it in a markers.ext if you need it to come out as a `note`.
+
+- *Filtering extension markers*
+
+  All `\z` markers are filtered as a single group, named `user-extension`, rather than one marker at a time. See the `ZNAMESPACES` option in [Filtering on USJ](#filtering-on-usj).
+
+- *A worked example*
+
+  With this `markers.ext`
+  ```
+  \marker zheading
+  \category title
+  \description A custom section heading.
+
+  \marker zbadge
+  \category char
+  \description A custom character marker.
+
+  \marker zalign-s
+  \category milestone
+  \description A custom alignment milestone.
+  ```
+  this USFM
+  ```
+  \id GEN
+  \c 1
+  \p
+  \v 1 Text with \zbadge inline content\zbadge* and more.
+  \zheading A line of its own
+  \v 2 Aligned \zalign-s|x-strong="G25320"\* text.
+  ```
+  gives these USJ nodes
+  ```json
+  {"type": "char", "content": [" inline content"], "marker": "zbadge"}
+  {"type": "para", "content": [" A line of its own\n"], "marker": "zheading"}
+  {"type": "ms", "content": [], "marker": "zalign-s", "x-strong": "G25320"}
+  ```
+  and this USX
+  ```xml
+  <usx version="3.1.2">
+    <book code="GEN" style="id"/>
+    <chapter number="1" style="c" sid="GEN 1"/>
+    <para style="p"><verse number="1" style="v" sid="GEN 1:1"/>Text with <char style="zbadge"> inline content</char> and more.
+  <para style="zheading"> A line of its own
+  </para><verse eid="GEN 1:1"/><verse number="2" style="v" sid="GEN 1:2"/>Aligned <ms style="zalign-s" x-strong="G25320"/> text.
+  <verse eid="GEN 1:2"/></para>
+    <chapter eid="GEN 1"/>
+  </usx>
+  ```
+
+
 ## Contributing
 Contributions are welcome! If you find any issues or have suggestions for improvements, feel free to open an issue or create a pull request on [GitHub](https://github.com/Bridgeconn/usfm-grammar).