twl-generator 1.2.15 → 1.3.1

package/README.md

@@ -1,105 +1,330 @@
-# twl-generator
+# TWL Generator
 
-Generate term-to-article lists from unfoldingWord en_tw archive for Bible books. Works in both Node.js (CLI) and React.js (browser) environments with intelligent caching.
+A Node.js library and CLI tool for generating Translation Word Links (TWL) TSV files from Door43 USFM data and Translation Words (TW) metadata. This tool intelligently matches biblical terms with their corresponding Translation Words articles using Strong's numbers, morphological analysis, and contextual matching.
 
-## Features
+## Installation
 
-- ✅ **Universal**: Works in Node.js and browser environments
-- ✅ **Smart Caching**: File system (Node.js) or localStorage/sessionStorage (browser)
-- ✅ **Performance**: Optimized matching with PrefixTrie algorithm
-- ✅ **Case Sensitivity**: Proper God/god distinction (God→kt/god, god→kt/falsegod)
-- ✅ **Morphological Variants**: Handles plurals, possessives, verb forms
-- ✅ **Parentheses Normalization**: "Joseph (OT)" → "Joseph" for better coverage
+### Global CLI Installation
+```bash
+npm install -g twl-generator
+```
 
----
+### Library Installation
+```bash
+npm install twl-generator
+```
 
 ## Usage
 
-### CLI
+### Command Line Interface
 
-Install globally:
+Generate TWL for a specific book:
+```bash
+twl-generator --book rut
+# Creates: rut.twl.tsv and rut.no-match.twl.tsv
+```
 
+Generate TWL for all books:
 ```bash
-npm install -g twl-generator
+twl-generator --all --out-dir ./output
+# Creates TWL files for all 66 biblical books
 ```
 
-Generate a TWL TSV for a Bible book (downloads USFM from Door43):
+Specify custom output location:
+```bash
+twl-generator --book mat --out matthew.twl.tsv
+```
 
+Enable advanced verb conjugation matching:
 ```bash
-twl-generator --book rut
+twl-generator --book jhn --use-compromise
+# Uses compromise.js for better verb form detection
 ```
 
-Generate a TWL TSV from a local USFM file:
+#### CLI Options
+- `--book <code>`: Book code (e.g., gen, exo, mat, mrk, jhn, etc.)
+- `--all`: Generate TWL files for all biblical books
+- `--out <file>`: Specify output file path
+- `--out-dir <dir>`: Output directory (for --all option)
+- `--use-compromise`: Enable advanced morphological analysis using compromise.js
 
-```bash
-twl-generator --usfm ./myfile.usfm
+### Library Usage
+
+#### Basic Usage
+```javascript
+import { generateTwlByBook } from 'twl-generator';
+
+// Generate TWL for Ruth
+const result = await generateTwlByBook('rut');
+console.log(result.matchedTsv);    // Main TWL output
+console.log(result.noMatchTsv);    // Unmatched entries for analysis
 ```
 
-Specify output file:
+#### With Advanced Options
+```javascript
+import { generateTwlByBook } from 'twl-generator';
 
-```bash
-twl-generator --usfm ./myfile.usfm --output ./output.tsv
+// Use advanced morphological analysis
+const result = await generateTwlByBook('jhn', { 
+  useCompromise: true  // Enable compromise.js for better verb matching
+});
+
+// Save to files
+import fs from 'fs/promises';
+await fs.writeFile('john.twl.tsv', result.matchedTsv);
+await fs.writeFile('john.no-match.tsv', result.noMatchTsv);
+```
+
+#### Integration Example
+```javascript
+import { generateTwlByBook } from 'twl-generator';
+
+async function processBibleBook(bookCode) {
+  try {
+    const { matchedTsv, noMatchTsv } = await generateTwlByBook(bookCode);
+    
+    // Process the TSV data
+    const lines = matchedTsv.split('\n');
+    const header = lines[0];
+    const rows = lines.slice(1).filter(Boolean);
+    
+    console.log(`Generated ${rows.length} TWL entries for ${bookCode.toUpperCase()}`);
+    
+    // Further processing...
+    return { success: true, entries: rows.length };
+  } catch (error) {
+    console.error(`Failed to process ${bookCode}:`, error);
+    return { success: false, error: error.message };
+  }
+}
 ```
 
-You can also combine `--book` and `--usfm` (book is used for output filename and context):
+## How It Works
+
+The TWL Generator uses a sophisticated multi-stage process to create Translation Word Links:
+
+### 1. **Data Sources**
+- **Original Language USFM**: Hebrew (hbo_uhb) and Greek (el-x-koine_ugnt) texts from Door43
+- **English Bible**: unfoldingWord Literal Text (en_ult) for context matching  
+- **Translation Words**: Local `tw_strongs_list.json` containing Strong's mappings and term definitions
+- **Strong's Numbers**: Links between original language words and semantic concepts
+
+### 2. **Processing Pipeline**
+
+#### Stage 1: Extract Strong's Data
+- Parses USFM `\w` tags to extract Strong's numbers from original language texts
+- Builds initial TSV with Reference, Strong's ID, and surface words
+- Handles multi-word phrases that share Strong's number sequences
+
+#### Stage 2: Generate English Context
+- Uses `tsv-quote-converters` to find corresponding English text (GLQuote) in ULT
+- Adds GLQuote and GLOccurrence columns for contextual matching
+- Converts to OrigWords/Occurrence format for processing
+
+#### Stage 3: Intelligent Article Selection  
+For each Strong's number and its English context, the system:
+
+1. **Prioritizes candidate articles** based on:
+   - Articles whose slug appears in the GLQuote text
+   - Article type preference: kt/ (key terms) → names/ → other/
+   - Alphabetical sorting within each category
+
+2. **Performs 4-stage matching** (best match wins):
+   - **Stage 1**: Case-sensitive word boundary matching
+   - **Stage 2**: Case-insensitive word boundary matching  
+   - **Stage 3**: Case-sensitive substring matching
+   - **Stage 4**: Case-insensitive morphological variants
+
+3. **Morphological analysis** includes:
+   - Pluralization (dog → dogs, man → men)
+   - Verb conjugation (-ing, -ed forms)
+   - Irregular verb forms (go → went, see → saw)
+   - Optional advanced analysis with compromise.js
+
+#### Stage 4: Quality Assurance
+- Generates disambiguation info when multiple articles could match
+- Marks entries as "Variant of" when morphological variants are used
+- Creates separate files for matched and unmatched entries
+- Provides detailed statistics and sample unmatched entries
+
+### 3. **Output Format**
+
+The generated TSV contains these columns:
+
+| Column | Description |
+|--------|-------------|
+| Reference | Chapter:verse (e.g., "1:1") |
+| ID | Random 4-character ID starting with letter |
+| Tags | "keyterm", "name", or empty based on article type |
+| OrigWords | The matched word(s) from the text |
+| Occurrence | Which occurrence of this word in the verse |
+| TWLink | Link to Translation Words article (rc://*/tw/dict/bible/...) |
+| Strongs | Original Strong's number |
+| GLQuote | English text context from ULT |  
+| GLOccurrence | Occurrence number in English context |
+| Variant of | Original term if morphological variant was used |
+| Disambiguation | List of other possible articles |
+
+### 4. **Matching Examples**
 
+```
+Reference   OrigWords   GLQuote              TWLink                      Variant of
+1:17        grace       grace and truth      rc://*/tw/dict/bible/kt/grace
+1:17        gracious    gracious God         rc://*/tw/dict/bible/kt/grace   grace
+2:3         men         wise men came        rc://*/tw/dict/bible/other/man
+2:3         wisdom      with great wisdom    rc://*/tw/dict/bible/kt/wise    wise
+```
+
+## Development
+
+### Prerequisites
+- Node.js 18+ (uses native fetch)
+- Git access to Door43 repositories
+
+### Setup
 ```bash
-twl-generator --usfm ./myfile.usfm --book rut
+git clone https://github.com/unfoldingWord/node-twl-generator.git
+cd node-twl-generator
+npm install
 ```
 
----
+### Testing
+```bash
+# Test single book generation
+npm test
 
-### As a Library (Node.js/ESM/React)
+# Test specific book
+npm run cli -- --book rut
 
-Install as a dependency:
+# Test with advanced morphology
+npm run cli -- --book jhn --use-compromise
+```
 
+### Local Development
 ```bash
-npm install twl-generator
+# Run CLI locally
+node src/cli.js --book gen --out test-output.tsv
+
+# Test library integration
+node -e "import('./src/index.js').then(m => m.generateTwlByBook('rut').then(console.log))"
 ```
 
-#### Example: Generate TWL TSV from USFM string
+### Project Structure
+```
+src/
+├── cli.js                    # Command line interface
+├── index.js                  # Main library exports
+├── common/
+│   └── books.js             # Bible book metadata
+└── utils/
+    ├── twl-matcher.js       # Term matching algorithms (legacy)
+    ├── zipProcessor.js      # TW archive processing (legacy)
+    └── usfm-alignment-remover.js  # USFM parsing (legacy)
+tw_strongs_list.json         # Translation Words database
+```
 
-```js
-import { generateTWLWithUsfm } from 'twl-generator';
+## Data Files
+
+### `tw_strongs_list.json`
+This file contains the core mapping between Strong's numbers and Translation Words articles:
+
+```json
+{
+  "kt/god": {
+    "article": {
+      "terms": ["God", "god", "deity", "divine"]
+    },
+    "strongs": [
+      ["H430"],     // Single Strong's number
+      ["H410"],
+      ["G2316", "G2318"]  // Multiple Strong's for compound concepts
+    ]
+  }
+}
+```
 
-// USFM string (can be loaded from file, API, etc.)
-const usfmContent = `
-\\id MAT
-\\c 1
-\\v 1 In the beginning...
-`;
+## Contributing
 
-const book = 'mat';
+We welcome contributions! Here's how you can help:
 
-const tsv = await generateTWLWithUsfm(book, usfmContent);
-// tsv is a string in TSV format, ready to save or process
-console.log(tsv);
-```
+### Reporting Issues
+- **Missing matches**: If legitimate biblical terms aren't being matched
+- **False positives**: If non-terms are being incorrectly matched  
+- **Performance issues**: Slow processing or memory problems
+- **Data quality**: Incorrect Strong's mappings or term definitions
 
-#### Example: Generate TWL TSV by fetching USFM for a book
+### Enhancement Ideas
+- **Better morphological analysis**: Improve verb conjugation and irregular forms
+- **Multi-language support**: Extend beyond English GLQuotes
+- **Contextual disambiguation**: Use surrounding words for better article selection
+- **Performance optimization**: Faster processing for large corpora
 
-```js
-import { generateTWLWithUsfm } from 'twl-generator';
+### Development Workflow
+1. Fork the repository
+2. Create a feature branch: `git checkout -b feature-name`
+3. Make your changes with tests
+4. Run the test suite: `npm test`
+5. Submit a pull request with detailed description
 
-const book = 'rut'; // Book code
+### Testing Your Changes
+```bash
+# Test various scenarios
+npm run cli -- --book psa --use-compromise  # Large book with advanced features
+npm run cli -- --book phm                   # Short book for quick testing
+npm run cli -- --book rev                   # Symbolic language testing
+```
 
-const tsv = await generateTWLWithUsfm(book);
-// This will fetch the USFM for the book from Door43 and return the TSV string
-console.log(tsv);
+## Browser Compatibility
+
+While primarily designed for Node.js, core functionality works in modern browsers:
+
+```javascript
+// React/Browser usage example
+import { generateTwlByBook } from 'twl-generator';
+
+const MyComponent = () => {
+  const [tsvData, setTsvData] = useState(null);
+  
+  const generateTWL = async () => {
+    try {
+      const result = await generateTwlByBook('mat');
+      setTsvData(result.matchedTsv);
+    } catch (error) {
+      console.error('TWL generation failed:', error);
+    }
+  };
+  
+  return (
+    <div>
+      <button onClick={generateTWL}>Generate TWL for Matthew</button>
+      {tsvData && <pre>{tsvData}</pre>}
+    </div>
+  );
+};
 ```
 
----
+## Performance
 
-### API Reference
+Typical processing times:
+- **Short books** (Philemon, 2-3 John): < 5 seconds
+- **Medium books** (Ruth, Ephesians): 5-15 seconds  
+- **Large books** (Psalms, Matthew): 30-60 seconds
+- **All books**: 15-30 minutes depending on network speed
 
-#### `generateTWLWithUsfm(book, usfmContent?)`
+Memory usage scales with book size, typically 50-200MB peak.
 
-- `book`: (string) Book code (e.g., 'mat', 'rut'). Required if `usfmContent` is not provided.
-- `usfmContent`: (string, optional) USFM file content. If provided, this is used instead of fetching from Door43.
-- **Returns:** `Promise<string>` — TSV string of TWL matches.
+## License
 
----
+MIT License - see [LICENSE](LICENSE) file for details.
 
-## License
+## Support
+
+- **Issues**: https://github.com/unfoldingWord/node-twl-generator/issues
+- **Discussions**: https://github.com/unfoldingWord/node-twl-generator/discussions  
+- **Documentation**: https://github.com/unfoldingWord/node-twl-generator/wiki
+
+## Related Projects
 
-MIT
+- [tsv-quote-converters](https://www.npmjs.com/package/tsv-quote-converters) - GLQuote generation
+- [compromise](https://www.npmjs.com/package/compromise) - Advanced morphological analysis
+- [Door43 Content](https://git.door43.org/unfoldingWord) - Source biblical texts and resources

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "twl-generator",
-  "version": "1.2.15",
+  "version": "1.3.1",
   "description": "Generate term-to-article lists from unfoldingWord en_tw archive for Bible books. Works in both Node.js (CLI) and React.js (browser) environments.",
   "main": "src/index.js",
   "bin": {
@@ -46,7 +46,10 @@
     "node": ">=18.0.0"
   },
   "dependencies": {
-    "jszip": "^3.10.1"
+    "csv-parse": "^5.5.6",
+    "csv-stringify": "^6.5.0",
+    "compromise": "^14.14.2",
+    "tsv-quote-converters": "^1.1.13"
   },
   "peerDependencies": {
     "react": ">=16.8.0"

package/src/cli.js

@@ -1,86 +1,84 @@
 #!/usr/bin/env node
-import { generateTWLWithUsfm } from './index.js';
-import fs from 'fs';
-import path from 'path';
+import fs from 'node:fs/promises';
+import path from 'node:path';
+import { generateTwlByBook } from '../src/index.js';
+import { BibleBookData } from '../src/common/books.js';
 
-const args = process.argv.slice(2);
+const THIS_DIR = path.dirname(new URL(import.meta.url).pathname);
 
-function printHelp() {
-  console.log(`Usage: generate-twls [options]
-
-Options:
-  --book <book>           Specify the Bible book (e.g., rut)
-  --usfm <path>          Path to USFM file to process
-  --output <path>        Path to output TSV file
-  --help                 Show this help message
-
-Examples:
-  generate-twls --book rut
-  generate-twls --usfm ./41-MAT.usfm --output ./mat_twl.tsv
-  generate-twls --usfm ./file.usfm --book rut`);
-}
-
-let book = null;
-let usfmPath = null;
-let outputPath = null;
-
-for (let i = 0; i < args.length; i++) {
-  if (args[i] === '--book' && args[i + 1]) {
-    book = args[i + 1].toLowerCase();
-    i++;
-  } else if (args[i] === '--usfm' && args[i + 1]) {
-    usfmPath = args[i + 1];
-    i++;
-  } else if (args[i] === '--output' && args[i + 1]) {
-    outputPath = args[i + 1];
-    i++;
-  } else if (args[i] === '--help') {
-    printHelp();
-    process.exit(0);
+async function readBooksJs() {
+  const map = {};
+  for (const [code, meta] of Object.entries(BibleBookData)) {
+    map[code.toUpperCase()] = { usfm: meta.usfm, testament: meta.testament };
   }
+  return map;
 }
 
-// Validate arguments
-if (!book && !usfmPath) {
-  console.error('Error: Either --book or --usfm parameter is required');
-  printHelp();
-  process.exit(1);
-}
-
-if (usfmPath && !fs.existsSync(usfmPath)) {
-  console.error(`Error: USFM file not found: ${usfmPath}`);
-  process.exit(1);
+function parseArgs(argv) {
+  const args = { book: '', out: '', outDir: '', all: false, useCompromise: false };
+  for (let i = 2; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === '--book' || a === '-b') { args.book = argv[++i] || ''; }
+    else if (a === '--out' || a === '-o') { args.out = argv[++i] || ''; }
+    else if (a === '--out-dir' || a === '-O') { args.outDir = argv[++i] || ''; }
+    else if (a === '--all' || a === '-A') { args.all = true; }
+    else if (a === '--use-compromise') { args.useCompromise = true; }
+  }
+  return args;
 }
 
-(async () => {
-  try {
-    let usfmContent = null;
-    if (usfmPath) {
-      usfmContent = fs.readFileSync(usfmPath, 'utf8');
-      console.log(`Reading USFM from: ${usfmPath}`);
-    }
-
-    const tsv = await generateTWLWithUsfm(book, usfmContent);
-
-    // Determine output filename
-    let filename;
-    if (outputPath) {
-      filename = outputPath;
-    } else if (book) {
-      filename = `twl_${book.toUpperCase()}.tsv`;
-    } else if (usfmPath) {
-      const baseName = path.basename(usfmPath, path.extname(usfmPath));
-      filename = `${baseName}.tsv`;
-    } else {
-      filename = 'output.tsv';
+async function main() {
+  const { book, out, outDir, all, useCompromise } = parseArgs(process.argv);
+  if (all || (book && book.toLowerCase() === 'all')) {
+    const books = await readBooksJs();
+    const codes = Object.keys(books);
+    const destDir = outDir ? path.resolve(outDir) : path.resolve(THIS_DIR, '..'); // default to twl-generator dir
+    await fs.mkdir(destDir, { recursive: true });
+    console.error(`Generating TWL for ${codes.length} books to ${destDir} (useCompromise=${useCompromise})`);
+    for (const code of codes) {
+      try {
+        const { matchedTsv, noMatchTsv } = await generateTwlByBook(code, { useCompromise });
+        const fname = `${code.toLowerCase()}.twl.tsv`;
+        const outPath = path.join(destDir, fname);
+        await fs.writeFile(outPath, matchedTsv, 'utf8');
+        const nmPath = path.join(destDir, `${code.toLowerCase()}.no-match.twl.tsv`);
+        await fs.writeFile(nmPath, noMatchTsv, 'utf8');
+        console.error(`  ✓ ${code} -> ${fname}`);
+      } catch (err) {
+        console.error(`  ✗ ${code} failed:`, err.message || err);
+      }
     }
+    return;
+  }
 
-    // Save TSV to file
-    fs.writeFileSync(filename, tsv, 'utf8');
-    console.log(`TSV file saved as ${filename}`);
-    console.log(`Found ${tsv.split('\n').length - 1} matches`);
-  } catch (error) {
-    console.error('Error:', error.message);
+  if (!book) {
+    console.error('Usage: generate-twl --book <code>|all [--out <file.tsv> | --out-dir <dir>] [--use-compromise]');
     process.exit(1);
   }
-})();
+
+  const { matchedTsv, noMatchTsv } = await generateTwlByBook(book, { useCompromise });
+  if (out) {
+    const outPath = path.resolve(out);
+    await fs.writeFile(outPath, matchedTsv, 'utf8');
+    console.log(`Wrote ${out}`);
+    const dir = path.dirname(outPath);
+    const base = path.basename(outPath);
+    const nmPath = path.join(dir, base.replace(/\.twl\.tsv$/i, '.no-match.twl.tsv'));
+    await fs.writeFile(nmPath, noMatchTsv, 'utf8');
+    console.log(`Wrote ${nmPath}`);
+  } else if (outDir) {
+    const destDir = path.resolve(outDir);
+    await fs.mkdir(destDir, { recursive: true });
+    const outPath = path.join(destDir, `${book.toLowerCase()}.twl.tsv`);
+    await fs.writeFile(outPath, matchedTsv, 'utf8');
+    const nmPath = path.join(destDir, `${book.toLowerCase()}.no-match.twl.tsv`);
+    await fs.writeFile(nmPath, noMatchTsv, 'utf8');
+    console.log(`Wrote ${outPath}`);
+    console.log(`Wrote ${nmPath}`);
+  } else {
+    // When writing to stdout, output only the matched TSV to avoid mixing tables
+    process.stdout.write(matchedTsv);
+  }
+}
+
+main().catch(err => { console.error(err); process.exit(1); });