lindera-nodejs 3.0.0

package/README.md

@@ -0,0 +1,308 @@
+# lindera-nodejs
+
+Node.js binding for [Lindera](https://github.com/lindera/lindera), a Japanese morphological analysis engine.
+
+## Overview
+
+lindera-nodejs provides a comprehensive Node.js interface to the Lindera morphological analysis engine, supporting Japanese, Korean, and Chinese text analysis. This implementation includes all major features:
+
+- **Multi-language Support**: Japanese (IPADIC, IPADIC-NEologd, UniDic), Korean (ko-dic), Chinese (CC-CEDICT, Jieba)
+- **Character Filters**: Text preprocessing with mapping, regex, Unicode normalization, and Japanese iteration mark handling
+- **Token Filters**: Post-processing filters including lowercase, length filtering, stop words, and Japanese-specific filters
+- **Flexible Configuration**: Configurable tokenization modes and penalty settings
+- **Metadata Support**: Complete dictionary schema and metadata management
+- **TypeScript Support**: Full type definitions included out of the box
+
+## Features
+
+### Core Components
+
+- **TokenizerBuilder**: Fluent API for building customized tokenizers
+- **Tokenizer**: High-performance text tokenization with integrated filtering
+- **CharacterFilter**: Pre-processing filters for text normalization
+- **TokenFilter**: Post-processing filters for token refinement
+- **Metadata & Schema**: Dictionary structure and configuration management
+- **Training & Export** (optional): Train custom morphological analysis models from corpus data
+
+### Supported Dictionaries
+
+- **Japanese**: IPADIC, IPADIC-NEologd, UniDic
+- **Korean**: ko-dic
+- **Chinese**: CC-CEDICT, Jieba
+- **Custom**: User dictionary support
+
+Pre-built dictionaries are available from [GitHub Releases](https://github.com/lindera/lindera/releases).
+Download a dictionary archive (e.g. `lindera-ipadic-*.zip`) and specify the extracted path when loading.
+
+### Filter Types
+
+**Character Filters:**
+
+- Mapping filter (character replacement)
+- Regex filter (pattern-based replacement)
+- Unicode normalization (NFKC, etc.)
+- Japanese iteration mark normalization
+
+**Token Filters:**
+
+- Text case transformation (lowercase, uppercase)
+- Length filtering (min/max character length)
+- Stop words filtering
+- Japanese-specific filters (base form, reading form, etc.)
+- Korean-specific filters
+
+## Install project dependencies
+
+- Node.js 18+ : <https://nodejs.org/>
+- Rust : <https://www.rust-lang.org/tools/install>
+- @napi-rs/cli : `npm install -g @napi-rs/cli`
+
+## Setup repository
+
+```shell
+# Clone lindera project repository
+git clone git@github.com:lindera/lindera.git
+cd lindera
+```
+
+## Install lindera-nodejs
+
+This command builds the library with development settings (debug build).
+
+```shell
+cd lindera-nodejs
+npm install
+npm run build
+```
+
+## Quick Start
+
+### Basic Tokenization
+
+```javascript
+const { loadDictionary, Tokenizer } = require("lindera-nodejs");
+
+// Load dictionary
+// Load dictionary from a local path (download from GitHub Releases)
+const dictionary = loadDictionary("/path/to/ipadic");
+
+// Create a tokenizer
+const tokenizer = new Tokenizer(dictionary, "normal");
+
+// Tokenize Japanese text
+const text = "すもももももももものうち";
+const tokens = tokenizer.tokenize(text);
+
+for (const token of tokens) {
+  console.log(`Text: ${token.surface}, Position: ${token.byteStart}-${token.byteEnd}`);
+}
+```
+
+### Using Character Filters
+
+```javascript
+const { TokenizerBuilder } = require("lindera-nodejs");
+
+// Create tokenizer builder
+const builder = new TokenizerBuilder();
+builder.setMode("normal");
+builder.setDictionary("/path/to/ipadic");
+
+// Add character filters
+builder.appendCharacterFilter("mapping", { mapping: { "ー": "-" } });
+builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
+
+// Build tokenizer with filters
+const tokenizer = builder.build();
+const text = "テストー１２３";
+const tokens = tokenizer.tokenize(text); // Will apply filters automatically
+```
+
+### Using Token Filters
+
+```javascript
+const { TokenizerBuilder } = require("lindera-nodejs");
+
+// Create tokenizer builder
+const builder = new TokenizerBuilder();
+builder.setMode("normal");
+builder.setDictionary("/path/to/ipadic");
+
+// Add token filters
+builder.appendTokenFilter("lowercase");
+builder.appendTokenFilter("length", { min: 2, max: 10 });
+builder.appendTokenFilter("japanese_stop_tags", { tags: ["助詞", "助動詞"] });
+
+// Build tokenizer with filters
+const tokenizer = builder.build();
+const tokens = tokenizer.tokenize("テキストの解析");
+```
+
+### Integrated Pipeline
+
+```javascript
+const { TokenizerBuilder } = require("lindera-nodejs");
+
+// Build tokenizer with integrated filters
+const builder = new TokenizerBuilder();
+builder.setMode("normal");
+builder.setDictionary("/path/to/ipadic");
+
+// Add character filters
+builder.appendCharacterFilter("mapping", { mapping: { "ー": "-" } });
+builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
+
+// Add token filters
+builder.appendTokenFilter("lowercase");
+builder.appendTokenFilter("japanese_base_form");
+
+// Build and use
+const tokenizer = builder.build();
+const tokens = tokenizer.tokenize("コーヒーショップ");
+```
+
+### Working with Metadata
+
+```javascript
+const { Metadata } = require("lindera-nodejs");
+
+// Create metadata with default values
+const metadata = new Metadata();
+console.log(`Name: ${metadata.name}`);
+console.log(`Encoding: ${metadata.encoding}`);
+
+// Create metadata from a JSON file
+const loaded = Metadata.fromJsonFile("metadata.json");
+console.log(loaded.toObject());
+```
+
+## Advanced Usage
+
+### Filter Configuration Examples
+
+Character filters and token filters accept configuration as object arguments:
+
+```javascript
+const { TokenizerBuilder } = require("lindera-nodejs");
+
+const builder = new TokenizerBuilder();
+builder.setDictionary("/path/to/ipadic");
+
+// Character filters with object configuration
+builder.appendCharacterFilter("unicode_normalize", { kind: "nfkc" });
+builder.appendCharacterFilter("japanese_iteration_mark", {
+  normalize_kanji: true,
+  normalize_kana: true,
+});
+builder.appendCharacterFilter("mapping", {
+  mapping: { "リンデラ": "lindera", "トウキョウ": "東京" },
+});
+
+// Token filters with object configuration
+builder.appendTokenFilter("japanese_katakana_stem", { min: 3 });
+builder.appendTokenFilter("length", { min: 2, max: 10 });
+builder.appendTokenFilter("japanese_stop_tags", {
+  tags: ["助詞", "助動詞", "記号"],
+});
+
+// Filters without configuration can omit the object
+builder.appendTokenFilter("lowercase");
+builder.appendTokenFilter("japanese_base_form");
+
+const tokenizer = builder.build();
+```
+
+See `examples/` directory for comprehensive examples including:
+
+- `tokenize.js`: Basic tokenization
+- `tokenize_with_filters.js`: Using character and token filters
+- `tokenize_with_userdict.js`: Custom user dictionary
+- `train_and_export.js`: Train and export custom dictionaries (requires `train` feature)
+- `tokenize_with_decompose.js`: Decompose mode tokenization
+
+## Dictionary Support
+
+### Japanese
+
+- **IPADIC**: Default Japanese dictionary, good for general text
+- **UniDic**: Academic dictionary with detailed morphological information
+
+### Korean
+
+- **ko-dic**: Standard Korean dictionary for morphological analysis
+
+### Chinese
+
+- **CC-CEDICT**: Community-maintained Chinese-English dictionary
+
+### Custom Dictionaries
+
+- User dictionary support for domain-specific terms
+- CSV format for easy customization
+
+## Dictionary Training (Experimental)
+
+lindera-nodejs supports training custom morphological analysis models from annotated corpus data when built with the `train` feature.
+
+### Building with Training Support
+
+```shell
+npm run build -- --features train
+```
+
+### Training a Model
+
+```javascript
+const { train } = require("lindera-nodejs");
+
+// Train a model from corpus
+train({
+  seed: "path/to/seed.csv",
+  corpus: "path/to/corpus.txt",
+  charDef: "path/to/char.def",
+  unkDef: "path/to/unk.def",
+  featureDef: "path/to/feature.def",
+  rewriteDef: "path/to/rewrite.def",
+  output: "model.dat",
+  lambda: 0.01,
+  maxIter: 100,
+});
+```
+
+### Exporting Dictionary Files
+
+```javascript
+const { exportModel } = require("lindera-nodejs");
+
+// Export trained model to dictionary files
+exportModel({
+  model: "model.dat",
+  output: "exported_dict/",
+  metadata: "metadata.json",
+});
+```
+
+This will create:
+
+- `lex.csv`: Lexicon file
+- `matrix.def`: Connection cost matrix
+- `unk.def`: Unknown word definitions
+- `char.def`: Character definitions
+- `metadata.json`: Dictionary metadata (if provided)
+
+See `examples/train_and_export.js` for a complete example.
+
+## API Reference
+
+### Core Classes
+
+- `TokenizerBuilder`: Fluent builder for tokenizer configuration
+- `Tokenizer`: Main tokenization engine
+- `Token`: Individual token with text, position, and linguistic features
+- `Metadata`: Dictionary metadata and configuration
+- `Schema`: Dictionary schema definition
+
+### Training Functions (requires `train` feature)
+
+- `train()`: Train a morphological analysis model from corpus
+- `exportModel()`: Export trained model to dictionary files

package/package.json

@@ -0,0 +1,63 @@
+{
+  "name": "lindera-nodejs",
+  "version": "3.0.0",
+  "description": "Node.js bindings for Lindera morphological analysis engine",
+  "main": "index.js",
+  "types": "index.d.ts",
+  "exports": {
+    ".": {
+      "require": "./index.js",
+      "types": "./index.d.ts"
+    }
+  },
+  "napi": {
+    "binaryName": "lindera-nodejs",
+    "package": {
+      "name": "lindera-nodejs"
+    },
+    "targets": [
+      "x86_64-apple-darwin",
+      "aarch64-apple-darwin",
+      "x86_64-unknown-linux-gnu",
+      "aarch64-unknown-linux-gnu",
+      "x86_64-pc-windows-msvc",
+      "aarch64-pc-windows-msvc"
+    ]
+  },
+  "license": "MIT",
+  "scripts": {
+    "build": "napi build --platform --release -p lindera-nodejs",
+    "build:debug": "napi build --platform -p lindera-nodejs",
+    "artifacts": "napi artifacts",
+    "test": "node --test tests/test_*.js"
+  },
+  "files": [
+    "index.js",
+    "index.d.ts"
+  ],
+  "optionalDependencies": {
+    "lindera-nodejs-darwin-arm64": "3.0.0",
+    "lindera-nodejs-darwin-x64": "3.0.0",
+    "lindera-nodejs-linux-arm64-gnu": "3.0.0",
+    "lindera-nodejs-linux-x64-gnu": "3.0.0",
+    "lindera-nodejs-win32-arm64-msvc": "3.0.0",
+    "lindera-nodejs-win32-x64-msvc": "3.0.0"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/lindera/lindera"
+  },
+  "keywords": [
+    "morphological",
+    "analysis",
+    "tokenizer",
+    "japanese",
+    "korean",
+    "chinese",
+    "lindera",
+    "napi"
+  ],
+  "devDependencies": {
+    "@napi-rs/cli": "^3.6.0"
+  }
+}