cjk_index 0.1.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0d4df8b57af0d9d2e2755bf87cc9fdb60313d87f68ab17160f5ff4f9f28460d9
-  data.tar.gz: 3bc33035945c48d226c328c58eab9c012f7a93874edde8a3adc2c5a92d9972f3
+  metadata.gz: 9c9ffe634562d740738f6d9b9cf97ade5f5ab6d32a80b6bfd349496435d9e88b
+  data.tar.gz: 6cc0aebdedd14b07cdcf9dda3dcc9b849b80d135149444f057daed4cc620c004
 SHA512:
-  metadata.gz: 3016fee0464ea949e62360c4d4bfbaf3a4f1a7271c711efe1096b4958ce754b5bf4059566395eb11980fc92a8c55d953989e6562b0fa0429e1e25036a7d277e7
-  data.tar.gz: f7dade68d837d456467e6d1c8fe6e8fbba3335432c7735414df161627942137d626a7a8f270423085957b162e0fa75c3aa709dd7279635e1c4a6c0bb47a9ff2e
+  metadata.gz: ee41c34d99efe6e17f8936f3211b62aaab6fdafa54b714f685458c641a56263a46b506de9f703a4cd1c6f470c32e0d336b1914547655a433641ecbf9cdc5fcc0
+  data.tar.gz: 608805cba3a84e8c1bb8425e09c8ff3bdcb2bd4da6443f6ca63eeb6d2e3952e75b0166db871e853383d3531cb40d7f48052cc70540f2ae7fd8bd5ae372424af7

data/CHANGELOG.md

@@ -0,0 +1,125 @@
+# Changelog
+
+## 1.0.0 (2026-10-28)
+
+Indexes built with 0.x should be rebuilt: the `ja` table changed, and the
+index now records a digest of its tables (see below). Formats 2 and 3 are
+still read.
+
+Large collections:
+
+- Index format `cjk_index/4`: posting lists as compact strings (about a third
+  smaller gzipped, three times smaller before compression).
+- Split indexes: `Builder#write_split(dir, shard_size:)` and
+  `Builder#each_split_file` write a manifest and shards of about 32 KB; the
+  browser fetches shards as queries need them (`index.searchAsync`,
+  `index.prepare`). `CJKIndex.load` and `Searcher.load` read a split index;
+  Jekyll: `split: true`.
+- `Builder#write(path)` writes one file without building the JSON in memory;
+  `Builder.read(path)` reads a file or a split index.
+- Building keeps posting lists encoded and normalizes faster: 3,000 full
+  texts take 0.56 GB instead of 2.3 GB at peak, and 37 s instead of 62 s.
+- Updating a loaded index is slower without YJIT (one update about 0.25 s at
+  17,000 records); one update and its write in `IndexFile` cost about the
+  same as before.
+
+Variant tables:
+
+- `ja`: 409 pairs (was 291), generated from the Jōyō Kanji Table, the
+  Jinmeiyō kanji list and Unihan kZVariant. 籐/藤, 聯/連 and 彎/湾 no longer
+  fold.
+- New opt-in tables: `zh-safe` (Traditional/Simplified pairs that are
+  one-to-one both ways, so 後/后 and 發/髮 stay apart) and `ko` (old Hangul
+  in the Hanyang Private Use Area to standard jamo). `zh` is unchanged.
+- `script/variant_tables.rb` generates the tables from their sources.
+- The index records `variant_digest`, a digest of what its tables fold; Ruby
+  and the browser script refuse an index whose digest differs.
+- The browser highlights matches in conjoining jamo, half-width kana with
+  sound marks and old Hangul in the Private Use Area.
+
+More ways to build an index:
+
+- `cjk_index` command: `build` (CSV, JSON or JSON Lines), `search`, `runtime`.
+- `CJKIndex::RakeTask`.
+- Bridgetown 2 plugin (`require "cjk_index/bridgetown"`).
+- Examples for Bridgetown and Rails (`examples/`), notes in
+  `docs/notes/adapters.md`.
+- Migration guide from lunr.js, in English and Japanese
+  (`docs/migrating-from-lunr.md`).
+
+Filters:
+
+- A `value` filter compares value ids instead of folded strings, in Ruby
+  and in the browser (#14): filtering 17,000 records by one author takes
+  1.7 ms instead of 2.7 ms.
+
+Fixes:
+
+- A `year` filter on a Time or Date (Jekyll gives a post's `date` as a Time)
+  gave no year.
+- The Jekyll plugin gave the builder only the index fields of a collection
+  document, so a filter on another front-matter field was empty.
+- A query word named like a JavaScript object member ("constructor") scored
+  differently in the browser than in Ruby.
+
+## 0.2.0 (2026-10-28)
+
+- Filters: `Builder.new(filters:)` with `value` and `year` filters;
+  `search(query, filter:)` in Ruby and `search(query, { filter })` in the
+  browser; a blank query with a filter lists every document that passes;
+  `facets(field, results)` counts values. Ruby and the browser give the same
+  results and counts.
+- Index updates: `Builder#remove`, `Builder#update`, `Builder.load`. The
+  changed index is the same as one rebuilt from scratch.
+- `Searcher.new(builder)` follows later changes to the builder (it used to
+  copy the index).
+- `IndexFile`: an index file shared by several processes (e.g. Rails
+  workers). Updates take a lock and are written with a rename, so none is
+  lost and readers never see a partly written file; readers load the file
+  again when it changes. A killed writer leaves the old index intact;
+  `lock_timeout:` bounds the wait for another writer.
+- Index format `cjk_index/3` (adds `filters`). The browser script and
+  `Searcher` still read `cjk_index/2`.
+- Jekyll: `filters:` for an index; the CollectionBuilder preset filters on
+  `metadata-facets-fields` and the year of `date`.
+
+## 0.1.1 (2026-10-18)
+
+No change to the library. This release fixes the state of the repository
+cited in a paper.
+
+- Retrieval quality on two real datasets (69 catalog records from cb-ja-demo,
+  16,951 Aozora Bunko works; 15 queries each), against lunr.js with its
+  default and Japanese settings: [bench/retrieval/](bench/retrieval/).
+  Queries, relevance judgments (committed before the runs), judging rules,
+  scripts and raw outputs are included.
+- Load and query times in Chrome and Safari for 1,000, 5,000 and 16,951
+  records: [bench/browser/](bench/browser/).
+- README and docs/comparison.md: the recommended size is now the range we
+  measured (up to 16,951 records), not "about 20,000".
+- CITATION.cff for citing the software.
+- CODE_OF_CONDUCT.md (Contributor Covenant 2.1), CONTRIBUTING.md, SECURITY.md,
+  issue forms and a pull request template.
+
+## 0.1.0 (2026-10-02)
+
+First release on RubyGems.
+
+- Normalizer in pure Ruby: NFKC, lowercase, katakana → hiragana, 291 old-form
+  kanji → modern forms, the iteration mark 々. Optional Traditional → Simplified
+  Chinese table (3,202 characters, from OpenCC, Apache-2.0). `CJKIndex.normalize`.
+- Tokenizer: positional bigrams for CJK runs, including hangul and old-hangul
+  jamo; Latin words as whole tokens.
+- `CJKIndex::Builder`: BM25 index with field boosts, written as JSON.
+- `CJKIndex::Searcher`: exact substring matching, BM25 ranking, highlighting and
+  excerpts in pure Ruby; prefix matching and one-typo tolerance for Latin words.
+- Browser runtime with no dependencies (`CJKIndex::Runtime.source`), generated
+  from the same tables; the test suite requires identical results in Ruby and
+  under Node.
+- Jekyll plugin (`cjk_index/jekyll`) with a CollectionBuilder preset.
+
+In the repository (not packaged in the gem):
+
+- `examples/search_csv.rb`: search a CSV catalog from the command line, without Jekyll.
+- Benchmarks on Aozora Bunko (up to 16,951 works): [bench/README.md](bench/README.md).
+- Comparison with other search tools: [docs/comparison.md](docs/comparison.md).

data/LICENSE-Unicode

@@ -0,0 +1,39 @@
+UNICODE LICENSE V3
+
+COPYRIGHT AND PERMISSION NOTICE
+
+Copyright © 1991-2026 Unicode, Inc.
+
+NOTICE TO USER: Carefully read the following legal agreement. BY
+DOWNLOADING, INSTALLING, COPYING OR OTHERWISE USING DATA FILES, AND/OR
+SOFTWARE, YOU UNEQUIVOCALLY ACCEPT, AND AGREE TO BE BOUND BY, ALL OF THE
+TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE, DO NOT
+DOWNLOAD, INSTALL, COPY, DISTRIBUTE OR USE THE DATA FILES OR SOFTWARE.
+
+Permission is hereby granted, free of charge, to any person obtaining a
+copy of data files and any associated documentation (the "Data Files") or
+software and any associated documentation (the "Software") to deal in the
+Data Files or Software without restriction, including without limitation
+the rights to use, copy, modify, merge, publish, distribute, and/or sell
+copies of the Data Files or Software, and to permit persons to whom the
+Data Files or Software are furnished to do so, provided that either (a)
+this copyright and permission notice appear with all copies of the Data
+Files or Software, or (b) this copyright and permission notice appear in
+associated Documentation.
+
+THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+THIRD PARTY RIGHTS.
+
+IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE
+BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES,
+OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
+WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
+ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA
+FILES OR SOFTWARE.
+
+Except as contained in this notice, the name of a copyright holder shall
+not be used in advertising or otherwise to promote the sale, use or other
+dealings in these Data Files or Software without prior written
+authorization of the copyright holder.

data/NOTICE

@@ -5,3 +5,20 @@ data/zh-hant-hans.tsv is derived from OpenCC (https://github.com/BYVoid/OpenCC),
 data/dictionary/TSCharacters.txt, Copyright (c) 2010-2026 Carbo Kuo and contributors,
 licensed under the Apache License, Version 2.0 (see LICENSE-OpenCC).
 Changes are described at the top of that file.
+
+data/zh-hant-hans-safe.tsv is derived from OpenCC data/dictionary/TSCharacters.txt and
+STCharacters.txt, under the same license and copyright as above.
+
+data/ja-kyujitai.tsv is derived in part from the Unicode Han Database (Unihan), Unicode 18.0.0
+(https://www.unicode.org/Public/18.0.0/ucd/Unihan.zip), Copyright (c) 1991-2026 Unicode, Inc.,
+licensed under the Unicode License v3 (see LICENSE-Unicode). The fields used (kJapaneseOldVariant,
+kJoyoKanji, kJinmeiyoKanji, kZVariant) record the Joyo Kanji Table (Cabinet Notice No. 2 of 2010)
+and the Jinmeiyo Kanji of the Family Register Act Enforcement Regulations, which as official
+notices and ordinances are not subject to copyright in Japan (Copyright Act, art. 13).
+
+data/ko-hanyang-pua.tsv is converted from the table in the hypua crate by kiwiyou
+(https://github.com/kiwiyou/hypua, commit 7e47655730d4cfea89564768ebb277b86072eef9), which its
+author released under the Unlicense (a public domain dedication). The mapping is the same as that
+of hypua2jamo by mete0r (LGPL-3.0-or-later code); both are based on the KTUG Hanyang PUA Table
+Project (http://faq.ktug.org/faq/HanyangPuaTableProject), which states no license. See
+docs/notes/variant-tables.md.

data/README.md

@@ -1,5 +1,7 @@
 # cjk_index
 
+日本語の説明: [docs/ja/README.md](docs/ja/README.md)
+
 Dictionary-free full-text search for Chinese, Japanese and Korean text, in
 pure Ruby. Build the index in Ruby, then query it in Ruby (`CJKIndex::Searcher`)
 or in the browser with a small script that has no dependencies. Both return
@@ -24,7 +26,8 @@ even when it is an exact title. cjk_index instead:
   native extension (the same idea as [kensaku](https://github.com/whispcat/kensaku));
 - folds variants on both sides: NFKC, lowercase, katakana → hiragana,
   old-form kanji → modern form (「圖」→「図」), 々 expanded (「佐々木」→「佐佐木」),
-  and optionally Traditional ↔ Simplified Chinese (from OpenCC);
+  and optionally Traditional ↔ Simplified Chinese (from OpenCC) and old
+  Hangul in the Private Use Area;
 - ranks with BM25 per field, with field boosts;
 - marks matches in the original text (`highlight`, `excerpt`), even when the
   query and the text differ only by these variants;
@@ -32,18 +35,42 @@ even when it is an exact title. cjk_index instead:
   one typo;
 - generates the browser script from the same tables as the Ruby code, so
   indexing and querying cannot drift apart (checked by the test suite under
-  Node; the script also refuses an index built with other tables).
+  Node; the script also refuses an index built with other tables);
+- splits a large index into a small manifest and shards that the browser
+  fetches only when a query needs them.
 
-The browser script is about 6.5 KB gzipped with the Japanese table, 24 KB
-with the Chinese table as well.
+The browser script is about 9 KB gzipped with the Japanese table, 24 KB
+with the Chinese table as well. Index it from Ruby, Jekyll, Bridgetown, a
+Rake task or the command line.
 
-How this compares with Pagefind, kensaku, lunr.js and others:
+How this compares with Pagefind, kensaku, lunr.js, rroonga, tantiny and others:
 [docs/comparison.md](docs/comparison.md).
 Index size and query time on up to 16,951 works of Aozora Bunko:
-[bench/README.md](bench/README.md).
+[bench/README.md](bench/README.md). Retrieval quality against lunr.js:
+[bench/retrieval/](bench/retrieval/). Load and query times in Chrome and
+Safari: [bench/browser/](bench/browser/).
+
+When to use it, and when not to:
+
+- **Good fit:** catalogs, collection records, blog posts and full texts on
+  static sites, command-line tools, scripts and small apps; anywhere you want
+  nothing to install outside Ruby, or the same results in Ruby and the
+  browser. In one file, the 16,951-record Aozora Bunko catalog is 12.9 MB
+  gzipped; as a [split index](#large-collections-split-indexes), the browser
+  loads a 72 KB manifest and fetches about 40 KB per query, and 1,000 full
+  texts (33 MB gzipped in one file) need about the same.
+- **Not a good fit:** frequent updates to a large index, or search served
+  continuously by a server. An update rewrites the whole index, and the index
+  is held in memory: building 3,000 full texts (35.6 million characters) takes
+  37 s and 0.56 GB. A native engine (rroonga, tantiny) or a database feature
+  (pg_bigm) is the better choice there.
 
 ## Ruby
 
+```console
+$ gem install cjk_index
+```
+
 ```ruby
 require "cjk_index"
 
@@ -71,22 +98,109 @@ Matching, BM25 ranking, highlighting and excerpts follow the browser script
 step for step. The test suite runs both on the same index and checks that
 refs, scores, matched fields, highlights and excerpts are identical.
 
-### Without Jekyll
+### Filters
 
-[examples/search_csv.rb](examples/search_csv.rb) searches a CSV catalogue from
-the command line, in about 40 lines of plain Ruby:
+Fields to filter on are given when the index is built. A `value` filter
+matches whole values, folded like the query (地圖 matches 地図); a `year`
+filter keeps the first four-digit number of a value (`1900-01-01`,
+`１９３１年`; era dates such as 明治33年 give no year).
 
-```console
-$ ruby examples/search_csv.rb items.csv 实录 --ref objectid --variants ja,zh
-12 results for 实录
-jitsu_51ed985b       宣祖[実録] 第1帙 第1冊 巻2戊辰元年(1568)
-...
-$ ruby examples/search_csv.rb items.csv 帝国大学 --ref objectid
-3 results for 帝国大学
-agri_6d032fb5        東京[帝國大學]農學部建物位置圖
-...
+```ruby
+builder = CJKIndex::Builder.new(fields: %w[title], filters: {
+  "type" => "value", "subject" => { "kind" => "value", "separator" => ";" }, "date" => "year"
+})
+builder.add("item1.html", "title" => "…鳥瞰圖", "type" => "地図", "subject" => "建物;農学部", "date" => "1900")
+
+searcher.search("鳥瞰図", filter: { "type" => "地図", "date" => 1890..1910 })
+searcher.search("", filter: { "subject" => %w[建物 植物] })  # no query: every item that passes
+searcher.facets("type", results)  # => [{ value: "地図", count: 12 }, ...]
 ```
 
+Conditions on different fields must all hold; a list matches any of its
+values. A filter only removes results: the scores of the others do not change.
+
+### Updating an index
+
+```ruby
+builder.update("item1.html", "title" => "…", "type" => "写真")  # in place, or added if new
+builder.remove("item2.html")
+
+builder = CJKIndex::Builder.load(File.read("search.json"))    # change a written index
+```
+
+After any changes, `to_h` is the same as for an index built from scratch
+with the remaining documents in the same order (the test suite checks this
+on random sequences of changes). A `Searcher` made from a `Builder` answers
+for its current documents.
+
+### Sharing an index file between processes
+
+A Rails app usually runs several worker processes, and each holds its own
+copy of the index in memory. `IndexFile` keeps them in step through one file:
+
+```ruby
+# config/initializers/cjk_index.rb
+SEARCH = CJKIndex::IndexFile.new(Rails.root.join("db/search-index.json"))
+
+SEARCH.write(builder)                                   # a full build, e.g. in a Rake task
+SEARCH.update { |b| b.update(item.id, item.to_doc) }    # after a record changes, in any process
+SEARCH.searcher.search("鳥瞰図")                        # loads the file again if it has changed
+```
+
+`update` takes a lock, reads the file, runs the block and writes the result,
+so updates from different processes are applied one after another. The new
+index is written to a temporary file and renamed over the old one: a reader
+gets the old index or the new one, never a partly written file. A reader
+keeps the index it loaded until the file changes; `interval:` limits how
+often it checks. Put several changes in one block to write once.
+
+On the Aozora Bunko catalog (16,951 records), a check of an unchanged file
+takes 0.09 ms, and an update about 0.5 s (about 2 s when another process
+wrote last, as the file is read again).
+[bench/index_file.rb](bench/index_file.rb). The lock is `flock`: use a local
+disk, not NFS.
+
+The file is never changed in place, so a writer that is killed leaves the
+previous index intact and nothing to repair. A writer waits at most
+`lock_timeout:` seconds (default 60) for another one, then raises
+`CJKIndex::IndexFile::LockTimeout`.
+
+An update rewrites the whole file, so its cost grows with the index: 0.08 s
+for 1,000 records, 0.24 s for 5,000, 0.7 s for 16,951. A record edited now
+and then needs nothing special; put a few changes in one block; for an
+import, build the index again and `write` it (11 s for 16,951 records).
+
+## Large collections: split indexes
+
+Indexes are written in format `cjk_index/4`: each posting list is a compact
+string, about a third smaller gzipped than format 3 and three times smaller
+before compression. Ruby and the browser script still read formats 2 and 3.
+
+For full text or tens of thousands of records, write a split index: a small
+manifest (refs, lengths, filters) and shards of about 32 KB that the browser
+fetches only when a query needs them.
+
+```ruby
+builder.write_split("assets/search")   # index.json and the shards
+```
+
+```js
+const index = await CJKIndex.load("/assets/search/index.json");
+const results = await index.searchAsync("銀河鉄道", { filter: { type: "小説" } });
+// or: await index.prepare(query); index.search(query, options)
+```
+
+`searchAsync` also works on an index in one file, so a page can use it for
+both. A blank query with a filter, `facets`, `highlight` and `excerpt` need
+no shard. On 1,000 Aozora Bunko full texts (33 MB gzipped as one file), the
+manifest is 28 KB and a query fetches 43 KB (median).
+
+In Ruby, `CJKIndex::Searcher.load("assets/search")` reads shards as queries
+need them, and `CJKIndex::Builder.read("assets/search")` loads a split index
+to change it. Shard names are digests of their content, so they can be cached
+for long; `write_split` removes the shards the new manifest no longer lists.
+`IndexFile` always writes one file. Measurements: [bench/README.md](bench/README.md).
+
 ## Browser
 
 ```html
@@ -96,7 +210,11 @@ agri_6d032fb5        東京[帝國大學]農學部建物位置圖
     index.search("鳥瞰図");  // => [{ ref: "item1.html", score: ..., fields: ["title"] }]
     CJKIndex.highlight("東京帝國大學", "帝国大学"); // => "東京<mark>帝國大學</mark>"
     CJKIndex.excerpt(longText, "鳥瞰図", 80);    // highlighted window around the first match
+
+    var results = index.search("鳥瞰図", { filter: { type: "地図", date: { min: 1890, max: 1910 } } });
+    index.facets("type", results);              // => [{ value: "地図", count: 12 }, ...]
   });
+  // a split index: await index.searchAsync(query, options), see above
 </script>
 ```
 
@@ -123,9 +241,18 @@ cjk_index:
       data: metadata              # rows of _data/metadata.csv
       ref: objectid
       fields: [title, creator, subject]
+      filters:                    # optional
+        type: value
+        subject: { kind: value, separator: ";" }
+        date: year
+    - name: texts
+      collection: texts
+      fields: [title, content]
+      split: true                 # -> assets/cjk-index/texts/index.json and shards
 ```
 
-The runtime is written to `assets/cjk-index/cjk-index.js`.
+The runtime is written to `assets/cjk-index/cjk-index.js`. Moving a site from
+lunr.js: [docs/migrating-from-lunr.md](docs/migrating-from-lunr.md).
 
 ### CollectionBuilder
 
@@ -139,25 +266,125 @@ cjk_index:
 The preset follows CollectionBuilder's own search settings: fields with
 `index: true` in `_data/config-search.csv`, and child objects only when
 `search-child-objects` is on. Result refs are the same `id` values as in
-`assets/js/lunr-store.js`. A working example:
+`assets/js/lunr-store.js`. Filters default to the theme's
+`metadata-facets-fields` (values separated by `;`) and a year from the `date`
+column; set `filters:` to change them. A working example:
 https://github.com/nakamura196/cb-ja-demo
 
+## Bridgetown
+
+Tested with Bridgetown 2.2.2. Node is not needed.
+
+```ruby
+# config/initializers.rb
+require "cjk_index/bridgetown"
+
+Bridgetown.configure do |config|
+  init :cjk_index, indexes: [
+    { name: "posts", collection: "posts", fields: %w[title content], boosts: { title: 3 },
+      filters: { category: "value", date: "year" } }
+  ]
+end
+```
+
+This writes `output/cjk-index/posts.json` and `output/cjk-index/cjk-index.js`.
+Use `data:` and `ref:` instead of `collection:` for rows of
+`src/_data/*.csv`. See [examples/bridgetown](examples/bridgetown).
+
+## Command line and Rake
+
+```console
+$ cjk_index build items.csv --ref id --fields title,creator --boost title=3 --filter type --filter date=year -o index.json --runtime cjk-index.js
+$ cjk_index search index.json 鳥瞰図 --filter type=地図 --filter date=1890..1910
+$ cjk_index runtime --variants ja,zh-safe -o cjk-index.js
+```
+
+`build` reads CSV, a JSON array or JSON Lines, from a file or from standard
+input (`-`). `--filter subject=value:";"` splits values on `;`. Add `--json`
+to `search` for machine-readable results. CSV needs the `csv` gem, which is
+no longer a default gem from Ruby 3.4.
+
+```ruby
+# Rakefile
+require "cjk_index/rake_task"
+
+CJKIndex::RakeTask.new(:search_index) do |t|
+  t.source  = "data/items.csv"            # or t.records = -> { Item.all.map(&:to_search) }
+  t.output  = "public/search/items.json"
+  t.runtime = "public/search/cjk-index.js"
+  t.fields  = %w[title creator]
+  t.boosts  = { "title" => 3 }
+  t.filters = { "type" => "value", "date" => "year" }
+end
+```
+
+### A script with no dependencies
+
+[examples/search_csv.rb](examples/search_csv.rb) searches a CSV catalog from
+the command line, in about 40 lines of plain Ruby:
+
+```console
+$ ruby examples/search_csv.rb items.csv 实录 --ref objectid --variants ja,zh
+12 results for 实录
+jitsu_51ed985b       宣祖[実録] 第1帙 第1冊 巻2戊辰元年(1568)
+...
+$ ruby examples/search_csv.rb items.csv 帝国大学 --ref objectid
+3 results for 帝国大学
+agri_6d032fb5        東京[帝國大學]農學部建物位置圖
+...
+```
+
+## Rails
+
+[examples/rails](examples/rails) (Rails 8.1.4, tested with Puma and four
+workers) keeps one index for all worker processes with `IndexFile`: an
+initializer, a model concern that updates the index after each commit, a
+search action with filters, and a Rake task that rebuilds the index.
+Notes on all the adapters: [docs/notes/adapters.md](docs/notes/adapters.md).
+
 ## Variant tables
 
 | name | folds | source |
 |---|---|---|
-| `ja` (default) | 291 Japanese old forms → modern forms | written for cjk_index |
-| `zh` | 3,202 Traditional → Simplified Chinese | OpenCC `TSCharacters.txt` (Apache-2.0); ambiguous entries dropped |
+| `ja` (default) | 409 Japanese old forms and glyph variants → one form (圖→図, 證→証, 禱/祷, 剝/剥) | Jōyō Kanji Table (2010), Jinmeiyō kanji list and kZVariant, via Unihan 18.0 (Unicode License v3) |
+| `zh-safe` | 2,767 Traditional → Simplified pairs that are one-to-one both ways | OpenCC (Apache-2.0) |
+| `zh` | 3,202 Traditional → Simplified (as in 0.1) | OpenCC `TSCharacters.txt` (Apache-2.0); ambiguous entries dropped |
+| `ko` | 5,660 old Hangul characters in the Hanyang Private Use Area → standard jamo | hypua table (Unlicense), from the KTUG Hanyang PUA Table Project |
 
 With several tables, characters linked by any of them fold to one form,
-preferring the table listed first (`[ja, zh]`: 圖, 図, 图 → 図). The `zh`
-table also merges characters that are distinct in Japanese (後 and 后), so
-it is opt-in.
+preferring the table listed first (`[ja, zh-safe]`: 圖, 図, 图 → 図).
+
+`zh` also merges characters that are distinct in Japanese and in
+Traditional Chinese: 後 and 后, 發 and 髮 (both 发), 機 and 机. Combined with
+`ja`, it merges 45 such groups of Jōyō and Jinmeiyō kanji. `zh-safe` leaves
+those pairs out, so a Simplified query 发展 does not find 發展. Use `zh-safe`
+for Japanese sites that should also match Simplified queries, and `zh` for
+Chinese sites.
+
+Hangul needs no table: precomposed syllables and conjoining jamo are folded
+by Unicode normalization. `ko` is for old Hangul stored in the Private Use
+Area by HWP and the Hanyang fonts; a query typed in standard jamo then finds
+and highlights it.
+
+Browser script size, gzipped: no table 6.3 KB, `ja` 9.0 KB, `ja, zh-safe`
+24.1 KB, `ja, zh` 25.9 KB, `ja, ko` 36.1 KB.
+
+The tables are UTF-8 TSV files in `data/` ("variant<TAB>folded", an optional
+comment column, `#` comments); `script/variant_tables.rb` generates them from
+their sources. Sources, rules and counts:
+[docs/notes/variant-tables.md](docs/notes/variant-tables.md). An index
+records a digest of what its tables fold, and Ruby and the browser script
+refuse an index built with other table contents: rebuild the index after
+changing tables or upgrading from 0.x.
 
 ## Status
 
-0.1, early. Not yet on RubyGems. Planned: index sharding for large sites, a
-Bridgetown adapter, a CLI, and size and speed benchmarks.
+1.0. On RubyGems: `gem install cjk_index`
+(https://rubygems.org/gems/cjk_index). Changes: [CHANGELOG.md](CHANGELOG.md).
+Development is supported by the Ruby Association Grant 2026.
+
+Not yet checked by a native reader: the `ko` table and the Korean examples
+([docs/review/](docs/review/)). Reports are welcome.
 
 ## Development
 
@@ -166,6 +393,9 @@ bundle install
 bundle exec rake test   # the runtime tests need node
 ```
 
+Bug reports and pull requests: see [CONTRIBUTING.md](CONTRIBUTING.md).
+Security problems: [SECURITY.md](SECURITY.md).
+
 ## License
 
 MIT