cjk_index 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0d4df8b57af0d9d2e2755bf87cc9fdb60313d87f68ab17160f5ff4f9f28460d9
-  data.tar.gz: 3bc33035945c48d226c328c58eab9c012f7a93874edde8a3adc2c5a92d9972f3
+  metadata.gz: 843127d7f858d567b1e4eb0d37975111612feab1bdf63fe9d8bc6e4dbaf0e0b1
+  data.tar.gz: 78c8ab89bbaf4477aa0e8b51f72de64503bb71e2401ed13c013eeed40bc3f41c
 SHA512:
-  metadata.gz: 3016fee0464ea949e62360c4d4bfbaf3a4f1a7271c711efe1096b4958ce754b5bf4059566395eb11980fc92a8c55d953989e6562b0fa0429e1e25036a7d277e7
-  data.tar.gz: f7dade68d837d456467e6d1c8fe6e8fbba3335432c7735414df161627942137d626a7a8f270423085957b162e0fa75c3aa709dd7279635e1c4a6c0bb47a9ff2e
+  metadata.gz: f8aef83a37b2b0644cab2d22a0bea337a467ea9c89a142c4a4fe2c702e49cfb4afcf1a9ef4146aa9a656b54ec95ebb9055d53a0ec3439808ea970533ba1588c9
+  data.tar.gz: d88440121eb7f3ebc1239ae28d12b80b7ac706a70065b66c160b0ceaf33f6e434d093bb53e8a1aca6e8f26cbed1ce79556e48eba8837132c05c04c42352e51fa

data/CHANGELOG.md

@@ -0,0 +1,63 @@
+# Changelog
+
+## 0.2.0 (2026-10-28)
+
+- Filters: `Builder.new(filters:)` with `value` and `year` filters;
+  `search(query, filter:)` in Ruby and `search(query, { filter })` in the
+  browser; a blank query with a filter lists every document that passes;
+  `facets(field, results)` counts values. Ruby and the browser give the same
+  results and counts.
+- Index updates: `Builder#remove`, `Builder#update`, `Builder.load`. The
+  changed index is the same as one rebuilt from scratch.
+- `Searcher.new(builder)` follows later changes to the builder (it used to
+  copy the index).
+- `IndexFile`: an index file shared by several processes (e.g. Rails
+  workers). Updates take a lock and are written with a rename, so none is
+  lost and readers never see a partly written file; readers load the file
+  again when it changes. A killed writer leaves the old index intact;
+  `lock_timeout:` bounds the wait for another writer.
+- Index format `cjk_index/3` (adds `filters`). The browser script and
+  `Searcher` still read `cjk_index/2`.
+- Jekyll: `filters:` for an index; the CollectionBuilder preset filters on
+  `metadata-facets-fields` and the year of `date`.
+
+## 0.1.1 (2026-10-18)
+
+No change to the library. This release fixes the state of the repository
+cited in a paper.
+
+- Retrieval quality on two real datasets (69 catalog records from cb-ja-demo,
+  16,951 Aozora Bunko works; 15 queries each), against lunr.js with its
+  default and Japanese settings: [bench/retrieval/](bench/retrieval/).
+  Queries, relevance judgments (committed before the runs), judging rules,
+  scripts and raw outputs are included.
+- Load and query times in Chrome and Safari for 1,000, 5,000 and 16,951
+  records: [bench/browser/](bench/browser/).
+- README and docs/comparison.md: the recommended size is now the range we
+  measured (up to 16,951 records), not "about 20,000".
+- CITATION.cff for citing the software.
+- CODE_OF_CONDUCT.md (Contributor Covenant 2.1), CONTRIBUTING.md, SECURITY.md,
+  issue forms and a pull request template.
+
+## 0.1.0 (2026-10-02)
+
+First release on RubyGems.
+
+- Normalizer in pure Ruby: NFKC, lowercase, katakana → hiragana, 291 old-form
+  kanji → modern forms, the iteration mark 々. Optional Traditional → Simplified
+  Chinese table (3,202 characters, from OpenCC, Apache-2.0). `CJKIndex.normalize`.
+- Tokenizer: positional bigrams for CJK runs, including hangul and old-hangul
+  jamo; Latin words as whole tokens.
+- `CJKIndex::Builder`: BM25 index with field boosts, written as JSON.
+- `CJKIndex::Searcher`: exact substring matching, BM25 ranking, highlighting and
+  excerpts in pure Ruby; prefix matching and one-typo tolerance for Latin words.
+- Browser runtime with no dependencies (`CJKIndex::Runtime.source`), generated
+  from the same tables; the test suite requires identical results in Ruby and
+  under Node.
+- Jekyll plugin (`cjk_index/jekyll`) with a CollectionBuilder preset.
+
+In the repository (not packaged in the gem):
+
+- `examples/search_csv.rb`: search a CSV catalog from the command line, without Jekyll.
+- Benchmarks on Aozora Bunko (up to 16,951 works): [bench/README.md](bench/README.md).
+- Comparison with other search tools: [docs/comparison.md](docs/comparison.md).

data/README.md

@@ -37,13 +37,33 @@ even when it is an exact title. cjk_index instead:
 The browser script is about 6.5 KB gzipped with the Japanese table, 24 KB
 with the Chinese table as well.
 
-How this compares with Pagefind, kensaku, lunr.js and others:
+How this compares with Pagefind, kensaku, lunr.js, rroonga, tantiny and others:
 [docs/comparison.md](docs/comparison.md).
 Index size and query time on up to 16,951 works of Aozora Bunko:
-[bench/README.md](bench/README.md).
+[bench/README.md](bench/README.md). Retrieval quality against lunr.js:
+[bench/retrieval/](bench/retrieval/). Load and query times in Chrome and
+Safari: [bench/browser/](bench/browser/).
+
+When to use it, and when not to:
+
+- **Good fit:** catalogs, collection records or blog posts, from a few
+  thousand records up to the 16,951 we measured (19.3 MB gzipped at that size,
+  a heavy download on slow connections; above about 10,000 records, consider
+  splitting the index); static sites, command-line tools, scripts and small apps; anywhere
+  you want nothing to install outside Ruby, or the same results in Ruby and
+  the browser.
+- **Not a good fit (yet):** large amounts of full text, frequent updates, or
+  search served continuously by a server. The whole index is held in memory
+  and shipped as one file: about 5 bytes per character gzipped, and 3 GB of
+  memory to build 35.6 million characters. A native engine (rroonga, tantiny)
+  or a database feature (pg_bigm) is the better choice there.
 
 ## Ruby
 
+```console
+$ gem install cjk_index
+```
+
 ```ruby
 require "cjk_index"
 
@@ -71,9 +91,81 @@ Matching, BM25 ranking, highlighting and excerpts follow the browser script
 step for step. The test suite runs both on the same index and checks that
 refs, scores, matched fields, highlights and excerpts are identical.
 
+### Filters
+
+Fields to filter on are given when the index is built. A `value` filter
+matches whole values, folded like the query (地圖 matches 地図); a `year`
+filter keeps the first four-digit number of a value (`1900-01-01`,
+`１９３１年`; era dates such as 明治33年 give no year).
+
+```ruby
+builder = CJKIndex::Builder.new(fields: %w[title], filters: {
+  "type" => "value", "subject" => { "kind" => "value", "separator" => ";" }, "date" => "year"
+})
+builder.add("item1.html", "title" => "…鳥瞰圖", "type" => "地図", "subject" => "建物;農学部", "date" => "1900")
+
+searcher.search("鳥瞰図", filter: { "type" => "地図", "date" => 1890..1910 })
+searcher.search("", filter: { "subject" => %w[建物 植物] })  # no query: every item that passes
+searcher.facets("type", results)  # => [{ value: "地図", count: 12 }, ...]
+```
+
+Conditions on different fields must all hold; a list matches any of its
+values. A filter only removes results: the scores of the others do not change.
+
+### Updating an index
+
+```ruby
+builder.update("item1.html", "title" => "…", "type" => "写真")  # in place, or added if new
+builder.remove("item2.html")
+
+builder = CJKIndex::Builder.load(File.read("search.json"))    # change a written index
+```
+
+After any changes, `to_h` is the same as for an index built from scratch
+with the remaining documents in the same order (the test suite checks this
+on random sequences of changes). A `Searcher` made from a `Builder` answers
+for its current documents.
+
+### Sharing an index file between processes
+
+A Rails app usually runs several worker processes, and each holds its own
+copy of the index in memory. `IndexFile` keeps them in step through one file:
+
+```ruby
+# config/initializers/cjk_index.rb
+SEARCH = CJKIndex::IndexFile.new(Rails.root.join("db/search-index.json"))
+
+SEARCH.write(builder)                                   # a full build, e.g. in a Rake task
+SEARCH.update { |b| b.update(item.id, item.to_doc) }    # after a record changes, in any process
+SEARCH.searcher.search("鳥瞰図")                        # loads the file again if it has changed
+```
+
+`update` takes a lock, reads the file, runs the block and writes the result,
+so updates from different processes are applied one after another. The new
+index is written to a temporary file and renamed over the old one: a reader
+gets the old index or the new one, never a partly written file. A reader
+keeps the index it loaded until the file changes; `interval:` limits how
+often it checks. Put several changes in one block to write once.
+
+On the Aozora Bunko catalog (16,951 records, 61 MB file), a check of an
+unchanged file takes 0.09 ms, loading a changed file 0.6–0.7 s, and an update
+0.6 s (2.1 s when another process wrote last, as the file is read again).
+[bench/index_file.rb](bench/index_file.rb). The lock is `flock`: use a local
+disk, not NFS.
+
+The file is never changed in place, so a writer that is killed leaves the
+previous index intact and nothing to repair. A writer waits at most
+`lock_timeout:` seconds (default 60) for another one, then raises
+`CJKIndex::IndexFile::LockTimeout`.
+
+An update rewrites the whole file, so its cost grows with the index: 0.08 s
+for 1,000 records, 0.24 s for 5,000, 0.7 s for 16,951. A record edited now
+and then needs nothing special; put a few changes in one block; for an
+import, build the index again and `write` it (11 s for 16,951 records).
+
 ### Without Jekyll
 
-[examples/search_csv.rb](examples/search_csv.rb) searches a CSV catalogue from
+[examples/search_csv.rb](examples/search_csv.rb) searches a CSV catalog from
 the command line, in about 40 lines of plain Ruby:
 
 ```console
@@ -96,6 +188,9 @@ agri_6d032fb5        東京[帝國大學]農學部建物位置圖
     index.search("鳥瞰図");  // => [{ ref: "item1.html", score: ..., fields: ["title"] }]
     CJKIndex.highlight("東京帝國大學", "帝国大学"); // => "東京<mark>帝國大學</mark>"
     CJKIndex.excerpt(longText, "鳥瞰図", 80);    // highlighted window around the first match
+
+    var results = index.search("鳥瞰図", { filter: { type: "地図", date: { min: 1890, max: 1910 } } });
+    index.facets("type", results);              // => [{ value: "地図", count: 12 }, ...]
   });
 </script>
 ```
@@ -123,6 +218,10 @@ cjk_index:
       data: metadata              # rows of _data/metadata.csv
       ref: objectid
       fields: [title, creator, subject]
+      filters:                    # optional
+        type: value
+        subject: { kind: value, separator: ";" }
+        date: year
 ```
 
 The runtime is written to `assets/cjk-index/cjk-index.js`.
@@ -139,7 +238,9 @@ cjk_index:
 The preset follows CollectionBuilder's own search settings: fields with
 `index: true` in `_data/config-search.csv`, and child objects only when
 `search-child-objects` is on. Result refs are the same `id` values as in
-`assets/js/lunr-store.js`. A working example:
+`assets/js/lunr-store.js`. Filters default to the theme's
+`metadata-facets-fields` (values separated by `;`) and a year from the `date`
+column; set `filters:` to change them. A working example:
 https://github.com/nakamura196/cb-ja-demo
 
 ## Variant tables
@@ -156,8 +257,11 @@ it is opt-in.
 
 ## Status
 
-0.1, early. Not yet on RubyGems. Planned: index sharding for large sites, a
-Bridgetown adapter, a CLI, and size and speed benchmarks.
+0.2. On RubyGems: `gem install cjk_index`
+(https://rubygems.org/gems/cjk_index). 0.2 adds filters, index updates and
+an index file shared by several processes. Planned for 1.0: index sharding
+for large sites (see the [benchmarks](bench/README.md)), a Bridgetown
+adapter, a CLI, and variant tables from official sources.
 
 ## Development
 
@@ -166,6 +270,9 @@ bundle install
 bundle exec rake test   # the runtime tests need node
 ```
 
+Bug reports and pull requests: see [CONTRIBUTING.md](CONTRIBUTING.md).
+Security problems: [SECURITY.md](SECURITY.md).
+
 ## License
 
 MIT

data/lib/cjk_index/builder.rb

@@ -5,82 +5,331 @@ require "json"
 module CJKIndex
   # Builds a prebuilt inverted index that the browser runtime loads as JSON.
   #
-  #   builder = CJKIndex::Builder.new(fields: %w[title creator], boosts: { "title" => 3 })
-  #   builder.add("item1.html", "title" => "東京帝國大學", "creator" => "…")
+  #   builder = CJKIndex::Builder.new(fields: %w[title creator], boosts: { "title" => 3 },
+  #                                   filters: { "type" => "value", "date" => "year" })
+  #   builder.add("item1.html", "title" => "東京帝國大學", "creator" => "…", "type" => "地図", "date" => "1900")
   #   File.write("search-index.json", builder.to_json)
   #
-  # Format (version 2):
+  # Format (version 3):
   #   {
-  #     "format":  "cjk_index/2",
+  #     "format":  "cjk_index/3",
   #     "variants": ["ja"],
   #     "fields":  ["title", "creator"],
   #     "boosts":  [3, 1],
   #     "refs":    ["item1.html", ...],
   #     "lengths": [len(doc0.title), len(doc0.creator), len(doc1.title), ...],
-  #     "tokens":  { "東京": [doc, field, n, pos1, ..., posn, doc, field, n, ...], ... }
+  #     "tokens":  { "東京": [doc, field, n, pos1, ..., posn, doc, field, n, ...], ... },
+  #     "filters": {
+  #       "type": { "kind": "value", "values": ["地図", ...], "docs": [[0], [], ...] },
+  #       "date": { "kind": "year", "docs": [1900, null, ...] }
+  #     }
   #   }
   # doc and field are positions in "refs" and "fields"; n is the number of
   # occurrences and pos are character offsets in the normalized field text.
   # Offsets let the runtime require the bigrams of a query to be adjacent
   # (exact substring match); lengths feed BM25 ranking.
+  # "filters" is present only when filters were configured. A value filter
+  # stores each distinct value once (by its folded form, as first written)
+  # and lists the values of each document by number; a year filter stores one
+  # year or null per document. Version 2 is the same without "filters".
+  #
+  # Documents can be removed and replaced (remove, update), also in an index
+  # read back with Builder.load. to_h is then the same as for an index built
+  # from scratch with the remaining documents in the same order.
   class Builder
-    FORMAT = "cjk_index/2"
+    FORMAT = "cjk_index/3"
+    FORMATS = ["cjk_index/2", FORMAT].freeze
 
     # Gap between the values of a multi-valued field, so that a phrase cannot
     # match across two values.
     VALUE_GAP = 1
 
-    attr_reader :fields, :refs, :normalizer
+    FILTER_KINDS = %w[value year].freeze
+
+    # Space, tab and line breaks, written out so that Ruby and JavaScript agree.
+    WHITESPACE = /[ \t\n\r\f\v]+/
+
+    attr_reader :fields, :normalizer, :filters, :version
 
     # normalizer must match the one the runtime script was generated with
     # (Runtime.source(normalizer:)); the index records its tables and the
     # runtime refuses an index built with different ones.
-    def initialize(fields:, boosts: {}, normalizer: Normalizer.default)
+    # filters: field name => "value" or "year", or { "kind" => "value", "separator" => ";" }
+    # for several values written in one string; an Array of names means "value" for each.
+    def initialize(fields:, boosts: {}, normalizer: Normalizer.default, filters: {})
       raise ArgumentError, "fields must not be empty" if fields.empty?
 
       @normalizer = normalizer
       @fields = fields.map(&:to_s)
       @boosts = @fields.map { |f| (boosts[f] || boosts[f.to_sym] || 1).to_f }
+      @filters = Builder.filter_specs(filters)
+      # Indexed by slot; a removed document leaves nil until to_h renumbers.
       @refs = []
       @lengths = []
+      @filter_values = []
+      @doc_tokens = []
+      @doc_tokens_complete = true
       @postings = Hash.new { |h, k| h[k] = [] }
+      @slots = Hash.new { |h, k| h[k] = [] }
+      @count = 0
+      @version = 0
+    end
+
+    def self.filter_specs(filters)
+      filters = filters.to_h { |name| [name, "value"] } if filters.is_a?(Array)
+      filters.to_h do |name, spec|
+        spec = { "kind" => spec } unless spec.is_a?(Hash)
+        spec = spec.to_h { |k, v| [k.to_s, v.to_s] }
+        unless FILTER_KINDS.include?(spec["kind"])
+          raise ArgumentError, "cjk_index: filter #{name} has kind #{spec['kind'].inspect}; use value or year"
+        end
+
+        [name.to_s, spec.slice("kind", "separator")]
+      end
+    end
+
+    # Reads an index written by to_json (String) or to_h back, to update it.
+    # A value filter keeps the form of each value that the index stored.
+    def self.load(index)
+      data = index.is_a?(String) ? JSON.parse(index) : index
+      unless data.is_a?(Hash) && FORMATS.include?(data["format"])
+        raise ArgumentError, "cjk_index: unsupported index format #{data.is_a?(Hash) ? data['format'].inspect : data.class}"
+      end
+
+      filters = (data["filters"] || {}).transform_values { |f| f.slice("kind", "separator") }
+      builder = new(fields: data["fields"], boosts: data["fields"].zip(data["boosts"]).to_h,
+                    normalizer: Normalizer.new(tables: data["variants"]), filters: filters)
+      builder.send(:restore, data)
+      builder
     end
 
+    # Folded form of a value filter's value, used to compare values.
+    def self.filter_key(value, normalizer)
+      normalizer.normalize(value.to_s).gsub(WHITESPACE, " ").delete_prefix(" ").delete_suffix(" ")
+    end
+
+    # The first four-digit number in a value (after folding full-width digits).
+    def self.year(value, normalizer)
+      return value.to_i if value.is_a?(Numeric)
+
+      normalizer.normalize(value.to_s)[/(?<!\d)\d{4}(?!\d)/]&.to_i
+    end
+
+    # Refs of the documents in the index, in index order.
+    def refs = holes? ? @refs.compact : @refs
+
+    def size = @count
+
+    def include?(ref) = !@slots[ref.to_s].empty?
+
     # doc is a Hash of field name => value. Array values are indexed as
     # separate strings (e.g. multi-valued subject fields).
     def add(ref, doc)
-      doc_index = @refs.length
+      slot = @refs.length
       @refs << ref.to_s
-      @fields.each_with_index do |field, field_index|
-        value = doc[field] || doc[field.to_sym]
-        positions = Hash.new { |h, k| h[k] = [] }
-        base = 0
-        Array(value).each do |v|
-          text = v.to_s
-          Tokenizer.tokens_with_offsets(text, unigrams: true, normalizer: @normalizer)
-                   .each { |t, offset| positions[t] << (base + offset) }
-          base += @normalizer.normalize(text).length + VALUE_GAP
-        end
-        @lengths << [base - VALUE_GAP, 0].max
-        positions.each { |token, offsets| @postings[token].push(doc_index, field_index, offsets.length, *offsets) }
-      end
+      @slots[ref.to_s] << slot
+      @count += 1
+      index(slot, doc)
+      self
+    end
+
+    # Removes the first document added with ref. Returns false if there is none.
+    def remove(ref)
+      slot = @slots[ref.to_s].shift or return false
+
+      unindex(slot)
+      @refs[slot] = nil
+      @lengths.fill(nil, slot * @fields.length, @fields.length)
+      @filter_values[slot] = nil
+      @count -= 1
+      true
+    end
+
+    # Replaces the document with ref in place, or adds it if there is none.
+    def update(ref, doc)
+      slot = @slots[ref.to_s].first or return add(ref, doc)
+
+      unindex(slot)
+      index(slot, doc)
       self
     end
 
     def to_h
-      {
+      live = holes? ? @refs.each_index.reject { |slot| @refs[slot].nil? } : @refs.each_index.to_a
+      h = {
         "format" => FORMAT,
         "variants" => @normalizer.tables,
         "fields" => @fields,
         "boosts" => @boosts.map { |b| b == b.to_i ? b.to_i : b },
-        "refs" => @refs,
-        "lengths" => @lengths,
-        "tokens" => @postings.sort.to_h
+        "refs" => live.map { |slot| @refs[slot] },
+        "lengths" => live.flat_map { |slot| @lengths[slot * @fields.length, @fields.length] },
+        "tokens" => holes? ? renumbered_postings(live) : @postings.sort.to_h
       }
+      h["filters"] = @filters.to_h { |name, spec| [name, filter_to_h(name, spec, live)] } unless @filters.empty?
+      h
     end
 
     def to_json(*_args)
       JSON.generate(to_h)
     end
+
+    # For Searcher: the current documents by slot, with nil for removed ones.
+    def live_data # :nodoc:
+      {
+        "variants" => @normalizer.tables, "fields" => @fields, "boosts" => @boosts,
+        "refs" => @refs, "lengths" => @lengths, "tokens" => @postings, "count" => @count,
+        "filters" => @filters.to_h do |name, spec|
+          [name, { "kind" => spec["kind"], "docs" => @filter_values.map { |v| v && v[name] } }]
+        end
+      }
+    end
+
+    private
+
+    def index(slot, doc)
+      in_place = slot < @refs.length - 1
+      tokens = []
+      @fields.each_with_index do |field, field_index|
+        positions = Hash.new { |h, k| h[k] = [] }
+        base = 0
+        Array(value_of(doc, field)).each do |v|
+          text = v.to_s
+          Tokenizer.tokens_with_offsets(text, unigrams: true, normalizer: @normalizer)
+                   .each { |t, offset| positions[t] << (base + offset) }
+          base += @normalizer.normalize(text).length + VALUE_GAP
+        end
+        @lengths[(slot * @fields.length) + field_index] = [base - VALUE_GAP, 0].max
+        positions.each do |token, offsets|
+          postings = @postings[token]
+          # A document replaced in place goes before the later slots, so that
+          # every list stays in the order a fresh build writes.
+          at = in_place ? insertion_point(postings, slot, field_index) : postings.length
+          postings.insert(at, slot, field_index, offsets.length, *offsets)
+        end
+        tokens.concat(positions.keys)
+      end
+      @doc_tokens[slot] = tokens.uniq
+      @filter_values[slot] = @filters.to_h { |name, spec| [name, filter_value(spec, value_of(doc, name))] }
+      @version += 1
+    end
+
+    def unindex(slot)
+      doc_tokens(slot).each do |token|
+        postings = @postings[token]
+        kept = []
+        i = 0
+        while i < postings.length
+          n = 3 + postings[i + 2]
+          kept.concat(postings[i, n]) unless postings[i] == slot
+          i += n
+        end
+        if kept.empty? then @postings.delete(token)
+        else @postings[token] = kept
+        end
+      end
+      @doc_tokens[slot] = nil
+      @version += 1
+    end
+
+    def value_of(doc, field) = doc[field] || doc[field.to_sym]
+
+    # Value filter: [[key, value as written], ...]; year filter: Integer or nil.
+    def filter_value(spec, raw)
+      if spec["kind"] == "year"
+        Array(raw).each do |v|
+          year = Builder.year(v, @normalizer)
+          return year if year
+        end
+        return nil
+      end
+
+      Array(raw).flat_map { |v| spec["separator"] ? v.to_s.split(spec["separator"]) : [v.to_s] }
+                .map { |v| v.gsub(WHITESPACE, " ").strip }.reject(&:empty?)
+                .map { |v| [Builder.filter_key(v, @normalizer), v] }.uniq(&:first)
+    end
+
+    def filter_to_h(name, spec, live)
+      return { "kind" => "year", "docs" => live.map { |slot| @filter_values[slot][name] } } if spec["kind"] == "year"
+
+      ids = {}
+      values = []
+      docs = live.map do |slot|
+        @filter_values[slot][name].map do |key, value|
+          ids.fetch(key) do
+            values << value
+            ids[key] = values.length - 1
+          end
+        end
+      end
+      { "kind" => "value", "values" => values, "docs" => docs }.merge(spec.slice("separator"))
+    end
+
+    def holes? = @count < @refs.length
+
+    # Where an entry for slot and field goes in a list ordered by slot, then field.
+    def insertion_point(postings, slot, field_index)
+      i = 0
+      while i < postings.length && (postings[i] < slot || (postings[i] == slot && postings[i + 1] < field_index))
+        i += 3 + postings[i + 2]
+      end
+      i
+    end
+
+    # Postings with slots renumbered to positions among the live documents.
+    # Removed slots appear in no list any more, and renumbering keeps the
+    # order, so sorted lists stay sorted.
+    def renumbered_postings(live)
+      number = []
+      live.each_with_index { |slot, doc| number[slot] = doc }
+      @postings.sort.to_h do |token, postings|
+        postings = postings.dup
+        i = 0
+        while i < postings.length
+          postings[i] = number[postings[i]]
+          i += 3 + postings[i + 2]
+        end
+        [token, postings]
+      end
+    end
+
+    # Tokens of a slot. An index read with load does not record them, so the
+    # first change scans the postings once.
+    def doc_tokens(slot)
+      unless @doc_tokens_complete
+        @postings.each do |token, postings|
+          i = 0
+          while i < postings.length
+            (@doc_tokens[postings[i]] ||= []) << token
+            i += 3 + postings[i + 2]
+          end
+        end
+        @doc_tokens.each { |tokens| tokens&.uniq! }
+        @doc_tokens_complete = true
+      end
+      @doc_tokens[slot] || []
+    end
+
+    def restore(data)
+      @refs = data["refs"].dup
+      @refs.each_with_index { |ref, slot| @slots[ref] << slot }
+      @count = @refs.length
+      @lengths = data["lengths"].dup
+      unless @lengths.length == @count * @fields.length
+        raise ArgumentError, "cjk_index: index has #{@lengths.length} lengths for #{@count} documents"
+      end
+
+      data["tokens"].each { |token, postings| @postings[token] = postings.dup }
+      filters = data["filters"] || {}
+      @filter_values = Array.new(@count) do |slot|
+        filters.to_h do |name, f|
+          value = f["docs"][slot]
+          next [name, value] if f["kind"] == "year"
+
+          [name, value.map { |id| [Builder.filter_key(f["values"][id], @normalizer), f["values"][id]] }]
+        end
+      end
+      @doc_tokens = []
+      @doc_tokens_complete = false
+    end
   end
 end

data/lib/cjk_index/index_file.rb

@@ -0,0 +1,199 @@
+# frozen_string_literal: true
+
+module CJKIndex
+  # An index file shared by several processes, such as the workers of a Rails
+  # app: any number of readers, and one writer at a time.
+  #
+  #   INDEX = CJKIndex::IndexFile.new("db/search-index.json")
+  #   INDEX.write(builder)                                  # a full build
+  #   INDEX.update { |builder| builder.update(ref, doc) }   # from any process
+  #   INDEX.searcher.search("東京")                         # sees the last update
+  #
+  # update takes an exclusive lock on "<path>.lock", reads the index, runs the
+  # block and writes the result, so updates from different processes are
+  # applied one after another and none is lost. The lock is released when the
+  # process ends, also when it crashes. An index is written to a temporary
+  # file and renamed over the old one, so a reader gets the old index or the
+  # new one, never a partly written file.
+  #
+  # Each reader keeps the index it loaded (a snapshot) and loads the file
+  # again when it has changed: checked on each call to searcher, or once every
+  # interval seconds. While one thread reloads, the others go on with the old
+  # snapshot. The process that wrote an update sees it on its next call.
+  #
+  # The file is never changed in place. A writer killed at any point leaves
+  # the previous index intact, the OS releases its lock, and its temporary
+  # file is removed by the next writer; there is nothing to recover.
+  #
+  # Locks use flock, which is reliable on a local disk; do not share the file
+  # over NFS.
+  class IndexFile
+    # Raised by update and write when another writer holds the lock too long.
+    class LockTimeout < StandardError; end
+
+    attr_reader :path
+
+    # lock_timeout: seconds update and write wait for another writer.
+    # options are passed to Searcher (prefix:, typos:).
+    def initialize(path, interval: 0, lock_timeout: 60, **options)
+      @path = path.to_s
+      @lock_path = "#{@path}.lock"
+      @interval = interval
+      @lock_timeout = lock_timeout
+      @options = options
+      @read_mutex = Mutex.new
+      @write_mutex = Mutex.new
+      @searcher = nil
+      @searcher_stamp = nil
+      @checked_at = nil
+      @builder = nil
+      @builder_stamp = nil
+    end
+
+    # A Searcher for the current file, loaded again if the file has changed.
+    def searcher
+      return @searcher if @searcher && !due?
+
+      if @searcher
+        # Another thread is checking or loading: use the snapshot we have.
+        return @searcher unless @read_mutex.try_lock
+
+        begin
+          reload
+        ensure
+          @read_mutex.unlock
+        end
+      else
+        @read_mutex.synchronize { reload }
+      end
+      @searcher
+    end
+
+    # Applies the block to a Builder holding the index in the file, and writes
+    # the result. The file must exist (see write). If the block raises,
+    # nothing is written.
+    def update
+      @write_mutex.synchronize do
+        lock do
+          # The lock is held, so the file cannot change between stat and read.
+          @builder = Builder.load(File.read(@path, encoding: "UTF-8")) unless @builder && stamp == @builder_stamp
+          done = false
+          begin
+            yield @builder
+            done = true
+          ensure
+            @builder = nil unless done # half-applied changes; read the file again next time
+          end
+          @builder_stamp = replace(@builder.to_json)
+        end
+      end
+      @checked_at = nil
+      self
+    end
+
+    # Replaces the index in the file with builder's (e.g. after a full build).
+    def write(builder)
+      @write_mutex.synchronize do
+        lock do
+          replace(builder.to_json)
+          @builder = nil
+        end
+      end
+      @checked_at = nil
+      self
+    end
+
+    private
+
+    def now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+    def due? = @checked_at.nil? || now - @checked_at >= @interval
+
+    # What identifies one written version of the file. rename gives the new
+    # file a new inode; mtime and size cover systems without inodes. Always
+    # from File.stat: on Windows the stat of an open file differs from it.
+    def stamp
+      stat = File.stat(@path)
+      [stat.ino, stat.dev, stat.size, stat.mtime.to_r]
+    end
+
+    def reload
+      return unless due?
+
+      @checked_at = now
+      # stat does not keep the file open; on Windows an open file cannot be
+      # renamed over, so open it only when it has changed.
+      current = stamp
+      return if @searcher && current == @searcher_stamp
+
+      # If a writer renames between stat and read, this reads the newer file
+      # under the older stamp, and the next check loads it once more.
+      @searcher = Searcher.new(windows_retry { File.read(@path, encoding: "UTF-8") }, **@options)
+      @searcher_stamp = current
+    end
+
+    # Waits for the lock for at most lock_timeout seconds, so that a writer
+    # that hangs while holding it cannot block the others for ever.
+    def lock
+      File.open(@lock_path, File::RDWR | File::CREAT, 0o644) do |f|
+        deadline = now + @lock_timeout
+        until f.flock(File::LOCK_EX | File::LOCK_NB)
+          raise LockTimeout, "cjk_index: #{@lock_path} still locked after #{@lock_timeout} s" if now >= deadline
+
+          sleep 0.01
+        end
+        remove_stale_files
+        yield
+      end
+    end
+
+    # Temporary files are made only while the lock is held, so any found now
+    # were left by a writer that was killed.
+    def remove_stale_files
+      dir = File.dirname(@path)
+      prefix = "#{File.basename(@path)}."
+      Dir.each_child(dir) do |name|
+        File.delete(File.join(dir, name)) if name.start_with?(prefix) && name.end_with?(".tmp")
+      end
+    end
+
+    # Writes json next to the file and renames it over the file. Returns the
+    # stamp of the new file.
+    def replace(json)
+      tmp = "#{@path}.#{Process.pid}.#{Thread.current.object_id}.tmp"
+      File.open(tmp, "w:UTF-8") do |f|
+        f.write(json)
+        f.flush
+        f.fsync
+      end
+      windows_retry { File.rename(tmp, @path) }
+      sync_dir
+      stamp
+    ensure
+      File.delete(tmp) if tmp && File.exist?(tmp)
+    end
+
+    # The rename itself survives a power loss only once the directory is
+    # synced too (Linux ext4, xfs). Windows cannot open a directory; skip it.
+    def sync_dir
+      File.open(File.dirname(@path), &:fsync)
+    rescue SystemCallError, IOError
+      nil
+    end
+
+    # On Windows, renaming over a file fails while a reader has it open, and
+    # opening it can fail while it is being replaced. Both last only a
+    # moment, so try again for up to 5 s.
+    def windows_retry
+      deadline = now + 5
+      begin
+        yield
+      rescue Errno::EACCES, Errno::EPERM
+        raise if now >= deadline
+
+        sleep 0.01
+        retry
+      end
+    end
+  end
+end

data/lib/cjk_index/jekyll.rb

@@ -19,6 +19,10 @@ module CJKIndex
   #         data: metadata                # ...rows of _data/metadata.csv
   #         ref: objectid                 # column used as the result's ref
   #         fields: [title, creator, subject]
+  #         filters:                      # fields to filter on (optional)
+  #           type: value
+  #           subject: { kind: value, separator: ";" }
+  #           date: year
   #       - name: search
   #         preset: collectionbuilder     # reads CollectionBuilder's own config
   #
@@ -70,7 +74,7 @@ module CJKIndex
         raise ::Jekyll::Errors::FatalException, "cjk_index: unknown preset #{spec['preset']}" if spec["preset"]
 
         builder = Builder.new(fields: Array(spec.fetch("fields")), boosts: spec.fetch("boosts", {}),
-                              normalizer: normalizer)
+                              normalizer: normalizer, filters: spec.fetch("filters", {}))
         if spec["collection"]
           docs = site.collections.fetch(spec["collection"]) do
             raise ::Jekyll::Errors::FatalException, "cjk_index: no collection #{spec['collection']}"
@@ -97,6 +101,9 @@ module CJKIndex
       # items with an objectid, child objects only when the theme asks for them,
       # fields marked index=true in _data/config-search.csv, and the same "id"
       # values, so results can be looked up in the existing search store.
+      # Filters default to the theme's metadata-facets-fields (values
+      # separated by ";", as CollectionBuilder writes them) and a year from
+      # the date column.
       module CollectionBuilder
         module_function
 
@@ -107,7 +114,8 @@ module CJKIndex
                    .select { |f| f["index"].to_s == "true" }
                    .map { |f| f["field"] }
           boosts = spec.fetch("boosts") { fields.first ? { fields.first => 3 } : {} }
-          builder = Builder.new(fields: fields, boosts: boosts, normalizer: normalizer)
+          builder = Builder.new(fields: fields, boosts: boosts, normalizer: normalizer,
+                                filters: spec.fetch("filters") { default_filters(site, rows) })
           rows.each do |row|
             next if row["objectid"].to_s.empty?
             next if !row["parentid"].to_s.empty? && !children
@@ -117,6 +125,13 @@ module CJKIndex
           builder
         end
 
+        def default_filters(site, rows)
+          facets = site.data.dig("theme", "metadata-facets-fields").to_s.split(",").map(&:strip).reject(&:empty?)
+          filters = facets.to_h { |f| [f, { "kind" => "value", "separator" => ";" }] }
+          filters["date"] = "year" if rows.any? { |row| row.key?("date") } && !filters.key?("date")
+          filters
+        end
+
         def ref_for(row)
           parent = row["parentid"].to_s
           parent.empty? ? "#{row['objectid']}.html" : "#{parent}.html##{row['objectid']}"

data/lib/cjk_index/runtime.js

@@ -11,7 +11,7 @@
   var TABLES = /*@TABLES@*/[];
   var CJK = new RegExp(/*@CJK@*/'', 'u');
   var SEPARATOR = new RegExp(/*@SEPARATOR@*/'', 'u');
-  var FORMAT = 'cjk_index/2';
+  var FORMATS = ['cjk_index/2', 'cjk_index/3'];
 
   // BM25 parameters (the usual defaults, as in Lucene and Pagefind).
   var K1 = 1.2;
@@ -89,6 +89,11 @@
 
   // Substring test on normalized text, ignoring whitespace. Useful for
   // filters that do not need the index (e.g. a browse page).
+  // Folded form of a filter value: as Builder.filter_key in Ruby.
+  function filterKey(value) {
+    return normalize(value).replace(/[ \t\n\r\f\v]+/g, ' ').replace(/^ /, '').replace(/ $/, '');
+  }
+
   function includes(haystack, needle) {
     var strip = function (s) { return normalize(s).replace(/\s+/g, ''); };
     return strip(haystack).indexOf(strip(needle)) !== -1;
@@ -107,7 +112,7 @@
   }
 
   function Index(data, options) {
-    if (!data || data.format !== FORMAT) {
+    if (!data || FORMATS.indexOf(data.format) < 0) {
       throw new Error('cjk_index: unsupported index format ' + (data && data.format));
     }
     if (JSON.stringify(data.variants) !== JSON.stringify(TABLES)) {
@@ -127,8 +132,89 @@
     for (var i = 0; i < this.lengths.length; i++) avg[i % nf] += this.lengths[i];
     this.avgLength = avg.map(function (s) { return s / Math.max(1, data.refs.length) || 1; });
     this.latinTokens = Object.keys(this.tokens).filter(isLatin).sort();
+    this.slotOf = new Map();
+    for (var d = 0; d < this.refs.length; d++) if (!this.slotOf.has(this.refs[d])) this.slotOf.set(this.refs[d], d);
+    // Filter values by document: keys (value filters) or a year, with the
+    // form to show for each key and its rank of first appearance.
+    this.filters = {};
+    var filters = data.filters || {};
+    Object.keys(filters).forEach(function (name) {
+      var f = filters[name];
+      var docs = f.docs;
+      var display = new Map();
+      if (f.kind !== 'year') {
+        var keys = f.values.map(filterKey);
+        f.values.forEach(function (v, i) { if (!display.has(keys[i])) display.set(keys[i], v); });
+        docs = f.docs.map(function (ids) { return ids.map(function (id) { return keys[id]; }); });
+      }
+      var rank = new Map();
+      docs.forEach(function (value) {
+        (f.kind === 'year' ? (value === null ? [] : [value]) : value).forEach(function (key) {
+          if (!rank.has(key)) rank.set(key, rank.size);
+        });
+      });
+      if (f.kind === 'year') rank.forEach(function (r, year) { display.set(year, year); });
+      this.filters[name] = { kind: f.kind, docs: docs, display: display, rank: rank };
+    }, this);
   }
 
+  // [[filter, test], ...] from the filter option of search.
+  Index.prototype._compileFilter = function (filter) {
+    var self = this;
+    var out = [];
+    Object.keys(filter).forEach(function (name) {
+      var f = self.filters[name];
+      if (!f) throw new Error('cjk_index: no filter ' + name);
+      var c = filter[name];
+      if (c === null || c === undefined || (Array.isArray(c) && !c.length)) return;
+      if (f.kind === 'year') {
+        var min = typeof c === 'object' ? c.min : c;
+        var max = typeof c === 'object' ? c.max : c;
+        out.push([f, function (year) {
+          return year !== null && (min === null || min === undefined || year >= min) &&
+            (max === null || max === undefined || year <= max);
+        }]);
+      } else {
+        var keys = new Set((Array.isArray(c) ? c : [c]).map(filterKey));
+        out.push([f, function (values) { return values.some(function (k) { return keys.has(k); }); }]);
+      }
+    });
+    return out;
+  };
+
+  Index.prototype._pass = function (doc, conditions) {
+    for (var i = 0; i < conditions.length; i++) if (!conditions[i][1](conditions[i][0].docs[doc])) return false;
+    return true;
+  };
+
+  // Counts of the values of a filter, for a facet list: over results (as
+  // returned by search, or refs) or over the whole index. Returns
+  // [{value, count}], the most frequent first, ties in index order.
+  Index.prototype.facets = function (field, results) {
+    var self = this;
+    var f = this.filters[field];
+    if (!f) throw new Error('cjk_index: no filter ' + field);
+    var docs = [];
+    if (results) {
+      results.forEach(function (r) {
+        var d = self.slotOf.get(typeof r === 'object' ? r.ref : String(r));
+        if (d !== undefined) docs.push(d);
+      });
+    } else {
+      for (var d = 0; d < this.refs.length; d++) docs.push(d);
+    }
+    var counts = new Map();
+    docs.forEach(function (doc) {
+      var v = f.docs[doc];
+      (f.kind === 'year' ? (v === null ? [] : [v]) : v).forEach(function (key) {
+        counts.set(key, (counts.get(key) || 0) + 1);
+      });
+    });
+    var out = Array.from(counts.entries());
+    out.sort(function (a, b) { return b[1] - a[1] || f.rank.get(a[0]) - f.rank.get(b[0]); });
+    return out.map(function (e) { return { value: f.display.get(e[0]), count: e[1] }; });
+  };
+
   // Map of "doc,field" key -> Set of positions, merged over the alternatives.
   Index.prototype._positions = function (tokens) {
     var nf = this.fields.length;
@@ -168,12 +254,23 @@
   // Every phrase of the query must occur in the document (in any field),
   // with its tokens adjacent and in order. Returns [{ref, score, fields}]
   // with the best match first; fields lists the names of the fields that matched.
-  Index.prototype.search = function (query) {
+  // options.filter: { type: '地図', subject: ['植物', '動物'], date: { min: 1900, max: 1950 } }
+  // keeps the documents that meet every condition (see Searcher#search in Ruby).
+  // With a filter, a blank query returns every document that passes it.
+  Index.prototype.search = function (query, options) {
     var self = this;
     var nf = this.fields.length;
     var n = this.refs.length;
+    var conditions = options && options.filter ? this._compileFilter(options.filter) : null;
     var qp = phrases(query, false);
-    if (!qp.length) return [];
+    if (!qp.length) {
+      if (!conditions) return [];
+      var all = [];
+      for (var d = 0; d < n; d++) {
+        if (this._pass(d, conditions)) all.push({ ref: this.refs[d], score: 0, fields: [] });
+      }
+      return all;
+    }
     var scores = new Map();
     var matchedFields = new Map();
     var docsSoFar = null;
@@ -225,6 +322,7 @@
 
     var results = [];
     docsSoFar.forEach(function (doc) {
+      if (conditions && !self._pass(doc, conditions)) return;
       results.push({ ref: self.refs[doc], score: scores.get(doc), fields: Array.from(matchedFields.get(doc)), doc: doc });
     });
     results.sort(function (a, b) { return b.score - a.score || a.doc - b.doc; });
@@ -310,7 +408,7 @@
   }
 
   var api = {
-    tables: TABLES, normalize: normalize, tokenize: tokenize, phrases: phrases, includes: includes,
+    tables: TABLES, normalize: normalize, filterKey: filterKey, tokenize: tokenize, phrases: phrases, includes: includes,
     highlight: highlight, excerpt: excerpt, Index: Index, load: load
   };
   if (typeof module !== 'undefined' && module.exports) module.exports = api;

data/lib/cjk_index/searcher.rb

@@ -21,9 +21,11 @@ module CJKIndex
     # Score multiplier for matches found by prefix or typo expansion.
     EXPANDED = 0.6
 
-    attr_reader :fields, :refs, :normalizer
+    attr_reader :fields, :normalizer
 
     # index is a Builder, its to_h, or the JSON it wrote (String or parsed).
+    # A Searcher made from a Builder answers for the builder's current
+    # documents, also after later add, remove and update.
     # prefix: the last Latin word of a query also matches longer words.
     # typos: Latin words of 5+ letters also match words one edit away.
     def self.load(path, **options)
@@ -31,13 +33,16 @@ module CJKIndex
     end
 
     def initialize(index, prefix: true, typos: true, normalizer: nil)
-      data = case index
-             when Builder then index.to_h
-             when String then JSON.parse(index)
-             else index
-             end
-      unless data.is_a?(Hash) && data["format"] == Builder::FORMAT
-        raise ArgumentError, "cjk_index: unsupported index format #{data.is_a?(Hash) ? data['format'].inspect : data.class}"
+      @prefix = prefix
+      @typos = typos
+      if index.is_a?(Builder)
+        @builder = index
+        data = index.live_data
+      else
+        data = index.is_a?(String) ? JSON.parse(index) : index
+        unless data.is_a?(Hash) && Builder::FORMATS.include?(data["format"])
+          raise ArgumentError, "cjk_index: unsupported index format #{data.is_a?(Hash) ? data['format'].inspect : data.class}"
+        end
       end
 
       @normalizer = normalizer || Normalizer.new(tables: data["variants"])
@@ -46,28 +51,43 @@ module CJKIndex
                              "but this normalizer folds with #{@normalizer.tables.inspect}; rebuild both together"
       end
 
-      @prefix = prefix
-      @typos = typos
-      @fields = data["fields"]
-      @boosts = data["boosts"]
-      @refs = data["refs"]
-      @lengths = data["lengths"]
-      @tokens = data["tokens"]
-      nf = @fields.length
-      avg = Array.new(nf, 0.0)
-      @lengths.each_with_index { |len, i| avg[i % nf] += len }
-      @avg_length = avg.map { |s| (v = s / [1, @refs.length].max).zero? ? 1 : v }
-      @latin_tokens = @tokens.keys.select { |t| latin?(t) }.sort
+      setup(data)
+    end
+
+    # Refs of the documents in the index, in index order.
+    def refs
+      refresh
+      @refs.compact
+    end
+
+    # Names of the filters the index was built with.
+    def filters
+      refresh
+      @filters.keys
     end
 
     # Every phrase of the query must occur in the document (in any field),
     # with its tokens adjacent and in order. Returns [{ ref:, score:, fields: }]
     # with the best match first; fields lists the fields that matched.
-    def search(query)
+    #
+    # filter: { "type" => "地図", "subject" => ["植物", "動物"], "date" => 1900..1950 }
+    # keeps the documents that meet every condition: a value filter matches
+    # any of the given values (folded like the query), a year filter takes a
+    # Range, { "min" => 1900, "max" => 1950 } or one year. Scores do not
+    # depend on the filter. With a filter, a blank query returns every
+    # document that passes it, in index order, with score 0.
+    def search(query, filter: nil)
+      refresh
       nf = @fields.length
-      n = @refs.length
+      n = @count
+      conditions = filter && compile_filter(filter)
       qp = Tokenizer.phrases(query, normalizer: @normalizer)
-      return [] if qp.empty?
+      if qp.empty?
+        return [] unless conditions
+
+        return @refs.each_index.select { |doc| @refs[doc] && pass?(doc, conditions) }
+                    .map { |doc| { ref: @refs[doc], score: 0.0, fields: [] } }
+      end
 
       scores = Hash.new(0.0)
       matched_fields = Hash.new { |h, k| h[k] = [] }
@@ -108,16 +128,108 @@ module CJKIndex
         end
       end
 
+      docs_so_far = docs_so_far.select { |doc| pass?(doc, conditions) } if conditions
       docs_so_far.sort_by { |doc| [-scores[doc], doc] }
                  .map { |doc| { ref: @refs[doc], score: scores[doc], fields: matched_fields[doc] } }
     end
 
+    # Counts of the values of a filter, for a facet list: over results (as
+    # returned by search, or refs) or over the whole index. Returns
+    # [{ value:, count: }], the most frequent first, ties in index order.
+    # Year filters count each year.
+    def facets(field, results = nil)
+      refresh
+      filter = @filters.fetch(field.to_s) { raise ArgumentError, "cjk_index: no filter #{field}" }
+      docs = if results
+               results.filter_map { |r| @slot_of[r.is_a?(Hash) ? (r[:ref] || r["ref"]) : r.to_s] }
+             else
+               @refs.each_index.select { |doc| @refs[doc] }
+             end
+      counts = Hash.new(0)
+      docs.each do |doc|
+        value = filter[:docs][doc]
+        if filter[:kind] == "year" then counts[value] += 1 unless value.nil?
+        else value.each { |key| counts[key] += 1 }
+        end
+      end
+      counts.sort_by { |key, count| [-count, filter[:rank][key]] }
+            .map { |key, count| { value: filter[:display][key], count: count } }
+    end
+
     def highlight(text, query) = CJKIndex.highlight(text, query, normalizer: @normalizer)
 
     def excerpt(text, query, length = 80) = CJKIndex.excerpt(text, query, length, normalizer: @normalizer)
 
     private
 
+    def setup(data)
+      @version = @builder&.version
+      @fields = data["fields"]
+      @boosts = data["boosts"]
+      @refs = data["refs"]
+      @lengths = data["lengths"]
+      @tokens = data["tokens"]
+      @count = data["count"] || @refs.length
+      nf = @fields.length
+      avg = Array.new(nf, 0.0)
+      @lengths.each_with_index { |len, i| avg[i % nf] += len if len }
+      @avg_length = avg.map { |s| (v = s / [1, @count].max).zero? ? 1 : v }
+      @latin_tokens = @tokens.keys.select { |t| latin?(t) }.sort
+      @slot_of = {}
+      @refs.each_with_index { |ref, doc| @slot_of[ref] ||= doc if ref }
+      @filters = (data["filters"] || {}).to_h { |name, f| [name, setup_filter(f)] }
+    end
+
+    # Filter values by document: keys (value filters) or a year, with the
+    # form to show for each key and its rank of first appearance.
+    def setup_filter(f)
+      docs = if f["kind"] == "year" then f["docs"]
+             elsif f["values"]
+               keys = f["values"].map { |v| Builder.filter_key(v, @normalizer) }
+               f["docs"].map { |ids| ids.map { |id| keys[id] } }
+             else
+               f["docs"].map { |pairs| pairs&.map(&:first) }
+             end
+      display = {}
+      if f["values"] then f["values"].each { |v| display[Builder.filter_key(v, @normalizer)] ||= v }
+      elsif f["kind"] == "value" then f["docs"].each { |pairs| pairs&.each { |key, v| display[key] ||= v } }
+      end
+      rank = {}
+      docs.each_with_index do |value, doc|
+        next unless @refs[doc]
+
+        Array(value).each { |key| rank[key] ||= rank.size }
+      end
+      display = rank.keys.to_h { |year| [year, year] } if f["kind"] == "year"
+      { kind: f["kind"], docs: docs, display: display, rank: rank }
+    end
+
+    def refresh
+      setup(@builder.live_data) if @builder && @builder.version != @version
+    end
+
+    # [[filter, test], ...] from a filter argument of search.
+    def compile_filter(filter)
+      filter.filter_map do |name, condition|
+        f = @filters.fetch(name.to_s) { raise ArgumentError, "cjk_index: no filter #{name}" }
+        next if condition.nil? || (condition.is_a?(Array) && condition.empty?)
+
+        if f[:kind] == "year"
+          range = case condition
+                  when Range then condition
+                  when Hash then (condition["min"] || condition[:min])..(condition["max"] || condition[:max])
+                  else Integer(condition)..Integer(condition)
+                  end
+          [f, ->(year) { !year.nil? && range.cover?(year) }]
+        else
+          keys = Array(condition).map { |v| Builder.filter_key(v, @normalizer) }.to_set
+          [f, ->(values) { values.any? { |key| keys.include?(key) } }]
+        end
+      end
+    end
+
+    def pass?(doc, conditions) = conditions.all? { |f, test| test.call(f[:docs][doc]) }
+
     def latin?(token) = !token.match?(Tokenizer::CJK)
 
     # "doc * nf + field" => Set of positions, merged over the alternatives.
@@ -125,7 +237,7 @@ module CJKIndex
       nf = @fields.length
       map = {}
       tokens.each do |token|
-        postings = @tokens[token] or next
+        postings = @tokens.fetch(token, nil) or next
         i = 0
         while i < postings.length
           count = postings[i + 2]

data/lib/cjk_index/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module CJKIndex
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/cjk_index.rb

@@ -6,6 +6,7 @@ require_relative "cjk_index/tokenizer"
 require_relative "cjk_index/builder"
 require_relative "cjk_index/runtime"
 require_relative "cjk_index/searcher"
+require_relative "cjk_index/index_file"
 
 # Dictionary-free full-text search for Chinese, Japanese and Korean text:
 # build the index in Ruby, query it in Ruby (Searcher) or in the browser

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: cjk_index
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Satoru Nakamura
@@ -16,6 +16,7 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- CHANGELOG.md
 - LICENSE
 - LICENSE-OpenCC
 - NOTICE
@@ -24,6 +25,7 @@ files:
 - data/zh-hant-hans.tsv
 - lib/cjk_index.rb
 - lib/cjk_index/builder.rb
+- lib/cjk_index/index_file.rb
 - lib/cjk_index/jekyll.rb
 - lib/cjk_index/normalizer.rb
 - lib/cjk_index/runtime.js
@@ -37,6 +39,9 @@ licenses:
 - Apache-2.0
 metadata:
   source_code_uri: https://github.com/nakamura196/cjk_index
+  documentation_uri: https://github.com/nakamura196/cjk_index#readme
+  changelog_uri: https://github.com/nakamura196/cjk_index/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/nakamura196/cjk_index/issues
   rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths: