nind 2.2.0__tar.gz → 2.2.1__tar.gz

nind-2.2.1/.gitignore

@@ -0,0 +1,9 @@
+*.kdev4
+build/
+*~
+*.pyc
+Dist*/
+docs/_build/
+bench/results/
+bench/msmarco/*
+!bench/msmarco/*.py

{nind-2.2.0 → nind-2.2.1}/CLAUDE.md

@@ -105,7 +105,7 @@ Parallels the C++ layering with lowercase-suffixed module names for the index cl
 - `NindFile.py` — low-level binary codec. Historically read-only-oriented (the `litXxx` getters have long existed for every format), but now has a full writer counterpart too: `ejcritNombre1/3/4/5`, `ejcritNombreULat`/`ejcritNombreSLat` (both varint encodings, full tier range matching the readers and the C++ implementation), `ejcritChaine`, `ejcritZejros`.
 - `NindPadFile.py` — the shared "pad file" envelope (header, indirection block(s), specifics, identification trailer); read-only, no writer (mirrors `NindBasics::NindPadFile`, but nothing in Python subclasses it as a writer the way `NindIndex`/`NindRetrolexicon` do in C++).
 - `NindRetrolexicon.py`, `NindLexiconindex.py`, `NindTermindex.py`, `NindLocalindex.py` — ergonomic read-only Python wrappers around the corresponding `NindIndex`-family C++ classes; their hot-path lookups delegate to `nind._native`, as does the shared structural-diagnostic base (`analyseFichierPadFile`/`analyseFichierIndex`, inherited from `NindPadFile.py`/`NindIndex.py`). The remaining, per-format diagnostic/analysis methods (`dumpeFichier`, `debogueIndex`, `donneCollisions`, `donneMax`, `donneClef`, `afficheTerme`, `afficheDocument`, and each class's own `analyseFichierXxx` beyond the shared base) remain hand-rolled pure-Python parsing, since `nind._native` doesn't expose that introspection surface yet (see "Planned follow-up work" below). `NindLexiconindex` is hash-bucketed (`clefB(word) % nombreIndirection`), the other two are directly indexed by id.
-- `nind_engine.py` — `NindIndexer` (writer) and `NindEngine` (BM25-style search) built on top of the above. `NindIndexer` is the *only* Python writer for the index-family formats (`.nindlexiconindex`/`.nindtermindex`/`.nindlocalindex`) — it builds them via `nind._native`'s `is_writer=True` constructors. It does not write `.nindretrolexicon` (not needed by `NindEngine`, which only supports simple, non-compound words). Indexing is parallel but every file keeps a single writer: files are tokenized in worker processes (`workers` argument of `index_files`; in-process below `PARALLEL_MIN_FILES` files), then the term and local index files are written concurrently from two threads through the bulk bindings `NindTermIndex.set_postings`/`NindLocalIndex.set_local_def_from_term_ids`, which take plain ints (building `Document`/`Term`/`Localisation` objects in Python cost ~20× the write itself) and release the GIL. `set_local_def_from_term_ids` splits a term's positions into entries of ≤255 (`<nbreLocalisations>` is one byte).
+- `nind_engine.py` — `NindIndexer` (writer) and `NindEngine` (BM25-style search) built on top of the above. `NindIndexer` is the *only* Python writer for the index-family formats (`.nindlexiconindex`/`.nindtermindex`/`.nindlocalindex`) — it builds them via `nind._native`'s `is_writer=True` constructors. It does not write `.nindretrolexicon` (not needed by `NindEngine`, which only supports simple, non-compound words). Indexing is parallel but every file keeps a single writer: files are tokenized in worker processes (`workers` argument of `index_files`; in-process below `PARALLEL_MIN_FILES` files), then the term and local index files are written concurrently from two threads through the flat-buffer bulk bindings `NindTermIndex.set_term_defs_arrays`/`NindLocalIndex.set_local_defs_arrays`, which take `array('I')` buffers (building `Document`/`Term`/`Localisation` objects in Python cost ~20× the write itself) and release the GIL. The `*_local_def*_arrays` writers split a term's positions into entries of ≤255 (`<nbreLocalisations>` is one byte).
 - `Nind_*.py` at the package root — standalone CLI scripts (dump a document, convert corpora, search, diagnostics)
 - `amose/` — Amose-specific conversion/parsing scripts (Lucene dump, XML-CLEF, sample corpora)
 

{nind-2.2.0 → nind-2.2.1}/CMakeLists.txt

@@ -113,7 +113,7 @@ endif ()
 
 ########### setting project verion number ###############
 
-set(NIND_VERSION "2.2.0")
+set(NIND_VERSION "2.2.1")
 
 string(REGEX REPLACE "^([0-9]+)\\.[0-9]+\\.[0-9]+$" "\\1" NIND_VERSION_MAJOR "${NIND_VERSION}")
 string(REGEX REPLACE "^[0-9]+\\.([0-9]+)\\.[0-9]+$" "\\1" NIND_VERSION_MINOR "${NIND_VERSION}")

{nind-2.2.0 → nind-2.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: nind
-Version: 2.2.0
+Version: 2.2.1
 Summary: Python frontend for the Nind Indexing Module, with its native C++ index stack built in
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown

nind-2.2.1/bench/README.md

@@ -0,0 +1,189 @@
+# nind performance benchmarks
+
+nind is a flat-file indexer, not a full search engine, so it's benchmarked
+here like a storage/indexing engine rather than with an anserini/pyserini-style
+retrieval-effectiveness harness (nDCG/MAP against relevance judgments) — see
+the design discussion this came out of for why. Three things are measured:
+
+1. **Indexing throughput & on-disk index size**, across a corpus-size sweep —
+   using nind's efficient *bulk-build* write pattern.
+2. **The cost of incremental, per-document indexing** into a growing index —
+   a different, also-real usage pattern, with a real and important caveat
+   (see below).
+3. **Term/document lookup latency**, through the same `nind._native`-backed
+   Python reader classes real callers use.
+
+Retrieval effectiveness (nDCG/MAP vs. qrels) is out of scope: `NindEngine`
+would first need stable external document ids and real inverted-list-driven
+candidate generation instead of its current O(N·|Q|) full scan.
+
+## Corpus: MS MARCO Passage Ranking
+
+Freely downloadable, no license/registration required, and its passage ids
+are already a dense integer sequence from 0 — a good match for nind's
+document-identifier model. Get `collection.tsv` (one passage per line,
+`pid<TAB>text`, ~8.84M rows) from the official MS MARCO Passage Ranking
+release — see <https://github.com/microsoft/MSMARCO-Passage-Ranking> for the
+current download instructions/links (verify the link is still current;
+dataset hosting has moved before).
+
+## Prerequisites
+
+```bash
+# Python side: builds nind._native and installs the reader/writer classes.
+# This alone is enough for the bulk-build benchmark (1) and lookup latency (3).
+uv sync
+
+# C++ side, only needed for the incremental-indexing benchmark (2):
+NIND_DIST=/path/to/dist ./gbuild.sh -m Release   # Release, not Debug - Debug timings are misleading
+# if cmake/ninja aren't installed system-wide, `uv tool install cmake ninja` works too
+```
+
+## 1. Indexing throughput & size (bulk build)
+
+```bash
+./bench/run_bulk_indexing_benchmark.sh /path/to/collection.tsv 10000,100000,1000000
+```
+
+For each size in the comma-separated list, this converts the first N
+passages to nind's corpus text format (`bench/msmarco/convert_to_nind.py`,
+offsetting `pid` by +1 since nind treats document id 0 as a sentinel
+elsewhere), then runs `bench/index_bulk.py`, which writes each term's
+definition and each document's local definition to disk **exactly once**,
+via `nind._native`'s `NindLexiconIndex`/`NindTermIndex`/`NindLocalIndex`
+writers (the same write-once pattern `nind.nind_engine.NindIndexer` uses),
+reading from the compact bulk corpus file instead of one file per document.
+
+The corpus is split into line-aligned byte ranges handled by a pool of worker
+processes (`--workers`, default: all cores); the main process only drives
+the nind writers, which are single-file and single-threaded:
+
+1. *pass 1* (workers): vocabulary and highest document id per chunk; the
+   main process then writes the lexicon, terms added in sorted order;
+2. *pass 2* (workers): parsing and term→id mapping. Each chunk's documents
+   come back as flat `array('I')` buffers, and the main process writes them
+   in document order with one `set_local_defs_arrays` call per chunk. Each
+   worker also spills its chunk's postings to a temporary file next to the
+   index, sorted by term id and split into term-id ranges (SPIMI-style), so
+   no process holds the whole inverted index;
+3. *pass 3* (workers): one term-id range each, merging that range's slice
+   of every spill file; the main process writes the terms in id order with
+   one `set_term_defs_arrays` call per range.
+
+The output is byte-identical to a serial accumulate-in-memory build, apart
+from the lexicon creation timestamp in each file's identification trailer.
+Wall-clock time (`time.perf_counter`), peak RSS of the main process and of
+the largest worker (`resource.getrusage`), and the index file sizes go into
+one row of `bench/results/indexing_bulk.csv`; per-phase timings are printed
+to stderr.
+
+Measured on the actual MS MARCO data (48-core machine; the `workers` column
+of the CSV records the pool size of each run):
+
+| docs | workers | wall | docs/s | main peak RSS | largest worker RSS | index size |
+|---:|---:|---:|---:|---:|---:|---:|
+| 10K | 24 | 1.0 s | 10,400 | 33 MB | 23 MB | 5.5 MB |
+| 100K | 24 | 5.0 s | 20,100 | 87 MB | 42 MB | 46 MB |
+| 1M | 24 | 42 s | 23,800 | 675 MB | 174 MB | 455 MB |
+| 8.84M (full) | 48 | 255 s | 34,700 | 3.1 GB | 573 MB | 4.1 GB |
+
+For comparison, the earlier serial version of this script (whole corpus
+accumulated in RAM as Python objects) did ~2,700-3,000 docs/s and needed
+7.9 GB at 1M docs; its rows are kept in
+`bench/results/indexing_bulk_serial_v1.csv`.
+
+The main process is the bottleneck: at 1M docs, 16 and 48 workers take the
+same time. Its time goes almost entirely into the C++ writers, which make
+every entry durable on its own (~13 system calls per definition written:
+seeks, reads of the indirection entry, `fwrite`+`fflush`, and signal-handler
+swaps around each Ctrl-C-safe critical section). A faster bulk load would
+need a C++-side bulk mode (e.g. buffered indirection and sequential writes).
+The main process's memory is mostly worker results waiting to be written;
+at most `2 × workers` chunks (~8 MB of corpus each) are in flight.
+
+Don't run two builds of the same index at once: they'd delete and
+overwrite each other's files, so `index_bulk.py` takes a lock on
+`<index-base>.lock` and refuses to start if it's held.
+
+Start with a small size (e.g. 10000) as a smoke test before scaling up.
+
+## 2. Incremental per-document indexing cost
+
+```bash
+NIND_DIST=/path/to/dist ./bench/run_indexing_benchmark.sh \
+    /path/to/collection.tsv 10000,100000
+```
+
+This drives `Nind_indexeCorpus` instead, which indexes one document at a
+time against the on-disk files as it reads them — a different, also-real
+usage pattern (e.g. appending new documents to an already-large index over
+time, rather than building it once from scratch). **Its scaling is much
+worse: on the actual MS MARCO data, going from 10K to 100K documents made it
+~89x slower, not ~10x.**
+
+The reason is structural, not a bug in this particular tool:
+`NindTermIndex::setTermDef` (`src/cpp/NindIndex/NindTermIndex.h:80-88`) only
+supports writing a term's *entire* postings list, no incremental append.
+`Nind_indexeCorpus` calls `getTermDef`+`setTermDef` for every term in every
+document (`tst/cpp/NindAmose/Nind_indexeCorpus.cpp:142-150`), so a term
+appearing in *df* documents has its (growing) postings list read and
+rewritten *df* times — total cost is dominated by common terms and scales
+roughly O(N²) in corpus size. **Don't run this one past ~100K documents** —
+it won't finish in reasonable time and the quadratic trend is already clear
+by then. It exists to document a real limitation (streaming/incremental
+indexing into a large existing base gets expensive), not as nind's headline
+throughput number — use benchmark 1 for that.
+
+Same output shape as before: one row per size in
+`bench/results/indexing.csv`.
+
+## 3. Lookup latency
+
+Run against an index built by either benchmark above (prefer benchmark 1's
+`docs_bulk` index — it's the one built the way nind is actually meant to be
+built at scale):
+
+```bash
+uv run python3 bench/lookup_latency_bench.py \
+    bench/results/msmarco_10000/docs.txt \
+    bench/results/msmarco_10000/docs_bulk \
+    10000
+```
+
+Samples random terms and document ids from the corpus text file, times
+repeated calls to `NindLexiconindex.donneIdentifiant` (term→id),
+`NindTermindex.donneListeTermesCG` (id→postings), and
+`NindLocalindex.donneListeTermes` (doc→local terms), and appends
+mean/median/p95/p99 latency and throughput to `bench/results/lookup.csv`.
+Run it once per size point to see how lookup latency scales with index size.
+
+By default each operation's sample set is run once untimed first
+(`--warmup 1`), so the numbers are steady-state lookup cost. With
+`--warmup 0` you mostly measure first-touch page-cache misses on a freshly
+opened file instead: on MS MARCO that made term→id look like it grew from
+~3µs (10K docs) to ~175µs median (1M docs). Don't run it concurrently with an
+indexing benchmark: I/O contention skews the cold numbers further.
+
+Warm, on the bulk-built MS MARCO indexes (median / p99, µs):
+
+| docs | term→id | id→postings | doc→local terms |
+|---:|---:|---:|---:|
+| 10K | 2.2 / 3.9 | 2.7 / 9.6 | 15.4 / 28.7 |
+| 100K | 2.7 / 5.4 | 3.2 / 22 | 16.7 / 31.3 |
+| 1M | 3.0 / 5.6 | 3.8 / 107 | 19.5 / 34.0 |
+| 8.84M | 3.1 / 5.1 | 3.7 / 284 | 19.9 / 36.1 |
+
+Lookups stay essentially flat from 10K to 8.84M documents: the lexicon's
+hash-bucket count is sized to the vocabulary, so buckets don't lengthen, and
+the term and local indexes are addressed directly by id. The id→postings tail
+is the few very frequent terms in the sample, whose postings lists are long.
+
+## Notes
+
+- `gbuild.sh` installs binaries with an empty RPATH (see the top-level
+  `CMakeLists.txt`), so `run_indexing_benchmark.sh` adds `<bin-dir>/../lib`
+  to `LD_LIBRARY_PATH` itself — no manual setup needed.
+- `bench/results/` is git-ignored; it's regenerated by these scripts. Both
+  indexing benchmarks reuse the same `bench/results/msmarco_<size>/docs.txt`
+  when present, so running benchmark 2 after benchmark 1 (or vice versa)
+  doesn't reconvert the corpus.

nind-2.2.1/bench/index_bulk.py

@@ -0,0 +1,328 @@
+#!/usr/bin/env python3
+"""Bulk-build indexing throughput benchmark, using nind's efficient write
+pattern: each term's definition and each document's local definition is
+written to disk exactly once (as nind.nind_engine.NindIndexer does) - as
+opposed to bench/run_indexing_benchmark.sh, which drives Nind_indexeCorpus
+and does a read-modify-write of each term's growing postings list on every
+document (see bench/README.md for why that shows ~quadratic scaling).
+
+The corpus is split into line-aligned byte ranges processed by a pool of
+worker processes; the main process only drives the (single-file,
+single-threaded) nind writers, handing them a whole chunk's documents or a
+whole term-id range's terms per call, as flat buffers (nind._native's
+set_local_defs_arrays/set_term_defs_arrays, which write with the GIL released,
+so the main process keeps receiving worker results meanwhile):
+
+- pass 1 (workers): per-chunk vocabulary and highest document id;
+- main: builds the lexicon, terms added in sorted order;
+- pass 2 (workers): parse and map terms to ids; each document's occurrences
+  come back as flat arrays and the main process writes its local definition
+  right away, in document order. Each worker also spills its chunk's
+  postings to a temporary file, sorted by term id and split into term-id
+  ranges (SPIMI-style), so no process ever holds the whole inverted index;
+- pass 3 (workers): one term-id range each, merging that range's slice of
+  every chunk file into per-term postings; the main process writes them in
+  term id order.
+
+The output is byte-identical to a serial accumulate-then-write build (apart
+from the lexicon creation timestamp in each file's identification trailer).
+
+Reads the same "one line per document" corpus format produced by
+msmarco/convert_to_nind.py (so it works on the compact bulk file, not
+one-file-per-document like NindIndexer.index_files expects).
+
+usage: index_bulk.py <docs.txt> <index-base> <size-label> [--workers N] [--csv PATH]
+  <index-base> is the .nind* file path prefix to write, e.g. .../docs
+"""
+import argparse
+from array import array
+import bisect
+import fcntl
+import gc
+import multiprocessing
+import os
+import resource
+import sys
+import tempfile
+import time
+from collections import defaultdict, deque
+from itertools import islice
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src" / "py" / "nind"))
+from nind import _native as native
+
+# state inherited by forked workers (set in the main process before the pool
+# that needs it is created, so it's shared copy-on-write, not pickled)
+_term_to_id = None
+_spill_dir = None
+_range_bounds = None     # term id lower bounds of each pass-3 range, ascending
+
+CHUNK_BYTES = 8 * 1024 * 1024    # target corpus bytes per chunk
+
+
+def bounded_imap(pool, fn, tasks, window):
+    """Like pool.imap (results in task order), but with at most `window`
+    tasks in flight: the main process, which consumes the results, is the
+    bottleneck, and pool.imap would otherwise queue every finished result
+    in its memory."""
+    tasks = iter(tasks)
+    pending = deque(pool.apply_async(fn, (task,)) for task in islice(tasks, window))
+    while pending:
+        result = pending.popleft().get()
+        for task in islice(tasks, 1):
+            pending.append(pool.apply_async(fn, (task,)))
+        yield result
+
+
+def parse_lines(text):
+    """Yields (doc_id, [(term, position, length), ...]) per non-empty line."""
+    for line in text.splitlines():
+        fields = line.split()
+        if not fields:
+            continue
+        occurrences = []
+        for term, loc in zip(fields[1::2], fields[2::2]):
+            pos_str, len_str = loc.split(",")
+            occurrences.append((term, int(pos_str), int(len_str)))
+        yield int(fields[0]), occurrences
+
+
+def read_chunk(docs_path, start, end):
+    with open(docs_path, "rb") as f:
+        f.seek(start)
+        return f.read(end - start).decode("utf-8")
+
+
+def split_chunks(docs_path, chunks_nb):
+    """Line-aligned byte ranges covering the whole file, in file order."""
+    size = os.path.getsize(docs_path)
+    bounds = [0]
+    with open(docs_path, "rb") as f:
+        for i in range(1, chunks_nb):
+            f.seek(max(bounds[-1], size * i // chunks_nb))
+            f.readline()
+            pos = f.tell()
+            if pos >= size:
+                break
+            if pos > bounds[-1]:
+                bounds.append(pos)
+    bounds.append(size)
+    return list(zip(bounds[:-1], bounds[1:]))
+
+
+def pass1_worker(task):
+    docs_path, start, end = task
+    vocab = set()
+    max_doc_id = 0
+    for doc_id, occurrences in parse_lines(read_chunk(docs_path, start, end)):
+        max_doc_id = max(max_doc_id, doc_id)
+        vocab.update(term for term, _pos, _len in occurrences)
+    return vocab, max_doc_id
+
+
+def pass2_worker(task):
+    """Returns the chunk's documents as flat arrays for set_local_def_arrays
+    and spills its postings, as (term_id, doc_id, freq) triples sorted by
+    term id, to <spill_dir>/<chunk_no>; triple offsets of each pass-3 range
+    are returned so pass 3 can read just its slice."""
+    chunk_no, docs_path, start, end = task
+    doc_ids = array("I")
+    doc_ends = array("I")                # end offset of each doc in the three arrays below
+    term_ids, positions, lengths = array("I"), array("I"), array("I")
+    postings = []                        # (term_id, doc_id, freq)
+    tokens = 0
+    for doc_id, occurrences in parse_lines(read_chunk(docs_path, start, end)):
+        counts = defaultdict(int)
+        for term, pos, length in occurrences:
+            tid = _term_to_id[term]
+            counts[tid] += 1
+            term_ids.append(tid)
+            positions.append(pos)
+            lengths.append(length)
+        tokens += len(occurrences)
+        doc_ids.append(doc_id)
+        doc_ends.append(len(term_ids))
+        postings.extend((tid, doc_id, freq) for tid, freq in counts.items())
+    # documents are in ascending id order within the file, so a stable sort
+    # on term id keeps each term's documents ascending
+    postings.sort(key=lambda p: p[0])
+    triples = array("I")
+    for p in postings:
+        triples.extend(p)
+    with open(os.path.join(_spill_dir, str(chunk_no)), "wb") as f:
+        triples.tofile(f)
+    sorted_tids = [p[0] for p in postings]
+    range_offsets = [bisect.bisect_left(sorted_tids, bound) for bound in _range_bounds]
+    range_offsets.append(len(postings))
+    return doc_ids, doc_ends, term_ids, positions, lengths, tokens, range_offsets
+
+
+def pass3_worker(task):
+    """Merges one term-id range across all chunk spill files, in chunk
+    (= document) order; returns the range's terms (ascending) as flat arrays
+    for set_term_defs_arrays: term ids, end offset of each term's postings,
+    and the concatenated postings' doc ids and frequencies."""
+    range_no, chunk_slices = task
+    by_term = {}
+    for chunk_no, first, last in chunk_slices:
+        if last == first:
+            continue
+        triples = array("I")
+        with open(os.path.join(_spill_dir, str(chunk_no)), "rb") as f:
+            f.seek(first * 3 * triples.itemsize)
+            triples.fromfile(f, (last - first) * 3)
+        for i in range(0, len(triples), 3):
+            tid = triples[i]
+            entry = by_term.get(tid)
+            if entry is None:
+                entry = by_term[tid] = (array("I"), array("I"))
+            entry[0].append(triples[i + 1])
+            entry[1].append(triples[i + 2])
+    term_ids, term_ends, doc_ids, freqs = array("I"), array("I"), array("I"), array("I")
+    for tid in sorted(by_term):
+        docs, term_freqs = by_term.pop(tid)
+        term_ids.append(tid)
+        doc_ids.extend(docs)
+        freqs.extend(term_freqs)
+        term_ends.append(len(doc_ids))
+    return term_ids, term_ends, doc_ids, freqs
+
+
+def build_index(docs_path, index_base, workers):
+    global _term_to_id, _spill_dir, _range_bounds
+    ctx = multiprocessing.get_context("fork")
+    phase_start = time.perf_counter()
+
+    def phase_done(name):
+        nonlocal phase_start
+        now = time.perf_counter()
+        print(f"  {name}: {now - phase_start:.2f}s", file=sys.stderr)
+        phase_start = now
+    chunks = split_chunks(docs_path, max(workers * 4, os.path.getsize(docs_path) // CHUNK_BYTES))
+    window = workers * 2
+
+    # pass 1: vocabulary (NindLexiconIndex needs the entry count up front) and
+    # the highest document id (so NindLocalIndex can be opened before pass 2)
+    vocab = set()
+    max_doc_id = 0
+    with ctx.Pool(workers) as pool:
+        for chunk_vocab, chunk_max in pool.imap_unordered(
+                pass1_worker, [(docs_path, s, e) for s, e in chunks]):
+            vocab |= chunk_vocab
+            max_doc_id = max(max_doc_id, chunk_max)
+    terms_sorted = sorted(vocab)
+    del vocab
+    phase_done("pass 1 (vocabulary)")
+
+    lexicon_writer = native.NindLexiconIndex(index_base, is_writer=True,
+                                              indirection_bloc_size=max(1, len(terms_sorted)))
+    _term_to_id = {term: lexicon_writer.add_word([term]) for term in terms_sorted}
+    del terms_sorted
+    vocab_size = len(_term_to_id)
+    lexicon_identification = lexicon_writer.get_identification()
+    del lexicon_writer
+    phase_done("lexicon write")
+
+    max_term_id = max(_term_to_id.values(), default=0)
+    ranges_nb = workers * 4
+    _range_bounds = [1 + (max_term_id * r) // ranges_nb for r in range(ranges_nb)]
+
+    docs = 0
+    tokens_total = 0
+    local_writer = native.NindLocalIndex(index_base, is_writer=True,
+                                          lexicon_identification=lexicon_identification,
+                                          indirection_bloc_size=max_doc_id + 1)
+    term_writer = native.NindTermIndex(index_base, is_writer=True,
+                                        lexicon_identification=lexicon_identification,
+                                        specifics_number=0,
+                                        indirection_bloc_size=max_term_id + 1)
+    spill_parent = os.path.dirname(os.path.abspath(index_base))
+    with tempfile.TemporaryDirectory(prefix="nind_spill_", dir=spill_parent) as spill_dir:
+        _spill_dir = spill_dir
+        # keep the inherited lexicon out of the (forked) workers' GC scans, so
+        # it isn't copied page by page into each of them
+        gc.freeze()
+        with ctx.Pool(workers) as pool:
+            # pass 2: chunks come back in file order, so local definitions are
+            # written in document order
+            all_range_offsets = []
+            for doc_ids, doc_ends, term_ids, positions, lengths, tokens, range_offsets in bounded_imap(
+                    pool, pass2_worker, [(i, docs_path, s, e) for i, (s, e) in enumerate(chunks)], window):
+                local_writer.set_local_defs_arrays(doc_ids, doc_ends, term_ids, positions, lengths,
+                                                   lexicon_identification)
+                docs += len(doc_ids)
+                tokens_total += tokens
+                all_range_offsets.append(range_offsets)
+            del local_writer
+            phase_done("pass 2 (local index write + postings spill)")
+
+            # pass 3: ranges come back in term id order
+            tasks = [(r, [(c, offsets[r], offsets[r + 1]) for c, offsets in enumerate(all_range_offsets)])
+                     for r in range(ranges_nb)]
+            for term_ids, term_ends, doc_ids, freqs in bounded_imap(pool, pass3_worker, tasks, window):
+                term_writer.set_term_defs_arrays(term_ids, term_ends, doc_ids, freqs, lexicon_identification)
+            del term_writer
+            phase_done("pass 3 (postings merge + term index write)")
+        gc.unfreeze()
+
+    return docs, tokens_total, vocab_size
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("docs_file")
+    parser.add_argument("index_base")
+    parser.add_argument("size_label")
+    parser.add_argument("--workers", type=int, default=os.cpu_count() or 1)
+    parser.add_argument("--csv", default=None)
+    args = parser.parse_args()
+
+    csv_path = Path(args.csv) if args.csv else Path(__file__).resolve().parent / "results" / "indexing_bulk.csv"
+    csv_path.parent.mkdir(parents=True, exist_ok=True)
+    new_file = not csv_path.exists()
+
+    # two runs writing the same index would delete/overwrite each other's
+    # files and fail (or worse) in the middle; refuse to start instead
+    lock_file = open(args.index_base + ".lock", "w")
+    try:
+        fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        sys.exit(f"index_bulk.py: another run is already writing {args.index_base}.* (lock held on "
+                 f"{args.index_base}.lock)")
+
+    docs_file_size = Path(args.docs_file).stat().st_size
+    start = time.perf_counter()
+    docs, tokens_total, vocab = build_index(args.docs_file, args.index_base, max(1, args.workers))
+    wall_seconds = time.perf_counter() - start
+    # process-lifetime peak RSS of the main process, and of the largest
+    # worker; accurate for this benchmark since it's run as a fresh
+    # subprocess per corpus size, doing nothing else beforehand
+    peak_rss_kb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    peak_worker_rss_kb = resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss
+
+    index_bytes = 0
+    for ext in (".nindlexiconindex", ".nindtermindex", ".nindlocalindex"):
+        p = Path(args.index_base + ext)
+        if p.exists():
+            index_bytes += p.stat().st_size
+
+    docs_per_sec = docs / wall_seconds if wall_seconds > 0 else docs
+    mb_per_sec = (docs_file_size / 1048576) / wall_seconds if wall_seconds > 0 else docs_file_size / 1048576
+
+    with open(csv_path, "a", encoding="utf-8") as f:
+        if new_file:
+            f.write("size,workers,docs,tokens,vocab,input_bytes,wall_seconds,docs_per_sec,mb_per_sec,"
+                     "peak_rss_kb,peak_worker_rss_kb,index_bytes\n")
+        f.write(f"{args.size_label},{args.workers},{docs},{tokens_total},{vocab},{docs_file_size},"
+                f"{wall_seconds:.3f},{docs_per_sec:.3f},{mb_per_sec:.5f},"
+                f"{peak_rss_kb},{peak_worker_rss_kb},{index_bytes}\n")
+
+    print(f"docs={docs} workers={args.workers} wall={wall_seconds:.2f}s docs/sec={docs_per_sec:.1f} "
+          f"MB/sec={mb_per_sec:.3f} peak_rss={peak_rss_kb}KB peak_worker_rss={peak_worker_rss_kb}KB "
+          f"index_bytes={index_bytes}", file=sys.stderr)
+    print(f"Results appended to {csv_path}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()