nind 2.2.0__tar.gz → 2.3.0__tar.gz

nind-2.3.0/.gitignore

@@ -0,0 +1,9 @@
+*.kdev4
+build/
+*~
+*.pyc
+Dist*/
+docs/_build/
+bench/results/
+bench/msmarco/*
+!bench/msmarco/*.py

{nind-2.2.0 → nind-2.3.0}/CLAUDE.md

@@ -33,7 +33,7 @@ To run tests manually instead of via the full `gbuild.sh` cycle: `ctest --output
 
 C++ unit tests live in `tst/cpp/unit/<Module>/` (one executable per library, one ctest entry each) and use **doctest**, vendored as the single header `tst/cpp/unit/doctest.h` (v2.4.12) — no system package or download needed, so they are always built. Each executable accepts doctest options, e.g. `NindIndexUnitTests -tc='NindIndexRobustness*' -s`. Corrupt/hostile-input tests live in the `*RobustnessTest.cpp` files: they patch or byte-by-byte corrupt files produced by the writers and require every read to either succeed or throw a `FileException` — run them under `-a ON` so ASan/UBSan/LSan turn any memory error, UB or leak into a failure. (`-a ON` needs the GCC sanitizer runtimes — `libasan`, `libubsan`, `liblsan` packages on RHEL-like systems.)
 
-Memory-safety conventions in the C++ core (keep them when touching it): readers must never trust a length/offset/count read from a file — bound it against the buffer (`NindFile::get*` check before advancing) or the real file size (`NindFile::getCurrentFileSize()`); byte assembly is done in `uint32_t`/`uint64_t` (never `int`, which overflows); chains read from files (indirection blocks, compound words) must be provably finite; `NindFile`/`NindPadFile` are non-copyable; critical sections use the RAII `NindCriticalSection`, never raw `beginCriticalSection`/`endCriticalSection`.
+Memory-safety conventions in the C++ core (keep them when touching it): readers must never trust a length/offset/count read from a file — bound it against the buffer (`NindFile::get*` check before advancing) or the real file size (`NindFile::getCurrentFileSize()`); byte assembly is done in `uint32_t`/`uint64_t` (never `int`, which overflows); chains read from files (indirection blocks, compound words) must be provably finite; `NindFile`/`NindPadFile` are non-copyable; critical sections use the RAII `NindCriticalSection`, never raw `beginCriticalSection`/`endCriticalSection`; a file has at most one writer — `NindPadFile`/`NindLexiconFile` writers take a non-blocking exclusive `flock` (`NindFile::lockExclusive`, no-op on Windows) and create files with `"w+bx"` (never truncating a file another writer just created), so a second writer, even in-process, gets an exception instead of corrupting the file (readers never lock).
 
 `-DDEBUG_NIND` is added automatically in `Debug` and `RelWithDebInfo` modes (and in `Release` if configured with `-DWITH_DEBUG_MESSAGES=ON`) to enable conditional debug output.
 
@@ -80,9 +80,11 @@ NindIndex (deps: NindBasics, NindRetrolexicon)
    — NindTermIndex: the inverted file itself, as an index file
    — NindLocalIndex: per-document local index file
    — NindIndex: ties the above into a full inverted/local index file
+   — NindIndexMerge: offline merge of index shards (disjoint documents) into one standard index
 
 NindAmose (deps: NindIndex, NindRetrolexicon)
    — adapts nind to Amose's richer term-type model (see internal doc refs "LAT2015.JYS.448")
+   — NindAmoseMerge: NindIndexMerge for Amose indexes (retro lexicon, recomputed term counters)
 
 NindPy (deps: NindBasics, NindRetrolexicon, NindLexicon, NindIndex sources, compiled in directly)
    — pybind11 bindings, built only through the Python wheel (scikit-build-core), not gbuild.sh
@@ -105,7 +107,8 @@ Parallels the C++ layering with lowercase-suffixed module names for the index cl
 - `NindFile.py` — low-level binary codec. Historically read-only-oriented (the `litXxx` getters have long existed for every format), but now has a full writer counterpart too: `ejcritNombre1/3/4/5`, `ejcritNombreULat`/`ejcritNombreSLat` (both varint encodings, full tier range matching the readers and the C++ implementation), `ejcritChaine`, `ejcritZejros`.
 - `NindPadFile.py` — the shared "pad file" envelope (header, indirection block(s), specifics, identification trailer); read-only, no writer (mirrors `NindBasics::NindPadFile`, but nothing in Python subclasses it as a writer the way `NindIndex`/`NindRetrolexicon` do in C++).
 - `NindRetrolexicon.py`, `NindLexiconindex.py`, `NindTermindex.py`, `NindLocalindex.py` — ergonomic read-only Python wrappers around the corresponding `NindIndex`-family C++ classes; their hot-path lookups delegate to `nind._native`, as does the shared structural-diagnostic base (`analyseFichierPadFile`/`analyseFichierIndex`, inherited from `NindPadFile.py`/`NindIndex.py`). The remaining, per-format diagnostic/analysis methods (`dumpeFichier`, `debogueIndex`, `donneCollisions`, `donneMax`, `donneClef`, `afficheTerme`, `afficheDocument`, and each class's own `analyseFichierXxx` beyond the shared base) remain hand-rolled pure-Python parsing, since `nind._native` doesn't expose that introspection surface yet (see "Planned follow-up work" below). `NindLexiconindex` is hash-bucketed (`clefB(word) % nombreIndirection`), the other two are directly indexed by id.
-- `nind_engine.py` — `NindIndexer` (writer) and `NindEngine` (BM25-style search) built on top of the above. `NindIndexer` is the *only* Python writer for the index-family formats (`.nindlexiconindex`/`.nindtermindex`/`.nindlocalindex`) — it builds them via `nind._native`'s `is_writer=True` constructors. It does not write `.nindretrolexicon` (not needed by `NindEngine`, which only supports simple, non-compound words). Indexing is parallel but every file keeps a single writer: files are tokenized in worker processes (`workers` argument of `index_files`; in-process below `PARALLEL_MIN_FILES` files), then the term and local index files are written concurrently from two threads through the bulk bindings `NindTermIndex.set_postings`/`NindLocalIndex.set_local_def_from_term_ids`, which take plain ints (building `Document`/`Term`/`Localisation` objects in Python cost ~20× the write itself) and release the GIL. `set_local_def_from_term_ids` splits a term's positions into entries of ≤255 (`<nbreLocalisations>` is one byte).
+- `nind_engine.py` — `NindIndexer` (writer) and `NindEngine` (BM25-style search) built on top of the above. `NindIndexer` is the *only* Python writer for the index-family formats (`.nindlexiconindex`/`.nindtermindex`/`.nindlocalindex`) — it builds them via `nind._native`'s `is_writer=True` constructors. It does not write `.nindretrolexicon` (not needed by `NindEngine`, which only supports simple, non-compound words). Indexing is parallel but every file keeps a single writer: files are tokenized in worker processes (`workers` argument of `index_files`; in-process below `PARALLEL_MIN_FILES` files), then the term and local index files are written concurrently from two threads through the flat-buffer bulk bindings `NindTermIndex.set_term_defs_arrays`/`NindLocalIndex.set_local_defs_arrays`, which take `array('I')` buffers (building `Document`/`Term`/`Localisation` objects in Python cost ~20× the write itself) and release the GIL. The `*_local_def*_arrays` writers split a term's positions into entries of ≤255 (`<nbreLocalisations>` is one byte). Every build goes into a new generation directory published atomically (see `_index_store.py` below), so rebuilding never disturbs readers. With `shards=N` it instead builds N independent indexes over contiguous parts of the files in worker processes (in a temporary directory inside the new generation), then merges them with `nind._native.merge_indexes`; opt-in because the merge costs about as much as writing the index again (see "Sharded indexing" below), so it only pays off when per-document work dominates.
+- `_index_store.py` — the on-disk layout that makes every `NindIndexer`/`NindEngine` operation safe without the caller doing anything: each build writes a fresh, never-modified generation directory `<prefix>.gen-<n>-<hex>/`, flushed to disk (`fsync`) then published by atomically replacing the pointer file `<prefix>.CURRENT` (a pointer file rather than a symlink: atomic on Windows too). Readers open every file of the generation `CURRENT` names under a shared `flock` on its `generation.lock` (only while opening: open files stay readable after deletion on POSIX); `NindEngine` checks `CURRENT` (one `stat`) at the start of each public call and switches to a newly published build, each call answering from one snapshot (`_Snapshot`), calls serialized by an `RLock` (the native readers aren't thread-safe). Writers of one prefix are serialized by `<prefix>.lock`; after publishing, the writer deletes older generations nobody is opening, crashed builds' leftovers, and flat files of indexes written before generations existed (still readable: no `CURRENT` = flat layout). A failed build never touches `CURRENT`. Lock fds are closed in forked children (`os.register_at_fork`) — otherwise a worker outliving a killed writer would keep the index locked forever — and `NindIndexer`'s pool workers exit when their parent dies. Windows: `msvcrt` locks are exclusive only, so readers don't lock and rely on retrying; generations still open can't be deleted and are retried at the next build.
 - `Nind_*.py` at the package root — standalone CLI scripts (dump a document, convert corpora, search, diagnostics)
 - `amose/` — Amose-specific conversion/parsing scripts (Lucene dump, XML-CLEF, sample corpora)
 
@@ -124,7 +127,11 @@ Binary file formats are specified in EBNF grammar (see README). The Python packa
 
 Not part of this: `.nindretrolexicon` writing (`NindIndexer` still doesn't write this format — unrelated, no change planned).
 
-**Sharded indexing (planned next, after the in-process parallelism above).** Nind files are single-writer by design (no locking, readers resync lazily; see `NindPadFile::getEntryPos`), and every write rewrites the shared specifics/identification tail at EOF, so concurrent writers on one file would serialize anyway — sharing a file between writers is explicitly *not* the plan. Instead, scale writes across N independent shards (each its own lexicon + term + local files, one single-threaded writer each, over a partition of the documents), then either:
-- merge offline: union the lexicons into a new one, remap every shard's term ids, concatenate posting lists with doc ids kept in increasing order (the term index stores relative doc ids), and remap term ids inside each shard's local index; or
-- search across shards: `NindEngine` fans out per shard, but BM25 needs corpus-wide statistics (N, df, avgdl) summed over all shards, not per-shard values.
-Expected cost ~1–2 weeks including tests; zero overhead on the write path.
+**Sharded indexing — offline merge implemented; follow-ups below.** Nind files are single-writer by design (no locking, readers resync lazily; see `NindPadFile::getEntryPos`), and every write rewrites the shared specifics/identification tail at EOF, so concurrent writers on one file would serialize anyway — sharing a file between writers is explicitly *not* the plan. Writes scale instead across N independent shards (each its own lexicon + term + local files, one single-threaded writer each, over a partition of the documents), merged offline into one standard index that every existing reader uses unchanged (search across shards was considered and rejected: every reader, Amose's included, would have to learn about shards, and BM25 would need corpus-wide statistics summed over them).
+
+How the merge works (`NindIndexMerge`, `src/cpp/NindIndex/`): shard lexicons are enumerated with `NindLexiconIndex::getEntries` (walks the hash buckets; no retro lexicon needed) and unioned with words canonicalised as (union id of the prefix, last component) — which also gives the exact merged size, used as the lexicon's hash modulo; documents are listed with `NindLocalIndex::getDocIdents` and checked disjoint *before anything is written*; each merged term's postings are gathered from every shard (per term, so memory stays small) and each document is copied with its term ids remapped (local term order is kept, so Amose's per-occurrence reading order survives). A failed merge removes the target files. `NindAmoseMerge` (`src/cpp/NindAmose/`) forces a retro lexicon and recomputes the 8 term counters, a term's type being derived from its components as in `NindLexiconAmose::getWord` (unique-term counts aren't additive across shards); they're written once at the end with `NindTermIndex::setSpecificWords`. The plain merge refuses shards with term specifics.
+
+Follow-ups:
+- **Merge cost is I/O-syscall bound, and so are all writes.** Merging 10k documents / 58k terms from 4 shards makes ~2.3M `lseek`, ~1.5M `read`, ~280k `write` and ~250k `rt_sigaction` calls: `NindIndex::getDefinition` discards the stdio buffer (`flush`) then seeks+reads twice (indirection, then data) on every read, `NindFile::writeBuffer` flushes every write, and every critical section installs/restores a signal handler. A direct build pays the same per definition, so a merge costs about a full rewrite. Reading with `pread` (one syscall per read, no buffer discard) and a bulk mode for a writer creating a fresh file (no per-write flush, tail written once) would speed up merges, direct builds and search alike — mind the reader-resync semantics above when touching this.
+- **CLI for Amose/LIMA** multi-process indexing: a small C++ program (like the legacy `tst/cpp/NindAmose/` tools) running `NindAmoseMerge` over shard prefixes. `NindAmoseMerge` isn't bound in `nind._native` (the wheel doesn't compile `NindAmose`).
+- `NindLexiconAmose::getWord` dereferences past the end of the component list for the lexicon word `§` alone (the first prefix of every named entity): undefined behaviour, seen as a `bad_alloc`. Normal lookups never ask for that id; the merge doesn't call `getWord`.

{nind-2.2.0 → nind-2.3.0}/CMakeLists.txt

@@ -113,7 +113,7 @@ endif ()
 
 ########### setting project verion number ###############
 
-set(NIND_VERSION "2.2.0")
+set(NIND_VERSION "2.3.0")
 
 string(REGEX REPLACE "^([0-9]+)\\.[0-9]+\\.[0-9]+$" "\\1" NIND_VERSION_MAJOR "${NIND_VERSION}")
 string(REGEX REPLACE "^[0-9]+\\.([0-9]+)\\.[0-9]+$" "\\1" NIND_VERSION_MINOR "${NIND_VERSION}")

{nind-2.2.0 → nind-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: nind
-Version: 2.2.0
+Version: 2.3.0
 Summary: Python frontend for the Nind Indexing Module, with its native C++ index stack built in
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown

nind-2.3.0/bench/README.md

@@ -0,0 +1,189 @@
+# nind performance benchmarks
+
+nind is a flat-file indexer, not a full search engine, so it's benchmarked
+here like a storage/indexing engine rather than with an anserini/pyserini-style
+retrieval-effectiveness harness (nDCG/MAP against relevance judgments) — see
+the design discussion this came out of for why. Three things are measured:
+
+1. **Indexing throughput & on-disk index size**, across a corpus-size sweep —
+   using nind's efficient *bulk-build* write pattern.
+2. **The cost of incremental, per-document indexing** into a growing index —
+   a different, also-real usage pattern, with a real and important caveat
+   (see below).
+3. **Term/document lookup latency**, through the same `nind._native`-backed
+   Python reader classes real callers use.
+
+Retrieval effectiveness (nDCG/MAP vs. qrels) is out of scope: `NindEngine`
+would first need stable external document ids and real inverted-list-driven
+candidate generation instead of its current O(N·|Q|) full scan.
+
+## Corpus: MS MARCO Passage Ranking
+
+Freely downloadable, no license/registration required, and its passage ids
+are already a dense integer sequence from 0 — a good match for nind's
+document-identifier model. Get `collection.tsv` (one passage per line,
+`pid<TAB>text`, ~8.84M rows) from the official MS MARCO Passage Ranking
+release — see <https://github.com/microsoft/MSMARCO-Passage-Ranking> for the
+current download instructions/links (verify the link is still current;
+dataset hosting has moved before).
+
+## Prerequisites
+
+```bash
+# Python side: builds nind._native and installs the reader/writer classes.
+# This alone is enough for the bulk-build benchmark (1) and lookup latency (3).
+uv sync
+
+# C++ side, only needed for the incremental-indexing benchmark (2):
+NIND_DIST=/path/to/dist ./gbuild.sh -m Release   # Release, not Debug - Debug timings are misleading
+# if cmake/ninja aren't installed system-wide, `uv tool install cmake ninja` works too
+```
+
+## 1. Indexing throughput & size (bulk build)
+
+```bash
+./bench/run_bulk_indexing_benchmark.sh /path/to/collection.tsv 10000,100000,1000000
+```
+
+For each size in the comma-separated list, this converts the first N
+passages to nind's corpus text format (`bench/msmarco/convert_to_nind.py`,
+offsetting `pid` by +1 since nind treats document id 0 as a sentinel
+elsewhere), then runs `bench/index_bulk.py`, which writes each term's
+definition and each document's local definition to disk **exactly once**,
+via `nind._native`'s `NindLexiconIndex`/`NindTermIndex`/`NindLocalIndex`
+writers (the same write-once pattern `nind.nind_engine.NindIndexer` uses),
+reading from the compact bulk corpus file instead of one file per document.
+
+The corpus is split into line-aligned byte ranges handled by a pool of worker
+processes (`--workers`, default: all cores); the main process only drives
+the nind writers, which are single-file and single-threaded:
+
+1. *pass 1* (workers): vocabulary and highest document id per chunk; the
+   main process then writes the lexicon, terms added in sorted order;
+2. *pass 2* (workers): parsing and term→id mapping. Each chunk's documents
+   come back as flat `array('I')` buffers, and the main process writes them
+   in document order with one `set_local_defs_arrays` call per chunk. Each
+   worker also spills its chunk's postings to a temporary file next to the
+   index, sorted by term id and split into term-id ranges (SPIMI-style), so
+   no process holds the whole inverted index;
+3. *pass 3* (workers): one term-id range each, merging that range's slice
+   of every spill file; the main process writes the terms in id order with
+   one `set_term_defs_arrays` call per range.
+
+The output is byte-identical to a serial accumulate-in-memory build, apart
+from the lexicon creation timestamp in each file's identification trailer.
+Wall-clock time (`time.perf_counter`), peak RSS of the main process and of
+the largest worker (`resource.getrusage`), and the index file sizes go into
+one row of `bench/results/indexing_bulk.csv`; per-phase timings are printed
+to stderr.
+
+Measured on the actual MS MARCO data (48-core machine; the `workers` column
+of the CSV records the pool size of each run):
+
+| docs | workers | wall | docs/s | main peak RSS | largest worker RSS | index size |
+|---:|---:|---:|---:|---:|---:|---:|
+| 10K | 24 | 1.0 s | 10,400 | 33 MB | 23 MB | 5.5 MB |
+| 100K | 24 | 5.0 s | 20,100 | 87 MB | 42 MB | 46 MB |
+| 1M | 24 | 42 s | 23,800 | 675 MB | 174 MB | 455 MB |
+| 8.84M (full) | 48 | 255 s | 34,700 | 3.1 GB | 573 MB | 4.1 GB |
+
+For comparison, the earlier serial version of this script (whole corpus
+accumulated in RAM as Python objects) did ~2,700-3,000 docs/s and needed
+7.9 GB at 1M docs; its rows are kept in
+`bench/results/indexing_bulk_serial_v1.csv`.
+
+The main process is the bottleneck: at 1M docs, 16 and 48 workers take the
+same time. Its time goes almost entirely into the C++ writers, which make
+every entry durable on its own (~13 system calls per definition written:
+seeks, reads of the indirection entry, `fwrite`+`fflush`, and signal-handler
+swaps around each Ctrl-C-safe critical section). A faster bulk load would
+need a C++-side bulk mode (e.g. buffered indirection and sequential writes).
+The main process's memory is mostly worker results waiting to be written;
+at most `2 × workers` chunks (~8 MB of corpus each) are in flight.
+
+Don't run two builds of the same index at once: they'd delete and
+overwrite each other's files, so `index_bulk.py` takes a lock on
+`<index-base>.lock` and refuses to start if it's held.
+
+Start with a small size (e.g. 10000) as a smoke test before scaling up.
+
+## 2. Incremental per-document indexing cost
+
+```bash
+NIND_DIST=/path/to/dist ./bench/run_indexing_benchmark.sh \
+    /path/to/collection.tsv 10000,100000
+```
+
+This drives `Nind_indexeCorpus` instead, which indexes one document at a
+time against the on-disk files as it reads them — a different, also-real
+usage pattern (e.g. appending new documents to an already-large index over
+time, rather than building it once from scratch). **Its scaling is much
+worse: on the actual MS MARCO data, going from 10K to 100K documents made it
+~89x slower, not ~10x.**
+
+The reason is structural, not a bug in this particular tool:
+`NindTermIndex::setTermDef` (`src/cpp/NindIndex/NindTermIndex.h:80-88`) only
+supports writing a term's *entire* postings list, no incremental append.
+`Nind_indexeCorpus` calls `getTermDef`+`setTermDef` for every term in every
+document (`tst/cpp/NindAmose/Nind_indexeCorpus.cpp:142-150`), so a term
+appearing in *df* documents has its (growing) postings list read and
+rewritten *df* times — total cost is dominated by common terms and scales
+roughly O(N²) in corpus size. **Don't run this one past ~100K documents** —
+it won't finish in reasonable time and the quadratic trend is already clear
+by then. It exists to document a real limitation (streaming/incremental
+indexing into a large existing base gets expensive), not as nind's headline
+throughput number — use benchmark 1 for that.
+
+Same output shape as before: one row per size in
+`bench/results/indexing.csv`.
+
+## 3. Lookup latency
+
+Run against an index built by either benchmark above (prefer benchmark 1's
+`docs_bulk` index — it's the one built the way nind is actually meant to be
+built at scale):
+
+```bash
+uv run python3 bench/lookup_latency_bench.py \
+    bench/results/msmarco_10000/docs.txt \
+    bench/results/msmarco_10000/docs_bulk \
+    10000
+```
+
+Samples random terms and document ids from the corpus text file, times
+repeated calls to `NindLexiconindex.donneIdentifiant` (term→id),
+`NindTermindex.donneListeTermesCG` (id→postings), and
+`NindLocalindex.donneListeTermes` (doc→local terms), and appends
+mean/median/p95/p99 latency and throughput to `bench/results/lookup.csv`.
+Run it once per size point to see how lookup latency scales with index size.
+
+By default each operation's sample set is run once untimed first
+(`--warmup 1`), so the numbers are steady-state lookup cost. With
+`--warmup 0` you mostly measure first-touch page-cache misses on a freshly
+opened file instead: on MS MARCO that made term→id look like it grew from
+~3µs (10K docs) to ~175µs median (1M docs). Don't run it concurrently with an
+indexing benchmark: I/O contention skews the cold numbers further.
+
+Warm, on the bulk-built MS MARCO indexes (median / p99, µs):
+
+| docs | term→id | id→postings | doc→local terms |
+|---:|---:|---:|---:|
+| 10K | 2.2 / 3.9 | 2.7 / 9.6 | 15.4 / 28.7 |
+| 100K | 2.7 / 5.4 | 3.2 / 22 | 16.7 / 31.3 |
+| 1M | 3.0 / 5.6 | 3.8 / 107 | 19.5 / 34.0 |
+| 8.84M | 3.1 / 5.1 | 3.7 / 284 | 19.9 / 36.1 |
+
+Lookups stay essentially flat from 10K to 8.84M documents: the lexicon's
+hash-bucket count is sized to the vocabulary, so buckets don't lengthen, and
+the term and local indexes are addressed directly by id. The id→postings tail
+is the few very frequent terms in the sample, whose postings lists are long.
+
+## Notes
+
+- `gbuild.sh` installs binaries with an empty RPATH (see the top-level
+  `CMakeLists.txt`), so `run_indexing_benchmark.sh` adds `<bin-dir>/../lib`
+  to `LD_LIBRARY_PATH` itself — no manual setup needed.
+- `bench/results/` is git-ignored; it's regenerated by these scripts. Both
+  indexing benchmarks reuse the same `bench/results/msmarco_<size>/docs.txt`
+  when present, so running benchmark 2 after benchmark 1 (or vice versa)
+  doesn't reconvert the corpus.

nind-2.3.0/bench/index_bulk.py

@@ -0,0 +1,328 @@
+#!/usr/bin/env python3
+"""Bulk-build indexing throughput benchmark, using nind's efficient write
+pattern: each term's definition and each document's local definition is
+written to disk exactly once (as nind.nind_engine.NindIndexer does) - as
+opposed to bench/run_indexing_benchmark.sh, which drives Nind_indexeCorpus
+and does a read-modify-write of each term's growing postings list on every
+document (see bench/README.md for why that shows ~quadratic scaling).
+
+The corpus is split into line-aligned byte ranges processed by a pool of
+worker processes; the main process only drives the (single-file,
+single-threaded) nind writers, handing them a whole chunk's documents or a
+whole term-id range's terms per call, as flat buffers (nind._native's
+set_local_defs_arrays/set_term_defs_arrays, which write with the GIL released,
+so the main process keeps receiving worker results meanwhile):
+
+- pass 1 (workers): per-chunk vocabulary and highest document id;
+- main: builds the lexicon, terms added in sorted order;
+- pass 2 (workers): parse and map terms to ids; each document's occurrences
+  come back as flat arrays and the main process writes its local definition
+  right away, in document order. Each worker also spills its chunk's
+  postings to a temporary file, sorted by term id and split into term-id
+  ranges (SPIMI-style), so no process ever holds the whole inverted index;
+- pass 3 (workers): one term-id range each, merging that range's slice of
+  every chunk file into per-term postings; the main process writes them in
+  term id order.
+
+The output is byte-identical to a serial accumulate-then-write build (apart
+from the lexicon creation timestamp in each file's identification trailer).
+
+Reads the same "one line per document" corpus format produced by
+msmarco/convert_to_nind.py (so it works on the compact bulk file, not
+one-file-per-document like NindIndexer.index_files expects).
+
+usage: index_bulk.py <docs.txt> <index-base> <size-label> [--workers N] [--csv PATH]
+  <index-base> is the .nind* file path prefix to write, e.g. .../docs
+"""
+import argparse
+from array import array
+import bisect
+import fcntl
+import gc
+import multiprocessing
+import os
+import resource
+import sys
+import tempfile
+import time
+from collections import defaultdict, deque
+from itertools import islice
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src" / "py" / "nind"))
+from nind import _native as native
+
+# state inherited by forked workers (set in the main process before the pool
+# that needs it is created, so it's shared copy-on-write, not pickled)
+_term_to_id = None
+_spill_dir = None
+_range_bounds = None     # term id lower bounds of each pass-3 range, ascending
+
+CHUNK_BYTES = 8 * 1024 * 1024    # target corpus bytes per chunk
+
+
+def bounded_imap(pool, fn, tasks, window):
+    """Like pool.imap (results in task order), but with at most `window`
+    tasks in flight: the main process, which consumes the results, is the
+    bottleneck, and pool.imap would otherwise queue every finished result
+    in its memory."""
+    tasks = iter(tasks)
+    pending = deque(pool.apply_async(fn, (task,)) for task in islice(tasks, window))
+    while pending:
+        result = pending.popleft().get()
+        for task in islice(tasks, 1):
+            pending.append(pool.apply_async(fn, (task,)))
+        yield result
+
+
+def parse_lines(text):
+    """Yields (doc_id, [(term, position, length), ...]) per non-empty line."""
+    for line in text.splitlines():
+        fields = line.split()
+        if not fields:
+            continue
+        occurrences = []
+        for term, loc in zip(fields[1::2], fields[2::2]):
+            pos_str, len_str = loc.split(",")
+            occurrences.append((term, int(pos_str), int(len_str)))
+        yield int(fields[0]), occurrences
+
+
+def read_chunk(docs_path, start, end):
+    with open(docs_path, "rb") as f:
+        f.seek(start)
+        return f.read(end - start).decode("utf-8")
+
+
+def split_chunks(docs_path, chunks_nb):
+    """Line-aligned byte ranges covering the whole file, in file order."""
+    size = os.path.getsize(docs_path)
+    bounds = [0]
+    with open(docs_path, "rb") as f:
+        for i in range(1, chunks_nb):
+            f.seek(max(bounds[-1], size * i // chunks_nb))
+            f.readline()
+            pos = f.tell()
+            if pos >= size:
+                break
+            if pos > bounds[-1]:
+                bounds.append(pos)
+    bounds.append(size)
+    return list(zip(bounds[:-1], bounds[1:]))
+
+
+def pass1_worker(task):
+    docs_path, start, end = task
+    vocab = set()
+    max_doc_id = 0
+    for doc_id, occurrences in parse_lines(read_chunk(docs_path, start, end)):
+        max_doc_id = max(max_doc_id, doc_id)
+        vocab.update(term for term, _pos, _len in occurrences)
+    return vocab, max_doc_id
+
+
+def pass2_worker(task):
+    """Returns the chunk's documents as flat arrays for set_local_def_arrays
+    and spills its postings, as (term_id, doc_id, freq) triples sorted by
+    term id, to <spill_dir>/<chunk_no>; triple offsets of each pass-3 range
+    are returned so pass 3 can read just its slice."""
+    chunk_no, docs_path, start, end = task
+    doc_ids = array("I")
+    doc_ends = array("I")                # end offset of each doc in the three arrays below
+    term_ids, positions, lengths = array("I"), array("I"), array("I")
+    postings = []                        # (term_id, doc_id, freq)
+    tokens = 0
+    for doc_id, occurrences in parse_lines(read_chunk(docs_path, start, end)):
+        counts = defaultdict(int)
+        for term, pos, length in occurrences:
+            tid = _term_to_id[term]
+            counts[tid] += 1
+            term_ids.append(tid)
+            positions.append(pos)
+            lengths.append(length)
+        tokens += len(occurrences)
+        doc_ids.append(doc_id)
+        doc_ends.append(len(term_ids))
+        postings.extend((tid, doc_id, freq) for tid, freq in counts.items())
+    # documents are in ascending id order within the file, so a stable sort
+    # on term id keeps each term's documents ascending
+    postings.sort(key=lambda p: p[0])
+    triples = array("I")
+    for p in postings:
+        triples.extend(p)
+    with open(os.path.join(_spill_dir, str(chunk_no)), "wb") as f:
+        triples.tofile(f)
+    sorted_tids = [p[0] for p in postings]
+    range_offsets = [bisect.bisect_left(sorted_tids, bound) for bound in _range_bounds]
+    range_offsets.append(len(postings))
+    return doc_ids, doc_ends, term_ids, positions, lengths, tokens, range_offsets
+
+
+def pass3_worker(task):
+    """Merges one term-id range across all chunk spill files, in chunk
+    (= document) order; returns the range's terms (ascending) as flat arrays
+    for set_term_defs_arrays: term ids, end offset of each term's postings,
+    and the concatenated postings' doc ids and frequencies."""
+    range_no, chunk_slices = task
+    by_term = {}
+    for chunk_no, first, last in chunk_slices:
+        if last == first:
+            continue
+        triples = array("I")
+        with open(os.path.join(_spill_dir, str(chunk_no)), "rb") as f:
+            f.seek(first * 3 * triples.itemsize)
+            triples.fromfile(f, (last - first) * 3)
+        for i in range(0, len(triples), 3):
+            tid = triples[i]
+            entry = by_term.get(tid)
+            if entry is None:
+                entry = by_term[tid] = (array("I"), array("I"))
+            entry[0].append(triples[i + 1])
+            entry[1].append(triples[i + 2])
+    term_ids, term_ends, doc_ids, freqs = array("I"), array("I"), array("I"), array("I")
+    for tid in sorted(by_term):
+        docs, term_freqs = by_term.pop(tid)
+        term_ids.append(tid)
+        doc_ids.extend(docs)
+        freqs.extend(term_freqs)
+        term_ends.append(len(doc_ids))
+    return term_ids, term_ends, doc_ids, freqs
+
+
+def build_index(docs_path, index_base, workers):
+    global _term_to_id, _spill_dir, _range_bounds
+    ctx = multiprocessing.get_context("fork")
+    phase_start = time.perf_counter()
+
+    def phase_done(name):
+        nonlocal phase_start
+        now = time.perf_counter()
+        print(f"  {name}: {now - phase_start:.2f}s", file=sys.stderr)
+        phase_start = now
+    chunks = split_chunks(docs_path, max(workers * 4, os.path.getsize(docs_path) // CHUNK_BYTES))
+    window = workers * 2
+
+    # pass 1: vocabulary (NindLexiconIndex needs the entry count up front) and
+    # the highest document id (so NindLocalIndex can be opened before pass 2)
+    vocab = set()
+    max_doc_id = 0
+    with ctx.Pool(workers) as pool:
+        for chunk_vocab, chunk_max in pool.imap_unordered(
+                pass1_worker, [(docs_path, s, e) for s, e in chunks]):
+            vocab |= chunk_vocab
+            max_doc_id = max(max_doc_id, chunk_max)
+    terms_sorted = sorted(vocab)
+    del vocab
+    phase_done("pass 1 (vocabulary)")
+
+    lexicon_writer = native.NindLexiconIndex(index_base, is_writer=True,
+                                              indirection_bloc_size=max(1, len(terms_sorted)))
+    _term_to_id = {term: lexicon_writer.add_word([term]) for term in terms_sorted}
+    del terms_sorted
+    vocab_size = len(_term_to_id)
+    lexicon_identification = lexicon_writer.get_identification()
+    del lexicon_writer
+    phase_done("lexicon write")
+
+    max_term_id = max(_term_to_id.values(), default=0)
+    ranges_nb = workers * 4
+    _range_bounds = [1 + (max_term_id * r) // ranges_nb for r in range(ranges_nb)]
+
+    docs = 0
+    tokens_total = 0
+    local_writer = native.NindLocalIndex(index_base, is_writer=True,
+                                          lexicon_identification=lexicon_identification,
+                                          indirection_bloc_size=max_doc_id + 1)
+    term_writer = native.NindTermIndex(index_base, is_writer=True,
+                                        lexicon_identification=lexicon_identification,
+                                        specifics_number=0,
+                                        indirection_bloc_size=max_term_id + 1)
+    spill_parent = os.path.dirname(os.path.abspath(index_base))
+    with tempfile.TemporaryDirectory(prefix="nind_spill_", dir=spill_parent) as spill_dir:
+        _spill_dir = spill_dir
+        # keep the inherited lexicon out of the (forked) workers' GC scans, so
+        # it isn't copied page by page into each of them
+        gc.freeze()
+        with ctx.Pool(workers) as pool:
+            # pass 2: chunks come back in file order, so local definitions are
+            # written in document order
+            all_range_offsets = []
+            for doc_ids, doc_ends, term_ids, positions, lengths, tokens, range_offsets in bounded_imap(
+                    pool, pass2_worker, [(i, docs_path, s, e) for i, (s, e) in enumerate(chunks)], window):
+                local_writer.set_local_defs_arrays(doc_ids, doc_ends, term_ids, positions, lengths,
+                                                   lexicon_identification)
+                docs += len(doc_ids)
+                tokens_total += tokens
+                all_range_offsets.append(range_offsets)
+            del local_writer
+            phase_done("pass 2 (local index write + postings spill)")
+
+            # pass 3: ranges come back in term id order
+            tasks = [(r, [(c, offsets[r], offsets[r + 1]) for c, offsets in enumerate(all_range_offsets)])
+                     for r in range(ranges_nb)]
+            for term_ids, term_ends, doc_ids, freqs in bounded_imap(pool, pass3_worker, tasks, window):
+                term_writer.set_term_defs_arrays(term_ids, term_ends, doc_ids, freqs, lexicon_identification)
+            del term_writer
+            phase_done("pass 3 (postings merge + term index write)")
+        gc.unfreeze()
+
+    return docs, tokens_total, vocab_size
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("docs_file")
+    parser.add_argument("index_base")
+    parser.add_argument("size_label")
+    parser.add_argument("--workers", type=int, default=os.cpu_count() or 1)
+    parser.add_argument("--csv", default=None)
+    args = parser.parse_args()
+
+    csv_path = Path(args.csv) if args.csv else Path(__file__).resolve().parent / "results" / "indexing_bulk.csv"
+    csv_path.parent.mkdir(parents=True, exist_ok=True)
+    new_file = not csv_path.exists()
+
+    # two runs writing the same index would delete/overwrite each other's
+    # files and fail (or worse) in the middle; refuse to start instead
+    lock_file = open(args.index_base + ".lock", "w")
+    try:
+        fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        sys.exit(f"index_bulk.py: another run is already writing {args.index_base}.* (lock held on "
+                 f"{args.index_base}.lock)")
+
+    docs_file_size = Path(args.docs_file).stat().st_size
+    start = time.perf_counter()
+    docs, tokens_total, vocab = build_index(args.docs_file, args.index_base, max(1, args.workers))
+    wall_seconds = time.perf_counter() - start
+    # process-lifetime peak RSS of the main process, and of the largest
+    # worker; accurate for this benchmark since it's run as a fresh
+    # subprocess per corpus size, doing nothing else beforehand
+    peak_rss_kb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    peak_worker_rss_kb = resource.getrusage(resource.RUSAGE_CHILDREN).ru_maxrss
+
+    index_bytes = 0
+    for ext in (".nindlexiconindex", ".nindtermindex", ".nindlocalindex"):
+        p = Path(args.index_base + ext)
+        if p.exists():
+            index_bytes += p.stat().st_size
+
+    docs_per_sec = docs / wall_seconds if wall_seconds > 0 else docs
+    mb_per_sec = (docs_file_size / 1048576) / wall_seconds if wall_seconds > 0 else docs_file_size / 1048576
+
+    with open(csv_path, "a", encoding="utf-8") as f:
+        if new_file:
+            f.write("size,workers,docs,tokens,vocab,input_bytes,wall_seconds,docs_per_sec,mb_per_sec,"
+                     "peak_rss_kb,peak_worker_rss_kb,index_bytes\n")
+        f.write(f"{args.size_label},{args.workers},{docs},{tokens_total},{vocab},{docs_file_size},"
+                f"{wall_seconds:.3f},{docs_per_sec:.3f},{mb_per_sec:.5f},"
+                f"{peak_rss_kb},{peak_worker_rss_kb},{index_bytes}\n")
+
+    print(f"docs={docs} workers={args.workers} wall={wall_seconds:.2f}s docs/sec={docs_per_sec:.1f} "
+          f"MB/sec={mb_per_sec:.3f} peak_rss={peak_rss_kb}KB peak_worker_rss={peak_worker_rss_kb}KB "
+          f"index_bytes={index_bytes}", file=sys.stderr)
+    print(f"Results appended to {csv_path}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()