dipo 0.2.0__tar.gz

dipo-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Luke Gessler
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dipo-0.2.0/PKG-INFO

@@ -0,0 +1,278 @@
+Metadata-Version: 2.4
+Name: dipo
+Version: 0.2.0
+Summary: Dipo, the Discourse Parsing Omnibus: a collection of easy-to-use discourse parsers and associated code.
+Author-email: Luke Gessler <lukegessler@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/larc-iu/dipo
+Project-URL: Repository, https://github.com/larc-iu/dipo
+Project-URL: Issues, https://github.com/larc-iu/dipo/issues
+Keywords: discourse parsing,rhetorical structure theory,RST,NLP,computational linguistics
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch
+Requires-Dist: transformers
+Requires-Dist: huggingface_hub
+Requires-Dist: tonga-config
+Requires-Dist: lxml
+Requires-Dist: rich
+Requires-Dist: sentencepiece
+Requires-Dist: sacremoses
+Requires-Dist: peft
+Requires-Dist: tensorboard
+Requires-Dist: setuptools<81
+Provides-Extra: dev
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# Dipo
+
+The **<u>Di</u>scourse <u>P</u>arsing <u>O</u>mnibus** (Dipo, said like "depot") is a collection of easy-to-use discourse parsers and other code related to discourse parsing.
+
+Dipo was formerly called IUDEX (`larc-iudex` on PyPI, with the command `iudex`).
+
+## Setup
+
+For the latest release:
+
+```
+pip install dipo
+```
+
+For the current state of `master`:
+
+```
+pip install git+https://github.com/larc-iu/dipo
+```
+
+Or for development:
+
+```
+git clone https://github.com/larc-iu/dipo && cd dipo
+pip install -e .
+```
+
+The old `iudex` command still works for now, as a deprecated alias of `dipo` that prints a notice. `import iudex` is now `import dipo`.
+
+## Quick Start with Inference
+
+Parse a sample document end-to-end with a pretrained DMRST model pulled from the HuggingFace Hub. From the command line:
+
+```bash
+dipo dmrst predict \
+    --hub-id larc-iu/dmrst-ettin-400m-gum-12.1.0 \
+    --text "Although the experiment was carefully designed, the results were inconclusive. We plan to repeat it tonight."
+```
+This yields the parsed tree in `.rs3` format printed to `stdout`:
+```xml
+<rst>
+  <relations><!-- ... --></relations>
+  <body>
+    <segment id="1" parent="2" relname="adversative-concession">Although the experiment was carefully # designed,</segment>
+    <segment id="2" parent="4" relname="span">the results were inconclusive.</segment>
+    <segment id="3" parent="5" relname="span">We plan to repeat it tonight.</segment>
+    <group id="4" type="span" parent="3" relname="adversative-antithesis"/>
+    <group id="5" type="span"/>
+  </body>
+</rst>
+```
+
+The same flow from Python:
+
+```python
+from dipo.rst.parsers.dmrst.modeling_dmrst import DMRSTParser
+parser = DMRSTParser.from_pretrained("larc-iu/dmrst-ettin-400m-gum-12.1.0")
+tree = parser.predict_from_text(
+    "Although the experiment was carefully designed, "
+    "the results were inconclusive. "
+    "We plan to repeat it tonight."
+)
+print(tree.to_rs4_string())
+```
+Yields:
+```xml
+<rst>
+  <relations><!-- ... --></relations>
+  <body>
+    <segment id="1" parent="2" relname="adversative-concession">Although the experiment was carefully # designed,</segment>
+    <segment id="2" parent="4" relname="span">the results were inconclusive.</segment>
+    <segment id="3" parent="5" relname="span">We plan to repeat it tonight.</segment>
+    <group id="4" type="span" parent="3" relname="adversative-antithesis"/>
+    <group id="5" type="span"/>
+  </body>
+</rst>
+```
+
+## Inference CLI
+
+To identify a model on the command line, you may use a configuration file (`--config`), a PyTorch checkpoint (`--checkpoint`), or a HuggingFace Hub repository (`--hub-id`).
+
+To provide input, you may specify an inline string (`--text`), a path to a raw text file or directory (`--text-file`, for parsers which support this), or an RS3/RS4 file or directory with gold EDUs already supplied (`--input`).
+
+For `--text-file` and `--input`, results are written to `--output-dir` as `.rs4` files.
+
+```
+# From the Hub, end-to-end on a directory of .txt files:
+dipo dmrst predict \
+    --hub-id larc-iu/dmrst-ettin-400m-gum-12.1.0 \
+    --text-file path/to/docs/ \
+    --output-dir out/ \
+    --device cuda
+
+# From an explicit checkpoint:
+dipo dmrst predict \
+    --checkpoint checkpoints/<run_id>/best_model.pt \
+    --text-file path/to/doc.txt \
+    --output-dir out/
+
+# From a trained run's config, parsing pre-segmented RS3/RS4 with gold EDUs:
+dipo topdown_biaffine predict \
+    --config configs/topdown_biaffine_rstdt.jsonnet \
+    --input data/rstdt/test \
+    --output-dir out/
+```
+
+## Available Models
+All official Dipo model releases are [tagged with `dipo` on the HuggingFace Hub](https://huggingface.co/models?other=dipo) (they also carry the old `iudex` tag for now).
+
+All of the models below parse raw text end to end (segmentation and structure) and were trained on untokenized text, so plain prose works as input.
+There is no need to tokenize it or to remove line breaks, and Chinese should be given unspaced.
+Scores are original Parseval Full F1 on each corpus's test set, end to end from raw text.
+
+| Hub ID | Parser | Backbone | Corpus | Test Full |
+| --- | --- | --- | --- | --- |
+| [`larc-iu/dmrst-ettin-400m-rstdt-coarse`](https://huggingface.co/larc-iu/dmrst-ettin-400m-rstdt-coarse) | `dmrst` | Ettin encoder 400M | RST-DT (English) | 53.2 |
+| [`larc-iu/gen-sr-t5gemma-2-1b-1b-rstdt-coarse`](https://huggingface.co/larc-iu/gen-sr-t5gemma-2-1b-1b-rstdt-coarse) | `gen` | T5Gemma 2 1B-1B | RST-DT (English) | 49.6 |
+| [`larc-iu/gen-sr-gemma-4-31b-it-rstdt-coarse`](https://huggingface.co/larc-iu/gen-sr-gemma-4-31b-it-rstdt-coarse) | `gen` | Gemma 4 31B | RST-DT (English) | 54.7 |
+| [`larc-iu/dmrst-ettin-400m-gum-12.1.0`](https://huggingface.co/larc-iu/dmrst-ettin-400m-gum-12.1.0) | `dmrst` | Ettin encoder 400M | GUM 12.1.0 (English) | 45.6 |
+| [`larc-iu/gen-sr-t5gemma-2-1b-1b-gum-12.1.0`](https://huggingface.co/larc-iu/gen-sr-t5gemma-2-1b-1b-gum-12.1.0) | `gen` | T5Gemma 2 1B-1B | GUM 12.1.0 (English) | 39.4 |
+| [`larc-iu/gen-sr-gemma-4-31b-it-gum-12.1.0`](https://huggingface.co/larc-iu/gen-sr-gemma-4-31b-it-gum-12.1.0) | `gen` | Gemma 4 31B | GUM 12.1.0 (English) | 45.8 |
+| [`larc-iu/dmrst-xlm-roberta-base-ert`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-ert) | `dmrst` | XLM-R base | RST Basque TreeBank | 27.6 |
+| [`larc-iu/dmrst-xlm-roberta-base-pcc-2.2`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-pcc-2.2) | `dmrst` | XLM-R base | Potsdam Commentary Corpus 2.2 (German) | 18.6 |
+| [`larc-iu/dmrst-xlm-roberta-base-prstc`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-prstc) | `dmrst` | XLM-R base | Persian RST Corpus | 32.2 |
+| [`larc-iu/dmrst-xlm-roberta-base-gcdt`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-gcdt) | `dmrst` | XLM-R base | GCDT (Chinese) | 29.4 |
+
+The `dmrst` models cannot see line breaks, so `predict_from_text` also starts a new segment at blank-line paragraph breaks and after short heading-like lines.
+Pass `break_at_paragraphs=False` to use the model's own segmentation alone.
+
+The `gen` models are built on a pretrained backbone, and loading one also downloads that backbone from the Hub.
+The two Gemma 4 models are small adapters (about 0.5 GB), so they require accepting Google's terms for `google/gemma-4-31B-it` and a GPU with roughly 70 GB of memory.
+The T5Gemma models store their full weights (about 4.4 GB).
+They are invoked like the others, e.g. `dipo gen predict --hub-id larc-iu/gen-sr-t5gemma-2-1b-1b-gum-12.1.0 --text "..."`.
+A document longer than a `gen` model's context window (16,384 tokens) is rejected with an error rather than silently truncated.
+
+## Training
+
+To train a new top-down biaffine parser on RSTDT:
+
+```
+dipo topdown_biaffine train configs/topdown_biaffine_rstdt.jsonnet
+```
+
+Note that `configs/topdown_biaffine_rstdt.jsonnet` is a configuration.
+You may either edit it directly or copy and modify it in a new location.
+
+### Training on Raw Text
+
+Several corpora distribute their EDUs word-tokenized (`the drug trade ,`), and a parser trained on that text sees punctuation spacing at inference that real input never has.
+`scripts/build_raw_text_data.py` writes untokenized copies of RST-DT, PCC, GCDT and the Persian corpus to `data/<corpus>_notok`, with identical trees, and the released models were trained on these.
+RST-DT's raw text comes from the LDC release, which must be present at `data/rst_discourse_treebank`.
+For unspaced Chinese, also set `split_cjk_tokens: true` in a DMRST config, so that every EDU boundary falls on a token boundary.
+
+### Grabbing Example Configurations
+
+Model configurations required for training are not bundled with the package distributed via PyPI.
+
+To get them you may visit [the associated directory](https://github.com/larc-iu/dipo/tree/master/configs) and download the configurations you're interested in manually.
+
+If you want to grab all of them at once, you can use the command line like so:
+
+**bash / zsh / macOS / Linux:**
+
+```bash
+curl -fL https://github.com/larc-iu/dipo/archive/refs/heads/master.tar.gz \
+  | tar -xz --strip-components=1 --wildcards '*/configs'
+```
+
+**Windows PowerShell:**
+
+```powershell
+Invoke-WebRequest https://github.com/larc-iu/dipo/archive/refs/heads/master.zip -OutFile dipo.zip
+Expand-Archive dipo.zip -DestinationPath .
+Move-Item dipo-master/configs configs
+Remove-Item -Recurse -Force dipo-master, dipo.zip
+```
+
+Either leaves you with a local `configs/` directory you can edit and pass to `dipo … train configs/<name>.jsonnet`.
+
+### Configuration Hashes
+
+Your configuration is used as the basis for a unique hash, which (by default) corresponds to a directory under `checkpoints/`.
+This hash is used for several purposes.
+For example, running the same config again resumes from the last epoch's checkpoint `last.pt` automatically if the run was interrupted.
+
+To view all runs and their status, you may run the `runs list` subcommand:
+
+```
+$ dipo runs list
+                                                            Runs in checkpoints                                                            
+┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ run_id       ┃ run_name ┃ parser           ┃ model_name                   ┃ train_dir             ┃  best_val ┃ step ┃ modified         ┃
+┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ 245b1d774676 │ -        │ dmrst            │ xlm-roberta-base             │ data/gum_12.1.0/train │    0.3099 │ 1704 │ 2026-05-18 18:02 │
+│ 41bc0fe1dd50 │ -        │ topdown_biaffine │ SpanBERT/spanbert-base-cased │ data/rstdt/train      │    0.7576 │ 2149 │ 2026-05-18 13:51 │
+│ 91525e48d63d │ -        │ topdown_biaffine │ SpanBERT/spanbert-base-cased │ data/gum_12.1.0/train │    0.6364 │ 1899 │ 2026-05-18 14:31 │
+│ ad934ca992d4 │ -        │ dmrst            │ xlm-roberta-base             │ data/rstdt/train      │    0.4665 │ 3090 │ 2026-05-18 16:46 │
+└──────────────┴──────────┴──────────────────┴──────────────────────────────┴───────────────────────┴───────────┴──────┴──────────────────┘
+```
+
+### Monitoring with TensorBoard
+
+Every run writes TensorBoard scalars (train loss, learning rate, gradient norm, and dev metrics) to `<run_dir>/tb/`. Point TensorBoard at your checkpoints directory to watch any run live or compare runs:
+
+```
+tensorboard --logdir checkpoints/
+```
+
+### Pushing Models to HF Hub
+You may host a trained model using each parser's `push` subcommand.
+Each uploads `best_model.pt`, `config.json`, and an auto-generated `README.md` in a single commit:
+
+```
+dipo topdown_biaffine push \
+    --config configs/topdown_biaffine_rstdt.jsonnet \
+    --repo-id larc-iu/topdown_biaffine-rstdt-coarse \
+    [--private] [--message "..."] [--token $HF_TOKEN]
+```
+
+## Citation
+
+If you use Dipo in your research, please cite it as:
+
+> Gessler, Luke. 2026. *Dipo: The Discourse Parsing Omnibus.* https://github.com/larc-iu/dipo.
+
+BibTeX:
+
+```bibtex
+@misc{gessler-dipo-2026,
+  author       = {Gessler, Luke},
+  title        = {{Dipo: The Discourse Parsing Omnibus}},
+  year         = {2026},
+  howpublished = {\url{https://github.com/larc-iu/dipo}},
+}
+```
+
+If you use one of the included parser re-implementations, please **also** cite the original paper (see each model's Hub card for the canonical reference).

dipo-0.2.0/README.md

@@ -0,0 +1,238 @@
+# Dipo
+
+The **<u>Di</u>scourse <u>P</u>arsing <u>O</u>mnibus** (Dipo, said like "depot") is a collection of easy-to-use discourse parsers and other code related to discourse parsing.
+
+Dipo was formerly called IUDEX (`larc-iudex` on PyPI, with the command `iudex`).
+
+## Setup
+
+For the latest release:
+
+```
+pip install dipo
+```
+
+For the current state of `master`:
+
+```
+pip install git+https://github.com/larc-iu/dipo
+```
+
+Or for development:
+
+```
+git clone https://github.com/larc-iu/dipo && cd dipo
+pip install -e .
+```
+
+The old `iudex` command still works for now, as a deprecated alias of `dipo` that prints a notice. `import iudex` is now `import dipo`.
+
+## Quick Start with Inference
+
+Parse a sample document end-to-end with a pretrained DMRST model pulled from the HuggingFace Hub. From the command line:
+
+```bash
+dipo dmrst predict \
+    --hub-id larc-iu/dmrst-ettin-400m-gum-12.1.0 \
+    --text "Although the experiment was carefully designed, the results were inconclusive. We plan to repeat it tonight."
+```
+This yields the parsed tree in `.rs3` format printed to `stdout`:
+```xml
+<rst>
+  <relations><!-- ... --></relations>
+  <body>
+    <segment id="1" parent="2" relname="adversative-concession">Although the experiment was carefully # designed,</segment>
+    <segment id="2" parent="4" relname="span">the results were inconclusive.</segment>
+    <segment id="3" parent="5" relname="span">We plan to repeat it tonight.</segment>
+    <group id="4" type="span" parent="3" relname="adversative-antithesis"/>
+    <group id="5" type="span"/>
+  </body>
+</rst>
+```
+
+The same flow from Python:
+
+```python
+from dipo.rst.parsers.dmrst.modeling_dmrst import DMRSTParser
+parser = DMRSTParser.from_pretrained("larc-iu/dmrst-ettin-400m-gum-12.1.0")
+tree = parser.predict_from_text(
+    "Although the experiment was carefully designed, "
+    "the results were inconclusive. "
+    "We plan to repeat it tonight."
+)
+print(tree.to_rs4_string())
+```
+Yields:
+```xml
+<rst>
+  <relations><!-- ... --></relations>
+  <body>
+    <segment id="1" parent="2" relname="adversative-concession">Although the experiment was carefully # designed,</segment>
+    <segment id="2" parent="4" relname="span">the results were inconclusive.</segment>
+    <segment id="3" parent="5" relname="span">We plan to repeat it tonight.</segment>
+    <group id="4" type="span" parent="3" relname="adversative-antithesis"/>
+    <group id="5" type="span"/>
+  </body>
+</rst>
+```
+
+## Inference CLI
+
+To identify a model on the command line, you may use a configuration file (`--config`), a PyTorch checkpoint (`--checkpoint`), or a HuggingFace Hub repository (`--hub-id`).
+
+To provide input, you may specify an inline string (`--text`), a path to a raw text file or directory (`--text-file`, for parsers which support this), or an RS3/RS4 file or directory with gold EDUs already supplied (`--input`).
+
+For `--text-file` and `--input`, results are written to `--output-dir` as `.rs4` files.
+
+```
+# From the Hub, end-to-end on a directory of .txt files:
+dipo dmrst predict \
+    --hub-id larc-iu/dmrst-ettin-400m-gum-12.1.0 \
+    --text-file path/to/docs/ \
+    --output-dir out/ \
+    --device cuda
+
+# From an explicit checkpoint:
+dipo dmrst predict \
+    --checkpoint checkpoints/<run_id>/best_model.pt \
+    --text-file path/to/doc.txt \
+    --output-dir out/
+
+# From a trained run's config, parsing pre-segmented RS3/RS4 with gold EDUs:
+dipo topdown_biaffine predict \
+    --config configs/topdown_biaffine_rstdt.jsonnet \
+    --input data/rstdt/test \
+    --output-dir out/
+```
+
+## Available Models
+All official Dipo model releases are [tagged with `dipo` on the HuggingFace Hub](https://huggingface.co/models?other=dipo) (they also carry the old `iudex` tag for now).
+
+All of the models below parse raw text end to end (segmentation and structure) and were trained on untokenized text, so plain prose works as input.
+There is no need to tokenize it or to remove line breaks, and Chinese should be given unspaced.
+Scores are original Parseval Full F1 on each corpus's test set, end to end from raw text.
+
+| Hub ID | Parser | Backbone | Corpus | Test Full |
+| --- | --- | --- | --- | --- |
+| [`larc-iu/dmrst-ettin-400m-rstdt-coarse`](https://huggingface.co/larc-iu/dmrst-ettin-400m-rstdt-coarse) | `dmrst` | Ettin encoder 400M | RST-DT (English) | 53.2 |
+| [`larc-iu/gen-sr-t5gemma-2-1b-1b-rstdt-coarse`](https://huggingface.co/larc-iu/gen-sr-t5gemma-2-1b-1b-rstdt-coarse) | `gen` | T5Gemma 2 1B-1B | RST-DT (English) | 49.6 |
+| [`larc-iu/gen-sr-gemma-4-31b-it-rstdt-coarse`](https://huggingface.co/larc-iu/gen-sr-gemma-4-31b-it-rstdt-coarse) | `gen` | Gemma 4 31B | RST-DT (English) | 54.7 |
+| [`larc-iu/dmrst-ettin-400m-gum-12.1.0`](https://huggingface.co/larc-iu/dmrst-ettin-400m-gum-12.1.0) | `dmrst` | Ettin encoder 400M | GUM 12.1.0 (English) | 45.6 |
+| [`larc-iu/gen-sr-t5gemma-2-1b-1b-gum-12.1.0`](https://huggingface.co/larc-iu/gen-sr-t5gemma-2-1b-1b-gum-12.1.0) | `gen` | T5Gemma 2 1B-1B | GUM 12.1.0 (English) | 39.4 |
+| [`larc-iu/gen-sr-gemma-4-31b-it-gum-12.1.0`](https://huggingface.co/larc-iu/gen-sr-gemma-4-31b-it-gum-12.1.0) | `gen` | Gemma 4 31B | GUM 12.1.0 (English) | 45.8 |
+| [`larc-iu/dmrst-xlm-roberta-base-ert`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-ert) | `dmrst` | XLM-R base | RST Basque TreeBank | 27.6 |
+| [`larc-iu/dmrst-xlm-roberta-base-pcc-2.2`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-pcc-2.2) | `dmrst` | XLM-R base | Potsdam Commentary Corpus 2.2 (German) | 18.6 |
+| [`larc-iu/dmrst-xlm-roberta-base-prstc`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-prstc) | `dmrst` | XLM-R base | Persian RST Corpus | 32.2 |
+| [`larc-iu/dmrst-xlm-roberta-base-gcdt`](https://huggingface.co/larc-iu/dmrst-xlm-roberta-base-gcdt) | `dmrst` | XLM-R base | GCDT (Chinese) | 29.4 |
+
+The `dmrst` models cannot see line breaks, so `predict_from_text` also starts a new segment at blank-line paragraph breaks and after short heading-like lines.
+Pass `break_at_paragraphs=False` to use the model's own segmentation alone.
+
+The `gen` models are built on a pretrained backbone, and loading one also downloads that backbone from the Hub.
+The two Gemma 4 models are small adapters (about 0.5 GB), so they require accepting Google's terms for `google/gemma-4-31B-it` and a GPU with roughly 70 GB of memory.
+The T5Gemma models store their full weights (about 4.4 GB).
+They are invoked like the others, e.g. `dipo gen predict --hub-id larc-iu/gen-sr-t5gemma-2-1b-1b-gum-12.1.0 --text "..."`.
+A document longer than a `gen` model's context window (16,384 tokens) is rejected with an error rather than silently truncated.
+
+## Training
+
+To train a new top-down biaffine parser on RSTDT:
+
+```
+dipo topdown_biaffine train configs/topdown_biaffine_rstdt.jsonnet
+```
+
+Note that `configs/topdown_biaffine_rstdt.jsonnet` is a configuration.
+You may either edit it directly or copy and modify it in a new location.
+
+### Training on Raw Text
+
+Several corpora distribute their EDUs word-tokenized (`the drug trade ,`), and a parser trained on that text sees punctuation spacing at inference that real input never has.
+`scripts/build_raw_text_data.py` writes untokenized copies of RST-DT, PCC, GCDT and the Persian corpus to `data/<corpus>_notok`, with identical trees, and the released models were trained on these.
+RST-DT's raw text comes from the LDC release, which must be present at `data/rst_discourse_treebank`.
+For unspaced Chinese, also set `split_cjk_tokens: true` in a DMRST config, so that every EDU boundary falls on a token boundary.
+
+### Grabbing Example Configurations
+
+Model configurations required for training are not bundled with the package distributed via PyPI.
+
+To get them you may visit [the associated directory](https://github.com/larc-iu/dipo/tree/master/configs) and download the configurations you're interested in manually.
+
+If you want to grab all of them at once, you can use the command line like so:
+
+**bash / zsh / macOS / Linux:**
+
+```bash
+curl -fL https://github.com/larc-iu/dipo/archive/refs/heads/master.tar.gz \
+  | tar -xz --strip-components=1 --wildcards '*/configs'
+```
+
+**Windows PowerShell:**
+
+```powershell
+Invoke-WebRequest https://github.com/larc-iu/dipo/archive/refs/heads/master.zip -OutFile dipo.zip
+Expand-Archive dipo.zip -DestinationPath .
+Move-Item dipo-master/configs configs
+Remove-Item -Recurse -Force dipo-master, dipo.zip
+```
+
+Either leaves you with a local `configs/` directory you can edit and pass to `dipo … train configs/<name>.jsonnet`.
+
+### Configuration Hashes
+
+Your configuration is used as the basis for a unique hash, which (by default) corresponds to a directory under `checkpoints/`.
+This hash is used for several purposes.
+For example, running the same config again resumes from the last epoch's checkpoint `last.pt` automatically if the run was interrupted.
+
+To view all runs and their status, you may run the `runs list` subcommand:
+
+```
+$ dipo runs list
+                                                            Runs in checkpoints                                                            
+┏━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ run_id       ┃ run_name ┃ parser           ┃ model_name                   ┃ train_dir             ┃  best_val ┃ step ┃ modified         ┃
+┡━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ 245b1d774676 │ -        │ dmrst            │ xlm-roberta-base             │ data/gum_12.1.0/train │    0.3099 │ 1704 │ 2026-05-18 18:02 │
+│ 41bc0fe1dd50 │ -        │ topdown_biaffine │ SpanBERT/spanbert-base-cased │ data/rstdt/train      │    0.7576 │ 2149 │ 2026-05-18 13:51 │
+│ 91525e48d63d │ -        │ topdown_biaffine │ SpanBERT/spanbert-base-cased │ data/gum_12.1.0/train │    0.6364 │ 1899 │ 2026-05-18 14:31 │
+│ ad934ca992d4 │ -        │ dmrst            │ xlm-roberta-base             │ data/rstdt/train      │    0.4665 │ 3090 │ 2026-05-18 16:46 │
+└──────────────┴──────────┴──────────────────┴──────────────────────────────┴───────────────────────┴───────────┴──────┴──────────────────┘
+```
+
+### Monitoring with TensorBoard
+
+Every run writes TensorBoard scalars (train loss, learning rate, gradient norm, and dev metrics) to `<run_dir>/tb/`. Point TensorBoard at your checkpoints directory to watch any run live or compare runs:
+
+```
+tensorboard --logdir checkpoints/
+```
+
+### Pushing Models to HF Hub
+You may host a trained model using each parser's `push` subcommand.
+Each uploads `best_model.pt`, `config.json`, and an auto-generated `README.md` in a single commit:
+
+```
+dipo topdown_biaffine push \
+    --config configs/topdown_biaffine_rstdt.jsonnet \
+    --repo-id larc-iu/topdown_biaffine-rstdt-coarse \
+    [--private] [--message "..."] [--token $HF_TOKEN]
+```
+
+## Citation
+
+If you use Dipo in your research, please cite it as:
+
+> Gessler, Luke. 2026. *Dipo: The Discourse Parsing Omnibus.* https://github.com/larc-iu/dipo.
+
+BibTeX:
+
+```bibtex
+@misc{gessler-dipo-2026,
+  author       = {Gessler, Luke},
+  title        = {{Dipo: The Discourse Parsing Omnibus}},
+  year         = {2026},
+  howpublished = {\url{https://github.com/larc-iu/dipo}},
+}
+```
+
+If you use one of the included parser re-implementations, please **also** cite the original paper (see each model's Hub card for the canonical reference).

dipo-0.2.0/dipo/__init__.py

@@ -0,0 +1,22 @@
+"""This module declares the project's CLI surface that lives above any one
+framework:
+
+  - `FRAMEWORKS`: dotted paths of framework modules (e.g. `dipo.rst`).
+     Each framework module exposes `PARSERS`, `PARSER_SCOPED_COMMANDS`,
+     and `GLOBAL_COMMANDS`. The dispatcher (`dipo/__main__.py`) imports
+     each and merges the three.
+  - `GLOBAL_COMMANDS`: `{cmd: module_path}` for project-level commands
+    that aren't owned by any single framework (e.g. `runs`, which walks
+    every framework's parser registry to tag rows by parser kind).
+
+To add a sibling framework, append its dotted path to `FRAMEWORKS` and
+give its `__init__.py` the three required attributes.
+"""
+
+__version__ = "0.2.0"
+
+FRAMEWORKS: list[str] = ["dipo.rst"]
+
+GLOBAL_COMMANDS: dict[str, str] = {
+    "runs": "dipo.runs",
+}

dipo-0.2.0/dipo/__main__.py

@@ -0,0 +1,103 @@
+"""dipo CLI dispatcher.
+
+`dipo <parser> <cmd>` or `dipo <cmd>` for globals. Routes by merging the
+registries declared by each framework in `dipo.FRAMEWORKS`. See
+`dipo/__init__.py` for the framework contract.
+"""
+
+import importlib
+import sys
+
+import dipo
+
+
+def _merge_frameworks() -> tuple[dict, dict, dict]:
+    """Merge each framework's three registry dicts atop project-level
+    `dipo.GLOBAL_COMMANDS`. Aborts on a name collision."""
+    parsers: dict = {}
+    parser_scoped: dict = {}
+    global_cmds: dict = dict(dipo.GLOBAL_COMMANDS)
+    for fw_path in dipo.FRAMEWORKS:
+        fw = importlib.import_module(fw_path)
+        _merge_no_collide(parsers, fw.PARSERS, fw_path, "parser name")
+        _merge_no_collide(parser_scoped, fw.PARSER_SCOPED_COMMANDS, fw_path, "parser-scoped command")
+        _merge_no_collide(global_cmds, fw.GLOBAL_COMMANDS, fw_path, "global command")
+    return parsers, parser_scoped, global_cmds
+
+
+def _merge_no_collide(dst: dict, src: dict, fw_path: str, kind: str) -> None:
+    for k, v in src.items():
+        if k in dst and dst[k] is not v:
+            sys.stderr.write(
+                f"dipo: {kind} {k!r} declared by both an earlier framework and {fw_path!r}. Rename one of them.\n"
+            )
+            sys.exit(2)
+        dst[k] = v
+
+
+PARSERS, PARSER_SCOPED_COMMANDS, GLOBAL_COMMANDS = _merge_frameworks()
+
+
+def main():
+    if len(sys.argv) > 1 and sys.argv[1] in ("-h", "--help"):
+        print(__doc__.strip())
+        print(f"\nKnown parsers: {', '.join(sorted(PARSERS))}")
+        print(f"Global commands: {', '.join(sorted(GLOBAL_COMMANDS))}")
+        sys.exit(0)
+
+    if len(sys.argv) < 2:
+        sys.stderr.write(__doc__.strip() + "\n\n")
+        sys.stderr.write(f"Known parsers: {', '.join(sorted(PARSERS))}\n")
+        sys.stderr.write(f"Global commands: {', '.join(sorted(GLOBAL_COMMANDS))}\n")
+        sys.exit(2)
+
+    head = sys.argv[1]
+
+    if head in GLOBAL_COMMANDS:
+        module = importlib.import_module(GLOBAL_COMMANDS[head])
+        sys.argv = [GLOBAL_COMMANDS[head]] + sys.argv[2:]
+        module.main()
+        return
+
+    if len(sys.argv) < 3:
+        sys.stderr.write(__doc__.strip() + "\n\n")
+        sys.stderr.write(f"Known parsers: {', '.join(sorted(PARSERS))}\n")
+        sys.exit(2)
+
+    parser_name = head
+    command = sys.argv[2]
+
+    if parser_name not in PARSERS:
+        sys.stderr.write(f"Unknown parser: {parser_name!r}\n")
+        sys.stderr.write(f"Known parsers: {', '.join(sorted(PARSERS))}\n")
+        sys.stderr.write(f"Global commands: {', '.join(sorted(GLOBAL_COMMANDS))}\n")
+        sys.exit(2)
+
+    if command in PARSER_SCOPED_COMMANDS:
+        shared_path = PARSER_SCOPED_COMMANDS[command]
+        module = importlib.import_module(shared_path)
+        sys.argv = [f"dipo {parser_name} {command}"] + sys.argv[3:]
+        module.main(parser_kind=parser_name)
+        return
+
+    module_path = f"{PARSERS[parser_name].package}.{command}_{parser_name}"
+    try:
+        module = importlib.import_module(module_path)
+    except ImportError as e:
+        sys.stderr.write(f"No such command {command!r} for parser {parser_name!r}\n")
+        sys.stderr.write(f"  (tried to import {module_path}: {e})\n")
+        sys.exit(2)
+
+    sys.argv = [module_path] + sys.argv[3:]
+    module.main()
+
+
+def main_iudex():
+    """Entry point of the deprecated `iudex` command, the old name of this project."""
+    sys.stderr.write("iudex: this command has been renamed to `dipo` and takes the same arguments. "
+                     "`iudex` will be removed in a future release.\n")
+    main()
+
+
+if __name__ == "__main__":
+    main()