lexograph 0.1.0__tar.gz

lexograph-0.1.0/PKG-INFO

@@ -0,0 +1,135 @@
+Metadata-Version: 2.3
+Name: lexograph
+Version: 0.1.0
+Summary: Spatialize linear text into pictures you can read: walks, spirals, dotplots, and concordances
+Keywords: nlp,text-visualization,information-design,concordance,recurrence-plot,dotplot,text-analysis,computational-humanities
+Author: Zoltan Varju, Orsolya Putz
+Author-email: Zoltan Varju <zoltan.varju@crowintelligence.org>, Orsolya Putz <orsolya.putz@crowintelligence.org>
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Visualization
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Dist: numpy>=1.26
+Requires-Dist: matplotlib>=3.8
+Requires-Dist: nltk>=3.8
+Requires-Dist: pytest>=8 ; extra == 'dev'
+Requires-Dist: pytest-cov ; extra == 'dev'
+Requires-Dist: hypothesis>=6.100 ; extra == 'dev'
+Requires-Dist: ruff>=0.4 ; extra == 'dev'
+Requires-Dist: ty ; extra == 'dev'
+Requires-Dist: mutmut>=3 ; extra == 'dev'
+Requires-Dist: mkdocs-material ; extra == 'docs'
+Requires-Dist: mkdocstrings[python] ; extra == 'docs'
+Requires-Dist: sentence-transformers>=2.2 ; extra == 'graph'
+Requires-Dist: scikit-learn>=1.3 ; extra == 'graph'
+Requires-Dist: networkx>=3.0 ; extra == 'graph'
+Requires-Dist: scipy>=1.10 ; extra == 'graph'
+Requires-Python: >=3.11
+Project-URL: Homepage, https://crowintelligence.org/
+Project-URL: Repository, https://github.com/crow-intelligence/lexograph
+Project-URL: Documentation, https://lexograph.readthedocs.io
+Provides-Extra: dev
+Provides-Extra: docs
+Provides-Extra: graph
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/crow-intelligence/lexograph/main/docs/assets/logo.png" alt="lexograph" width="480">
+</p>
+
+# lexograph
+
+Spatialize linear text into pictures you can read — in pure Python, rendered with
+matplotlib.
+
+lexograph is the visualization member of the corpus-lx family (alongside
+[chronowords](https://github.com/crow-intelligence/chronowords),
+[kenon](https://github.com/crow-intelligence/kenon), and
+[keyflux](https://github.com/crow-intelligence/keyflux)). It turns a text into a
+picture through one four-step spine — **segment → layout → encode → render** — and
+ships several presets that are each just a point on that spine:
+
+- **Punctuation spiral** — every non-alphanumeric mark, in order, along an Archimedean
+  spiral, coloured by symbol class.
+- **Text walk (2-D / 3-D)** — each sentence steps forward and turns 90°, space-filling;
+  size, colour, and glyph encode per-unit attributes. The 3-D variant lifts the walk
+  into a corkscrew.
+- **Recurrence dotplot** — the only preset that plots a text against *itself*: a
+  sentence × sentence self-similarity grid that exposes internal echo structure.
+- **Concordance** — a term's dispersion across the text (and across texts/time), with
+  optional KWIC.
+
+Every preset returns a matplotlib `Figure` and never calls `show()`, so it renders
+inline in Jupyter and saves cleanly with `fig.savefig(...)`. The core is headless and
+dependency-light; heavy analysis (sentence embeddings, graph centrality, communities)
+lives behind an optional `[graph]` extra.
+
+## Installation
+
+```bash
+uv add lexograph
+```
+
+The core depends only on `numpy`, `matplotlib`, and `nltk`.
+
+## Quickstart
+
+```python
+from lexograph import (
+    load_demo_text, punctuation_spiral, text_walk, recurrence_plot, concordance,
+)
+
+text = load_demo_text()  # Chapter 1 of Pride and Prejudice (public domain)
+
+punctuation_spiral(text)                              # marks on an Archimedean spiral
+text_walk(text)                                      # the 2-D space-filling walk
+text_walk(text, helix=True, z_step=4.0)              # the 3-D corkscrew
+recurrence_plot(text)                                # the text against itself
+concordance(text, ["Bennet", "Bingley", "wife"])    # term dispersion
+```
+
+Every call returns a matplotlib `Figure`. For PageRank-sized, community-coloured, or
+semantically-recurrent figures, install the `[graph]` extra and feed
+`lexograph.analyze.analyze_text`'s arrays into any preset.
+
+## The data contract
+
+Every visual channel is fed by a **plain per-unit array** — a scalar array for `size`,
+an array of labels or values for `colour`, an optional `glyph`/font. Nothing in the
+core knows where those numbers came from, so you can drive a spiral or a walk from
+`length`, `frequency`, or your own column with no analysis stack at all. The optional
+`analyze` layer and the `[kenon]` / `[chronowords]` integrations only *produce* arrays
+that satisfy this contract.
+
+## Documentation
+
+Full documentation — quickstart, a tutorial per preset, troubleshooting, and the API
+reference — is at [lexograph.readthedocs.io](https://lexograph.readthedocs.io). The
+sources live in `docs/`.
+
+## Roadmap
+
+Open modelling and packaging decisions are analysed in
+[`CHANGES_SUMMARY.md`](CHANGES_SUMMARY.md), and the failure modes in
+[`PRE-MORTEM.md`](PRE-MORTEM.md). The main not-yet-built pieces:
+
+- [ ] Vendor the four OFL/Apache handwriting fonts so the handwriting walk works out of
+      the box (the width-step already runs on any TTF).
+- [ ] `integrations/` — thin `[kenon]` and `[chronowords]` adapters over the data contract.
+- [ ] Optional interactive HTML/WebGL export (`render/html.py`), ported from the source
+      viewers and gated behind a stretch extra.
+- [ ] A space-filling / non-self-overlapping turn rule for very uniform texts.
+
+## Made by
+
+lexograph is made by [Crow Intelligence](https://crowintelligence.org/).
+
+## License
+
+MIT

lexograph-0.1.0/README.md

@@ -0,0 +1,94 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/crow-intelligence/lexograph/main/docs/assets/logo.png" alt="lexograph" width="480">
+</p>
+
+# lexograph
+
+Spatialize linear text into pictures you can read — in pure Python, rendered with
+matplotlib.
+
+lexograph is the visualization member of the corpus-lx family (alongside
+[chronowords](https://github.com/crow-intelligence/chronowords),
+[kenon](https://github.com/crow-intelligence/kenon), and
+[keyflux](https://github.com/crow-intelligence/keyflux)). It turns a text into a
+picture through one four-step spine — **segment → layout → encode → render** — and
+ships several presets that are each just a point on that spine:
+
+- **Punctuation spiral** — every non-alphanumeric mark, in order, along an Archimedean
+  spiral, coloured by symbol class.
+- **Text walk (2-D / 3-D)** — each sentence steps forward and turns 90°, space-filling;
+  size, colour, and glyph encode per-unit attributes. The 3-D variant lifts the walk
+  into a corkscrew.
+- **Recurrence dotplot** — the only preset that plots a text against *itself*: a
+  sentence × sentence self-similarity grid that exposes internal echo structure.
+- **Concordance** — a term's dispersion across the text (and across texts/time), with
+  optional KWIC.
+
+Every preset returns a matplotlib `Figure` and never calls `show()`, so it renders
+inline in Jupyter and saves cleanly with `fig.savefig(...)`. The core is headless and
+dependency-light; heavy analysis (sentence embeddings, graph centrality, communities)
+lives behind an optional `[graph]` extra.
+
+## Installation
+
+```bash
+uv add lexograph
+```
+
+The core depends only on `numpy`, `matplotlib`, and `nltk`.
+
+## Quickstart
+
+```python
+from lexograph import (
+    load_demo_text, punctuation_spiral, text_walk, recurrence_plot, concordance,
+)
+
+text = load_demo_text()  # Chapter 1 of Pride and Prejudice (public domain)
+
+punctuation_spiral(text)                              # marks on an Archimedean spiral
+text_walk(text)                                      # the 2-D space-filling walk
+text_walk(text, helix=True, z_step=4.0)              # the 3-D corkscrew
+recurrence_plot(text)                                # the text against itself
+concordance(text, ["Bennet", "Bingley", "wife"])    # term dispersion
+```
+
+Every call returns a matplotlib `Figure`. For PageRank-sized, community-coloured, or
+semantically-recurrent figures, install the `[graph]` extra and feed
+`lexograph.analyze.analyze_text`'s arrays into any preset.
+
+## The data contract
+
+Every visual channel is fed by a **plain per-unit array** — a scalar array for `size`,
+an array of labels or values for `colour`, an optional `glyph`/font. Nothing in the
+core knows where those numbers came from, so you can drive a spiral or a walk from
+`length`, `frequency`, or your own column with no analysis stack at all. The optional
+`analyze` layer and the `[kenon]` / `[chronowords]` integrations only *produce* arrays
+that satisfy this contract.
+
+## Documentation
+
+Full documentation — quickstart, a tutorial per preset, troubleshooting, and the API
+reference — is at [lexograph.readthedocs.io](https://lexograph.readthedocs.io). The
+sources live in `docs/`.
+
+## Roadmap
+
+Open modelling and packaging decisions are analysed in
+[`CHANGES_SUMMARY.md`](CHANGES_SUMMARY.md), and the failure modes in
+[`PRE-MORTEM.md`](PRE-MORTEM.md). The main not-yet-built pieces:
+
+- [ ] Vendor the four OFL/Apache handwriting fonts so the handwriting walk works out of
+      the box (the width-step already runs on any TTF).
+- [ ] `integrations/` — thin `[kenon]` and `[chronowords]` adapters over the data contract.
+- [ ] Optional interactive HTML/WebGL export (`render/html.py`), ported from the source
+      viewers and gated behind a stretch extra.
+- [ ] A space-filling / non-self-overlapping turn rule for very uniform texts.
+
+## Made by
+
+lexograph is made by [Crow Intelligence](https://crowintelligence.org/).
+
+## License
+
+MIT

lexograph-0.1.0/pyproject.toml

@@ -0,0 +1,112 @@
+[project]
+name = "lexograph"
+version = "0.1.0"
+description = "Spatialize linear text into pictures you can read: walks, spirals, dotplots, and concordances"
+readme = "README.md"
+authors = [
+    { name = "Zoltan Varju", email = "zoltan.varju@crowintelligence.org" },
+    { name = "Orsolya Putz", email = "orsolya.putz@crowintelligence.org" },
+]
+requires-python = ">=3.11"
+license = { text = "MIT" }
+keywords = [
+    "nlp",
+    "text-visualization",
+    "information-design",
+    "concordance",
+    "recurrence-plot",
+    "dotplot",
+    "text-analysis",
+    "computational-humanities",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Visualization",
+    "Topic :: Text Processing :: Linguistic",
+]
+dependencies = [
+    "numpy>=1.26",
+    "matplotlib>=3.8",
+    "nltk>=3.8",
+]
+
+[project.urls]
+Homepage = "https://crowintelligence.org/"
+Repository = "https://github.com/crow-intelligence/lexograph"
+Documentation = "https://lexograph.readthedocs.io"
+
+[project.optional-dependencies]
+# Heavy analysis layer (lexograph.analyze): sentence embeddings, kNN graph,
+# disparity-filter backbone, PageRank, and community detection. The core never
+# imports these.
+graph = [
+    "sentence-transformers>=2.2",
+    "scikit-learn>=1.3",
+    "networkx>=3.0",
+    "scipy>=1.10",
+]
+dev = [
+    "pytest>=8",
+    "pytest-cov",
+    "hypothesis>=6.100",
+    "ruff>=0.4",
+    "ty",
+    "mutmut>=3",
+]
+docs = [
+    "mkdocs-material",
+    "mkdocstrings[python]",
+]
+
+[build-system]
+requires = ["uv_build>=0.9.22,<0.10.0"]
+build-backend = "uv_build"
+
+[tool.ruff]
+line-length = 88
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "UP", "ANN", "D"]
+ignore = ["D105", "D107"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/**" = ["D100", "D102", "D103", "D104", "ANN"]
+"examples/**" = ["D100", "ANN"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.pytest.ini_options]
+addopts = "--doctest-modules --tb=short"
+testpaths = ["src", "tests"]
+
+# Mutation testing (dev-only, not run in CI). Scoped to the layout geometry — the
+# math-heavy core — and its fast, plotting-free test files. Run with
+# `uv run mutmut run`.
+[tool.mutmut]
+source_paths = ["src/lexograph"]
+only_mutate = [
+    "src/lexograph/layout/walk.py",
+    "src/lexograph/layout/walk3d.py",
+    "src/lexograph/layout/spiral.py",
+    "src/lexograph/layout/recurrence.py",
+    "src/lexograph/layout/dispersion.py",
+]
+pytest_add_cli_args_test_selection = [
+    "tests/test_walk.py",
+    "tests/test_walk3d.py",
+    "tests/test_spiral.py",
+    "tests/test_recurrence.py",
+    "tests/test_dispersion.py",
+    "tests/test_geometry.py",
+]
+pytest_add_cli_args = ["-p", "no:cacheprovider"]
+
+[tool.ty]

lexograph-0.1.0/src/lexograph/__init__.py

@@ -0,0 +1,65 @@
+"""lexograph — spatialize linear text into pictures you can read.
+
+lexograph turns a linear text into a picture through one four-step spine:
+**segment** the text into ordered units (characters, tokens, or sentences),
+**lay them out** in 2-D or 3-D space, **encode** per-unit attributes onto
+visual channels (size, colour, glyph), and **render** the result as a
+matplotlib :class:`~matplotlib.figure.Figure` that displays inline in Jupyter
+and saves cleanly with ``fig.savefig(...)``.
+"""
+
+__version__ = "0.1.0"
+
+from lexograph.datasets import load_demo_text
+from lexograph.encode import (
+    Channels,
+    categorical_colors,
+    continuous_colors,
+    normalize_size,
+)
+from lexograph.layout import (
+    kwic,
+    linear_layout,
+    rendered_widths,
+    spiral_layout,
+    term_offsets,
+    walk3d_layout,
+    walk_layout,
+)
+from lexograph.presets import (
+    concordance,
+    punctuation_spiral,
+    recurrence_plot,
+    text_walk,
+)
+from lexograph.render import frame_axes, render_path, render_path_3d, render_points
+from lexograph.scalars import frequencies, lengths, positions
+from lexograph.segment import segment
+
+__all__ = [
+    "segment",
+    "lengths",
+    "positions",
+    "frequencies",
+    "linear_layout",
+    "walk_layout",
+    "walk3d_layout",
+    "spiral_layout",
+    "rendered_widths",
+    "term_offsets",
+    "kwic",
+    "normalize_size",
+    "categorical_colors",
+    "continuous_colors",
+    "Channels",
+    "render_points",
+    "render_path",
+    "render_path_3d",
+    "frame_axes",
+    "punctuation_spiral",
+    "text_walk",
+    "recurrence_plot",
+    "concordance",
+    "load_demo_text",
+    "__version__",
+]

lexograph-0.1.0/src/lexograph/_types.py

@@ -0,0 +1,26 @@
+"""Shared type aliases for the lexograph package.
+
+The visual-channel arrays named here are the package's public data contract:
+``encode`` accepts plain per-unit arrays, and ``analyze``/``integrations`` only
+ever *produce* arrays that satisfy these aliases. Nothing in the core needs to
+know where the numbers came from.
+"""
+
+from __future__ import annotations
+
+from typing import Literal, TypeAlias
+
+import numpy as np
+import numpy.typing as npt
+
+Unit: TypeAlias = str
+"""A single segmented unit of text: a character, token, or sentence."""
+
+UnitKind: TypeAlias = Literal["chars", "tokens", "sentences"]
+"""Which kind of unit a segmenter emits."""
+
+FloatArray: TypeAlias = npt.NDArray[np.float64]
+"""A 1-D array of floats: a per-unit scalar channel (e.g. ``size``)."""
+
+Coords: TypeAlias = npt.NDArray[np.float64]
+"""An ``(N, 2)`` or ``(N, 3)`` array of layout coordinates, one row per unit."""

lexograph-0.1.0/src/lexograph/analyze/__init__.py

@@ -0,0 +1,139 @@
+"""Optional analysis layer: turn a text into per-sentence channel arrays.
+
+Behind the ``lexograph[graph]`` extra. It runs the Wittgenstein pipeline —
+sentence embeddings → cosine kNN graph → (optional disparity backbone) → PageRank
+and community detection — and hands back plain arrays that satisfy the
+``encode`` data contract: a ``size`` array (PageRank centrality), a ``community``
+array (colour labels), and a cosine ``distances`` matrix (for a semantic
+recurrence dotplot). The core never imports this; the arrow points inward.
+
+The core dependency-free scalars (length, position, frequency) live in
+:mod:`lexograph.scalars`.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+from lexograph.analyze.backbone import extract_backbone
+from lexograph.analyze.embeddings import embed_sentences
+from lexograph.analyze.graph import (
+    community_labels,
+    embedding_distances,
+    knn_graph,
+    pagerank_scores,
+)
+from lexograph.segment.units import sentences as split_sentences
+
+if TYPE_CHECKING:
+    import numpy as np
+
+    from lexograph._types import FloatArray
+    from lexograph.analyze.graph import CommunityMethod
+
+__all__ = [
+    "Analysis",
+    "analyze_text",
+    "embed_sentences",
+    "knn_graph",
+    "embedding_distances",
+    "pagerank_scores",
+    "community_labels",
+    "extract_backbone",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class Analysis:
+    """The per-sentence channel arrays derived from a text.
+
+    Attributes:
+        sentences: The segmented sentences (length ``N``).
+        embeddings: The ``(N, D)`` sentence embeddings.
+        size: PageRank centrality per sentence — the size channel.
+        community: Community id per sentence — the colour channel.
+        distances: The ``(N, N)`` cosine distance matrix — pass it to
+            ``recurrence_plot(distances=...)`` for a semantic dotplot.
+    """
+
+    sentences: list[str]
+    embeddings: FloatArray
+    size: FloatArray
+    community: np.ndarray
+    distances: FloatArray
+
+
+def analyze_text(
+    text: str,
+    *,
+    embeddings: FloatArray | None = None,
+    k: int = 5,
+    community: CommunityMethod = "louvain",
+    n_clusters: int = 10,
+    backbone: bool = False,
+    min_alpha_ptile: float = 0.5,
+    seed: int = 42,
+) -> Analysis:
+    """Run the analysis pipeline and return per-sentence channel arrays.
+
+    Args:
+        text: The source text.
+        embeddings: Precomputed ``(N, D)`` embeddings to use. If ``None``, the
+            sentences are embedded with the default model (a model download).
+        k: Neighbours per node in the kNN graph.
+        community: Community method — ``"louvain"`` (default) or ``"kmeans"``.
+        n_clusters: Number of clusters for the ``"kmeans"`` method.
+        backbone: If ``True``, sparsify the kNN graph with the disparity filter
+            before PageRank and community detection.
+        min_alpha_ptile: Disparity-filter threshold (used when ``backbone``).
+        seed: Random seed for reproducibility.
+
+    Returns:
+        An :class:`Analysis` whose arrays align to the segmented sentences.
+
+    Raises:
+        ValueError: If ``embeddings`` is given but its length does not match the
+            sentence count.
+
+    Example:
+        Drive a walk and a semantic dotplot from the analysis (not run as a
+        doctest — embedding downloads a model)::
+
+            from lexograph import text_walk, recurrence_plot, load_demo_text
+            from lexograph.analyze import analyze_text
+
+            a = analyze_text(load_demo_text())
+            walk = text_walk(load_demo_text(), colour=a.community,
+                             colour_kind="categorical", size=a.size)
+            dots = recurrence_plot(load_demo_text(), distances=a.distances,
+                                   threshold=0.4)
+    """
+    units = split_sentences(text)
+    n = len(units)
+    if embeddings is None:
+        embeddings = embed_sentences(units)
+    elif len(embeddings) != n:
+        msg = f"embeddings must have one row per sentence ({n}), got {len(embeddings)}"
+        raise ValueError(msg)
+
+    graph = knn_graph(embeddings, k=k)
+    if backbone:
+        graph = extract_backbone(graph, min_alpha_ptile=min_alpha_ptile)
+    size = pagerank_scores(graph, n)
+    labels = community_labels(
+        graph,
+        n,
+        method=community,
+        embeddings=embeddings,
+        n_clusters=n_clusters,
+        seed=seed,
+    )
+    distances = embedding_distances(embeddings)
+    return Analysis(
+        sentences=units,
+        embeddings=embeddings,
+        size=size,
+        community=labels,
+        distances=distances,
+    )