context-atlas 1.0.0__tar.gz

context_atlas-1.0.0/.gitignore

@@ -0,0 +1,13 @@
+.DS_Store
+/vendor/
+/node_modules/
+/js/dist/
+.env
+.phpunit.result.cache
+__pycache__/
+.pytest_cache/
+.ruff_cache/
+.venv/
+/dist/
+*.egg-info/
+/local/

context_atlas-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Fox Islam
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

context_atlas-1.0.0/PKG-INFO

@@ -0,0 +1,196 @@
+Metadata-Version: 2.5
+Name: context-atlas
+Version: 1.0.0
+Summary: Utility library to help process large context inputs in limited context window scenarios
+Project-URL: Homepage, https://github.com/Fox-Islam/context-atlas
+Project-URL: Issues, https://github.com/Fox-Islam/context-atlas/issues
+Author-email: Fox Islam <foxislam@outlook.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: chunking,jev,reranking,retrieval,system-one,typesafe
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: typesafe-sdk>=0.7
+Description-Content-Type: text/markdown
+
+# Context Atlas for Python
+
+A utility library to help process large context inputs in limited context window scenarios:
+it reads documents in the formats people give an LLM, structures each section by what kind of
+text it is, and, given a question, gathers the context that answers it instead of the whole
+document, using [Jev](https://docs.typesafe.ai) for the judgements. It reads documents the same
+way and sends Jev the same requests as the PHP and JavaScript packages; the `spec/` directory in the
+repository states the rules.
+
+## Install
+
+```sh
+pip install context-atlas      # or: uv add context-atlas
+```
+
+Python 3.10 or later. Jev is reached through the official
+[`typesafe-sdk`](https://pypi.org/project/typesafe-sdk/), which reads
+`TYPESAFE_API_KEY`, or `OPENROUTER_API_KEY` with `TYPESAFE_PROVIDER=openrouter`.
+
+Bad input (an unknown format or config key, a file that cannot be read, a malformed block
+list, retrieval without Jev) raises `ContextAtlasError`. A Jev request that fails never
+raises: it gives `None` or the structural result, as each section below says.
+
+## Chunking
+
+```python
+from typesafe_sdk import TypeSafeClient
+from context_atlas import Chunker, Config, parse_markdown
+
+client = TypeSafeClient()
+chunked = Chunker(client, Config(max_tokens=600)).chunk(parse_markdown(markdown))
+
+for chunk in chunked.chunks:
+    index.add(chunk.text, heading_path=chunk.heading_path)
+```
+
+`Chunker()` with no Jev, or a Jev that does not answer, cuts at headings and
+sizes only, and `chunked.used_jev` is false. `parse_text` reads plain text and
+`parse_blocks` a JSON list of `{"type", "text", "level"?, "meta"?}`.
+`chunked.to_dict()` is the JSON the command line prints. `Config.from_dict` takes
+the snake_case keys of `spec/config.schema.json` and refuses any other key.
+
+## Ranking
+
+```python
+from context_atlas import Passage, Reranker
+
+hits = index.search(query, limit=15)
+ranked = Reranker(client).rank(
+    [Passage(hit.id, hit.text, hit.heading_path) for hit in hits], query, limit=5
+)
+
+if ranked is None:
+    ranked_ids = [hit.id for hit in hits[:5]]  # Jev did not answer: keep the search order
+else:
+    ranked_ids = [item.passage.id for item in ranked]
+```
+
+## Sending the request yourself
+
+`Chunker`, `Reranker` and `Processor` also take a hook in place of the client. It is
+handed the request, `{"state": ..., "questions": {...}}` with SDK `Noul` and `Choice`
+objects, and returns the SDK's `SystemOneResponse`, or `None` for no answer. Caching and
+logging go here:
+
+```python
+from typesafe_sdk import SystemOneResponse
+from context_atlas import Chunker, JevRequest
+
+def hook(request: JevRequest) -> SystemOneResponse | None:
+    return client.system_one(request["state"], request["questions"])
+
+chunked = Chunker(hook).chunk(blocks)
+```
+
+An exception raised by the hook or the client counts as no answer.
+
+## Documents and retrieval
+
+`Processor` turns input into a `Document`, plain data you store and hand back as a `Source` at
+retrieval. `spec/document.md`, `spec/prose.md`, `spec/markup.md` and `spec/tabular.md` give
+the rules.
+
+### Ingesting
+
+```python
+from context_atlas import FileStore, Processor
+
+processor = Processor(client)
+store = FileStore("documents")
+
+store.put(processor.ingest_file("terms.html", "terms"))
+store.put(processor.ingest_file("spend.csv", "spend-2026-03", title="Spend over £25,000"))
+```
+
+| Method | Reads |
+| --- | --- |
+| `ingest_file(path, id, profile=None, title=None)` | a file, in the format its extension names, `.docx` and `.xlsx` included |
+| `ingest(text_or_bytes, id, format, profile=None, title=None)` | text or bytes in any format |
+| `ingest_blocks(blocks, id, profile=None)` | blocks you built, such as the headings and paragraphs of a PDF |
+| `ingest_markup(text, id, format, title=None)` | a data file, cut into path subtrees linked by `$ref`, YAML aliases and XML references, with no Jev request |
+| `ingest_csv(text_or_bytes, id, title)` | CSV already in memory |
+| `ingest_spreadsheet(paths, id, title=None)` | one CSV or XLSX file, or several (`{"Data": "population.csv", "Metadata": "regions.csv"}`) joined into one document |
+| `append(document, blocks, profile=None)` | blocks added to the end of a prose document |
+
+Document formats (`markdown`, `mdx`, `text`, `html`, `rst`, `asciidoc`, `org`, `latex`,
+`ipynb`, `vtt`, `srt`, `chat`, `docx`, `blocks`) become prose; data formats (`yaml`, `json`,
+`jsonl`, `xml`, `toml`) go to `ingest_markup`; `csv` and `xlsx` to the spreadsheet readers.
+
+- **Prose** gets a profile per section (`guidance`, `normative`, `narrative`, `procedural`,
+  `reference` or `conversational`), detected by Jev unless `profile` names one. A section whose
+  blocks carry `meta.speaker` is conversational without asking Jev.
+- **Chat logs**: `ingest_file` reads a `.json` or `.jsonl` file as a chat log when the chat
+  reader finds at least two messages in it, and as data otherwise. For a chat log in memory,
+  pass `"chat"` to `ingest`. For `append`,
+  `context_atlas.parsing.formats.blocks(new_lines, "chat")` reads new lines. The README's "Chat
+  and agent transcripts" covers attachments, `output` units and the `conversation_*` config
+  keys.
+- **DOCX** is binary: `ingest` takes its bytes, not text.
+- **Spreadsheets** are profiled without Jev; at retrieval Jev picks the table, measure, filters
+  and grouping, and the package computes the answer.
+- `context_atlas.parsing.formats.blocks(text, format)` gives any document format's blocks
+  without ingesting them, and `formats.of(path)` the format an extension names.
+
+### Storing
+
+- `FileStore(directory)` keeps one JSON file per document plus an append-only term index under
+  `index/`, in the layout the PHP and JavaScript packages share. `compact()` rewrites the index
+  without the lines of rewritten or removed documents.
+- `Collection()` holds documents in memory.
+- `put(document)` replaces a stored document with the same id; `remove(document_id)` deletes
+  one. Drop its vectors from your own store too.
+- For your own database, `document.to_dict()` is the JSON to keep and `Document.from_dict()`
+  reads it back. Any object with `postings`, `stats` and `document` methods can stand in for a
+  store (see `Source`).
+
+### Retrieving
+
+```python
+def nearest(query, limit, kinds, documents):
+    hits = vectors.search(embed(query), limit, document=documents)
+    return [{"document": hit.payload["document"], "unit": hit.payload["unit"]} for hit in hits]
+
+context = processor.retrieve(question, store, nearest, documents=allowed_file_ids)
+if context is not None and not context.abstained:
+    prompt = context.text
+```
+
+`retrieve()` searches by exact terms. `nearest` fuses in your vector search: embed each unit of
+`document.units` under its document and unit ids at ingestion, and return those ids (`embed`
+and `vectors` stand for your model and store). A `nearest` with three parameters is not given
+the documents. `documents` limits both searches to those documents.
+
+It returns None when Jev did not answer. A `Context` has `text`, `sources` (a
+`{"document", "unit"}` dict per part, with `rows` for a table and `superseded_by` for a
+corrected message), `abstained` and `trace`.
+
+## Command line
+
+```sh
+context-atlas chunk [FILE|-] [--format=markdown|text|blocks] [--no-jev] [--config=FILE] [--pretty]
+context-atlas rank QUERY [FILE|-] [--limit=N] [--config=FILE] [--pretty]
+context-atlas ingest FILE|- [--id=ID] [--format=FORMAT] [--profile=PROFILE] [--title=TITLE] [--store=DIR] [--no-jev] [--config=FILE] [--pretty]
+context-atlas retrieve QUERY --store=DIR [--documents=ID,...] [--config=FILE] [--pretty]
+python -m context_atlas ...
+```
+
+`chunk` reads Markdown, a `.txt` or `.text` file as plain text, and a `.json`
+file as blocks. `rank` reads a JSON list of passages or the output of `chunk`.
+`ingest` reads any supported format (by extension, or `--format`, which a chat
+log needs unless it is a `.json` or `.jsonl` file; stdin needs `--format` and `--id`) and prints the document, or with
+`--store=DIR` saves it there and prints a summary. `retrieve` answers QUERY from that store by exact
+terms, optionally limited to `--documents`.
+`--config` names a JSON file of the keys in `spec/config.schema.json`. Exit
+codes: 0 done, 1 bad input, 2 Jev did not answer. `TYPESAFE_PROVIDER=openrouter`
+with `OPENROUTER_API_KEY` reaches Jev through OpenRouter.

context_atlas-1.0.0/docs/python.md

@@ -0,0 +1,176 @@
+# Context Atlas for Python
+
+A utility library to help process large context inputs in limited context window scenarios:
+it reads documents in the formats people give an LLM, structures each section by what kind of
+text it is, and, given a question, gathers the context that answers it instead of the whole
+document, using [Jev](https://docs.typesafe.ai) for the judgements. It reads documents the same
+way and sends Jev the same requests as the PHP and JavaScript packages; the `spec/` directory in the
+repository states the rules.
+
+## Install
+
+```sh
+pip install context-atlas      # or: uv add context-atlas
+```
+
+Python 3.10 or later. Jev is reached through the official
+[`typesafe-sdk`](https://pypi.org/project/typesafe-sdk/), which reads
+`TYPESAFE_API_KEY`, or `OPENROUTER_API_KEY` with `TYPESAFE_PROVIDER=openrouter`.
+
+Bad input (an unknown format or config key, a file that cannot be read, a malformed block
+list, retrieval without Jev) raises `ContextAtlasError`. A Jev request that fails never
+raises: it gives `None` or the structural result, as each section below says.
+
+## Chunking
+
+```python
+from typesafe_sdk import TypeSafeClient
+from context_atlas import Chunker, Config, parse_markdown
+
+client = TypeSafeClient()
+chunked = Chunker(client, Config(max_tokens=600)).chunk(parse_markdown(markdown))
+
+for chunk in chunked.chunks:
+    index.add(chunk.text, heading_path=chunk.heading_path)
+```
+
+`Chunker()` with no Jev, or a Jev that does not answer, cuts at headings and
+sizes only, and `chunked.used_jev` is false. `parse_text` reads plain text and
+`parse_blocks` a JSON list of `{"type", "text", "level"?, "meta"?}`.
+`chunked.to_dict()` is the JSON the command line prints. `Config.from_dict` takes
+the snake_case keys of `spec/config.schema.json` and refuses any other key.
+
+## Ranking
+
+```python
+from context_atlas import Passage, Reranker
+
+hits = index.search(query, limit=15)
+ranked = Reranker(client).rank(
+    [Passage(hit.id, hit.text, hit.heading_path) for hit in hits], query, limit=5
+)
+
+if ranked is None:
+    ranked_ids = [hit.id for hit in hits[:5]]  # Jev did not answer: keep the search order
+else:
+    ranked_ids = [item.passage.id for item in ranked]
+```
+
+## Sending the request yourself
+
+`Chunker`, `Reranker` and `Processor` also take a hook in place of the client. It is
+handed the request, `{"state": ..., "questions": {...}}` with SDK `Noul` and `Choice`
+objects, and returns the SDK's `SystemOneResponse`, or `None` for no answer. Caching and
+logging go here:
+
+```python
+from typesafe_sdk import SystemOneResponse
+from context_atlas import Chunker, JevRequest
+
+def hook(request: JevRequest) -> SystemOneResponse | None:
+    return client.system_one(request["state"], request["questions"])
+
+chunked = Chunker(hook).chunk(blocks)
+```
+
+An exception raised by the hook or the client counts as no answer.
+
+## Documents and retrieval
+
+`Processor` turns input into a `Document`, plain data you store and hand back as a `Source` at
+retrieval. `spec/document.md`, `spec/prose.md`, `spec/markup.md` and `spec/tabular.md` give
+the rules.
+
+### Ingesting
+
+```python
+from context_atlas import FileStore, Processor
+
+processor = Processor(client)
+store = FileStore("documents")
+
+store.put(processor.ingest_file("terms.html", "terms"))
+store.put(processor.ingest_file("spend.csv", "spend-2026-03", title="Spend over £25,000"))
+```
+
+| Method | Reads |
+| --- | --- |
+| `ingest_file(path, id, profile=None, title=None)` | a file, in the format its extension names, `.docx` and `.xlsx` included |
+| `ingest(text_or_bytes, id, format, profile=None, title=None)` | text or bytes in any format |
+| `ingest_blocks(blocks, id, profile=None)` | blocks you built, such as the headings and paragraphs of a PDF |
+| `ingest_markup(text, id, format, title=None)` | a data file, cut into path subtrees linked by `$ref`, YAML aliases and XML references, with no Jev request |
+| `ingest_csv(text_or_bytes, id, title)` | CSV already in memory |
+| `ingest_spreadsheet(paths, id, title=None)` | one CSV or XLSX file, or several (`{"Data": "population.csv", "Metadata": "regions.csv"}`) joined into one document |
+| `append(document, blocks, profile=None)` | blocks added to the end of a prose document |
+
+Document formats (`markdown`, `mdx`, `text`, `html`, `rst`, `asciidoc`, `org`, `latex`,
+`ipynb`, `vtt`, `srt`, `chat`, `docx`, `blocks`) become prose; data formats (`yaml`, `json`,
+`jsonl`, `xml`, `toml`) go to `ingest_markup`; `csv` and `xlsx` to the spreadsheet readers.
+
+- **Prose** gets a profile per section (`guidance`, `normative`, `narrative`, `procedural`,
+  `reference` or `conversational`), detected by Jev unless `profile` names one. A section whose
+  blocks carry `meta.speaker` is conversational without asking Jev.
+- **Chat logs**: `ingest_file` reads a `.json` or `.jsonl` file as a chat log when the chat
+  reader finds at least two messages in it, and as data otherwise. For a chat log in memory,
+  pass `"chat"` to `ingest`. For `append`,
+  `context_atlas.parsing.formats.blocks(new_lines, "chat")` reads new lines. The README's "Chat
+  and agent transcripts" covers attachments, `output` units and the `conversation_*` config
+  keys.
+- **DOCX** is binary: `ingest` takes its bytes, not text.
+- **Spreadsheets** are profiled without Jev; at retrieval Jev picks the table, measure, filters
+  and grouping, and the package computes the answer.
+- `context_atlas.parsing.formats.blocks(text, format)` gives any document format's blocks
+  without ingesting them, and `formats.of(path)` the format an extension names.
+
+### Storing
+
+- `FileStore(directory)` keeps one JSON file per document plus an append-only term index under
+  `index/`, in the layout the PHP and JavaScript packages share. `compact()` rewrites the index
+  without the lines of rewritten or removed documents.
+- `Collection()` holds documents in memory.
+- `put(document)` replaces a stored document with the same id; `remove(document_id)` deletes
+  one. Drop its vectors from your own store too.
+- For your own database, `document.to_dict()` is the JSON to keep and `Document.from_dict()`
+  reads it back. Any object with `postings`, `stats` and `document` methods can stand in for a
+  store (see `Source`).
+
+### Retrieving
+
+```python
+def nearest(query, limit, kinds, documents):
+    hits = vectors.search(embed(query), limit, document=documents)
+    return [{"document": hit.payload["document"], "unit": hit.payload["unit"]} for hit in hits]
+
+context = processor.retrieve(question, store, nearest, documents=allowed_file_ids)
+if context is not None and not context.abstained:
+    prompt = context.text
+```
+
+`retrieve()` searches by exact terms. `nearest` fuses in your vector search: embed each unit of
+`document.units` under its document and unit ids at ingestion, and return those ids (`embed`
+and `vectors` stand for your model and store). A `nearest` with three parameters is not given
+the documents. `documents` limits both searches to those documents.
+
+It returns None when Jev did not answer. A `Context` has `text`, `sources` (a
+`{"document", "unit"}` dict per part, with `rows` for a table and `superseded_by` for a
+corrected message), `abstained` and `trace`.
+
+## Command line
+
+```sh
+context-atlas chunk [FILE|-] [--format=markdown|text|blocks] [--no-jev] [--config=FILE] [--pretty]
+context-atlas rank QUERY [FILE|-] [--limit=N] [--config=FILE] [--pretty]
+context-atlas ingest FILE|- [--id=ID] [--format=FORMAT] [--profile=PROFILE] [--title=TITLE] [--store=DIR] [--no-jev] [--config=FILE] [--pretty]
+context-atlas retrieve QUERY --store=DIR [--documents=ID,...] [--config=FILE] [--pretty]
+python -m context_atlas ...
+```
+
+`chunk` reads Markdown, a `.txt` or `.text` file as plain text, and a `.json`
+file as blocks. `rank` reads a JSON list of passages or the output of `chunk`.
+`ingest` reads any supported format (by extension, or `--format`, which a chat
+log needs unless it is a `.json` or `.jsonl` file; stdin needs `--format` and `--id`) and prints the document, or with
+`--store=DIR` saves it there and prints a summary. `retrieve` answers QUERY from that store by exact
+terms, optionally limited to `--documents`.
+`--config` names a JSON file of the keys in `spec/config.schema.json`. Exit
+codes: 0 done, 1 bad input, 2 Jev did not answer. `TYPESAFE_PROVIDER=openrouter`
+with `OPENROUTER_API_KEY` reaches Jev through OpenRouter.

context_atlas-1.0.0/pyproject.toml

@@ -0,0 +1,53 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "context-atlas"
+version = "1.0.0"
+description = "Utility library to help process large context inputs in limited context window scenarios"
+readme = "docs/python.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.10"
+keywords = ["typesafe", "jev", "chunking", "retrieval", "reranking", "system-one"]
+authors = [{ name = "Fox Islam", email = "foxislam@outlook.com" }]
+classifiers = [
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Text Processing",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Typing :: Typed",
+]
+dependencies = [
+    "typesafe-sdk>=0.7",
+]
+
+[project.urls]
+Homepage = "https://github.com/Fox-Islam/context-atlas"
+Issues = "https://github.com/Fox-Islam/context-atlas/issues"
+
+[project.scripts]
+context-atlas = "context_atlas.console:main"
+
+[dependency-groups]
+dev = ["pytest>=8", "ruff>=0.13"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["python/src/context_atlas"]
+
+[tool.hatch.build.targets.sdist]
+include = ["/python", "/spec", "/docs/python.md", "/LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["python/tests"]
+pythonpath = ["python/src"]
+
+[tool.ruff]
+line-length = 120
+src = ["python/src"]
+include = ["python/**/*.py"]
+
+[tool.ruff.lint]
+select = ["F", "E", "W", "I", "UP", "B", "C4", "SIM", "RET", "PTH", "RUF"]

context_atlas-1.0.0/python/src/context_atlas/__init__.py

@@ -0,0 +1,47 @@
+"""
+Split documents into passages at headings, size limits and, using Jev, where the topic
+changes; rank search hits by whether they answer a query; and ingest spreadsheets and prose into
+stored documents that retrieval answers questions from.
+"""
+from .blocks import Block, BlockType
+from .chunk import Chunk, Chunked
+from .chunker import Chunker
+from .config import Config
+from .context import Context
+from .document import Document, Link, Unit
+from .errors import ContextAtlasError
+from .index import Collection, FileStore, Source
+from .jev import Jev, JevHook, JevRequest
+from .parsing import parse_blocks, parse_markdown, parse_text
+from .passage import Passage, RankedPassage
+from .processor import Processor
+from .reranker import Reranker
+
+__version__ = '1.0.0'
+
+__all__ = [
+    'Block',
+    'BlockType',
+    'Chunk',
+    'Chunked',
+    'Chunker',
+    'Collection',
+    'Config',
+    'Context',
+    'ContextAtlasError',
+    'Document',
+    'FileStore',
+    'Jev',
+    'JevHook',
+    'JevRequest',
+    'Link',
+    'Passage',
+    'Processor',
+    'RankedPassage',
+    'Reranker',
+    'Source',
+    'Unit',
+    'parse_blocks',
+    'parse_markdown',
+    'parse_text',
+]

context_atlas-1.0.0/python/src/context_atlas/__main__.py

@@ -0,0 +1,4 @@
+"""`python -m context_atlas`, for a checkout with no console script on the PATH."""
+from .console import main
+
+raise SystemExit(main())

context_atlas-1.0.0/python/src/context_atlas/blocks.py

@@ -0,0 +1,82 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Any
+
+from .errors import ContextAtlasError
+
+
+class BlockType(str, Enum):
+    HEADING = 'heading'
+    PARAGRAPH = 'paragraph'
+    TABLE = 'table'
+    CODE = 'code'
+
+
+@dataclass(frozen=True)
+class Block:
+    """
+    One unit of a document.
+
+    A paragraph is the only kind split into sentences when it is too long; a table or a code
+    block is kept whole however long it runs. Only a heading has a `level`. `meta` is carried
+    through and never read, so a caller can map chunks back to pages or offsets.
+    """
+
+    type: BlockType
+    text: str
+    level: int | None = None
+    meta: dict[str, Any] | list[Any] = field(default_factory=dict)
+
+    @staticmethod
+    def heading(text: str, level: int, meta: dict[str, Any] | None = None) -> Block:
+        return Block(BlockType.HEADING, text, level, meta or {})
+
+    @staticmethod
+    def paragraph(text: str, meta: dict[str, Any] | None = None) -> Block:
+        return Block(BlockType.PARAGRAPH, text, None, meta or {})
+
+    @staticmethod
+    def from_dict(value: Any) -> Block:
+        """
+        Reads `{"type", "text", "level"?, "meta"?}`.
+
+        :raises ContextAtlasError: on an unknown type, a text that is not a string, or a
+            heading without an integer level
+        """
+        value = value if isinstance(value, dict) else {}
+        kind = next((member for member in BlockType if member.value == value.get('type')), None)
+        text = value.get('text')
+        level = value.get('level')
+
+        if kind is None or not isinstance(text, str):
+            raise ContextAtlasError(
+                'A block needs a "type" (heading, paragraph, table or code) and a "text".',
+            )
+        if kind is BlockType.HEADING and (not isinstance(level, int) or isinstance(level, bool)):
+            raise ContextAtlasError('A heading block needs an integer "level".')
+
+        return Block(kind, text, level if kind is BlockType.HEADING else None, _meta(value.get('meta')))
+
+    def is_heading(self) -> bool:
+        return self.type is BlockType.HEADING
+
+    def to_dict(self) -> dict[str, Any]:
+        """The block as `from_dict` reads it, without an empty `level` or `meta`."""
+        value: dict[str, Any] = {'type': self.type.value, 'text': self.text}
+
+        if self.level is not None:
+            value['level'] = self.level
+        if self.meta:
+            value['meta'] = self.meta
+
+        return value
+
+
+def _meta(value: Any) -> dict[str, Any] | list[Any]:
+    """`meta` as PHP's `(array)` cast leaves it: an object or list kept, another value wrapped in a list."""
+    if value is None:
+        return {}
+
+    return value if isinstance(value, (dict, list)) else [value]

context_atlas-1.0.0/python/src/context_atlas/chunk.py

@@ -0,0 +1,61 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, replace
+from typing import Any, TypedDict
+
+
+class Heading(TypedDict):
+    block: int
+    title: str
+    level: int
+
+
+@dataclass(frozen=True)
+class Chunk:
+    """
+    A passage ready to embed.
+
+    `text` is the heading path in brackets, any overlap carried from the previous chunk, then
+    the chunk's own text. `tokens` counts all of it and `overlap` the carried part. `blocks`
+    are indexes into the blocks given to `Chunker.chunk`.
+    """
+
+    text: str
+    blocks: list[int]
+    heading_path: list[str]
+    heading_level: int | None
+    section: int
+    tokens: int
+    overlap: int
+    topic_start: bool
+
+    def with_blocks(self, blocks: list[int]) -> Chunk:
+        return replace(self, blocks=blocks)
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            'text': self.text,
+            'blocks': list(self.blocks),
+            'heading_path': list(self.heading_path),
+            'heading_level': self.heading_level,
+            'section': self.section,
+            'tokens': self.tokens,
+            'overlap': self.overlap,
+            'topic_start': self.topic_start,
+        }
+
+
+@dataclass(frozen=True)
+class Chunked:
+    """The chunks of a document, with its outline: every heading in order."""
+
+    chunks: list[Chunk]
+    outline: list[Heading]
+    used_jev: bool
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            'chunks': [chunk.to_dict() for chunk in self.chunks],
+            'outline': [dict(heading) for heading in self.outline],
+            'used_jev': self.used_jev,
+        }