openmaterica 0.1.0.dev1__py3-none-any.whl

materica/README.md

@@ -0,0 +1,33 @@
+# `materica`
+
+The Python package. It implements the reader side of the OpenMaterica specification (`docs/specification/`).
+The specification is normative: where this code and it differ, the code is wrong.
+
+## Layers
+
+The package is built in layers (AD 7.1), and a module imports only from its own layer or a lower one.
+
+| Layer | Directory | What it holds |
+|---|---|---|
+| L0, schemas | `schema/`, `shapes.py` | the JSON Schemas (`schema/README.md`), and the JSON shapes as TypedDicts generated from them |
+| L1, data | `data/`, `schema/0.1/vocabulary/` | the specification's conformance cases and the inputs they supply (`data/conformance/`, with their own `README.md`), and its vocabulary, published with the schemas |
+| L2, core | `core/` | strict JSON, the canonical form and the content hash; identifiers, the envelope checks and the read up to step 5; Python's standard library only (`core/README.md`) |
+| L4, readers | `io/` | files into the core's types (`io/README.md`) |
+
+The authoring tools (L3), the sampling helpers (`sample/`, L5) and the command line (`tools/`, L6) arrive in
+later steps. `tests/unit/test_source_files.py` walks every module's imports: a module imports only its own
+layer or a lower one; `core/` imports only the standard library, itself and `shapes.py`, which imports only the
+standard library; and nothing imports `materica_authoring`.
+
+## What holds across the package
+
+- No run-time dependency (AD 7.2): the built wheel declares none (`scripts/check_wheel.py`).
+- It is typed (PEP 561) and ships `py.typed`. `mypy --strict` and pyright in strict mode pass over it, and a
+  consumer's code type-checks against the installed wheel (`tests/typing/`). `shapes.pyi`, which type checkers
+  read in place of `shapes.py`, marks what a read returns read-only (PEP 705).
+- Everything under `data/` and `schema/` ships in the wheel byte for byte, since the conformance inputs are
+  byte-exact and the reader loads the vocabulary (`scripts/check_wheel.py`).
+- Generated files are never edited by hand. `scripts/` regenerates them and CI fails on any difference: the
+  vocabulary from the specification's tables, the reader schema and the shapes from the writer schemas
+  (CONTRIBUTING, "Changing the schema").
+- `__init__.py` holds the public API built so far: `content_hash`, `load_records` and `read`.

materica/__init__.py

@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: Apache-2.0
+"""OpenMaterica: material records and the reader of their specification (`docs/specification/`)."""
+
+from materica.core.hashing import content_hash
+from materica.core.read import read
+from materica.io.plain import load_records
+
+__all__ = ["content_hash", "load_records", "read"]

materica/core/README.md

@@ -0,0 +1,72 @@
+# `materica.core`
+
+The core (layer L2, AD 7.1): the code that reads and checks records. It uses Python's standard library only
+(AD 7.2, CONTRIBUTING constraint 3), so it adds nothing to a consumer's dependencies; the shapes it types its
+results with (`materica.shapes`, generated) use the standard library only too.
+
+## The data paths
+
+The content hash:
+
+```
+bytes ─▶ jsonio.parse ─▶ value ─▶ hashing.hashable_view ─▶ view ─▶ canonical.canonicalize ─▶ bytes ─▶ SHA-256
+            │                       (hashing.content_hash runs these three, and writes "sha256:<hex>")
+            └─▶ Reason("malformed")
+```
+
+A read, steps 0 to 5 of `read_side_contract.md` section 3:
+
+```
+request ─▶ read.ReadRequest.from_json ─▶ step 0, the request on its own ─▶ reasons, or a ReadRequest
+                                                                                │
+sources ─▶ record.RecordStore.add ─▶ candidates by id and version,              ▼
+           and the unreadable sources                      resolve.resolve_record
+                                                           1 the candidate that states the pin
+                                                           2 its schema version     ┐
+                                                           3 envelope.py's checks   ├ kept on the Record
+                                                           5 its type               ┘
+                                                           4 the hash, with HS: recomputed on every read,
+                                                             between steps 3 and 5
+                                                                  │
+               read.ReadResult (status, reasons, labels) ◀────────┘   or NotImplementedError for steps 6 to 13
+```
+
+| Module | Implements |
+|---|---|
+| `reasons.py` | a reason as `read_side_contract.md` section 2 shapes it, each code and condition once; codes from the table in section 4 |
+| `status.py` | the three statuses of a read, and the worst of several (section 2) |
+| `jsonio.py` | `record_format.md` section 1 item 6, strict JSON; `JsonValue`, the type of what it reads |
+| `canonical.py` | `record_format.md` section 9 item 2: RFC 8785, with no Unicode normalisation |
+| `hashing.py` | `record_format.md` section 9 items 1 and 2: what the hash leaves out, and the hash |
+| `identifiers.py` | `record_format.md` sections 7 and 8: identifiers and versions, matched in full; material references |
+| `vocabulary.py` | the tokens of the specification's tables, read from `schema/0.1/vocabulary/` |
+| `envelope.py` | the reader schema's checks by hand (`record_format.md` section 1 item 2) |
+| `record.py` | the records a read resolves among, and the checks that depend on one record alone (steps 2, 3 and 5) |
+| `resolve.py` | steps 1 to 5: a material reference to the record it pins, or the reasons it is refused |
+| `read.py` | the request's own check (step 0), and the read; steps 6 to 13 are not built yet |
+
+The conformance cases HS-001 to HS-003 check the hash end to end; the plain-read cases, HS-010 and HS-011 the
+read up to step 5 (`tests/conformance/`).
+
+## Invariants
+
+- **Results, never exceptions, for content.** `parse` gives a value or `malformed` for any bytes, running out
+  of stack or memory included, and a read gives a result for any request and any records. An exception means
+  a misuse of the Python API (`canonicalize` given a value `parse` would never give; claims no reader makes, a
+  suite that does not exist or claims without plain read) or a step not built yet: a read whose answer a later
+  step would decide raises `NotImplementedError` rather than give one, whatever else is wrong with the request
+  (`tests/property/test_read_properties.py`, `tests/unit/test_read.py`).
+- **A request is JSON text or a value.** Text is a `str` or bytes-like (`bytes`, `bytearray`, `memoryview`),
+  read like a record's bytes; a `str` is always text, as a JSON string is never a request. Any other value is
+  read as the JSON text `json.dumps` writes of it.
+- **Every number is a float.** An integer reads as the nearest double; `JsonValue` has no `int`, and
+  `canonicalize` refuses one. A request given as a Python value is read through JSON text, so its integers too.
+  (The reader's properties compare types exactly; `tests/unit/test_canonical.py`.)
+- **Stored JSON is never mutated.** `hashable_view` builds new containers and leaves its argument as it was
+  (`tests/unit/test_hashing.py`); a `Record` keeps the value its bytes parse to as it is.
+- **Identifiers match in full**, by `re.fullmatch` with character classes written out
+  (`tests/property/test_version_properties.py`).
+- **One envelope check, in two forms.** `envelope.py` and `schema/0.1/record.reader.json` accept and refuse
+  the same records (`tests/spec_sync/test_envelope.py`, `tests/property/test_envelope_properties.py`).
+- **Standard library only**, with `materica.shapes`, itself standard library only, and, for type checkers
+  only, `typing_extensions`; relative imports are checked like absolute ones (`tests/unit/test_source_files.py`).

materica/core/__init__.py

@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+"""The core (layer L2, AD 7.1): Python's standard library only (AD 7.2, CONTRIBUTING constraint 3)."""

materica/core/canonical.py

@@ -0,0 +1,92 @@
+# SPDX-License-Identifier: Apache-2.0
+"""RFC 8785 canonical JSON, which the content hash is taken over (`record_format.md` section 9 item 2)."""
+
+import math
+
+from materica.core.jsonio import JsonValue
+
+_ESCAPES = {code: f"\\u{code:04x}" for code in range(0x20)} | {
+    ord('"'): '\\"',
+    ord("\\"): "\\\\",
+    ord("\b"): "\\b",
+    ord("\f"): "\\f",
+    ord("\n"): "\\n",
+    ord("\r"): "\\r",
+    ord("\t"): "\\t",
+}
+
+
+def canonicalize(value: JsonValue) -> bytes:
+    """`value`, as `parse` gives it, serialised by RFC 8785 in UTF-8.
+
+    Strings are written as they are, never Unicode-normalised. A value `parse` would never give raises rather
+    than being written: `TypeError` for an `int` or a tuple, say, and `ValueError` for a number that is not
+    finite or a lone surrogate.
+    """
+    parts: list[str] = []
+    _write(value, parts)
+    return "".join(parts).encode("utf-8")
+
+
+def _write(value: JsonValue, parts: list[str]) -> None:
+    if value is None:
+        parts.append("null")
+    elif isinstance(value, bool):
+        parts.append("true" if value else "false")
+    elif isinstance(value, float):
+        parts.append(_number(value))
+    elif isinstance(value, str):
+        parts.append(_string(value))
+    elif isinstance(value, list):
+        parts.append("[")
+        for index, item in enumerate(value):
+            if index:
+                parts.append(",")
+            _write(item, parts)
+        parts.append("]")
+    elif isinstance(value, dict):
+        parts.append("{")
+        for index, name in enumerate(sorted(value, key=_utf16_units)):
+            if index:
+                parts.append(",")
+            parts.append(_string(name))
+            parts.append(":")
+            _write(value[name], parts)
+        parts.append("}")
+    else:
+        raise TypeError(f"not a JSON value as parse gives it: {type(value).__name__}")
+
+
+def _number(number: float) -> str:
+    """ECMAScript's `Number.prototype.toString`, as RFC 8785 section 3.2.2.3 requires."""
+    if not math.isfinite(number):
+        raise ValueError(f"{number} has no JSON form")
+    if number == 0:
+        return "0"
+    # repr gives the fewest digits that read back as the same double, the nearest where several do, as
+    # ECMAScript does. Here the double is 0.<digits> times 10**point.
+    mantissa, _, exponent = repr(abs(number)).partition("e")
+    whole, _, fraction = mantissa.partition(".")
+    written = whole + fraction
+    digits = written.strip("0")
+    leading_zeros = len(written) - len(written.lstrip("0"))
+    point = len(whole) - leading_zeros + int(exponent or "0")
+    count = len(digits)
+    sign = "-" if number < 0 else ""
+    if count <= point <= 21:
+        return sign + digits + "0" * (point - count)
+    if 0 < point <= 21:
+        return sign + digits[:point] + "." + digits[point:]
+    if -6 < point <= 0:
+        return sign + "0." + "0" * -point + digits
+    fraction_part = "." + digits[1:] if count > 1 else ""
+    return f"{sign}{digits[0]}{fraction_part}e{point - 1:+d}"
+
+
+def _string(text: str) -> str:
+    return '"' + text.translate(_ESCAPES) + '"'
+
+
+def _utf16_units(name: str) -> bytes:
+    # Big-endian bytes order as the UTF-16 code units they encode.
+    return name.encode("utf-16-be")

materica/core/envelope.py

@@ -0,0 +1,93 @@
+# SPDX-License-Identifier: Apache-2.0
+"""The reader schema's checks, written by hand (`record_format.md` section 1 item 2), so that the core reads
+records with no JSON Schema validator.
+
+`schema/0.1/record.reader.json` states the same checks for other languages. Tests hold the two to agreement,
+on every conformance input and on records drawn near each boundary of the item.
+"""
+
+from materica.core.identifiers import CONTENT_HASH, RECORD_ID, VERSION, Version
+from materica.core.jsonio import JsonValue
+from materica.core.vocabulary import VOCABULARY
+
+SCHEMA_VERSION = Version("0", "1", "0")
+"""The schema version these checks are the reader schema of."""
+
+Members = dict[str, type[object]]
+
+_KINDS: dict[type[object], str] = {str: "a string", dict: "an object"}
+_ENVELOPE: Members = {
+    "schema_version": str,
+    "record_type": str,
+    "id": str,
+    "version": str,
+    "name": str,
+    "licence": dict,
+    "content_hash": str,
+    "body": dict,
+}
+_PATTERNS = {"id": RECORD_ID, "version": VERSION, "content_hash": CONTENT_HASH}
+
+
+def envelope_problems(record: JsonValue) -> list[str]:
+    """Each way `record` breaks the reader schema of schema 0.1, in words: none where it passes.
+
+    The body is checked by its `record_type`: a material's sections, as far as this version knows them, and an
+    overlay's members. A body of another type, a section of another name, `extensions`' contents, a section's
+    `specimen` and every value field are left unchecked.
+    """
+    if not isinstance(record, dict):
+        return ["the record is not an object"]
+    problems = _members(record, "", _ENVELOPE, {"card": str})
+    for name, pattern in _PATTERNS.items():
+        value = record.get(name)
+        if isinstance(value, str) and not pattern.fullmatch(value):
+            problems.append(f"{name} does not match its pattern: {value!r}")
+    licence, body = record.get("licence"), record.get("body")
+    if isinstance(licence, dict):
+        problems += _members(licence, "licence.", {"values": str, "evidence": str})
+    if isinstance(body, dict) and record.get("record_type") == "material":
+        problems += _material_body(body)
+    elif isinstance(body, dict) and record.get("record_type") == "overlay":
+        problems += _overlay_body(body)
+    return problems
+
+
+def _material_body(body: dict[str, JsonValue]) -> list[str]:
+    problems = _members(body, "body.", {"sections": dict}, {"extensions": dict})
+    sections = body.get("sections")
+    if isinstance(sections, dict):
+        for name in sorted(VOCABULARY.sections & sections.keys()):
+            section = sections[name]
+            if isinstance(section, dict):
+                problems += _members(section, f"body.sections.{name}.", {"fields": dict})
+            else:
+                problems.append(f"body.sections.{name} is not an object")
+    return problems
+
+
+def _overlay_body(body: dict[str, JsonValue]) -> list[str]:
+    problems = _members(body, "body.", {"purpose": str, "base": dict, "scope": dict, "replaces": dict})
+    base, scope = body.get("base"), body.get("scope")
+    if isinstance(base, dict):
+        problems += _members(base, "body.base.", {"id": str, "version": str, "content_hash": str})
+    if isinstance(scope, dict):
+        optional: Members = {"model": dict, "model_configuration_hash": str, "run": str}
+        problems += _members(scope, "body.scope.", {"consumer": str}, optional)
+        model = scope.get("model")
+        if isinstance(model, dict):
+            problems += _members(model, "body.scope.model.", {"name": str, "version": str})
+    return problems
+
+
+def _members(
+    value: dict[str, JsonValue], where: str, required: Members, optional: Members | None = None
+) -> list[str]:
+    problems: list[str] = []
+    for name, kind in (required | (optional or {})).items():
+        if name not in value:
+            if name in required:
+                problems.append(f"{where}{name} is missing")
+        elif not isinstance(value[name], kind):
+            problems.append(f"{where}{name} is not {_KINDS[kind]}")
+    return problems

materica/core/hashing.py

@@ -0,0 +1,38 @@
+# SPDX-License-Identifier: Apache-2.0
+"""The content hash of a record, or of any JSON value (`record_format.md` section 9)."""
+
+import hashlib
+
+from materica.core.canonical import canonicalize
+from materica.core.jsonio import JsonValue
+
+
+def hashable_view(value: JsonValue) -> JsonValue:
+    """`value` without the members `record_format.md` section 9 item 1 removes; `value` is left unchanged.
+
+    The top-level `content_hash` and `card` go only when `value` is an object. `review` and `copy` go at any
+    depth inside an object that is the value of a member named `provenance`, through objects and arrays.
+    """
+    if isinstance(value, dict):
+        value = {name: member for name, member in value.items() if name not in ("content_hash", "card")}
+    return _view(value, inside_provenance=False)
+
+
+def _view(value: JsonValue, *, inside_provenance: bool) -> JsonValue:
+    if isinstance(value, dict):
+        return {
+            name: _view(
+                member,
+                inside_provenance=inside_provenance or (name == "provenance" and isinstance(member, dict)),
+            )
+            for name, member in value.items()
+            if not (inside_provenance and name in ("review", "copy"))
+        }
+    if isinstance(value, list):
+        return [_view(item, inside_provenance=inside_provenance) for item in value]
+    return value
+
+
+def content_hash(value: JsonValue) -> str:
+    """`sha256:` and the lowercase hexadecimal SHA-256 of `value`'s hashable view in RFC 8785 form."""
+    return "sha256:" + hashlib.sha256(canonicalize(hashable_view(value))).hexdigest()

materica/core/identifiers.py

@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Identifiers and versions (`record_format.md` sections 7 and 8), and material references.
+
+Every pattern here is matched in full, with `re.fullmatch`, never searched: `$` also matches before a final
+newline. Character classes are written out, since `\\d` would match any Unicode digit.
+"""
+
+import functools
+import re
+from dataclasses import dataclass
+
+from materica import shapes
+
+NAMESPACE = re.compile(r"[a-z0-9_]+")
+RECORD_ID = re.compile(r"[a-z0-9_]+/[A-Za-z0-9_.-]+")
+FAMILY_ID = re.compile(r"[a-z0-9_]+(?:\.[a-z0-9_]+)+\.v[1-9][0-9]*")
+VERSION = re.compile(r"(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)")
+_COMPONENT = re.compile(r"0|[1-9][0-9]*")
+CONTENT_HASH = re.compile(r"sha256:[0-9a-f]{64}")
+# A request's field, `<section>.<field>` (`read_side_contract.md` section 1).
+FIELD = re.compile(r"[a-z0-9_]+\.[a-z0-9_]+")
+
+
+@functools.total_ordering
+@dataclass(frozen=True)
+class Version:
+    """`MAJOR.MINOR.PATCH`, its components kept as their digits.
+
+    Section 8 compares components as numbers. With no leading zeros, which construction refuses, comparing
+    their lengths and then their digits gives that order for any length, where `int` refuses more than 4300
+    digits.
+    """
+
+    major: str
+    minor: str
+    patch: str
+
+    def __post_init__(self) -> None:
+        for component in (self.major, self.minor, self.patch):
+            if not _COMPONENT.fullmatch(component):
+                raise ValueError(f"not a version component: {component!r}")
+
+    def __lt__(self, other: object) -> bool:
+        if not isinstance(other, Version):
+            return NotImplemented
+        return self._key() < other._key()
+
+    def _key(self) -> tuple[tuple[int, str], ...]:
+        return tuple((len(component), component) for component in (self.major, self.minor, self.patch))
+
+
+def parse_version(text: str) -> Version | None:
+    """The version `text` writes, or None where it does not match section 8's pattern in full."""
+    matched = VERSION.fullmatch(text)
+    return None if matched is None else Version(matched[1], matched[2], matched[3])
+
+
+@dataclass(frozen=True)
+class MaterialRef:
+    """A material reference: a record's `id` and `version`, and its `content_hash` as the pin."""
+
+    id: str
+    version: str
+    content_hash: str
+
+    def to_json(self) -> shapes.MaterialRef:
+        return {"id": self.id, "version": self.version, "content_hash": self.content_hash}

materica/core/jsonio.py

@@ -0,0 +1,102 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Strict JSON: bytes to a value, or the reason `malformed` (`record_format.md` section 1 item 6).
+
+The item applies to records, scene documents, engine crosswalks, read requests and conformance case files.
+"""
+
+import codecs
+import json
+import math
+import re
+from collections.abc import Iterator
+from typing import NoReturn, TypeAlias
+
+from materica.core.reasons import Reason
+
+JsonValue: TypeAlias = bool | float | str | list["JsonValue"] | dict[str, "JsonValue"] | None
+"""A value as `parse` gives it: every number is a float."""
+
+MAX_DEPTH = 64
+
+# A whole string, so that brackets inside one are skipped, or a bracket.
+_STRING_OR_BRACKET = re.compile(r'"[^"\\]*(?:\\.[^"\\]*)*"|[][{}]', re.DOTALL)
+_SURROGATE = re.compile("[\ud800-\udfff]")
+
+
+def parse(data: bytes) -> JsonValue | Reason:
+    """Read `data` by `record_format.md` section 1 item 6.
+
+    Gives the value, or `Reason("malformed")` with a detail for anything that item refuses. Never raises for
+    any bytes.
+    """
+    try:
+        return _parse(data)
+    except (ValueError, RecursionError, MemoryError) as error:
+        return Reason("malformed", detail=str(error) or type(error).__name__)
+
+
+def _parse(data: bytes) -> JsonValue:
+    if data.startswith(codecs.BOM_UTF8):
+        raise ValueError("a byte-order mark")
+    text = data.decode("utf-8")
+    # Counted before json runs, so that json never recurses deeper than this: 100000 levels would exhaust
+    # its stack.
+    if _nesting_exceeds(text, MAX_DEPTH):
+        raise ValueError(f"nested more than {MAX_DEPTH} deep")
+    value: JsonValue = json.loads(
+        text, object_pairs_hook=_object, parse_float=_double, parse_int=_double, parse_constant=_constant
+    )
+    # json joins a paired surrogate escape into one character and keeps an unpaired one; the strict decoding
+    # above has refused any surrogate written as bytes.
+    if any(_SURROGATE.search(string) for string in _strings(value)):
+        raise ValueError("an unpaired surrogate escape")
+    return value
+
+
+def _nesting_exceeds(text: str, limit: int) -> bool:
+    depth = 0
+    for token in _STRING_OR_BRACKET.finditer(text):
+        if token.group() in ("[", "{"):
+            depth += 1
+            if depth > limit:
+                return True
+        elif token.group() in ("]", "}"):
+            depth -= 1
+    return False
+
+
+def _object(pairs: list[tuple[str, JsonValue]]) -> dict[str, JsonValue]:
+    members: dict[str, JsonValue] = {}
+    for name, value in pairs:
+        if name in members:
+            raise ValueError(f"the member name {name!r} repeated")
+        members[name] = value
+    return members
+
+
+def _double(text: str) -> float:
+    # Integers come here too, as text: int() refuses more than 4300 digits, and float(int) overflows where
+    # float(text) rounds. json's pure-Python scanner, used where its C one is missing, matches any Unicode
+    # digit as a digit.
+    if not text.isascii():
+        raise ValueError(f"not a JSON number: {text!r}")
+    number = float(text)
+    if math.isinf(number):
+        raise ValueError("a number that rounds to an infinity")
+    return number
+
+
+def _constant(name: str) -> NoReturn:
+    raise ValueError(f"{name} is not JSON")
+
+
+def _strings(value: JsonValue) -> Iterator[str]:
+    if isinstance(value, str):
+        yield value
+    elif isinstance(value, list):
+        for item in value:
+            yield from _strings(item)
+    elif isinstance(value, dict):
+        for name, member in value.items():
+            yield name
+            yield from _strings(member)