protolib 0.2.2__tar.gz

protolib-0.2.2/PKG-INFO

@@ -0,0 +1,593 @@
+Metadata-Version: 2.4
+Name: protolib
+Version: 0.2.2
+Summary: Pure-Python, from-scratch declarative binary protocol (de)serializer — node-protodef style, for Minecraft, ClassiCube and any other binary protocol.
+Author: R
+License: MIT
+Keywords: protocol,binary,serialization,minecraft,classicube,protodef,networking,packet
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Networking
+Classifier: Topic :: Games/Entertainment
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: yaml
+Requires-Dist: pyyaml>=6.0; extra == "yaml"
+
+# protolib (Python)
+
+A from-scratch, pure-Python implementation of a
+[node-protodef](https://github.com/ProtoDef-io/ProtoDef)-style system:
+you describe the shape of a binary protocol's packets in a declarative
+file (`.yml` or `.json`), and the library takes care of parsing raw
+bytes into a Python `dict` and serializing a `dict` back into bytes —
+without you having to hand-write a parser for every single packet.
+
+Already tested against two real protocols, each with its own `.yml`
+(hand-written/maintained) and its equivalent `.json` (the native
+`["type", opts]` form, generated by the loader itself — so you can see
+exactly what the engine understands internally, and it's useful for
+consuming the protocol from something that doesn't want to parse YAML):
+- **Minecraft Java 1.8.9** (`examples/example_protocol.yml` /
+  `.json`) — advanced patterns: parametrizable switch, relative
+  `compareTo`, bitfield.
+- **Minecraft Classic / ClassiCube 0x07 + CPE**
+  (`examples/classicube_protocol.yml` / `.json`) — a complete
+  fixed-size protocol, no varints, with CPE extensions.
+
+---
+
+## 1. The idea in 30 seconds
+
+```python
+from protolib.core import Protocol
+
+proto = Protocol("my_protocol.yml")
+
+# bytes -> dict
+parsed = proto.parse_packet("play", "toClient", raw_data)
+print(parsed.name, parsed.params, parsed.bytes_read)
+
+# dict -> bytes
+data = proto.serialize_packet("play", "toClient", "spawn_player", {
+    "playerId": 5, "x": 320, "y": 2080, "z": 320,
+})
+```
+
+Everything else in this document is about how to declare
+`my_protocol.yml` so that `parse_packet`/`serialize_packet` know which
+fields to expect.
+
+---
+
+## 2. Install / use
+
+No dependencies beyond `pyyaml` (for the `.yml` format). Used as a
+local package, no need to install it:
+
+```python
+import sys
+sys.path.insert(0, "path/to/the/folder/containing/protolib")
+from protolib.core import Protocol
+```
+
+`Protocol(...)` accepts:
+- a path to a `.json`, `.yml`, or `.yaml` file
+- an in-memory string with JSON or YAML content
+- an already-parsed `dict` (the "native" node-protodef format)
+
+---
+
+## 3. Structure of a protocol file
+
+```yaml
+types:
+  # types valid across the ENTIRE protocol, regardless of state/direction
+  varint: native
+  myCustomType:
+    container:
+    - name: x
+      type: varint
+
+<state_name>:        # e.g.: "play", "login", "handshaking"...
+  toClient:
+    types:
+      # types that only exist for this direction/state, take
+      # PRIORITY over the ones above if there's a name collision
+      packet: ...
+  toServer:
+    types:
+      packet: ...
+```
+
+- **global `types:`** — shared types (primitives, common structs).
+- **`<state>.toClient.types` / `<state>.toServer.types`** — the real
+  entry point to parse a packet is always a type named `packet` inside
+  one of these blocks. `parse_packet(state, direction, data)` looks for
+  that `packet`, reads it, and returns `{name, params}` based on how
+  you built it (see the "packet container with mapper+switch" pattern
+  in section 7).
+
+> **Important:** `Protocol.parse_packet`/`serialize_packet` expect the
+> `packet` type to ALWAYS have exactly two fields named `name` and
+> `params` — that's what builds the `ParsedPacket(name=...,
+> params=...)` you get back. The standard pattern (used in both
+> example protocols) is:
+>
+> ```yaml
+> packet:
+>   container:
+>   - name: name
+>     type:
+>       mapper:               # wire integer -> packet name
+>         type: u8            # (or varint, depending on your protocol)
+>         mappings:
+>           '0x00': identification
+>   - name: params
+>     type:
+>       switch:                # packet name -> its payload type
+>         compareTo: name
+>         fields:
+>           identification: packet_identification
+> ```
+>
+> If your `packet` doesn't follow this exact shape, `parse_packet` will
+> fail with `KeyError: 'name'` — it's not a weird limitation of your
+> protocol, it's literally how this library knows what to return.
+
+`state` and `direction` don't have to be called that — they're simply
+the two keys you'll later use to call
+`parse_packet`/`serialize_packet`. A protocol without real "states"
+(like ClassiCube) still declares a single `play:` just to have
+somewhere to hang `toClient`/`toServer`.
+
+---
+
+## 4. The native `.json` format (without the YAML shorthand)
+
+Everything in section 3 can be written directly in JSON, without going
+through YAML — it's the format already used by `minecraft-data` and
+the original `node-protodef`, and this library understands it with no
+translation needed. The difference with the `.yml` shorthand is one of
+**shape**, not semantics: each composite type (`container`, `switch`,
+`array`, `bitfield`, etc.) is written as a 2-element list instead of a
+single-key mapping:
+
+```json
+["<base_type_name>", <options>]
+```
+
+The same `switch` from section 6, in YAML shorthand:
+
+```yaml
+type:
+  switch:
+    compareTo: name
+    fields:
+      identification: packet_identification
+```
+
+...is this in native JSON:
+
+```json
+{
+  "type": [
+    "switch",
+    {
+      "compareTo": "name",
+      "fields": { "identification": "packet_identification" }
+    }
+  ]
+}
+```
+
+And a `container` (which carries a list of fields instead of an
+options-dict) follows the same `["container", [...]]` pattern:
+
+```json
+"packet_identification": [
+  "container",
+  [
+    { "name": "protocolVersion", "type": "u8" },
+    { "name": "name", "type": "string64" }
+  ]
+]
+```
+
+**Why would you still want to write `.yml`?** Because by hand, several
+levels of nested 2-element lists are error-prone and hard to read (does
+that `]` close the `container` or the outer `switch`?). The loader
+(`protolib/loader.py`) automatically translates the `.yml` shorthand
+into this native form before `Protocol` uses it — that's why
+`example_protocol.json` and `example_protocol.yml` (or
+`classicube_protocol.json`/`.yml`) are **exactly equivalent**: they
+serialize and parse byte-for-byte the same, one is just more
+convenient to write/maintain by hand than the other.
+
+**So when should you use the raw `.json` directly?**
+- You already have a protocol from `minecraft-data` or another
+  `node-protodef` project in JSON and want to reuse it as-is, without
+  rewriting it.
+- You want to generate the protocol programmatically from another
+  language or script that has no business knowing the shorthand
+  syntax.
+- You want to see "what the engine actually understands internally"
+  with no ambiguity — useful for debugging a `.yml` that isn't loading
+  the way you expected: generate its `.json` (see below) and check if
+  the translation came out as you thought.
+
+**Generating `.json` from an existing `.yml`**, using the loader
+itself (this guarantees it's faithful to what `Protocol` will actually
+read, not a separately hand-made translation):
+
+```python
+import json
+from protolib.loader import load_protocol_dict
+
+d = load_protocol_dict("my_protocol.yml")
+with open("my_protocol.json", "w", encoding="utf-8") as f:
+    json.dump(d, f, indent=2, ensure_ascii=False)
+```
+
+This is how `examples/classicube_protocol.json` was generated, and
+`examples/example_protocol.json` already existed in this project.
+
+---
+
+## 5. `native`: how primitive types get registered
+
+`native` in the `.yml` means "don't define this here, look it up by
+name in `protolib/primitives.py`". Already registered:
+
+| Name        | Size          | Notes |
+|---|---|---|
+| `u8`/`i8`   | 1 byte        | unsigned/signed |
+| `u16`/`i16` | 2 bytes BE    | |
+| `u32`/`i32` | 4 bytes BE    | |
+| `u64`/`i64` | 8 bytes BE    | |
+| `f32`/`f64` | 4/8 bytes BE  | float/double |
+| `lu16`…`lf64` | same, little-endian | `l` prefix |
+| `varint`/`varlong` | variable | signed zigzag LEB128 — no, see `zigzag32`/`zigzag64` for those |
+| `bool`      | 1 byte        | |
+| `void`      | 0 bytes       | useful as an empty case in a `switch` |
+| `cstring`   | variable      | `\0`-terminated |
+| `UUID`      | 16 bytes      | |
+| `nbt`/`optionalNbt` | variable | Minecraft's NBT format |
+| `string64`  | 64 fixed bytes | CP437, space-padded (ClassiCube) |
+| `buffer1024`| 1024 fixed bytes | raw, `0x00`-padded (ClassiCube) |
+| `fixedCoord`/`fixedCoordDelta` | 2/1 bytes | fixed-point *32 (ClassiCube) |
+
+**If your protocol needs a new type that isn't a composition of the
+ones above** (say, a fixed-length string with a different encoding, or
+a buffer of another fixed size), add it in Python following the
+pattern of `make_fixed_cp437_string`/`make_fixed_buffer` in
+`primitives.py`, and register it in the `PRIMITIVES` dict at the end
+of the file. After that it's available as `native` in any `.yml`, for
+you and for anyone else using this library.
+
+Non-native types (`pstring`, `container`, `switch`, `array`,
+`bitfield`, `bitflags`, `buffer`, `mapper`, `option`, `count`) are
+resolved by the Python engine (`core.py`) according to their own logic
+— explained below with examples.
+
+---
+
+## 6. Composite types, one by one
+
+### `container` — groups named fields, in order
+
+```yaml
+packet_login:
+  container:
+  - name: protocolVersion
+    type: varint
+  - name: username
+    type: string
+```
+
+Reads as `{"protocolVersion": ..., "username": ...}`.
+
+**`anon: true` field**: instead of being stored under its own name, its
+sub-fields get merged directly into the parent container's `dict`.
+Useful for "breaking up" a bitfield or a switch into several loose
+fields at the same level (see bitfield below).
+
+**`condition` field**: the field is only read/written if the condition
+(an expression over already-read fields) is true. Syntax: same path
+language as `compareTo` (section 7).
+
+### `array` — list of N elements of the same type
+
+```yaml
+type:
+  array:
+    countType: varint    # reads a varint first, that's the length
+    # count: otherField  # alternative: uses the VALUE of another already-read field
+    type: myItemType
+```
+
+`countType` prepends an integer that's read/written automatically.
+`count`, on the other hand, references a sibling field already present
+(typically built with the `count` type, see section 9) — it doesn't
+write anything new, it just reads that existing value as the array's
+length.
+
+### `switch` — picks the field's type based on another field's value
+
+```yaml
+type:
+  switch:
+    compareTo: name        # path to an already-read field (see section 7)
+    fields:
+      valueA: typeForA
+      valueB: typeForB
+    default: void           # optional, if nothing matches
+```
+
+This is the central piece for "the `params` field depends on what the
+`name` field says" — the pattern of every network packet with an
+id/name.
+
+### `mapper` — integer ⟷ symbolic name
+
+```yaml
+type:
+  mapper:
+    type: varint
+    mappings:
+      '0x00': handshake
+      '0x01': status_request
+```
+
+When reading, a `0x00` in the stream becomes the string `"handshake"`
+in the resulting `dict` (and vice versa when writing). This way the
+`switch` above can compare against readable names (`compareTo: name`)
+instead of the raw packet id integer.
+
+### `bitfield` — several fields packed into less than 1 byte each
+
+```yaml
+type:
+  bitfield:
+  - name: type
+    size: 3
+    signed: false
+  - name: index
+    size: 5
+    signed: false
+```
+
+Declares sub-fields that each take up `size` bits, MSB first, within
+the minimum necessary bytes (if they don't fill an exact byte, padding
+is added). Almost always combined with `anon: true` on the container
+that holds it, so that `type`/`index` end up as loose fields instead of
+nested under an extra name.
+
+### `bitflags` — an integer as a named set of booleans
+
+```yaml
+type:
+  bitflags:
+    type: u8
+    flags: [air, water, lava, null, fire]   # position = bit number
+    # or, as a dict: {air: 0x01, water: 0x02, ...}
+```
+
+Returns `{"air": true, "water": false, ..., "_value": <raw integer>}`.
+
+### `buffer` — raw bytes (uninterpreted)
+
+```yaml
+type:
+  buffer:
+    count: 1024        # fixed size, or a reference to a field
+    # countType: varint  # alternative: prefixed length
+    # rest: true         # alternative: "whatever's left in the frame"
+```
+
+### `option` — present only if a preceding boolean flag says so
+
+```yaml
+type:
+  option: myType
+```
+
+Reads a `bool` first; if `true`, reads `myType` right after; if
+`false`, the value is `None` and nothing else is read.
+
+### `pstring` — string with a prefixed length
+
+```yaml
+type:
+  pstring:
+    countType: varint     # how many length-bytes come before it
+    # encoding: utf-8      # default
+```
+
+The `string:` shortcut already ships built with this
+(`countType: varint`, used in Minecraft Java). For ClassiCube,
+`string64` is used instead (ALWAYS fixed size, no prefix — see
+section 5), since that protocol doesn't prefix a length.
+
+---
+
+## 7. `compareTo` and field paths (`../`, `/`)
+
+`compareTo` (in a `switch`) and `condition` (on a `container` field)
+use the same tiny path language to refer to "a field that's already
+been read":
+
+| Syntax | Means |
+|---|---|
+| `fieldName` | sibling field, same container |
+| `../fieldName` | goes up one level: the container that holds this one |
+| `../../fieldName` | goes up two levels |
+| `/fieldName` | absolute, from the root of the whole packet |
+
+**Important gotcha, already documented with a full example in
+`examples/README.md`:** `../` only goes up one real level when the
+container is written **inline** (literally inside `array.type` or
+another container). If instead you reference a type **by name**
+(`type: myNamedType`), that named type always adds its own nesting
+level — so a `../field` inside it reaches, at most, the named
+container itself, not the real parent further up. If your switch needs
+to see a field higher up in the hierarchy, declare the container
+inline at the point of use instead of moving it into a separately
+named type.
+
+---
+
+## 8. Parametrizable types (`$arg`)
+
+A named type can declare a `$`-prefixed placeholder instead of a fixed
+value, and whoever uses it decides the real value at that point:
+
+```yaml
+types:
+  itemByType:
+    switch:
+      compareTo: $compareTo    # placeholder -- the name after
+      fields:                 # the $ is arbitrary, you choose it
+        '0': i8
+        '1': varint
+
+  myContainer:
+    container:
+    - name: type
+      type: u8
+    - name: value
+      type:
+      - itemByType             # referenced as [name, options]
+      - compareTo: type          # here the real $compareTo gets resolved
+```
+
+This defines a "generic" type once and reuses it with a different
+comparison field depending on context — it's exactly how
+`minecraft-data` defines `entityMetadataItem`.
+
+---
+
+## 9. `count` — a length-prefix declared apart from the array/buffer
+
+For the (rare, but real in some protocols) case where an array's
+length prefix isn't attached to that array but is instead its own
+field elsewhere in the container:
+
+```yaml
+container:
+- name: itemCount
+  type:
+    count:
+      type: varint
+      countFor: items      # when WRITING, ignores the given value and
+- name: items               # writes len(items) automatically
+  type:
+    array:
+      count: itemCount   # when READING, references that already-read field
+      type: myItem
+```
+
+---
+
+## 10. The 4 advanced Minecraft Java patterns
+
+Covered with working, commented examples in
+[`examples/README.md`](examples/README.md) +
+[`examples/example_protocol.yml`](examples/example_protocol.yml)
+(its `.json` equivalent, already translated to the native `["type",
+opts]` form, lives in
+[`examples/example_protocol.json`](examples/example_protocol.json)):
+
+1. Parametrizable named type with `$arg` (section 8 of this doc).
+2. `compareTo: ../field` and the inline-vs-named container gotcha
+   (section 7).
+3. Array of containers with a `switch` inside that looks at a field of
+   the array's parent container.
+4. `bitfield` packed into 1 byte + `switch` that uses that field to
+   pick the type of another sibling field (real pattern from
+   `entity_metadata` in Minecraft 1.8.9-1.16).
+
+Run `python examples/demo.py` to see them in action.
+
+---
+
+## 11. Full reference protocol: ClassiCube
+
+[`examples/classicube_protocol.yml`](examples/classicube_protocol.yml)
+(equivalent in
+[`examples/classicube_protocol.json`](examples/classicube_protocol.json),
+byte-for-byte identical when serializing/parsing — generated by the
+loader itself, not hand-written)
+has the complete 0x07 protocol (Identification, Level Init/Data/
+Finalize, Set Block, Spawn/Position/Despawn Player, Message,
+Disconnect, Update User Type) plus several CPE extensions (ExtInfo/
+ExtEntry, ClickDistance, CustomBlockSupportLevel, HeldBlock,
+ExtPlayerList) — a good example of a protocol **with no varints**, all
+fixed size, useful if your use case doesn't look like Minecraft Java's.
+
+`examples/servidor.py` is a real, minimal server (login + generates a
+world and sends it) built entirely on top of `Protocol.parse_packet` /
+`serialize_packet` — it's an example of how to integrate this library
+with real sockets (a `recv()` loop accumulating bytes until
+`parse_packet` stops throwing `BufferUnderrun`).
+
+```bash
+cd examples
+python servidor.py               # caves, port 25565
+python servidor.py --modo llano  # flat world
+```
+
+---
+
+## 12. Errors and what they mean
+
+All of them live in `protolib/errors.py`:
+
+- **`UnknownTypeError`** — a type name isn't in `native`
+  (`primitives.py`) nor defined in `types:` (global or scoped). Usually
+  a typo, or you forgot to declare it.
+- **`SwitchCaseNotFound`** — a `switch` has no `fields` entry for the
+  value `compareTo` produced, and there's no `default` either. Check
+  that the comparison value resolves to what you expect (does the
+  `mapper` building that field convert it to a string? does
+  `compareTo` point to the right field using the syntax from section
+  7?).
+- **`InvalidTypeDefinition`** — a composite type is missing a required
+  option (`array` without `count`/`countType`, `buffer` without
+  `count`/`countType`/`rest`, etc.).
+- **`BufferUnderrun`** (in `protolib/io.py`, not `errors.py`) — an
+  attempt was made to read more bytes than are available. In a real
+  server this is NOT necessarily a protocol error: it means "the full
+  packet hasn't arrived on the socket yet," and the correct pattern is
+  to catch it and wait for more bytes (see the loop in `servidor.py`).
+
+## Changelog
+
+### 0.2.1
+- New primitive `buffer64`: fixed 64-byte raw bytes with automatic
+  `\x00` padding (same semantics as `buffer1024`, built with the same
+  `make_fixed_buffer` mold). Added while migrating `PocketNet` to a
+  single `protocol.yml` — the `data` field of `PluginMessagePacket`
+  (0x35) needed exactly this behavior, and the generic `buffer` with a
+  fixed `count` is strict (fails if the value isn't exactly N bytes
+  long instead of truncating/padding).
+- New example: `examples/pocketnet_protocol.yml`, the complete
+  Minecraft Classic 0.30 + CPE protocol (55 packets) migrated from the
+  "one class per packet" style into a single declarative file. Tested
+  with byte-exact round-trips against the original `encode()`/
+  `decode()` on the most complex cases (packed bitfields, nested
+  arrays of containers, padded buffers).
+
+### 0.2.0
+- Base version: primitives, composites (`container`, `array`,
+  `switch`, `mapper`, `option`, `bitfield`, `bitflags`, `buffer`,
+  `pstring`, `count`, `entityMetadataLoop`,
+  `topBitSetTerminatedArray`), JSON and YAML support, native NBT and
+  UUID.