nemtus-catparser 3.2.0__tar.gz

nemtus_catparser-3.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

nemtus_catparser-3.2.0/PKG-INFO

@@ -0,0 +1,88 @@
+Metadata-Version: 2.4
+Name: nemtus-catparser
+Version: 3.2.0
+Summary: Symbol Catbuffer Parser
+License: MIT
+License-File: LICENSE
+Keywords: symbol,catbuffer,catparser,parser,catbuffer-parser
+Author: Symbol Contributors
+Author-email: contributors@symbol.dev
+Maintainer: Symbol Contributors
+Maintainer-email: contributors@symbol.dev
+Requires-Python: >=3.10,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Project-URL: Repository, https://github.com/nemtus/symbol/tree/dev/catbuffer/parser
+Description-Content-Type: text/markdown
+
+<!-- nemtus-mirror-notice -->
+> **Note (nemtus mirror):** This package is a content mirror of the catbuffer
+> parser from [`symbol/symbol`](https://github.com/symbol/symbol)
+> (`catbuffer/parser`, upstream PyPI name `catparser`). nemtus republishes it on
+> PyPI as [`nemtus-catparser`](https://pypi.org/project/nemtus-catparser/); the
+> only change from upstream is the published distribution name. The import module
+> name is unchanged — you still `import catparser`. For the canonical project, see
+> [symbol/symbol](https://github.com/symbol/symbol).
+<!-- /nemtus-mirror-notice -->
+
+# catbuffer-parser
+
+[![Build Status](https://api.travis-ci.com/symbol/catbuffer-parser.svg?branch=main)](https://travis-ci.com/symbol/catbuffer-parser)
+
+This project is the reference parser implementation for the CATS (Compact Affinitized Transfer Schema) DSL that is used by the Symbol blockchain. The parser converts a CATS file or files into an AST that can be used by a generator to produce serialization and deserialization code for defined objects.
+
+- A reference to the DSL format can be found [here](docs/cats_dsl.md).
+- For a sampling of real world schemas, those used by the Symbol blockchain can be found [here](https://github.com/symbol/catbuffer-schemas).
+
+## Requirements
+
+* Python >= 3.7
+
+## Installation
+
+```bash
+git clone https://github.com/symbol/catbuffer-parser
+pip3 install -r requirements.txt
+pip3 install -r lint_requirements.txt # optional
+```
+
+## Usage
+
+```
+usage: python -m catparser [-h] -s SCHEMA -i INCLUDE [-o OUTPUT]
+
+CATS code generator
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -s SCHEMA, --schema SCHEMA
+                        input CATS file
+  -i INCLUDE, --include INCLUDE
+                        schema root directory
+  -o OUTPUT, --output OUTPUT
+                        yaml output file
+  -g GENERATOR, --generator GENERATOR
+                        generator class to use to produce output files (defaults to YAML output)
+  -q, --quiet           do not print type descriptors to console
+```
+
+## Examples
+
+``catparser`` can be used on its own to parse input files, check their validity and optionally output a YAML file containing the parsed type descriptors:
+
+```bash
+# parse but don't output anything
+python3 -m catparser --schema ../schemas/symbol/transfer/transfer.cats --include ../schemas/symbol --quiet
+
+# parse and output the AST
+python3 -m catparser --schema ../schemas/symbol/transfer/transfer.cats --include ../schemas/symbol
+
+# parse and generate code using the specified generator (`generator.Generator`)
+python3 -m catparser --schema ../schemas/symbol/transfer/transfer.cats --include ../schemas/symbol --generator generator.Generator
+```
+

nemtus_catparser-3.2.0/README.md

@@ -0,0 +1,65 @@
+<!-- nemtus-mirror-notice -->
+> **Note (nemtus mirror):** This package is a content mirror of the catbuffer
+> parser from [`symbol/symbol`](https://github.com/symbol/symbol)
+> (`catbuffer/parser`, upstream PyPI name `catparser`). nemtus republishes it on
+> PyPI as [`nemtus-catparser`](https://pypi.org/project/nemtus-catparser/); the
+> only change from upstream is the published distribution name. The import module
+> name is unchanged — you still `import catparser`. For the canonical project, see
+> [symbol/symbol](https://github.com/symbol/symbol).
+<!-- /nemtus-mirror-notice -->
+
+# catbuffer-parser
+
+[![Build Status](https://api.travis-ci.com/symbol/catbuffer-parser.svg?branch=main)](https://travis-ci.com/symbol/catbuffer-parser)
+
+This project is the reference parser implementation for the CATS (Compact Affinitized Transfer Schema) DSL that is used by the Symbol blockchain. The parser converts a CATS file or files into an AST that can be used by a generator to produce serialization and deserialization code for defined objects.
+
+- A reference to the DSL format can be found [here](docs/cats_dsl.md).
+- For a sampling of real world schemas, those used by the Symbol blockchain can be found [here](https://github.com/symbol/catbuffer-schemas).
+
+## Requirements
+
+* Python >= 3.7
+
+## Installation
+
+```bash
+git clone https://github.com/symbol/catbuffer-parser
+pip3 install -r requirements.txt
+pip3 install -r lint_requirements.txt # optional
+```
+
+## Usage
+
+```
+usage: python -m catparser [-h] -s SCHEMA -i INCLUDE [-o OUTPUT]
+
+CATS code generator
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -s SCHEMA, --schema SCHEMA
+                        input CATS file
+  -i INCLUDE, --include INCLUDE
+                        schema root directory
+  -o OUTPUT, --output OUTPUT
+                        yaml output file
+  -g GENERATOR, --generator GENERATOR
+                        generator class to use to produce output files (defaults to YAML output)
+  -q, --quiet           do not print type descriptors to console
+```
+
+## Examples
+
+``catparser`` can be used on its own to parse input files, check their validity and optionally output a YAML file containing the parsed type descriptors:
+
+```bash
+# parse but don't output anything
+python3 -m catparser --schema ../schemas/symbol/transfer/transfer.cats --include ../schemas/symbol --quiet
+
+# parse and output the AST
+python3 -m catparser --schema ../schemas/symbol/transfer/transfer.cats --include ../schemas/symbol
+
+# parse and generate code using the specified generator (`generator.Generator`)
+python3 -m catparser --schema ../schemas/symbol/transfer/transfer.cats --include ../schemas/symbol --generator generator.Generator
+```

nemtus_catparser-3.2.0/catparser/AstPostProcessor.py

@@ -0,0 +1,97 @@
+from .ast import AstException, Struct, StructInlinePlaceholder
+
+
+class AstPostProcessor:
+	"""Post processes AST type descriptors."""
+
+	def __init__(self, type_descriptors):
+		self.raw_type_descriptors = type_descriptors
+		self.type_descriptor_map = {model.name: model for model in self.raw_type_descriptors}
+
+	@property
+	def type_descriptors(self):
+		# filter out inline structs
+		return [model for model in self.raw_type_descriptors if not hasattr(model, 'disposition') or 'inline' != model.disposition]
+
+	def apply_attributes(self):
+		"""Sets properties from attributes within all structures."""
+		for model in self._structs():
+			for field in model.fields:
+				if not hasattr(field, 'attributes') or not field.attributes:
+					continue
+
+				for attribute in field.attributes:
+					if not hasattr(field.field_type, attribute.name):
+						raise AstException(f'field {field.name} ({field.field_type}) does not have property {attribute.name}')
+
+					setattr(field.field_type, attribute.name, attribute.values)
+
+	def _structs(self):
+		return [model for _, model in self.type_descriptor_map.items() if isinstance(model, Struct)]
+
+	def expand_named_inlines(self):
+		"""Expands named inline fields within all structures."""
+
+		for model in self._structs_with_named_inlines():
+			original_fields = model.fields[:]
+			model.fields = []
+
+			for field in original_fields:
+				if not self._is_named_inline(field):
+					model.fields.append(field)
+					continue
+
+				if field.field_type not in self.type_descriptor_map:
+					raise AstException(f'struct {model.name} contains named inline of unknown type {field.field_type}')
+
+				referenced_type_model = self.type_descriptor_map[field.field_type]
+				model.fields.extend(referenced_type_model.apply_inline_template(field))
+
+	def _structs_with_named_inlines(self):
+		return [
+			model for _, model in self.type_descriptor_map.items()
+			if isinstance(model, Struct) and any(self._is_named_inline(field) for field in model.fields)
+		]
+
+	@staticmethod
+	def _is_named_inline(field):
+		return hasattr(field, 'disposition') and 'inline' == field.disposition and hasattr(field, 'name')
+
+	def expand_unnamed_inlines(self):
+		"""Expands unnamed inline fields within all structures."""
+
+		for model in self._structs_with_unnamed_inlines():
+			while self._has_unnamed_inline_field(model):
+				original_fields = model.fields[:]
+				model.fields = []
+
+				for field in original_fields:
+					if not isinstance(field, StructInlinePlaceholder):
+						model.fields.append(field)
+						continue
+
+					if field.inlined_typename not in self.type_descriptor_map:
+						raise AstException(f'struct {model.name} contains unnamed inline of unknown type {field.inlined_typename}')
+
+					referenced_type_model = self.type_descriptor_map[field.inlined_typename]
+					if 'abstract' == referenced_type_model.disposition:
+						model.factory_type = referenced_type_model.name
+					elif referenced_type_model.factory_type:
+						model.factory_type = referenced_type_model.factory_type
+
+					model.fields.extend(referenced_type_model.fields)
+					if referenced_type_model.attributes:
+						if not model.attributes:
+							model.attributes = []
+
+						model.attributes.extend(referenced_type_model.attributes)
+
+	def _structs_with_unnamed_inlines(self):
+		return [
+			model for _, model in self.type_descriptor_map.items()
+			if self._has_unnamed_inline_field(model)
+		]
+
+	@staticmethod
+	def _has_unnamed_inline_field(model):
+		return isinstance(model, Struct) and any(isinstance(field, StructInlinePlaceholder) for field in model.fields)

nemtus_catparser-3.2.0/catparser/AstValidator.py

@@ -0,0 +1,243 @@
+from enum import Enum
+
+from .ast import Array, Conditional, Enum, FixedSizeInteger, Struct, StructInlinePlaceholder
+
+
+class ErrorDescriptor:
+	"""Describes an AST validation error."""
+
+	def __init__(self, message, typename, field_names=None):
+		self.message = message
+		self.typename = typename
+		self.field_names = [field_names] if isinstance(field_names, str) else field_names
+
+	def __eq__(self, rhs):
+		return isinstance(rhs, ErrorDescriptor) and str(self) == str(rhs)
+
+	def __repr__(self):
+		if self.field_names:
+			joined_field_names = ', '.join(self.field_names)
+			return f'[{self.typename}::{{ {joined_field_names} }}] {self.message}'
+
+		return f'[{self.typename}] {self.message}'
+
+
+class AstValidator:
+	"""Validates AST type descriptors."""
+
+	class Mode(Enum):
+		"""Validation mode."""
+
+		PRE_EXPANSION = 1
+		POST_EXPANSION = 2
+
+	def __init__(self, type_descriptors):
+		self.raw_type_descriptors = type_descriptors
+		self.type_descriptor_map = {model.name: model for model in self.raw_type_descriptors}
+
+		self.mode = self.Mode.PRE_EXPANSION
+		self.errors = []
+
+	def set_validation_mode(self, mode):
+		"""Sets the validation mode."""
+		self.mode = mode
+
+	def validate(self):
+		"""Validates all types for correctness."""
+
+		for _, model in self.type_descriptor_map.items():
+			if isinstance(model, Enum):
+				self._validate_enum(model)
+
+			if isinstance(model, Struct):
+				self._validate_struct(model)
+
+	def _validate_enum(self, model):
+		duplicate_names = self._find_duplicate_names(model.values)
+		if duplicate_names:
+			self.errors.append(ErrorDescriptor('duplicate enum values', model.name, duplicate_names))
+
+	def _validate_struct(self, model):
+		duplicate_names = self._find_duplicate_names(model.fields)
+		if duplicate_names:
+			self.errors.append(ErrorDescriptor('duplicate struct fields', model.name, duplicate_names))
+
+		field_map = {field.name: field for field in model.fields if hasattr(field, 'name')}
+		for field in model.fields:
+			def create_error_descriptor(message, error_field=field):
+				return ErrorDescriptor(message, model.name, error_field.name)
+
+			if isinstance(field, StructInlinePlaceholder):
+				self._validate_unnamed_inline(field, lambda message: ErrorDescriptor(message, model.name))
+			else:
+				self._validate_struct_field(field, field_map, create_error_descriptor)
+
+		if self.Mode.PRE_EXPANSION != self.mode:
+			self._check_struct_attributes(model, field_map)
+
+	def _validate_unnamed_inline(self, field, create_error_descriptor):
+		if not self._is_known_type(field.inlined_typename):
+			self.errors.append(create_error_descriptor(f'reference to unknown inlined type "{field.inlined_typename}"'))
+		else:
+			# all dispositions are allowed as unnamed inline
+			pass
+
+	def _validate_struct_field(self, field, field_map, create_error_descriptor):
+		if not self._is_known_type(field.field_type):
+			self.errors.append(create_error_descriptor(f'reference to unknown type "{field.field_type}"'))
+		else:
+			if 'inline' == field.disposition and 'inline' != self.type_descriptor_map[field.field_type].disposition:
+				self.errors.append(create_error_descriptor(f'named inline field referencing non inline struct "{field.field_type}"'))
+
+		if isinstance(field.field_type, FixedSizeInteger):
+			self._validate_integer(field.field_type, field_map, create_error_descriptor)
+
+		if isinstance(field.field_type, Array):
+			self._validate_array(field.field_type, field_map, create_error_descriptor)
+
+		if field.value is not None:
+			if 'sizeof' == field.disposition:
+				self._validate_sizeof(field, field_map, create_error_descriptor)
+			elif isinstance(field.value, Conditional):
+				self._validate_conditional(field, field_map, create_error_descriptor)
+			else:
+				self._validate_in_range(field.field_type, field.value, create_error_descriptor)
+
+		if field.attributes:
+			for attribute in field.attributes:
+				if not hasattr(field.field_type, attribute.name):
+					self.errors.append(create_error_descriptor(f'inapplicable attribute "{attribute.name}"'))
+
+	def _validate_integer(self, field_type, field_map, create_error_descriptor):
+		sizeref = field_type.sizeref
+		if not sizeref:
+			return
+
+		if sizeref.property_name not in field_map:
+			self.errors.append(create_error_descriptor(f'reference to unknown sizeref property "{sizeref.property_name}"'))
+
+	def _validate_array(self, field_type, field_map, create_error_descriptor):
+		element_type = field_type.element_type
+		is_sort_key_valid = True
+		sort_key = field_type.sort_key
+		if not self._is_known_type(element_type):
+			self.errors.append(create_error_descriptor(f'reference to unknown element type "{element_type}"'))
+			is_sort_key_valid = not sort_key
+		else:
+			is_sort_key_valid = not sort_key or any(
+				sort_key == element_field.name for element_field in self.type_descriptor_map[element_type].fields
+			)
+
+		if not is_sort_key_valid:
+			self.errors.append(create_error_descriptor(f'reference to unknown sort_key property "{sort_key}"'))
+
+		size = field_type.size
+		if isinstance(size, str) and size not in field_map:
+			self.errors.append(create_error_descriptor(f'reference to unknown size property "{size}"'))
+
+	def _validate_sizeof(self, field, field_map, create_error_descriptor):
+		if field.value not in field_map:
+			self.errors.append(create_error_descriptor(f'reference to unknown sizeof property "{field.value}"'))
+			return
+
+		reference_typename = field_map[field.value].field_type
+		reference_type = None if not isinstance(reference_typename, str) else self.type_descriptor_map[reference_typename]
+		if not isinstance(reference_type, Struct):
+			self.errors.append(create_error_descriptor(f'sizeof property references fixed size type "{reference_typename}"'))
+		elif not reference_type.is_size_implicit:
+			message = f'sizeof property references type "{reference_typename}" without is_size_implicit attribute'
+			self.errors.append(create_error_descriptor(message))
+
+	def _validate_conditional(self, field, field_map, create_error_descriptor):
+		linked_field_name = field.value.linked_field_name
+
+		if linked_field_name not in field_map:
+			self.errors.append(create_error_descriptor(f'reference to unknown condition field "{linked_field_name}"'))
+		else:
+			self._validate_in_range(field_map[linked_field_name].field_type, field.value.value, create_error_descriptor)
+
+	def _validate_in_range(self, value_type, value, create_error_descriptor):
+		if isinstance(value_type, str):
+			if self._is_known_type(value_type):
+				value_type = self.type_descriptor_map[value_type]
+
+				if isinstance(value_type, Enum):
+					if not any(value == enum_value.name for enum_value in value_type.values):
+						self.errors.append(create_error_descriptor(f'field value "{value}" is not a valid enum value'))
+
+					return
+
+		if not isinstance(value, int):
+			self.errors.append(create_error_descriptor(f'field value "{value}" is not a valid numeric value'))
+
+	def _check_struct_attributes(self, model, field_map):
+		if self._check_known_field(model, field_map, 'size'):
+			if not isinstance(field_map[model.size].field_type, FixedSizeInteger):
+				self.errors.append(ErrorDescriptor(f'reference to "size" property "{model.size}" has unexpected type', model.name))
+
+		self._check_known_field(model, field_map, 'discriminator', True)
+
+		self._check_comparer(model, field_map)
+		self._check_initializers(model, field_map)
+
+	def _check_comparer(self, model, field_map):
+		if not model.comparer:
+			return
+
+		for (property_name, transform) in model.comparer:
+			if property_name not in field_map:
+				self.errors.append(ErrorDescriptor(f'reference to unknown "comparer" property "{property_name}"', model.name))
+			if transform not in (None, 'ripemd_keccak_256'):
+				self.errors.append(ErrorDescriptor(f'reference to unknown "comparer" transform "{transform}"', model.name))
+
+	def _check_initializers(self, model, field_map):
+		if not model.initializers:
+			return
+
+		for initializer in model.initializers:
+			is_valid = True
+			is_concrete = model.disposition not in ('abstract', 'inline')
+			for property_name, raise_error in [(initializer.target_property_name, True), (initializer.value, is_concrete)]:
+				if property_name not in field_map:
+					is_valid = False
+					if raise_error:
+						self.errors.append(ErrorDescriptor(
+							f'reference to unknown "intializes" property "{property_name}"',
+							model.name))
+
+			if not is_valid:
+				continue
+
+			# str compare is good enough as it lets us differentiate among subtypes of things like FixedSizeInteger
+			if str(field_map[initializer.target_property_name].field_type) != str(field_map[initializer.value].field_type):
+				self.errors.append(ErrorDescriptor(
+					f'property "{initializer.target_property_name}" has initializer "{initializer.value}" of different type',
+					model.name))
+
+	def _check_known_field(self, model, field_map, property_name, multi_value=False):
+		values = getattr(model, property_name)
+		if not values:
+			return False
+
+		if not multi_value:
+			values = [values]
+
+		has_error = False
+		for value in values:
+			if value not in field_map:
+				self.errors.append(ErrorDescriptor(f'reference to unknown "{property_name}" property "{value}"', model.name))
+				has_error = True
+
+		return not has_error
+
+	def _is_known_type(self, typename):
+		return not isinstance(typename, str) or typename in self.type_descriptor_map
+
+	@staticmethod
+	def _find_duplicate_names(items):
+		unique_names = set()
+		duplicate_names = set()
+		for item in filter(lambda item: hasattr(item, 'name'), items):
+			(unique_names if item.name not in unique_names else duplicate_names).add(item.name)
+
+		return duplicate_names