ssc_codegen 0.7.7__tar.gz → 0.8.0__tar.gz

{ssc_codegen-0.7.7 → ssc_codegen-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ssc_codegen
-Version: 0.7.7
+Version: 0.8.0
 Summary: Python-dsl code converter to html parser for web scraping 
 Project-URL: Documentation, https://github.com/vypivshiy/selector_schema_codegen#readme
 Project-URL: Issues, https://github.com/vypivshiy/selector_schema_codegen/issues
@@ -19,9 +19,11 @@ Classifier: Topic :: Software Development :: Code Generators
 Classifier: Topic :: Text Processing :: Markup :: HTML
 Classifier: Topic :: Utilities
 Requires-Python: >=3.10
+Requires-Dist: colorama>=0.4.6; sys_platform == 'win32'
 Requires-Dist: cssselect>=1.2.0
 Requires-Dist: httpx>=0.28.1
 Requires-Dist: ichrome>=4.0.4
+Requires-Dist: jinja2>=3.1.6
 Requires-Dist: lxml>=5.3.0
 Requires-Dist: parsel>=1.10.0
 Requires-Dist: typer>=0.15.1
@@ -32,12 +34,12 @@ Description-Content-Type: text/markdown
 
 ## Introduction
 
-ssc-gen - based-python DSL language for writing html parsers in dataclass style for converting to targeting language.
+ssc-gen - based-python DSL language for writing html parsers in dataclass style for translating to targeting language.
 
 Project solving next problems:
 
-- boilerplate code
-- create types (type annotations) and documentation 
+- decrease boilerplate code
+- create types (or type annotations if possible) and documentation 
 - simplify code support
 - portability to other languages
 
@@ -45,30 +47,28 @@ Project solving next problems:
 
 Current support converters
 
-
-| Language      | Library (html parser backend)                                | XPath Support | CSS Support | Generated types                          | Code formatter |
-|---------------|--------------------------------------------------------------|---------------|-------------|------------------------------------------|----------------|
-| Python (3.8+) | bs4                                                          | N             | Y           | TypedDict*, list, dict                   | ruff           |
-| ...           | parsel                                                       | Y             | Y           | ...                                      | -              |
-| ...           | selectolax (modest)                                          | N             | Y           | ...                                      | -              |
-| ...           | scrapy (possibly use parsel - pass Response.selector object) | Y             | Y           | ...                                      | -              |
-| Dart (3)      | universal_html                                               | N             | Y           | record, List, Map                        | dart format    |
-| js (ES6)      | pure (firefox/chrome)                                        | Y             | Y           | Array, Map**                             | -              |
-| go (1.10+)    | goquery                                                      | N             | Y           | struct(json anchors include), array, map | gofmt          |
+| Language      | Library (html parser backend) | XPath Support | CSS Support | Generated types                          | Code formatter |
+|---------------|-------------------------------|---------------|-------------|------------------------------------------|----------------|
+| Python (3.8+) | bs4                           | N             | Y           | TypedDict*, list, dict                   | ruff           |
+| ...           | parsel                        | Y             | Y           | ...                                      | -              |
+| ...           | selectolax (modest)           | N             | Y           | ...                                      | -              |
+| js (ES6)      | pure (firefox/chrome/nodejs)  | Y             | Y           | Array, Map**                             | -              |
+| go (1.10+)    | goquery                       | N             | Y           | struct(json anchors include), array, map | gofmt          |
 
 - *this annotation type was deliberately chosen as a compromise reasons. 
-Python has many ways of serialization: `dataclass, namedtuple, attrs, pydantic`
+Python has many ways of serialization: `namedtuple, dataclass, attrs, pydantic, msgspec, etc`
   - TypedDict is like a build-in dict, but with IDE and linter hint support, and you can easily implement an adapter for the required structure.
-- **js not exists build-in serialization methods
+- **js exclude build-in serialization methods
 
 ### Limitations
 
 For maximum portability of the configuration to the target language:
 
 - Use CSS selectors: they are guaranteed to be converted to XPATH
-- Unlike javascript, most html parse libs implement [CSS3 selectors standard](https://www.w3.org/TR/selectors-3/)
-  - basic selectors: (`tag`, `.class`, `#id`)
-  - combined: (`div p`, `ul > li`, `h2 +p`\[1])
+- Unlike javascript, most html parse libs implement [CSS3 selectors standard](https://www.w3.org/TR/selectors-3/).
+check in target lib complex selectors before implement:
+  - basic selectors: (`tag`, `.class`, `#id`, `tag1,tag2`)
+  - combined: (`div p`, `ul > li`, `h2 +p`, `title ~head`\[1])
   - attribute: (`a[href]`, `input[type='text']`)\[2]
   - pseudo classes: (`:nth-child(n)`, `:first-child`, `:last-child`)\[3]
   - **often, not support more complex, dynamic styles**: (`:has()`, `:nth-of-type()`, `:where()`, `:is()`)
@@ -177,15 +177,14 @@ Code output looks like this (code formatted by IDE):
 
 ```javascript
 // autogenerated by ssc-gen DO NOT_EDIT
-/**
- *
+/***
  *
  * {
  *     "title": "String",
  *     "a_hrefs": "Array<String>"
- * }
- */
+ * }*/
 class HelloWorld {
+
     constructor(doc) {
         if (typeof doc === 'string') {
             this._doc = new DOMParser().parseFromString(doc, 'text/html');
@@ -196,18 +195,21 @@ class HelloWorld {
         }
     }
 
-    _parseTitle(value) {
-        let value1 = value.querySelector('title');
-        return typeof value1.textContent === "undefined" ? value1.documentElement.textContent : value1.textContent;
+    _parseTitle(v) {
+        let v0 = v.querySelector('title');
+        return typeof v0.textContent === "undefined" ? v0.documentElement.textContent : v0.textContent;
     }
 
-    _parseAHrefs(value) {
-        let value1 = Array.from(value.querySelectorAll('a'));
-        return value1.map(e => e.getAttribute('href'));
+    _parseAHrefs(v) {
+        let v0 = Array.from(v.querySelectorAll('a'));
+        return v0.map(e => e.getAttribute('href'));
     }
 
     parse() {
-        return {title: this._parseTitle(this._doc), a_hrefs: this._parseAHrefs(this._doc)};
+        return {
+            title: this._parseTitle(this._doc),
+            a_hrefs: this._parseAHrefs(this._doc),
+        };
     }
 }
 ```

{ssc_codegen-0.7.7 → ssc_codegen-0.8.0}/README.md

@@ -2,12 +2,12 @@
 
 ## Introduction
 
-ssc-gen - based-python DSL language for writing html parsers in dataclass style for converting to targeting language.
+ssc-gen - based-python DSL language for writing html parsers in dataclass style for translating to targeting language.
 
 Project solving next problems:
 
-- boilerplate code
-- create types (type annotations) and documentation 
+- decrease boilerplate code
+- create types (or type annotations if possible) and documentation 
 - simplify code support
 - portability to other languages
 
@@ -15,30 +15,28 @@ Project solving next problems:
 
 Current support converters
 
-
-| Language      | Library (html parser backend)                                | XPath Support | CSS Support | Generated types                          | Code formatter |
-|---------------|--------------------------------------------------------------|---------------|-------------|------------------------------------------|----------------|
-| Python (3.8+) | bs4                                                          | N             | Y           | TypedDict*, list, dict                   | ruff           |
-| ...           | parsel                                                       | Y             | Y           | ...                                      | -              |
-| ...           | selectolax (modest)                                          | N             | Y           | ...                                      | -              |
-| ...           | scrapy (possibly use parsel - pass Response.selector object) | Y             | Y           | ...                                      | -              |
-| Dart (3)      | universal_html                                               | N             | Y           | record, List, Map                        | dart format    |
-| js (ES6)      | pure (firefox/chrome)                                        | Y             | Y           | Array, Map**                             | -              |
-| go (1.10+)    | goquery                                                      | N             | Y           | struct(json anchors include), array, map | gofmt          |
+| Language      | Library (html parser backend) | XPath Support | CSS Support | Generated types                          | Code formatter |
+|---------------|-------------------------------|---------------|-------------|------------------------------------------|----------------|
+| Python (3.8+) | bs4                           | N             | Y           | TypedDict*, list, dict                   | ruff           |
+| ...           | parsel                        | Y             | Y           | ...                                      | -              |
+| ...           | selectolax (modest)           | N             | Y           | ...                                      | -              |
+| js (ES6)      | pure (firefox/chrome/nodejs)  | Y             | Y           | Array, Map**                             | -              |
+| go (1.10+)    | goquery                       | N             | Y           | struct(json anchors include), array, map | gofmt          |
 
 - *this annotation type was deliberately chosen as a compromise reasons. 
-Python has many ways of serialization: `dataclass, namedtuple, attrs, pydantic`
+Python has many ways of serialization: `namedtuple, dataclass, attrs, pydantic, msgspec, etc`
   - TypedDict is like a build-in dict, but with IDE and linter hint support, and you can easily implement an adapter for the required structure.
-- **js not exists build-in serialization methods
+- **js exclude build-in serialization methods
 
 ### Limitations
 
 For maximum portability of the configuration to the target language:
 
 - Use CSS selectors: they are guaranteed to be converted to XPATH
-- Unlike javascript, most html parse libs implement [CSS3 selectors standard](https://www.w3.org/TR/selectors-3/)
-  - basic selectors: (`tag`, `.class`, `#id`)
-  - combined: (`div p`, `ul > li`, `h2 +p`\[1])
+- Unlike javascript, most html parse libs implement [CSS3 selectors standard](https://www.w3.org/TR/selectors-3/).
+check in target lib complex selectors before implement:
+  - basic selectors: (`tag`, `.class`, `#id`, `tag1,tag2`)
+  - combined: (`div p`, `ul > li`, `h2 +p`, `title ~head`\[1])
   - attribute: (`a[href]`, `input[type='text']`)\[2]
   - pseudo classes: (`:nth-child(n)`, `:first-child`, `:last-child`)\[3]
   - **often, not support more complex, dynamic styles**: (`:has()`, `:nth-of-type()`, `:where()`, `:is()`)
@@ -147,15 +145,14 @@ Code output looks like this (code formatted by IDE):
 
 ```javascript
 // autogenerated by ssc-gen DO NOT_EDIT
-/**
- *
+/***
  *
  * {
  *     "title": "String",
  *     "a_hrefs": "Array<String>"
- * }
- */
+ * }*/
 class HelloWorld {
+
     constructor(doc) {
         if (typeof doc === 'string') {
             this._doc = new DOMParser().parseFromString(doc, 'text/html');
@@ -166,18 +163,21 @@ class HelloWorld {
         }
     }
 
-    _parseTitle(value) {
-        let value1 = value.querySelector('title');
-        return typeof value1.textContent === "undefined" ? value1.documentElement.textContent : value1.textContent;
+    _parseTitle(v) {
+        let v0 = v.querySelector('title');
+        return typeof v0.textContent === "undefined" ? v0.documentElement.textContent : v0.textContent;
     }
 
-    _parseAHrefs(value) {
-        let value1 = Array.from(value.querySelectorAll('a'));
-        return value1.map(e => e.getAttribute('href'));
+    _parseAHrefs(v) {
+        let v0 = Array.from(v.querySelectorAll('a'));
+        return v0.map(e => e.getAttribute('href'));
     }
 
     parse() {
-        return {title: this._parseTitle(this._doc), a_hrefs: this._parseAHrefs(this._doc)};
+        return {
+            title: this._parseTitle(this._doc),
+            a_hrefs: this._parseAHrefs(this._doc),
+        };
     }
 }
 ```

{ssc_codegen-0.7.7 → ssc_codegen-0.8.0}/pyproject.toml

@@ -1,13 +1,15 @@
 [project]
 name = "ssc_codegen"
-version = "0.7.7"
+version = "0.8.0"
 description = "Python-dsl code converter to html parser for web scraping "
 readme = "README.md"
 requires-python = ">=3.10"
 dependencies = [
+    "colorama>=0.4.6 ; sys_platform == 'win32'",
     "cssselect>=1.2.0",
     "httpx>=0.28.1",
     "ichrome>=4.0.4",
+    "jinja2>=3.1.6",
     "lxml>=5.3.0",
     "parsel>=1.10.0",
     "typer>=0.15.1",

{ssc_codegen-0.7.7 → ssc_codegen-0.8.0}/ssc_codegen/__init__.py

@@ -1,15 +1,22 @@
-from .document import HTMLDocument, StringDocument, ArrayDocument, AssertDocument, NestedDocument, DefaultDocument, \
+from ssc_codegen.document import HTMLDocument, StringDocument, ArrayDocument, AssertDocument, NestedDocument, \
+    DefaultDocument, \
     NumericDocument, JsonDocument, BooleanDocument
-from .json_struct import Json
-from .schema import ItemSchema, DictSchema, ListSchema, FlatListSchema
+from ssc_codegen.json_struct import Json
+from ssc_codegen.logs import setup_logger
+from ssc_codegen.schema import ItemSchema, DictSchema, ListSchema, FlatListSchema
+
+setup_logger()
+
+VERSION = "0.8.0"
 
-VERSION = "0.7.7"
 
 class __MISSING(object):
     pass
 
+
 _NO_DEFAULT = __MISSING()
 
+
 class Document(
     HTMLDocument,
     StringDocument,
@@ -27,12 +34,12 @@ class Nested(HTMLDocument, NestedDocument, ArrayDocument, AssertDocument):
     pass
 
 
-def D(default: None | str | int | float | __MISSING =_NO_DEFAULT) -> Document:  # noqa
+def D(default: None | str | int | float | __MISSING = _NO_DEFAULT) -> Document:  # noqa
     """Shortcut as a Document() object
 
     :param default: .default() operator shortcut
     """
-    if default==_NO_DEFAULT:
+    if default == _NO_DEFAULT:
         return Document()
     return Document().default(value=default)  # type: ignore
 
@@ -42,12 +49,12 @@ def N() -> Nested:  # noqa
     return Nested()
 
 
-def R(default: None | str | int | float | __MISSING =_NO_DEFAULT) -> Document: # noqa
+def R(default: None | str | int | float | __MISSING = _NO_DEFAULT) -> Document:  # noqa
     """Shortcut as a Document().raw() object.
     For regex and format string operations
 
     :param default: .default() operator shortcut
     """
-    if default==_NO_DEFAULT:
+    if default == _NO_DEFAULT:
         return Document().raw()
     return Document().default(default).raw()  # type: ignore

ssc_codegen-0.8.0/ssc_codegen/ast_/__init__.py

@@ -0,0 +1,32 @@
+from .base import BaseAstNode
+from .nodes_array import ExprIndex, ExprToListLength, ExprListStringJoin
+from .nodes_cast import ExprNested, ExprJsonify, ExprToInt, ExprToFloat, ExprToListInt, ExprToListFloat, ExprToBool
+from .nodes_core import (ExprReturn,
+                         ExprNoReturn,
+                         ExprDefaultValueEnd,
+                         ExprDefaultValueStart,
+                         ExprDefaultValueWrapper,
+                         ModuleProgram,
+                         Docstring,
+                         ModuleImports,
+                         StructInitMethod,
+                         StructPreValidateMethod,
+                         StructFieldMethod,
+                         StructPartDocMethod,
+                         StartParseMethod,
+                         StructParser,
+                         JsonStruct,
+                         JsonStructField,
+                         TypeDef,
+                         TypeDefField,
+                         ExprCallStructMethod)
+from .nodes_selectors import ExprCss, ExprCssAll, ExprXpathAll, ExprXpath, ExprGetHtmlText, ExprGetHtmlRaw, \
+    ExprGetHtmlAttr, ExprGetHtmlAttrAll, ExprGetHtmlRawAll, ExprGetHtmlTextAll
+
+from .nodes_string import ExprStringTrim, ExprStringRegex, ExprStringSplit, ExprStringReplace, ExprStringLeftTrim, \
+    ExprStringFormat, ExprListStringTrim, ExprStringRegexAll, ExprStringRegexSub, ExprStringRightTrim, \
+    ExprListStringFormat, ExprListStringReplace, ExprListStringRightTrim, ExprListStringLeftTrim, \
+    ExprListStringRegexSub, \
+    ExprStringRmPrefix, ExprStringRmSuffix, ExprStringRmPrefixAndSuffix, ExprListStringRmPrefixAndSuffix, \
+    ExprListStringRmPrefix, ExprListStringRmSuffix
+from .nodes_validate import ExprIsCss, ExprIsEqual, ExprIsRegex, ExprIsXpath, ExprIsNotEqual, ExprIsContains

ssc_codegen-0.8.0/ssc_codegen/ast_/base.py

@@ -0,0 +1,88 @@
+from dataclasses import dataclass, field
+from typing import (
+    ClassVar,
+    Mapping,
+    TypeVar,
+    Generic,
+    Any,
+    TypedDict,
+    Optional,
+    MutableSequence,
+    Sequence,
+)
+
+from ssc_codegen.tokens import TokenType, VariableType
+
+T_ARGUMENTS = TypeVar("T_ARGUMENTS")
+T_MAPPING_FIELD = TypeVar("T_MAPPING_FIELD", bound=Mapping[str, Any])
+T_EMPTY_KWARGS = TypedDict("T_EMPTY_KWARGS", {})
+
+
+@dataclass(kw_only=True)
+class BaseAstNode(Generic[T_MAPPING_FIELD, T_ARGUMENTS]):
+    """base AST container"""
+
+    kind: ClassVar[TokenType]
+    kwargs: T_MAPPING_FIELD = field(default_factory=dict)  # type: ignore[assignment]
+
+    body: MutableSequence["BaseAstNode"] = field(default_factory=list)
+    parent: Optional["BaseAstNode"] = field(default=None, repr=False)
+
+    # used for type check expressions
+    accept_type: VariableType = VariableType.ANY
+    ret_type: VariableType = VariableType.ANY
+    exclude_types: Sequence[VariableType] = field(default_factory=tuple)
+
+    def unpack_args(self) -> T_ARGUMENTS:
+        """extract all values from kwargs field wout keys"""
+        return tuple(self.kwargs.values())  # type: ignore
+
+    # todo: cached property?
+    @property
+    def index(self) -> int:
+        if self.parent:
+            return self.parent.body.index(self)
+        return -1
+
+    @property
+    def index_next(self) -> int:
+        if self.parent and len(self.parent.body) > self.index:
+            return self.index + 1
+        return -1
+
+    @property
+    def index_prev(self) -> int:
+        if self.parent and self.index == 0:
+            return -1
+        elif self.index == -1:
+            return -1
+        return self.index - 1
+
+    def __getattr__(self, name: str) -> Any:
+        # modify for provide extract argument from kwargs container (like property)
+        try:
+            return super().__getattribute__(name)
+        except AttributeError as e:
+            if self.kwargs.get(name):
+                return self.kwargs[name]
+            raise e
+
+
+@dataclass(kw_only=True)
+class _DisableReprBody(BaseAstNode):
+    """special class for disable repr unnecessary node fields"""
+
+    body: MutableSequence[BaseAstNode] = field(default_factory=list, repr=False)
+
+
+@dataclass(kw_only=True)
+class _DisableReprAcceptAndRetType(BaseAstNode):
+    accept_type: VariableType = field(default=VariableType.ANY, repr=False)
+    ret_type: VariableType = field(default=VariableType.ANY, repr=False)
+
+
+@dataclass(kw_only=True)
+class _DisableRepr(_DisableReprBody, _DisableReprAcceptAndRetType):
+    body: MutableSequence[BaseAstNode] = field(default_factory=list, repr=False)
+    accept_type: VariableType = field(default=VariableType.ANY, repr=False)
+    ret_type: VariableType = field(default=VariableType.ANY, repr=False)

ssc_codegen-0.8.0/ssc_codegen/ast_/nodes_array.py

@@ -0,0 +1,33 @@
+from dataclasses import dataclass
+from typing import TypedDict, ClassVar
+
+from ssc_codegen.ast_.base import BaseAstNode, T_EMPTY_KWARGS
+from ssc_codegen.tokens import TokenType, VariableType
+
+KW_EXPR_INDEX = TypedDict("KW_EXPR_INDEX", {"index": int})
+ARGS_EXPR_INDEX = tuple[int]
+
+
+@dataclass(kw_only=True)
+class ExprIndex(BaseAstNode[KW_EXPR_INDEX, ARGS_EXPR_INDEX]):
+    kind: ClassVar[TokenType] = TokenType.EXPR_LIST_ANY_INDEX
+    accept_type: VariableType = VariableType.LIST_ANY
+    ret_type: VariableType = VariableType.ANY
+
+
+KW_EXPR_JOIN = TypedDict("KW_EXPR_JOIN", {"sep": str})
+ARGS_EXPR_JOIN = tuple[str]
+
+
+@dataclass(kw_only=True)
+class ExprListStringJoin(BaseAstNode[KW_EXPR_JOIN, ARGS_EXPR_JOIN]):
+    kind: ClassVar[TokenType] = TokenType.EXPR_LIST_JOIN
+    accept_type: VariableType = VariableType.LIST_STRING
+    ret_type: VariableType = VariableType.STRING
+
+
+@dataclass(kw_only=True)
+class ExprToListLength(BaseAstNode[T_EMPTY_KWARGS, tuple]):
+    kind: ClassVar[TokenType] = TokenType.EXPR_LIST_LEN
+    accept_type: VariableType = VariableType.LIST_ANY
+    ret_type: VariableType = VariableType.INT

ssc_codegen-0.8.0/ssc_codegen/ast_/nodes_cast.py

@@ -0,0 +1,70 @@
+from dataclasses import dataclass
+from typing import TypedDict, Type, TYPE_CHECKING, TypeAlias, ClassVar
+
+from ssc_codegen.tokens import TokenType, VariableType, StructType
+from ssc_codegen.ast_.base import BaseAstNode, T_EMPTY_KWARGS
+
+if TYPE_CHECKING:
+    from ssc_codegen.schema import BaseSchema
+
+
+@dataclass(kw_only=True)
+class ExprToInt(BaseAstNode[T_EMPTY_KWARGS, tuple]):
+    kind: ClassVar[TokenType] = TokenType.TO_INT
+    accept_type: VariableType = VariableType.STRING
+    ret_type: VariableType = VariableType.INT
+
+
+@dataclass(kw_only=True)
+class ExprToListInt(BaseAstNode[T_EMPTY_KWARGS, tuple]):
+    kind: ClassVar[TokenType] = TokenType.TO_INT_LIST
+    accept_type: VariableType = VariableType.LIST_STRING
+    ret_type: VariableType = VariableType.LIST_INT
+
+
+@dataclass(kw_only=True)
+class ExprToFloat(BaseAstNode[T_EMPTY_KWARGS, tuple]):
+    kind: ClassVar[TokenType] = TokenType.TO_FLOAT
+    accept_type: VariableType = VariableType.STRING
+    ret_type: VariableType = VariableType.FLOAT
+
+
+@dataclass(kw_only=True)
+class ExprToListFloat(BaseAstNode[T_EMPTY_KWARGS, tuple]):
+    kind: ClassVar[TokenType] = TokenType.TO_FLOAT_LIST
+    accept_type: VariableType = VariableType.LIST_STRING
+    ret_type: VariableType = VariableType.LIST_FLOAT
+
+
+@dataclass(kw_only=True)
+class ExprToBool(BaseAstNode[T_EMPTY_KWARGS, tuple]):
+    kind: ClassVar[TokenType] = TokenType.TO_BOOL
+    accept_type: VariableType = VariableType.ANY
+    ret_type: VariableType = VariableType.BOOL
+
+
+_T_SCHEMA: TypeAlias = Type["BaseSchema"]
+KW_EXPR_NESTED = TypedDict(
+    "KW_EXPR_NESTED", {"schema_name": str, "schema_type": StructType}
+)
+EXPR_NESTED_ARGS = tuple[str, StructType]
+
+
+@dataclass(kw_only=True)
+class ExprNested(BaseAstNode[KW_EXPR_NESTED, EXPR_NESTED_ARGS]):
+    kind: ClassVar[TokenType] = TokenType.EXPR_NESTED
+    accept_type: VariableType = VariableType.DOCUMENT
+    ret_type: VariableType = VariableType.NESTED
+
+
+KW_EXPR_JSONIFY = TypedDict(
+    "KW_EXPR_JSONIFY", {"json_struct_name": str, "is_array": bool}
+)
+ARGS_EXPR_JSONIFY = tuple[str, bool]
+
+
+@dataclass(kw_only=True)
+class ExprJsonify(BaseAstNode[KW_EXPR_JSONIFY, ARGS_EXPR_JSONIFY]):
+    kind: ClassVar[TokenType] = TokenType.TO_JSON
+    accept_type: VariableType = VariableType.STRING
+    ret_type: VariableType = VariableType.JSON