ssc_codegen 0.31.0__tar.gz → 0.32.0__tar.gz

{ssc_codegen-0.31.0 → ssc_codegen-0.32.0}/.gitignore

@@ -150,5 +150,10 @@ test_schemas/
 node_modules/
 package-lock.json
 
+# Go (test/research scratch)
+go.work
+go.work.sum
+/tests/go/scratch/
+
 # slopmachines
 .claude/*

{ssc_codegen-0.31.0 → ssc_codegen-0.32.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ssc_codegen
-Version: 0.31.0
+Version: 0.32.0
 Summary: Python-dsl code converter to html parser for web scraping 
 Project-URL: Documentation, https://github.com/vypivshiy/selector_schema_codegen#readme
 Project-URL: Issues, https://github.com/vypivshiy/selector_schema_codegen/issues
@@ -22,9 +22,9 @@ Requires-Python: >=3.10
 Requires-Dist: bs4>=0.0.2
 Requires-Dist: colorama>=0.4.6; sys_platform == 'win32'
 Requires-Dist: cssselect>=1.2.0
-Requires-Dist: kdlquery>=1.1.2
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: kdlquery>=1.3.0
 Requires-Dist: lxml>=5.3.0
-Requires-Dist: pyyaml>=6.0
 Requires-Dist: soupsieve>=2.6
 Requires-Dist: typer>0.26
 Requires-Dist: typing-extensions; python_version < '3.11'
@@ -69,7 +69,7 @@ uv tool install ssc_codegen
 Generate Python parser:
 
 ```bash
-ssc-gen generate books.kdl -t py-bs4 -o ./output
+ssc-gen generate books.kdl -l python -L bs4 -o ./output
 ```
 
 ## Usage
@@ -77,17 +77,24 @@ ssc-gen generate books.kdl -t py-bs4 -o ./output
 ### Generate code
 
 ```bash
-# single file
-ssc-gen generate schema.kdl -t py-bs4 -o ./output
+# single file (Python + bs4)
+ssc-gen generate schema.kdl -l python -L bs4 -o ./output
 
-# all .kdl files in a directory
-ssc-gen generate examples/ -t js-pure -o ./output
+# all .kdl files in a directory (JavaScript)
+ssc-gen generate examples/ -l js -o ./output
 
 # with custom package name
-ssc-gen generate schema.kdl -t py-bs4 -o ./parsers --package scraper
+ssc-gen generate schema.kdl -l python -L bs4 -o ./parsers --package scraper
+
+# with @request support (REST/HTTP codegen)
+ssc-gen generate schema.kdl -l python -L bs4 -o ./out --http-client httpx
+
+# extract helper functions into a separate runtime module
+ssc-gen generate schema.kdl -l python -L bs4 -o ./out -R
 ```
 
-Targets: `py-bs4`, `py-lxml`, `py-parsel`, `py-slax`, `js-pure`
+Languages (`--lang / -l`): `python`, `js`.
+HTML libraries (`--lib / -L`, Python only): `bs4` (default), `lxml`, `parsel`, `slax`.
 
 ### Lint schemas
 
@@ -106,10 +113,10 @@ ssc-gen check examples/
 
 ```bash
 # from file
-ssc-gen run examples/booksToScrape.kdl:MainCatalogue -t py-bs4 -i page.html
+ssc-gen run examples/booksToScrape.kdl:MainCatalogue -l python -L bs4 -i page.html
 
 # from stdin
-curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -t py-bs4
+curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -l python -L bs4
 ```
 
 ### Health check (verify selectors match elements)
@@ -124,27 +131,35 @@ curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:Mai
 
 ## Documentation
 
-- [Quick start](docs2/guide.md)
-- [Syntax and file structure](docs2/syntax.md)
-- [Type system](docs2/types.md)
-- [Pipeline operations](docs2/operations.md)
-- [Predicates and logic](docs2/predicates.md)
-- [JSON schemas and jsonify](docs2/json.md)
-- [Transforms and dsl blocks](docs2/transforms.md)
-- [LLM-compact reference](docs2/llm.txt) -- full DSL spec in one file for LLM context
+- [Quick start](docs/guide.md)
+- [Syntax and file structure](docs/syntax.md)
+- [Type system](docs/types.md)
+- [Pipeline operations](docs/operations.md)
+- [Predicates and logic](docs/predicates.md)
+- [JSON schemas and jsonify](docs/json.md)
+- [@request — HTTP constructor + REST clients](docs/learn/10-request.md)
+- [LLM-compact reference](docs/llm.txt) -- full DSL spec in one file for LLM context
 - [Examples](examples/)
 
 ## LLM integration
 
-LLM agents can generate and validate `.kdl` schemas automatically using the linter feedback loop.
+LLM agents can generate and validate `.kdl` schemas automatically using the
+linter feedback loop.
 
 ### In chats (ChatGPT, Claude, etc.)
 
-Use [SYSTEM_PROMPT.md](SYSTEM_PROMPT.md) as system prompt. After generation, run `ssc-gen check -f json` and send errors back to the LLM for correction.
+Use [SYSTEM_PROMPT.md](SYSTEM_PROMPT.md) as system prompt. After generation,
+run `ssc-gen check -f json` and send errors back to the LLM for correction.
+
+### In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)
 
-### In AI-powered IDEs (Claude Code, Cursor, etc.)
+Use the skills under [.agents/skills/](.agents/skills/):
 
-Use the [kdl-schema-dsl](.agents/skills/kdl-schema-dsl) skill for automatic generation, validation, and iteration.
+- **`sscgen-dsl`** — HTML scraping schemas (`css`, `text`, `(item)struct`,
+  `(list)struct`, `(table)struct`, predicates, `jsonify`).
+- **`sscgen-rest`** — REST/JSON API clients (`(rest)struct`, `@request`,
+  `@error`, typed placeholders).
+- **`sscgen-openapi`** — convert OpenAPI/Swagger specs into `.kdl`.
 
 ## Development
 

{ssc_codegen-0.31.0 → ssc_codegen-0.32.0}/README.md

@@ -37,7 +37,7 @@ uv tool install ssc_codegen
 Generate Python parser:
 
 ```bash
-ssc-gen generate books.kdl -t py-bs4 -o ./output
+ssc-gen generate books.kdl -l python -L bs4 -o ./output
 ```
 
 ## Usage
@@ -45,17 +45,24 @@ ssc-gen generate books.kdl -t py-bs4 -o ./output
 ### Generate code
 
 ```bash
-# single file
-ssc-gen generate schema.kdl -t py-bs4 -o ./output
+# single file (Python + bs4)
+ssc-gen generate schema.kdl -l python -L bs4 -o ./output
 
-# all .kdl files in a directory
-ssc-gen generate examples/ -t js-pure -o ./output
+# all .kdl files in a directory (JavaScript)
+ssc-gen generate examples/ -l js -o ./output
 
 # with custom package name
-ssc-gen generate schema.kdl -t py-bs4 -o ./parsers --package scraper
+ssc-gen generate schema.kdl -l python -L bs4 -o ./parsers --package scraper
+
+# with @request support (REST/HTTP codegen)
+ssc-gen generate schema.kdl -l python -L bs4 -o ./out --http-client httpx
+
+# extract helper functions into a separate runtime module
+ssc-gen generate schema.kdl -l python -L bs4 -o ./out -R
 ```
 
-Targets: `py-bs4`, `py-lxml`, `py-parsel`, `py-slax`, `js-pure`
+Languages (`--lang / -l`): `python`, `js`.
+HTML libraries (`--lib / -L`, Python only): `bs4` (default), `lxml`, `parsel`, `slax`.
 
 ### Lint schemas
 
@@ -74,10 +81,10 @@ ssc-gen check examples/
 
 ```bash
 # from file
-ssc-gen run examples/booksToScrape.kdl:MainCatalogue -t py-bs4 -i page.html
+ssc-gen run examples/booksToScrape.kdl:MainCatalogue -l python -L bs4 -i page.html
 
 # from stdin
-curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -t py-bs4
+curl https://books.toscrape.com/ | ssc-gen run examples/booksToScrape.kdl:MainCatalogue -l python -L bs4
 ```
 
 ### Health check (verify selectors match elements)
@@ -92,27 +99,35 @@ curl https://books.toscrape.com/ | ssc-gen health examples/booksToScrape.kdl:Mai
 
 ## Documentation
 
-- [Quick start](docs2/guide.md)
-- [Syntax and file structure](docs2/syntax.md)
-- [Type system](docs2/types.md)
-- [Pipeline operations](docs2/operations.md)
-- [Predicates and logic](docs2/predicates.md)
-- [JSON schemas and jsonify](docs2/json.md)
-- [Transforms and dsl blocks](docs2/transforms.md)
-- [LLM-compact reference](docs2/llm.txt) -- full DSL spec in one file for LLM context
+- [Quick start](docs/guide.md)
+- [Syntax and file structure](docs/syntax.md)
+- [Type system](docs/types.md)
+- [Pipeline operations](docs/operations.md)
+- [Predicates and logic](docs/predicates.md)
+- [JSON schemas and jsonify](docs/json.md)
+- [@request — HTTP constructor + REST clients](docs/learn/10-request.md)
+- [LLM-compact reference](docs/llm.txt) -- full DSL spec in one file for LLM context
 - [Examples](examples/)
 
 ## LLM integration
 
-LLM agents can generate and validate `.kdl` schemas automatically using the linter feedback loop.
+LLM agents can generate and validate `.kdl` schemas automatically using the
+linter feedback loop.
 
 ### In chats (ChatGPT, Claude, etc.)
 
-Use [SYSTEM_PROMPT.md](SYSTEM_PROMPT.md) as system prompt. After generation, run `ssc-gen check -f json` and send errors back to the LLM for correction.
+Use [SYSTEM_PROMPT.md](SYSTEM_PROMPT.md) as system prompt. After generation,
+run `ssc-gen check -f json` and send errors back to the LLM for correction.
+
+### In AI-powered IDEs (Claude Code, Cursor, opencode, etc.)
 
-### In AI-powered IDEs (Claude Code, Cursor, etc.)
+Use the skills under [.agents/skills/](.agents/skills/):
 
-Use the [kdl-schema-dsl](.agents/skills/kdl-schema-dsl) skill for automatic generation, validation, and iteration.
+- **`sscgen-dsl`** — HTML scraping schemas (`css`, `text`, `(item)struct`,
+  `(list)struct`, `(table)struct`, predicates, `jsonify`).
+- **`sscgen-rest`** — REST/JSON API clients (`(rest)struct`, `@request`,
+  `@error`, typed placeholders).
+- **`sscgen-openapi`** — convert OpenAPI/Swagger specs into `.kdl`.
 
 ## Development
 

{ssc_codegen-0.31.0 → ssc_codegen-0.32.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ssc_codegen"
-version = "0.31.0"
+version = "0.32.0"
 description = "Python-dsl code converter to html parser for web scraping "
 readme = "README.md"
 requires-python = ">=3.10"
@@ -12,8 +12,8 @@ dependencies = [
     "soupsieve>=2.6",
     "typer>0.26",
     "typing_extensions;python_version<'3.11'",
-    "pyyaml>=6.0",
-    "kdlquery>=1.1.2",
+    "kdlquery>=1.3.0",
+    "httpx>=0.28.1",
 ]
 
 classifiers = [
@@ -54,6 +54,8 @@ packages = ["ssc_codegen"]
 
 [dependency-groups]
 dev = [
+    "aiohttp>=3.9,<3.12",  # aioresponses 0.7.x breaks on 3.12+ (stream_writer)
+    "aioresponses>=0.7",
     "bs4>=0.0.2",
     "coverage>=7.6.12",
     "httpx>=0.28.1",
@@ -63,6 +65,7 @@ dev = [
     "parsel>=1.9.1",
     "pytest>=8.3.4",
     "respx>=0.23.1",
+    "responses>=0.25",
     "ruff>=0.9.3",
     "selectolax>=0.3.27",
 ]
@@ -88,6 +91,11 @@ disallow_untyped_defs = false
 module = ["colorama", "lxml", "lxml.*"]
 ignore_missing_imports = true
 
+[[tool.mypy.overrides]]
+# Large visitor classes with match statements may need relaxed return checks
+module = ["ssc_codegen.targets.python.visitor", "ssc_codegen.targets.javascript.visitor"]
+disable_error_code = ["return", "return-value"]
+
 
 [tool.coverage.report]
 exclude_also = [

ssc_codegen-0.32.0/ssc_codegen/ast/__init__.py

@@ -0,0 +1,268 @@
+# mypy: ignore-errors
+"""
+AST nodes for the KDL Schema DSL.
+
+Import everything from here:
+    from kdl_ast import Module, Field, CssSelect, ...
+"""
+
+from .types import VariableType, StructType, TypeInfo, VT
+
+from kdlquery.parser import Span
+
+from .base import Node
+
+from .module import (
+    Module,
+    CodeStartHook,
+    CodeEndHook,
+    Docstring,
+    Utilities,
+)
+
+from .typedef import TypeDef, TypeDefField
+
+from .jsondef import JsonDef, JsonDefField
+
+from .rest import (
+    ResultVariantDef,
+    ResultAliasDef,
+    MatcherEntry,
+    MatcherListDef,
+)
+
+from .struct import (
+    StructBase,
+    Struct,
+    StructRest,
+    StructDocstring,
+    PreValidate,
+    CheckMethod,
+    Init,
+    InitFieldCall,
+    InitField,
+    SplitDoc,
+    Key,
+    Value,
+    TableConfig,
+    TableRows,
+    TableMatchKey,
+    RequestHttp,
+    MethodBase,
+    MethodFetch,
+    MethodRest,
+    ErrorResponse,
+    PlaceholderSpec,
+    PlaceholderTemplate,
+    Field,
+    StartParse,
+)
+
+from .selectors import (
+    CssSelect,
+    CssSelectAll,
+    XpathSelect,
+    XpathSelectAll,
+    CssRemove,
+    XpathRemove,
+)
+
+from .extract import Text, Raw, Attr
+
+from .string import (
+    Trim,
+    Ltrim,
+    Rtrim,
+    NormalizeSpace,
+    RmPrefix,
+    RmSuffix,
+    RmPrefixSuffix,
+    Fmt,
+    Repl,
+    ReplMap,
+    Lower,
+    Upper,
+    Split,
+    Join,
+    Unescape,
+)
+
+from .regex import Re, ReAll, ReSub
+
+from .array import Index, Slice, Len, Unique
+
+from .cast import ToInt, ToFloat, ToBool, Jsonify, Nested
+
+from .control import Self, Fallback, Return
+
+from .predicate_containers import Filter, Assert, Match
+
+from .predicate_ops import (
+    PredEq,
+    PredNe,
+    PredStarts,
+    PredEnds,
+    PredContains,
+    PredRe,
+    PredReAny,
+    PredReAll,
+    PredCss,
+    PredXpath,
+    PredHasAttr,
+    PredCountEq,
+    PredCountGt,
+    PredCountLt,
+    PredCountNe,
+    PredCountGe,
+    PredCountLe,
+    PredCountRange,
+    PredAttrEnds,
+    PredAttrEq,
+    PredAttrNe,
+    PredAttrRe,
+    PredAttrStarts,
+    PredAttrContains,
+    PredTextContains,
+    PredTextEnds,
+    PredTextRe,
+    PredTextStarts,
+    LogicNot,
+    LogicAnd,
+    LogicOr,
+)
+
+__all__ = [
+    # types
+    "VariableType",
+    "StructType",
+    "TypeInfo",
+    "VT",
+    # source location
+    "Span",
+    # base
+    "Node",
+    # module
+    "Module",
+    "CodeStartHook",
+    "CodeEndHook",
+    "Docstring",
+    "Utilities",
+    # typedef
+    "TypeDef",
+    "TypeDefField",
+    # jsondef
+    "JsonDef",
+    "JsonDefField",
+    # struct
+    "StructBase",
+    "Struct",
+    "StructRest",
+    "StructDocstring",
+    "PreValidate",
+    "CheckMethod",
+    "Init",
+    "InitFieldCall",
+    "InitField",
+    "SplitDoc",
+    "Key",
+    "Value",
+    "TableConfig",
+    "TableRows",
+    "TableMatchKey",
+    "RequestHttp",
+    "MethodBase",
+    "MethodFetch",
+    "MethodRest",
+    "ErrorResponse",
+    "PlaceholderSpec",
+    "PlaceholderTemplate",
+    "Field",
+    "StartParse",
+    # REST result artifacts (synthesized)
+    "ResultVariantDef",
+    "ResultAliasDef",
+    "MatcherEntry",
+    "MatcherListDef",
+    # selectors
+    "CssSelect",
+    "CssSelectAll",
+    "XpathSelect",
+    "XpathSelectAll",
+    "CssRemove",
+    "XpathRemove",
+    # extract
+    "Text",
+    "Raw",
+    "Attr",
+    # string
+    "Trim",
+    "Ltrim",
+    "Rtrim",
+    "NormalizeSpace",
+    "RmPrefix",
+    "RmSuffix",
+    "RmPrefixSuffix",
+    "Fmt",
+    "Repl",
+    "ReplMap",
+    "Lower",
+    "Upper",
+    "Split",
+    "Join",
+    "Unescape",
+    # regex
+    "Re",
+    "ReAll",
+    "ReSub",
+    # array
+    "Index",
+    "Slice",
+    "Len",
+    "Unique",
+    # cast
+    "ToInt",
+    "ToFloat",
+    "ToBool",
+    "Jsonify",
+    "Nested",
+    # control
+    "Self",
+    "Fallback",
+    "Return",
+    # predicate containers
+    "Filter",
+    "Assert",
+    "Match",
+    # predicate ops
+    "PredEq",
+    "PredNe",
+    "PredStarts",
+    "PredEnds",
+    "PredContains",
+    "PredRe",
+    "PredReAny",
+    "PredReAll",
+    "PredCss",
+    "PredXpath",
+    "PredHasAttr",
+    "PredAttrEq",
+    "PredAttrNe",
+    "PredAttrStarts",
+    "PredAttrEnds",
+    "PredAttrContains",
+    "PredAttrRe",
+    "PredTextStarts",
+    "PredTextEnds",
+    "PredTextContains",
+    "PredTextRe",
+    "PredCountEq",
+    "PredCountGt",
+    "PredCountLt",
+    "PredCountNe",
+    "PredCountGe",
+    "PredCountLe",
+    "PredCountRange",
+    "LogicNot",
+    "LogicAnd",
+    "LogicOr",
+]

ssc_codegen-0.32.0/ssc_codegen/ast/array.py

@@ -0,0 +1,76 @@
+from __future__ import annotations
+from dataclasses import dataclass, field
+
+from .base import Node
+from .types import TypeInfo, VariableType
+
+# Index, First, Last, Slice accept LIST_AUTO / return AUTO or LIST_AUTO.
+# Concrete types are resolved by the builder from the cursor type via
+# VariableType.scalar / VariableType.as_list helpers.
+
+
+@dataclass
+class Index(Node):
+    """
+    Returns element at position i.
+    Negative index counts from end.
+    Accepts list, returns scalar.
+    """
+
+    i: int = 0
+    accept_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO)
+    )
+    ret_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO)
+    )
+
+
+@dataclass
+class Slice(Node):
+    """
+    Returns sublist [start:end].
+    Accepts list, returns list.
+    """
+
+    start: int = 0
+    end: int = 0
+    accept_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO)
+    )
+    ret_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO, is_array=True)
+    )
+    is_array: bool = True
+
+
+@dataclass
+class Len(Node):
+    """Returns list length as INT."""
+
+    accept_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO)
+    )
+    ret_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.INT)
+    )
+
+
+@dataclass
+class Unique(Node):
+    """
+    Removes duplicate strings from list.
+    keep_order=True — preserves original order (default: False).
+    STRING → STRING with is_array=True.
+    """
+
+    keep_order: bool = False
+    accept_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.STRING)
+    )
+    ret_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(
+            base=VariableType.STRING, is_array=True
+        )
+    )
+    is_array: bool = True

ssc_codegen-0.32.0/ssc_codegen/ast/base.py

@@ -0,0 +1,52 @@
+from __future__ import annotations
+from dataclasses import dataclass, field
+
+from kdlquery.parser import Span
+
+from .types import TypeInfo, VariableType
+
+
+@dataclass
+class Node:
+    """
+    Base AST node.
+
+    Canonical type metadata:
+        ret_type_info    — resolved type of the value this node RETURNS.
+        accept_type_info — resolved type this node ACCEPTS as input.
+    is_array           — list context flag for pipeline expression nodes
+                         (kept in sync with ret_type_info.is_array by the builder).
+    parent             — back-reference, excluded from repr to avoid cycles.
+    body               — child nodes (pipeline body, struct body, etc.).
+    span               — optional source location carried from KdlNode.span.
+                         Populated lazily by parsers that need position info
+                         in codegen (Assert, Re). None for nodes that never
+                         use source location.
+
+    The scalar ``accept`` / ``ret`` views and the legacy ``type_info`` name are
+    exposed as read-only backport properties derived from the TypeInfo fields.
+    """
+
+    ret_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO)
+    )
+    accept_type_info: TypeInfo = field(
+        default_factory=lambda: TypeInfo(base=VariableType.AUTO)
+    )
+    is_array: bool = False
+    parent: Node | None = field(default=None, repr=False)
+    body: list[Node] = field(default_factory=list)
+    span: Span | None = field(default=None, repr=False, compare=False)
+
+    # ── backport properties (deprecated: prefer the TypeInfo fields) ───────────
+    @property
+    def ret(self) -> VariableType:
+        return self.ret_type_info.base
+
+    @property
+    def accept(self) -> VariableType:
+        return self.accept_type_info.base
+
+    @property
+    def type_info(self) -> TypeInfo:
+        return self.ret_type_info