avrotize 3.0.2__tar.gz → 3.1.1__tar.gz

avrotize-3.0.2/README.md → avrotize-3.1.1/PKG-INFO

@@ -1,3 +1,64 @@
+Metadata-Version: 2.4
+Name: avrotize
+Version: 3.1.1
+Summary: Tools to convert from and to Avro Schema from various other schema languages.
+Author-email: Clemens Vasters <clemensv@microsoft.com>
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+License-File: LICENSE
+Requires-Dist: jsonschema>=4.23.0
+Requires-Dist: lark>=1.1.9
+Requires-Dist: pyarrow>=22.0.0
+Requires-Dist: asn1tools>=0.167.0
+Requires-Dist: jsonpointer>=3.0.0
+Requires-Dist: jsonpath-ng>=1.6.1
+Requires-Dist: jsoncomparison>=1.1.0
+Requires-Dist: requests>=2.32.3
+Requires-Dist: azure-kusto-data>=5.0.5
+Requires-Dist: azure-identity>=1.17.1
+Requires-Dist: datapackage>=1.15.4
+Requires-Dist: jinja2>=3.1.4
+Requires-Dist: pyiceberg>=0.10.0
+Requires-Dist: pandas>=2.2.2
+Requires-Dist: docker>=7.1.0
+Requires-Dist: cddlparser>=0.5.0
+Requires-Dist: json-structure>=0.1.8
+Requires-Dist: psycopg2-binary>=2.9.9 ; extra == "all-sql"
+Requires-Dist: pymysql>=1.1.1 ; extra == "all-sql"
+Requires-Dist: pyodbc>=5.1.0 ; extra == "all-sql"
+Requires-Dist: oracledb>=2.3.0 ; extra == "all-sql"
+Requires-Dist: pytest>=8.3.2 ; extra == "dev"
+Requires-Dist: fastavro>=1.9.5 ; extra == "dev"
+Requires-Dist: xmlschema>=3.3.2 ; extra == "dev"
+Requires-Dist: xmlunittest>=1.0.1 ; extra == "dev"
+Requires-Dist: pylint>=3.2.6 ; extra == "dev"
+Requires-Dist: dataclasses_json>=0.6.7 ; extra == "dev"
+Requires-Dist: dataclasses>=0.8 ; extra == "dev"
+Requires-Dist: pydantic>=2.8.2 ; extra == "dev"
+Requires-Dist: avro>=1.12.0 ; extra == "dev"
+Requires-Dist: testcontainers>=4.7.2 ; extra == "dev"
+Requires-Dist: pymysql>=1.1.1 ; extra == "dev"
+Requires-Dist: psycopg2-binary>=2.9.9 ; extra == "dev"
+Requires-Dist: pyodbc>=5.1.0 ; extra == "dev"
+Requires-Dist: pymongo>=4.8.0 ; extra == "dev"
+Requires-Dist: oracledb>=2.3.0 ; extra == "dev"
+Requires-Dist: cassandra-driver>=3.29.1 ; extra == "dev"
+Requires-Dist: sqlalchemy>=2.0.32 ; extra == "dev"
+Requires-Dist: graphql-core>=3.2.0 ; extra == "dev"
+Requires-Dist: pymysql>=1.1.1 ; extra == "mysql"
+Requires-Dist: oracledb>=2.3.0 ; extra == "oracle"
+Requires-Dist: psycopg2-binary>=2.9.9 ; extra == "postgres"
+Requires-Dist: pyodbc>=5.1.0 ; extra == "sqlserver"
+Provides-Extra: all-sql
+Provides-Extra: dev
+Provides-Extra: mysql
+Provides-Extra: oracle
+Provides-Extra: postgres
+Provides-Extra: sqlserver
+
 # Avrotize & Structurize
 
 [![PyPI version](https://img.shields.io/pypi/v/avrotize)](https://pypi.org/project/avrotize/)
@@ -27,6 +88,22 @@ You can install Avrotize from PyPI, [having installed Python 3.10 or later](http
 pip install avrotize
 ```
 
+For SQL database support (`sql2a` command), install the optional database drivers:
+
+```bash
+# PostgreSQL
+pip install avrotize[postgres]
+
+# MySQL
+pip install avrotize[mysql]
+
+# SQL Server
+pip install avrotize[sqlserver]
+
+# All SQL databases
+pip install avrotize[all-sql]
+```
+
 ## Usage
 
 Avrotize provides several commands for converting schema formats via Avrotize Schema.
@@ -38,6 +115,11 @@ Converting to Avrotize Schema:
 - [`avrotize x2a`](#convert-xml-schema-xsd-to-avrotize-schema) - Convert XML schema to Avrotize Schema.
 - [`avrotize asn2a`](#convert-asn1-schema-to-avrotize-schema) - Convert ASN.1 to Avrotize Schema.
 - [`avrotize k2a`](#convert-kusto-table-definition-to-avrotize-schema) - Convert Kusto table definitions to Avrotize Schema.
+- [`avrotize sql2a`](#convert-sql-database-schema-to-avrotize-schema) - Convert SQL database schema to Avrotize Schema.
+- [`avrotize json2a`](#infer-avro-schema-from-json-files) - Infer Avro schema from JSON files.
+- [`avrotize json2s`](#infer-json-structure-schema-from-json-files) - Infer JSON Structure schema from JSON files.
+- [`avrotize xml2a`](#infer-avro-schema-from-xml-files) - Infer Avro schema from XML files.
+- [`avrotize xml2s`](#infer-json-structure-schema-from-xml-files) - Infer JSON Structure schema from XML files.
 - [`avrotize pq2a`](#convert-parquet-schema-to-avrotize-schema) - Convert Parquet schema to Avrotize Schema.
 - [`avrotize csv2a`](#convert-csv-file-to-avrotize-schema) - Convert CSV file to Avrotize Schema.
 - [`avrotize kstruct2a`](#convert-kafka-connect-schema-to-avrotize-schema) - Convert Kafka Connect Schema to Avrotize Schema.
@@ -105,6 +187,7 @@ Direct JSON Structure conversions:
 Other commands:
 
 - [`avrotize pcf`](#create-the-parsing-canonical-form-pcf-of-an-avrotize-schema) - Create the Parsing Canonical Form (PCF) of an Avrotize Schema.
+- [`avrotize validate`](#validate-json-instances-against-schemas) - Validate JSON instances against Avro or JSON Structure schemas.
 
 JSON Structure conversions:
 
@@ -378,6 +461,150 @@ Conversion notes:
 - For `dynamic` columns, the tool will sample the data in the table to determine the structure of the dynamic column. The tool will map the dynamic column to an Avro record type with fields that correspond to the fields found in the dynamic column. If the dynamic column contains nested dynamic columns, the tool will recursively map those to Avro record types. If records with conflicting structures are found in the dynamic column, the tool will emit a union of record types for the dynamic column.
 - If the `--emit-cloudevents-xregistry` option is set, the tool will emit an [xRegistry](http://xregistry.io) registry manifest file with a CloudEvent message definition for each table in the Kusto database and a separate Avro Schema for each table in the embedded schema registry. If one or more tables are found to contain CloudEvent data (as indicated by the presence of the CloudEvents attribute columns), the tool will inspect the content of the `type` (or `__type` or `__type`) columns to determine which CloudEvent types have been stored in the table and will emit a CloudEvent definition and schema for each unique type.
 
+### Convert SQL database schema to Avrotize Schema
+
+```bash
+avrotize sql2a --connection-string <connection_string> [--username <user>] [--password <pass>] [--dialect <dialect>] [--database <database>] [--table-name <table>] [--out <path_to_avro_schema_file>] [--namespace <namespace>] [--infer-json] [--infer-xml] [--sample-size <n>] [--emit-cloudevents] [--emit-xregistry]
+```
+
+Parameters:
+
+- `--connection-string`: The database connection string. Supports SSL/TLS and integrated authentication options (see examples below).
+- `--username`: (optional) Database username. Overrides any username in the connection string. Use this to avoid credentials in command history.
+- `--password`: (optional) Database password. Overrides any password in the connection string. Use this to avoid credentials in command history.
+- `--dialect`: (optional) The SQL dialect: `postgres` (default), `mysql`, `sqlserver`, `oracle`, or `sqlite`.
+- `--database`: (optional) The database name if not specified in the connection string.
+- `--table-name`: (optional) A specific table to convert. If omitted, all tables are converted.
+- `--out`: The path to the Avrotize Schema file. If omitted, output goes to stdout.
+- `--namespace`: (optional) The Avro namespace for the generated schema.
+- `--infer-json`: (optional, default: true) Infer schema for JSON/JSONB columns by sampling data.
+- `--infer-xml`: (optional, default: true) Infer schema for XML columns by sampling data.
+- `--sample-size`: (optional, default: 100) Number of rows to sample for JSON/XML schema inference.
+- `--emit-cloudevents`: (optional) Detect CloudEvents tables and emit CloudEvents declarations.
+- `--emit-xregistry`: (optional) Emit an xRegistry manifest instead of a single schema file.
+
+Connection string examples:
+
+```bash
+# PostgreSQL with separate credentials (preferred for security)
+avrotize sql2a --connection-string "postgresql://host:5432/mydb?sslmode=require" --username myuser --password mypass --out schema.avsc
+
+# PostgreSQL with SSL (credentials in URL)
+avrotize sql2a --connection-string "postgresql://user:pass@host:5432/mydb?sslmode=require" --out schema.avsc
+
+# MySQL with SSL
+avrotize sql2a --connection-string "mysql://user:pass@host:3306/mydb?ssl=true" --dialect mysql --out schema.avsc
+
+# SQL Server with Windows Authentication (omit user/password)
+avrotize sql2a --connection-string "mssql://@host:1433/mydb" --dialect sqlserver --out schema.avsc
+
+# SQL Server with TLS encryption
+avrotize sql2a --connection-string "mssql://user:pass@host:1433/mydb?encrypt=true" --dialect sqlserver --out schema.avsc
+
+# SQLite file
+avrotize sql2a --connection-string "/path/to/database.db" --dialect sqlite --out schema.avsc
+```
+
+Conversion notes:
+
+- The tool connects to a live database and reads the schema from the information schema or system catalogs.
+- Type mappings for each dialect:
+  - **PostgreSQL**: All standard types including `uuid`, `jsonb`, `xml`, arrays, and custom types.
+  - **MySQL**: Standard types including `json`, `enum`, `set`, and spatial types.
+  - **SQL Server**: Standard types including `uniqueidentifier`, `xml`, `money`, and `hierarchyid`.
+  - **Oracle**: Standard types including `number`, `clob`, `blob`, and Oracle-specific types.
+  - **SQLite**: Dynamic typing mapped based on declared type affinity.
+- For JSON/JSONB columns (PostgreSQL, MySQL) and XML columns, the tool samples data to infer the structure. Fields that appear in some but not all records are folded together. If field types conflict across records, the tool emits a union of record types.
+- For columns with keys that cannot be valid Avro identifiers (UUIDs, URLs, special characters), the tool generates `map<string, T>` types instead of record types.
+- Table and column comments are preserved as Avro `doc` attributes where available.
+- Primary key columns are noted in the schema's `unique` attribute.
+
+### Infer Avro schema from JSON files
+
+```bash
+avrotize json2a <json_files...> [--out <path_to_avro_schema_file>] [--type-name <name>] [--namespace <namespace>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<json_files...>`: One or more JSON files to analyze. Supports JSON arrays, single objects, and JSONL (JSON Lines) format.
+- `--out`: The path to the Avro schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--namespace`: (optional) Avro namespace for generated types.
+- `--sample-size`: (optional) Maximum number of records to sample (0 = all, default: 0).
+
+Example:
+
+```bash
+# Infer schema from multiple JSON files
+avrotize json2a data1.json data2.json --out schema.avsc --type-name Event --namespace com.example
+
+# Infer schema from JSONL file
+avrotize json2a events.jsonl --out events.avsc --type-name LogEntry
+```
+
+### Infer JSON Structure schema from JSON files
+
+```bash
+avrotize json2s <json_files...> [--out <path_to_jstruct_schema_file>] [--type-name <name>] [--base-id <uri>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<json_files...>`: One or more JSON files to analyze.
+- `--out`: The path to the JSON Structure schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--base-id`: (optional) Base URI for $id generation (default: "https://example.com/").
+- `--sample-size`: (optional) Maximum number of records to sample (0 = all, default: 0).
+
+Example:
+
+```bash
+avrotize json2s data.json --out schema.jstruct.json --type-name Person --base-id https://myapi.example.com/schemas/
+```
+
+### Infer Avro schema from XML files
+
+```bash
+avrotize xml2a <xml_files...> [--out <path_to_avro_schema_file>] [--type-name <name>] [--namespace <namespace>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<xml_files...>`: One or more XML files to analyze.
+- `--out`: The path to the Avro schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--namespace`: (optional) Avro namespace for generated types.
+- `--sample-size`: (optional) Maximum number of documents to sample (0 = all, default: 0).
+
+Example:
+
+```bash
+avrotize xml2a config.xml --out config.avsc --type-name Configuration --namespace com.example.config
+```
+
+### Infer JSON Structure schema from XML files
+
+```bash
+avrotize xml2s <xml_files...> [--out <path_to_jstruct_schema_file>] [--type-name <name>] [--base-id <uri>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<xml_files...>`: One or more XML files to analyze.
+- `--out`: The path to the JSON Structure schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--base-id`: (optional) Base URI for $id generation (default: "https://example.com/").
+- `--sample-size`: (optional) Maximum number of documents to sample (0 = all, default: 0).
+
+Conversion notes (applies to all inference commands):
+
+- XML attributes are converted to fields prefixed with `@` (normalized to valid identifiers).
+- Text content in mixed-content elements becomes a `#text` field.
+- Repeated elements are inferred as arrays.
+- Multiple files with different structures are merged into a unified schema.
+- Sparse data (fields that appear in some but not all records) is folded into a single type.
+
 ### Convert Avrotize Schema to Kusto table declaration
 
 ```bash
@@ -1260,6 +1487,45 @@ Conversion notes:
 - The tool generates the Parsing Canonical Form (PCF) of the Avrotize Schema. The PCF is a normalized form of the schema that is used for schema comparison and compatibility checking.
 - The PCF is a JSON object that is written to stdout.
 
+### Validate JSON instances against schemas
+
+```bash
+avrotize validate <json_files...> --schema <schema_file> [--schema-type <type>] [--quiet]
+```
+
+Parameters:
+
+- `<json_files...>`: One or more JSON files to validate. Supports single JSON objects, JSON arrays, and JSONL (newline-delimited JSON) formats.
+- `--schema <schema_file>`: Path to the schema file (`.avsc` for Avro, `.jstruct.json` for JSON Structure).
+- `--schema-type`: (optional) Schema type: `avro` or `jstruct`. Auto-detected from file extension if omitted.
+- `--quiet`: (optional) Suppress output. Exit code 0 if all instances are valid, 1 if any are invalid.
+
+Validation notes:
+
+- Validates JSON instances against Avro schemas per the [Avrotize Schema specification](specs/avrotize-schema.md).
+- Supports all Avro primitive types: null, boolean, int, long, float, double, bytes, string.
+- Supports all Avro complex types: record, enum, array, map, fixed.
+- Supports logical types with both native and string encodings: decimal, uuid, date, time-millis, time-micros, timestamp-millis, timestamp-micros, duration.
+- Supports field `altnames` for JSON field name mapping.
+- Supports enum `altsymbols` for JSON symbol mapping.
+- For JSON Structure validation, requires the `json-structure` package.
+
+Example:
+
+```bash
+# Validate JSON file against Avro schema
+avrotize validate data.json --schema schema.avsc
+
+# Validate multiple files
+avrotize validate file1.json file2.json --schema schema.avsc
+
+# Validate JSONL file against JSON Structure schema
+avrotize validate events.jsonl --schema events.jstruct.json
+
+# Quiet mode for CI/CD pipelines (exit code only)
+avrotize validate data.json --schema schema.avsc --quiet
+```
+
 ### Convert JSON Structure schema to GraphQL schema
 
 ```bash
@@ -1327,3 +1593,4 @@ cat myschema.avsc | avrotize a2graphql > myschema.graphql
 ```
 
 This document provides an overview of the usage and functionality of Avrotize. For more detailed information, please refer to the [Avrotize Schema documentation](specs/avrotize-schema.md) and the individual command help messages.
+

avrotize-3.0.2/PKG-INFO → avrotize-3.1.1/README.md

@@ -1,51 +1,3 @@
-Metadata-Version: 2.4
-Name: avrotize
-Version: 3.0.2
-Summary: Tools to convert from and to Avro Schema from various other schema languages.
-Author-email: Clemens Vasters <clemensv@microsoft.com>
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-License-File: LICENSE
-Requires-Dist: jsonschema>=4.23.0
-Requires-Dist: lark>=1.1.9
-Requires-Dist: pyarrow>=22.0.0
-Requires-Dist: asn1tools>=0.167.0
-Requires-Dist: jsonpointer>=3.0.0
-Requires-Dist: jsonpath-ng>=1.6.1
-Requires-Dist: jsoncomparison>=1.1.0
-Requires-Dist: requests>=2.32.3
-Requires-Dist: azure-kusto-data>=5.0.5
-Requires-Dist: azure-identity>=1.17.1
-Requires-Dist: datapackage>=1.15.4
-Requires-Dist: jinja2>=3.1.4
-Requires-Dist: pyiceberg>=0.10.0
-Requires-Dist: pandas>=2.2.2
-Requires-Dist: docker>=7.1.0
-Requires-Dist: cddlparser>=0.5.0
-Requires-Dist: json-structure>=0.1.8
-Requires-Dist: pytest>=8.3.2 ; extra == "dev"
-Requires-Dist: fastavro>=1.9.5 ; extra == "dev"
-Requires-Dist: xmlschema>=3.3.2 ; extra == "dev"
-Requires-Dist: xmlunittest>=1.0.1 ; extra == "dev"
-Requires-Dist: pylint>=3.2.6 ; extra == "dev"
-Requires-Dist: dataclasses_json>=0.6.7 ; extra == "dev"
-Requires-Dist: dataclasses>=0.8 ; extra == "dev"
-Requires-Dist: pydantic>=2.8.2 ; extra == "dev"
-Requires-Dist: avro>=1.12.0 ; extra == "dev"
-Requires-Dist: testcontainers>=4.7.2 ; extra == "dev"
-Requires-Dist: pymysql>=1.1.1 ; extra == "dev"
-Requires-Dist: psycopg2>=2.9.9 ; extra == "dev"
-Requires-Dist: pyodbc>=5.1.0 ; extra == "dev"
-Requires-Dist: pymongo>=4.8.0 ; extra == "dev"
-Requires-Dist: oracledb>=2.3.0 ; extra == "dev"
-Requires-Dist: cassandra-driver>=3.29.1 ; extra == "dev"
-Requires-Dist: sqlalchemy>=2.0.32 ; extra == "dev"
-Requires-Dist: graphql-core>=3.2.0 ; extra == "dev"
-Provides-Extra: dev
-
 # Avrotize & Structurize
 
 [![PyPI version](https://img.shields.io/pypi/v/avrotize)](https://pypi.org/project/avrotize/)
@@ -75,6 +27,22 @@ You can install Avrotize from PyPI, [having installed Python 3.10 or later](http
 pip install avrotize
 ```
 
+For SQL database support (`sql2a` command), install the optional database drivers:
+
+```bash
+# PostgreSQL
+pip install avrotize[postgres]
+
+# MySQL
+pip install avrotize[mysql]
+
+# SQL Server
+pip install avrotize[sqlserver]
+
+# All SQL databases
+pip install avrotize[all-sql]
+```
+
 ## Usage
 
 Avrotize provides several commands for converting schema formats via Avrotize Schema.
@@ -86,6 +54,11 @@ Converting to Avrotize Schema:
 - [`avrotize x2a`](#convert-xml-schema-xsd-to-avrotize-schema) - Convert XML schema to Avrotize Schema.
 - [`avrotize asn2a`](#convert-asn1-schema-to-avrotize-schema) - Convert ASN.1 to Avrotize Schema.
 - [`avrotize k2a`](#convert-kusto-table-definition-to-avrotize-schema) - Convert Kusto table definitions to Avrotize Schema.
+- [`avrotize sql2a`](#convert-sql-database-schema-to-avrotize-schema) - Convert SQL database schema to Avrotize Schema.
+- [`avrotize json2a`](#infer-avro-schema-from-json-files) - Infer Avro schema from JSON files.
+- [`avrotize json2s`](#infer-json-structure-schema-from-json-files) - Infer JSON Structure schema from JSON files.
+- [`avrotize xml2a`](#infer-avro-schema-from-xml-files) - Infer Avro schema from XML files.
+- [`avrotize xml2s`](#infer-json-structure-schema-from-xml-files) - Infer JSON Structure schema from XML files.
 - [`avrotize pq2a`](#convert-parquet-schema-to-avrotize-schema) - Convert Parquet schema to Avrotize Schema.
 - [`avrotize csv2a`](#convert-csv-file-to-avrotize-schema) - Convert CSV file to Avrotize Schema.
 - [`avrotize kstruct2a`](#convert-kafka-connect-schema-to-avrotize-schema) - Convert Kafka Connect Schema to Avrotize Schema.
@@ -153,6 +126,7 @@ Direct JSON Structure conversions:
 Other commands:
 
 - [`avrotize pcf`](#create-the-parsing-canonical-form-pcf-of-an-avrotize-schema) - Create the Parsing Canonical Form (PCF) of an Avrotize Schema.
+- [`avrotize validate`](#validate-json-instances-against-schemas) - Validate JSON instances against Avro or JSON Structure schemas.
 
 JSON Structure conversions:
 
@@ -426,6 +400,150 @@ Conversion notes:
 - For `dynamic` columns, the tool will sample the data in the table to determine the structure of the dynamic column. The tool will map the dynamic column to an Avro record type with fields that correspond to the fields found in the dynamic column. If the dynamic column contains nested dynamic columns, the tool will recursively map those to Avro record types. If records with conflicting structures are found in the dynamic column, the tool will emit a union of record types for the dynamic column.
 - If the `--emit-cloudevents-xregistry` option is set, the tool will emit an [xRegistry](http://xregistry.io) registry manifest file with a CloudEvent message definition for each table in the Kusto database and a separate Avro Schema for each table in the embedded schema registry. If one or more tables are found to contain CloudEvent data (as indicated by the presence of the CloudEvents attribute columns), the tool will inspect the content of the `type` (or `__type` or `__type`) columns to determine which CloudEvent types have been stored in the table and will emit a CloudEvent definition and schema for each unique type.
 
+### Convert SQL database schema to Avrotize Schema
+
+```bash
+avrotize sql2a --connection-string <connection_string> [--username <user>] [--password <pass>] [--dialect <dialect>] [--database <database>] [--table-name <table>] [--out <path_to_avro_schema_file>] [--namespace <namespace>] [--infer-json] [--infer-xml] [--sample-size <n>] [--emit-cloudevents] [--emit-xregistry]
+```
+
+Parameters:
+
+- `--connection-string`: The database connection string. Supports SSL/TLS and integrated authentication options (see examples below).
+- `--username`: (optional) Database username. Overrides any username in the connection string. Use this to avoid credentials in command history.
+- `--password`: (optional) Database password. Overrides any password in the connection string. Use this to avoid credentials in command history.
+- `--dialect`: (optional) The SQL dialect: `postgres` (default), `mysql`, `sqlserver`, `oracle`, or `sqlite`.
+- `--database`: (optional) The database name if not specified in the connection string.
+- `--table-name`: (optional) A specific table to convert. If omitted, all tables are converted.
+- `--out`: The path to the Avrotize Schema file. If omitted, output goes to stdout.
+- `--namespace`: (optional) The Avro namespace for the generated schema.
+- `--infer-json`: (optional, default: true) Infer schema for JSON/JSONB columns by sampling data.
+- `--infer-xml`: (optional, default: true) Infer schema for XML columns by sampling data.
+- `--sample-size`: (optional, default: 100) Number of rows to sample for JSON/XML schema inference.
+- `--emit-cloudevents`: (optional) Detect CloudEvents tables and emit CloudEvents declarations.
+- `--emit-xregistry`: (optional) Emit an xRegistry manifest instead of a single schema file.
+
+Connection string examples:
+
+```bash
+# PostgreSQL with separate credentials (preferred for security)
+avrotize sql2a --connection-string "postgresql://host:5432/mydb?sslmode=require" --username myuser --password mypass --out schema.avsc
+
+# PostgreSQL with SSL (credentials in URL)
+avrotize sql2a --connection-string "postgresql://user:pass@host:5432/mydb?sslmode=require" --out schema.avsc
+
+# MySQL with SSL
+avrotize sql2a --connection-string "mysql://user:pass@host:3306/mydb?ssl=true" --dialect mysql --out schema.avsc
+
+# SQL Server with Windows Authentication (omit user/password)
+avrotize sql2a --connection-string "mssql://@host:1433/mydb" --dialect sqlserver --out schema.avsc
+
+# SQL Server with TLS encryption
+avrotize sql2a --connection-string "mssql://user:pass@host:1433/mydb?encrypt=true" --dialect sqlserver --out schema.avsc
+
+# SQLite file
+avrotize sql2a --connection-string "/path/to/database.db" --dialect sqlite --out schema.avsc
+```
+
+Conversion notes:
+
+- The tool connects to a live database and reads the schema from the information schema or system catalogs.
+- Type mappings for each dialect:
+  - **PostgreSQL**: All standard types including `uuid`, `jsonb`, `xml`, arrays, and custom types.
+  - **MySQL**: Standard types including `json`, `enum`, `set`, and spatial types.
+  - **SQL Server**: Standard types including `uniqueidentifier`, `xml`, `money`, and `hierarchyid`.
+  - **Oracle**: Standard types including `number`, `clob`, `blob`, and Oracle-specific types.
+  - **SQLite**: Dynamic typing mapped based on declared type affinity.
+- For JSON/JSONB columns (PostgreSQL, MySQL) and XML columns, the tool samples data to infer the structure. Fields that appear in some but not all records are folded together. If field types conflict across records, the tool emits a union of record types.
+- For columns with keys that cannot be valid Avro identifiers (UUIDs, URLs, special characters), the tool generates `map<string, T>` types instead of record types.
+- Table and column comments are preserved as Avro `doc` attributes where available.
+- Primary key columns are noted in the schema's `unique` attribute.
+
+### Infer Avro schema from JSON files
+
+```bash
+avrotize json2a <json_files...> [--out <path_to_avro_schema_file>] [--type-name <name>] [--namespace <namespace>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<json_files...>`: One or more JSON files to analyze. Supports JSON arrays, single objects, and JSONL (JSON Lines) format.
+- `--out`: The path to the Avro schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--namespace`: (optional) Avro namespace for generated types.
+- `--sample-size`: (optional) Maximum number of records to sample (0 = all, default: 0).
+
+Example:
+
+```bash
+# Infer schema from multiple JSON files
+avrotize json2a data1.json data2.json --out schema.avsc --type-name Event --namespace com.example
+
+# Infer schema from JSONL file
+avrotize json2a events.jsonl --out events.avsc --type-name LogEntry
+```
+
+### Infer JSON Structure schema from JSON files
+
+```bash
+avrotize json2s <json_files...> [--out <path_to_jstruct_schema_file>] [--type-name <name>] [--base-id <uri>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<json_files...>`: One or more JSON files to analyze.
+- `--out`: The path to the JSON Structure schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--base-id`: (optional) Base URI for $id generation (default: "https://example.com/").
+- `--sample-size`: (optional) Maximum number of records to sample (0 = all, default: 0).
+
+Example:
+
+```bash
+avrotize json2s data.json --out schema.jstruct.json --type-name Person --base-id https://myapi.example.com/schemas/
+```
+
+### Infer Avro schema from XML files
+
+```bash
+avrotize xml2a <xml_files...> [--out <path_to_avro_schema_file>] [--type-name <name>] [--namespace <namespace>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<xml_files...>`: One or more XML files to analyze.
+- `--out`: The path to the Avro schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--namespace`: (optional) Avro namespace for generated types.
+- `--sample-size`: (optional) Maximum number of documents to sample (0 = all, default: 0).
+
+Example:
+
+```bash
+avrotize xml2a config.xml --out config.avsc --type-name Configuration --namespace com.example.config
+```
+
+### Infer JSON Structure schema from XML files
+
+```bash
+avrotize xml2s <xml_files...> [--out <path_to_jstruct_schema_file>] [--type-name <name>] [--base-id <uri>] [--sample-size <n>]
+```
+
+Parameters:
+
+- `<xml_files...>`: One or more XML files to analyze.
+- `--out`: The path to the JSON Structure schema file. If omitted, output goes to stdout.
+- `--type-name`: (optional) Name for the root type (default: "Document").
+- `--base-id`: (optional) Base URI for $id generation (default: "https://example.com/").
+- `--sample-size`: (optional) Maximum number of documents to sample (0 = all, default: 0).
+
+Conversion notes (applies to all inference commands):
+
+- XML attributes are converted to fields prefixed with `@` (normalized to valid identifiers).
+- Text content in mixed-content elements becomes a `#text` field.
+- Repeated elements are inferred as arrays.
+- Multiple files with different structures are merged into a unified schema.
+- Sparse data (fields that appear in some but not all records) is folded into a single type.
+
 ### Convert Avrotize Schema to Kusto table declaration
 
 ```bash
@@ -1308,6 +1426,45 @@ Conversion notes:
 - The tool generates the Parsing Canonical Form (PCF) of the Avrotize Schema. The PCF is a normalized form of the schema that is used for schema comparison and compatibility checking.
 - The PCF is a JSON object that is written to stdout.
 
+### Validate JSON instances against schemas
+
+```bash
+avrotize validate <json_files...> --schema <schema_file> [--schema-type <type>] [--quiet]
+```
+
+Parameters:
+
+- `<json_files...>`: One or more JSON files to validate. Supports single JSON objects, JSON arrays, and JSONL (newline-delimited JSON) formats.
+- `--schema <schema_file>`: Path to the schema file (`.avsc` for Avro, `.jstruct.json` for JSON Structure).
+- `--schema-type`: (optional) Schema type: `avro` or `jstruct`. Auto-detected from file extension if omitted.
+- `--quiet`: (optional) Suppress output. Exit code 0 if all instances are valid, 1 if any are invalid.
+
+Validation notes:
+
+- Validates JSON instances against Avro schemas per the [Avrotize Schema specification](specs/avrotize-schema.md).
+- Supports all Avro primitive types: null, boolean, int, long, float, double, bytes, string.
+- Supports all Avro complex types: record, enum, array, map, fixed.
+- Supports logical types with both native and string encodings: decimal, uuid, date, time-millis, time-micros, timestamp-millis, timestamp-micros, duration.
+- Supports field `altnames` for JSON field name mapping.
+- Supports enum `altsymbols` for JSON symbol mapping.
+- For JSON Structure validation, requires the `json-structure` package.
+
+Example:
+
+```bash
+# Validate JSON file against Avro schema
+avrotize validate data.json --schema schema.avsc
+
+# Validate multiple files
+avrotize validate file1.json file2.json --schema schema.avsc
+
+# Validate JSONL file against JSON Structure schema
+avrotize validate events.jsonl --schema events.jstruct.json
+
+# Quiet mode for CI/CD pipelines (exit code only)
+avrotize validate data.json --schema schema.avsc --quiet
+```
+
 ### Convert JSON Structure schema to GraphQL schema
 
 ```bash
@@ -1375,4 +1532,3 @@ cat myschema.avsc | avrotize a2graphql > myschema.graphql
 ```
 
 This document provides an overview of the usage and functionality of Avrotize. For more detailed information, please refer to the [Avrotize Schema documentation](specs/avrotize-schema.md) and the individual command help messages.
-

{avrotize-3.0.2 → avrotize-3.1.1}/avrotize/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '3.0.2'
-__version_tuple__ = version_tuple = (3, 0, 2)
+__version__ = version = '3.1.1'
+__version_tuple__ = version_tuple = (3, 1, 1)
 
-__commit_id__ = commit_id = 'g3ee5f8f67'
+__commit_id__ = commit_id = 'ge20c22879'

{avrotize-3.0.2 → avrotize-3.1.1}/avrotize/avrotize.py

@@ -38,6 +38,10 @@ def create_subparsers(subparsers, commands):
                 kwargs['choices'] = arg['choices']
             if 'default' in arg:
                 kwargs['default'] = arg['default']
+            # Handle dest for optional arguments only (positional args can't have dest)
+            arg_is_positional = not arg['name'].startswith('-')
+            if 'dest' in arg and not arg_is_positional:
+                kwargs['dest'] = arg['dest']
             if arg['type'] == 'bool':
                 kwargs['action'] = 'store_true'
                 del kwargs['type']

{avrotize-3.0.2 → avrotize-3.1.1}/avrotize/avrotots/class_core.ts.jinja

@@ -8,7 +8,7 @@ import { jsonObject, jsonMember, TypedJSON } from 'typedjson';
 {%- endif %}
 {%- endif %}
 {%- if avro_annotation %}
-import avro from 'avro-js';
+import avro, { type Type } from 'avro-js';
 {%- endif %}
 {%- for import_type, import_path in imports.items() %}
 import { {{ import_type }} } from '{{ import_path }}';
@@ -22,7 +22,7 @@ import pako from 'pako';
 {%- endif %}
 export class {{ class_name }} {
     {%- if avro_annotation %}
-    public static AvroType: avro.Type = avro.parse({{ avro_schema_json }});
+    public static AvroType: Type = avro.parse({{ avro_schema_json }});
     {%- endif %}
 
     {%- for field in fields %}