avrotize 3.0.1__py3-none-any.whl → 3.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- avrotize/_version.py +2 -2
- avrotize/avrotize.py +4 -0
- avrotize/avrotoiceberg.py +111 -13
- avrotize/avrotots/class_core.ts.jinja +2 -2
- avrotize/avrotots.py +62 -7
- avrotize/avrovalidator.py +518 -0
- avrotize/commands.json +485 -2
- avrotize/dependencies/cs/net90/dependencies.csproj +4 -4
- avrotize/dependencies/java/jdk21/pom.xml +6 -6
- avrotize/dependencies/typescript/node22/package.json +1 -1
- avrotize/jsontoschema.py +151 -0
- avrotize/schema_inference.py +825 -0
- avrotize/sqltoavro.py +1159 -0
- avrotize/structuretodb.py +1 -1
- avrotize/structuretoiceberg.py +113 -13
- avrotize/validate.py +242 -0
- avrotize/xmltoschema.py +122 -0
- {avrotize-3.0.1.dist-info → avrotize-3.1.0.dist-info}/METADATA +224 -4
- {avrotize-3.0.1.dist-info → avrotize-3.1.0.dist-info}/RECORD +22 -16
- {avrotize-3.0.1.dist-info → avrotize-3.1.0.dist-info}/WHEEL +0 -0
- {avrotize-3.0.1.dist-info → avrotize-3.1.0.dist-info}/entry_points.txt +0 -0
- {avrotize-3.0.1.dist-info → avrotize-3.1.0.dist-info}/licenses/LICENSE +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: avrotize
|
|
3
|
-
Version: 3.0
|
|
3
|
+
Version: 3.1.0
|
|
4
4
|
Summary: Tools to convert from and to Avro Schema from various other schema languages.
|
|
5
5
|
Author-email: Clemens Vasters <clemensv@microsoft.com>
|
|
6
6
|
Requires-Python: >=3.10
|
|
@@ -26,6 +26,10 @@ Requires-Dist: pandas>=2.2.2
|
|
|
26
26
|
Requires-Dist: docker>=7.1.0
|
|
27
27
|
Requires-Dist: cddlparser>=0.5.0
|
|
28
28
|
Requires-Dist: json-structure>=0.1.8
|
|
29
|
+
Requires-Dist: psycopg2-binary>=2.9.9 ; extra == "all-sql"
|
|
30
|
+
Requires-Dist: pymysql>=1.1.1 ; extra == "all-sql"
|
|
31
|
+
Requires-Dist: pyodbc>=5.1.0 ; extra == "all-sql"
|
|
32
|
+
Requires-Dist: oracledb>=2.3.0 ; extra == "all-sql"
|
|
29
33
|
Requires-Dist: pytest>=8.3.2 ; extra == "dev"
|
|
30
34
|
Requires-Dist: fastavro>=1.9.5 ; extra == "dev"
|
|
31
35
|
Requires-Dist: xmlschema>=3.3.2 ; extra == "dev"
|
|
@@ -37,14 +41,23 @@ Requires-Dist: pydantic>=2.8.2 ; extra == "dev"
|
|
|
37
41
|
Requires-Dist: avro>=1.12.0 ; extra == "dev"
|
|
38
42
|
Requires-Dist: testcontainers>=4.7.2 ; extra == "dev"
|
|
39
43
|
Requires-Dist: pymysql>=1.1.1 ; extra == "dev"
|
|
40
|
-
Requires-Dist: psycopg2>=2.9.9 ; extra == "dev"
|
|
44
|
+
Requires-Dist: psycopg2-binary>=2.9.9 ; extra == "dev"
|
|
41
45
|
Requires-Dist: pyodbc>=5.1.0 ; extra == "dev"
|
|
42
46
|
Requires-Dist: pymongo>=4.8.0 ; extra == "dev"
|
|
43
47
|
Requires-Dist: oracledb>=2.3.0 ; extra == "dev"
|
|
44
48
|
Requires-Dist: cassandra-driver>=3.29.1 ; extra == "dev"
|
|
45
49
|
Requires-Dist: sqlalchemy>=2.0.32 ; extra == "dev"
|
|
46
50
|
Requires-Dist: graphql-core>=3.2.0 ; extra == "dev"
|
|
51
|
+
Requires-Dist: pymysql>=1.1.1 ; extra == "mysql"
|
|
52
|
+
Requires-Dist: oracledb>=2.3.0 ; extra == "oracle"
|
|
53
|
+
Requires-Dist: psycopg2-binary>=2.9.9 ; extra == "postgres"
|
|
54
|
+
Requires-Dist: pyodbc>=5.1.0 ; extra == "sqlserver"
|
|
55
|
+
Provides-Extra: all-sql
|
|
47
56
|
Provides-Extra: dev
|
|
57
|
+
Provides-Extra: mysql
|
|
58
|
+
Provides-Extra: oracle
|
|
59
|
+
Provides-Extra: postgres
|
|
60
|
+
Provides-Extra: sqlserver
|
|
48
61
|
|
|
49
62
|
# Avrotize & Structurize
|
|
50
63
|
|
|
@@ -75,6 +88,22 @@ You can install Avrotize from PyPI, [having installed Python 3.10 or later](http
|
|
|
75
88
|
pip install avrotize
|
|
76
89
|
```
|
|
77
90
|
|
|
91
|
+
For SQL database support (`sql2a` command), install the optional database drivers:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
# PostgreSQL
|
|
95
|
+
pip install avrotize[postgres]
|
|
96
|
+
|
|
97
|
+
# MySQL
|
|
98
|
+
pip install avrotize[mysql]
|
|
99
|
+
|
|
100
|
+
# SQL Server
|
|
101
|
+
pip install avrotize[sqlserver]
|
|
102
|
+
|
|
103
|
+
# All SQL databases
|
|
104
|
+
pip install avrotize[all-sql]
|
|
105
|
+
```
|
|
106
|
+
|
|
78
107
|
## Usage
|
|
79
108
|
|
|
80
109
|
Avrotize provides several commands for converting schema formats via Avrotize Schema.
|
|
@@ -86,6 +115,11 @@ Converting to Avrotize Schema:
|
|
|
86
115
|
- [`avrotize x2a`](#convert-xml-schema-xsd-to-avrotize-schema) - Convert XML schema to Avrotize Schema.
|
|
87
116
|
- [`avrotize asn2a`](#convert-asn1-schema-to-avrotize-schema) - Convert ASN.1 to Avrotize Schema.
|
|
88
117
|
- [`avrotize k2a`](#convert-kusto-table-definition-to-avrotize-schema) - Convert Kusto table definitions to Avrotize Schema.
|
|
118
|
+
- [`avrotize sql2a`](#convert-sql-database-schema-to-avrotize-schema) - Convert SQL database schema to Avrotize Schema.
|
|
119
|
+
- [`avrotize json2a`](#infer-avro-schema-from-json-files) - Infer Avro schema from JSON files.
|
|
120
|
+
- [`avrotize json2s`](#infer-json-structure-schema-from-json-files) - Infer JSON Structure schema from JSON files.
|
|
121
|
+
- [`avrotize xml2a`](#infer-avro-schema-from-xml-files) - Infer Avro schema from XML files.
|
|
122
|
+
- [`avrotize xml2s`](#infer-json-structure-schema-from-xml-files) - Infer JSON Structure schema from XML files.
|
|
89
123
|
- [`avrotize pq2a`](#convert-parquet-schema-to-avrotize-schema) - Convert Parquet schema to Avrotize Schema.
|
|
90
124
|
- [`avrotize csv2a`](#convert-csv-file-to-avrotize-schema) - Convert CSV file to Avrotize Schema.
|
|
91
125
|
- [`avrotize kstruct2a`](#convert-kafka-connect-schema-to-avrotize-schema) - Convert Kafka Connect Schema to Avrotize Schema.
|
|
@@ -153,6 +187,7 @@ Direct JSON Structure conversions:
|
|
|
153
187
|
Other commands:
|
|
154
188
|
|
|
155
189
|
- [`avrotize pcf`](#create-the-parsing-canonical-form-pcf-of-an-avrotize-schema) - Create the Parsing Canonical Form (PCF) of an Avrotize Schema.
|
|
190
|
+
- [`avrotize validate`](#validate-json-instances-against-schemas) - Validate JSON instances against Avro or JSON Structure schemas.
|
|
156
191
|
|
|
157
192
|
JSON Structure conversions:
|
|
158
193
|
|
|
@@ -426,6 +461,150 @@ Conversion notes:
|
|
|
426
461
|
- For `dynamic` columns, the tool will sample the data in the table to determine the structure of the dynamic column. The tool will map the dynamic column to an Avro record type with fields that correspond to the fields found in the dynamic column. If the dynamic column contains nested dynamic columns, the tool will recursively map those to Avro record types. If records with conflicting structures are found in the dynamic column, the tool will emit a union of record types for the dynamic column.
|
|
427
462
|
- If the `--emit-cloudevents-xregistry` option is set, the tool will emit an [xRegistry](http://xregistry.io) registry manifest file with a CloudEvent message definition for each table in the Kusto database and a separate Avro Schema for each table in the embedded schema registry. If one or more tables are found to contain CloudEvent data (as indicated by the presence of the CloudEvents attribute columns), the tool will inspect the content of the `type` (or `__type` or `__type`) columns to determine which CloudEvent types have been stored in the table and will emit a CloudEvent definition and schema for each unique type.
|
|
428
463
|
|
|
464
|
+
### Convert SQL database schema to Avrotize Schema
|
|
465
|
+
|
|
466
|
+
```bash
|
|
467
|
+
avrotize sql2a --connection-string <connection_string> [--username <user>] [--password <pass>] [--dialect <dialect>] [--database <database>] [--table-name <table>] [--out <path_to_avro_schema_file>] [--namespace <namespace>] [--infer-json] [--infer-xml] [--sample-size <n>] [--emit-cloudevents] [--emit-xregistry]
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
Parameters:
|
|
471
|
+
|
|
472
|
+
- `--connection-string`: The database connection string. Supports SSL/TLS and integrated authentication options (see examples below).
|
|
473
|
+
- `--username`: (optional) Database username. Overrides any username in the connection string. Use this to avoid credentials in command history.
|
|
474
|
+
- `--password`: (optional) Database password. Overrides any password in the connection string. Use this to avoid credentials in command history.
|
|
475
|
+
- `--dialect`: (optional) The SQL dialect: `postgres` (default), `mysql`, `sqlserver`, `oracle`, or `sqlite`.
|
|
476
|
+
- `--database`: (optional) The database name if not specified in the connection string.
|
|
477
|
+
- `--table-name`: (optional) A specific table to convert. If omitted, all tables are converted.
|
|
478
|
+
- `--out`: The path to the Avrotize Schema file. If omitted, output goes to stdout.
|
|
479
|
+
- `--namespace`: (optional) The Avro namespace for the generated schema.
|
|
480
|
+
- `--infer-json`: (optional, default: true) Infer schema for JSON/JSONB columns by sampling data.
|
|
481
|
+
- `--infer-xml`: (optional, default: true) Infer schema for XML columns by sampling data.
|
|
482
|
+
- `--sample-size`: (optional, default: 100) Number of rows to sample for JSON/XML schema inference.
|
|
483
|
+
- `--emit-cloudevents`: (optional) Detect CloudEvents tables and emit CloudEvents declarations.
|
|
484
|
+
- `--emit-xregistry`: (optional) Emit an xRegistry manifest instead of a single schema file.
|
|
485
|
+
|
|
486
|
+
Connection string examples:
|
|
487
|
+
|
|
488
|
+
```bash
|
|
489
|
+
# PostgreSQL with separate credentials (preferred for security)
|
|
490
|
+
avrotize sql2a --connection-string "postgresql://host:5432/mydb?sslmode=require" --username myuser --password mypass --out schema.avsc
|
|
491
|
+
|
|
492
|
+
# PostgreSQL with SSL (credentials in URL)
|
|
493
|
+
avrotize sql2a --connection-string "postgresql://user:pass@host:5432/mydb?sslmode=require" --out schema.avsc
|
|
494
|
+
|
|
495
|
+
# MySQL with SSL
|
|
496
|
+
avrotize sql2a --connection-string "mysql://user:pass@host:3306/mydb?ssl=true" --dialect mysql --out schema.avsc
|
|
497
|
+
|
|
498
|
+
# SQL Server with Windows Authentication (omit user/password)
|
|
499
|
+
avrotize sql2a --connection-string "mssql://@host:1433/mydb" --dialect sqlserver --out schema.avsc
|
|
500
|
+
|
|
501
|
+
# SQL Server with TLS encryption
|
|
502
|
+
avrotize sql2a --connection-string "mssql://user:pass@host:1433/mydb?encrypt=true" --dialect sqlserver --out schema.avsc
|
|
503
|
+
|
|
504
|
+
# SQLite file
|
|
505
|
+
avrotize sql2a --connection-string "/path/to/database.db" --dialect sqlite --out schema.avsc
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
Conversion notes:
|
|
509
|
+
|
|
510
|
+
- The tool connects to a live database and reads the schema from the information schema or system catalogs.
|
|
511
|
+
- Type mappings for each dialect:
|
|
512
|
+
- **PostgreSQL**: All standard types including `uuid`, `jsonb`, `xml`, arrays, and custom types.
|
|
513
|
+
- **MySQL**: Standard types including `json`, `enum`, `set`, and spatial types.
|
|
514
|
+
- **SQL Server**: Standard types including `uniqueidentifier`, `xml`, `money`, and `hierarchyid`.
|
|
515
|
+
- **Oracle**: Standard types including `number`, `clob`, `blob`, and Oracle-specific types.
|
|
516
|
+
- **SQLite**: Dynamic typing mapped based on declared type affinity.
|
|
517
|
+
- For JSON/JSONB columns (PostgreSQL, MySQL) and XML columns, the tool samples data to infer the structure. Fields that appear in some but not all records are folded together. If field types conflict across records, the tool emits a union of record types.
|
|
518
|
+
- For columns with keys that cannot be valid Avro identifiers (UUIDs, URLs, special characters), the tool generates `map<string, T>` types instead of record types.
|
|
519
|
+
- Table and column comments are preserved as Avro `doc` attributes where available.
|
|
520
|
+
- Primary key columns are noted in the schema's `unique` attribute.
|
|
521
|
+
|
|
522
|
+
### Infer Avro schema from JSON files
|
|
523
|
+
|
|
524
|
+
```bash
|
|
525
|
+
avrotize json2a <json_files...> [--out <path_to_avro_schema_file>] [--type-name <name>] [--namespace <namespace>] [--sample-size <n>]
|
|
526
|
+
```
|
|
527
|
+
|
|
528
|
+
Parameters:
|
|
529
|
+
|
|
530
|
+
- `<json_files...>`: One or more JSON files to analyze. Supports JSON arrays, single objects, and JSONL (JSON Lines) format.
|
|
531
|
+
- `--out`: The path to the Avro schema file. If omitted, output goes to stdout.
|
|
532
|
+
- `--type-name`: (optional) Name for the root type (default: "Document").
|
|
533
|
+
- `--namespace`: (optional) Avro namespace for generated types.
|
|
534
|
+
- `--sample-size`: (optional) Maximum number of records to sample (0 = all, default: 0).
|
|
535
|
+
|
|
536
|
+
Example:
|
|
537
|
+
|
|
538
|
+
```bash
|
|
539
|
+
# Infer schema from multiple JSON files
|
|
540
|
+
avrotize json2a data1.json data2.json --out schema.avsc --type-name Event --namespace com.example
|
|
541
|
+
|
|
542
|
+
# Infer schema from JSONL file
|
|
543
|
+
avrotize json2a events.jsonl --out events.avsc --type-name LogEntry
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
### Infer JSON Structure schema from JSON files
|
|
547
|
+
|
|
548
|
+
```bash
|
|
549
|
+
avrotize json2s <json_files...> [--out <path_to_jstruct_schema_file>] [--type-name <name>] [--base-id <uri>] [--sample-size <n>]
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
Parameters:
|
|
553
|
+
|
|
554
|
+
- `<json_files...>`: One or more JSON files to analyze.
|
|
555
|
+
- `--out`: The path to the JSON Structure schema file. If omitted, output goes to stdout.
|
|
556
|
+
- `--type-name`: (optional) Name for the root type (default: "Document").
|
|
557
|
+
- `--base-id`: (optional) Base URI for $id generation (default: "https://example.com/").
|
|
558
|
+
- `--sample-size`: (optional) Maximum number of records to sample (0 = all, default: 0).
|
|
559
|
+
|
|
560
|
+
Example:
|
|
561
|
+
|
|
562
|
+
```bash
|
|
563
|
+
avrotize json2s data.json --out schema.jstruct.json --type-name Person --base-id https://myapi.example.com/schemas/
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
### Infer Avro schema from XML files
|
|
567
|
+
|
|
568
|
+
```bash
|
|
569
|
+
avrotize xml2a <xml_files...> [--out <path_to_avro_schema_file>] [--type-name <name>] [--namespace <namespace>] [--sample-size <n>]
|
|
570
|
+
```
|
|
571
|
+
|
|
572
|
+
Parameters:
|
|
573
|
+
|
|
574
|
+
- `<xml_files...>`: One or more XML files to analyze.
|
|
575
|
+
- `--out`: The path to the Avro schema file. If omitted, output goes to stdout.
|
|
576
|
+
- `--type-name`: (optional) Name for the root type (default: "Document").
|
|
577
|
+
- `--namespace`: (optional) Avro namespace for generated types.
|
|
578
|
+
- `--sample-size`: (optional) Maximum number of documents to sample (0 = all, default: 0).
|
|
579
|
+
|
|
580
|
+
Example:
|
|
581
|
+
|
|
582
|
+
```bash
|
|
583
|
+
avrotize xml2a config.xml --out config.avsc --type-name Configuration --namespace com.example.config
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
### Infer JSON Structure schema from XML files
|
|
587
|
+
|
|
588
|
+
```bash
|
|
589
|
+
avrotize xml2s <xml_files...> [--out <path_to_jstruct_schema_file>] [--type-name <name>] [--base-id <uri>] [--sample-size <n>]
|
|
590
|
+
```
|
|
591
|
+
|
|
592
|
+
Parameters:
|
|
593
|
+
|
|
594
|
+
- `<xml_files...>`: One or more XML files to analyze.
|
|
595
|
+
- `--out`: The path to the JSON Structure schema file. If omitted, output goes to stdout.
|
|
596
|
+
- `--type-name`: (optional) Name for the root type (default: "Document").
|
|
597
|
+
- `--base-id`: (optional) Base URI for $id generation (default: "https://example.com/").
|
|
598
|
+
- `--sample-size`: (optional) Maximum number of documents to sample (0 = all, default: 0).
|
|
599
|
+
|
|
600
|
+
Conversion notes (applies to all inference commands):
|
|
601
|
+
|
|
602
|
+
- XML attributes are converted to fields prefixed with `@` (normalized to valid identifiers).
|
|
603
|
+
- Text content in mixed-content elements becomes a `#text` field.
|
|
604
|
+
- Repeated elements are inferred as arrays.
|
|
605
|
+
- Multiple files with different structures are merged into a unified schema.
|
|
606
|
+
- Sparse data (fields that appear in some but not all records) is folded into a single type.
|
|
607
|
+
|
|
429
608
|
### Convert Avrotize Schema to Kusto table declaration
|
|
430
609
|
|
|
431
610
|
```bash
|
|
@@ -676,7 +855,7 @@ Conversion notes:
|
|
|
676
855
|
### Convert Avrotize Schema to Iceberg schema
|
|
677
856
|
|
|
678
857
|
```bash
|
|
679
|
-
avrotize a2ib <path_to_avro_schema_file> [--out <path_to_iceberg_schema_file>] [--record-type <record-type-from-avro>] [--emit-cloudevents-columns]
|
|
858
|
+
avrotize a2ib <path_to_avro_schema_file> [--out <path_to_iceberg_schema_file>] [--record-type <record-type-from-avro>] [--emit-cloudevents-columns] [--format schema|arrow]
|
|
680
859
|
```
|
|
681
860
|
|
|
682
861
|
Parameters:
|
|
@@ -685,6 +864,7 @@ Parameters:
|
|
|
685
864
|
- `--out`: The path to the Iceberg schema file to write the conversion result to. If omitted, the output is directed to stdout.
|
|
686
865
|
- `--record-type`: (optional) The name of the Avro record type to convert to an Iceberg schema.
|
|
687
866
|
- `--emit-cloudevents-columns`: (optional) If set, the tool will add [CloudEvents](https://cloudevents.io) attribute columns to the Iceberg schema: `__id`, `__source`, `__subject`, `__type`, and `__time`.
|
|
867
|
+
- `--format`: (optional) Output format. `schema` (default) outputs JSON per the [Iceberg Table Spec Appendix C](https://iceberg.apache.org/spec/#appendix-c-json-serialization). `arrow` outputs a binary Arrow IPC serialized schema.
|
|
688
868
|
|
|
689
869
|
Conversion notes:
|
|
690
870
|
|
|
@@ -695,7 +875,7 @@ Conversion notes:
|
|
|
695
875
|
### Convert JSON Structure to Iceberg schema
|
|
696
876
|
|
|
697
877
|
```bash
|
|
698
|
-
avrotize s2ib <path_to_structure_schema_file> [--out <path_to_iceberg_schema_file>] [--record-type <record-type-from-structure>] [--emit-cloudevents-columns]
|
|
878
|
+
avrotize s2ib <path_to_structure_schema_file> [--out <path_to_iceberg_schema_file>] [--record-type <record-type-from-structure>] [--emit-cloudevents-columns] [--format schema|arrow]
|
|
699
879
|
```
|
|
700
880
|
|
|
701
881
|
Parameters:
|
|
@@ -704,6 +884,7 @@ Parameters:
|
|
|
704
884
|
- `--out`: The path to the Iceberg schema file to write the conversion result to. If omitted, the output is directed to stdout.
|
|
705
885
|
- `--record-type`: (optional) The name of the record type in definitions to convert to an Iceberg schema.
|
|
706
886
|
- `--emit-cloudevents-columns`: (optional) If set, the tool will add [CloudEvents](https://cloudevents.io) attribute columns to the Iceberg schema: `___id`, `___source`, `___subject`, `___type`, and `___time`.
|
|
887
|
+
- `--format`: (optional) Output format. `schema` (default) outputs JSON per the [Iceberg Table Spec Appendix C](https://iceberg.apache.org/spec/#appendix-c-json-serialization). `arrow` outputs a binary Arrow IPC serialized schema.
|
|
707
888
|
|
|
708
889
|
Conversion notes:
|
|
709
890
|
|
|
@@ -1306,6 +1487,45 @@ Conversion notes:
|
|
|
1306
1487
|
- The tool generates the Parsing Canonical Form (PCF) of the Avrotize Schema. The PCF is a normalized form of the schema that is used for schema comparison and compatibility checking.
|
|
1307
1488
|
- The PCF is a JSON object that is written to stdout.
|
|
1308
1489
|
|
|
1490
|
+
### Validate JSON instances against schemas
|
|
1491
|
+
|
|
1492
|
+
```bash
|
|
1493
|
+
avrotize validate <json_files...> --schema <schema_file> [--schema-type <type>] [--quiet]
|
|
1494
|
+
```
|
|
1495
|
+
|
|
1496
|
+
Parameters:
|
|
1497
|
+
|
|
1498
|
+
- `<json_files...>`: One or more JSON files to validate. Supports single JSON objects, JSON arrays, and JSONL (newline-delimited JSON) formats.
|
|
1499
|
+
- `--schema <schema_file>`: Path to the schema file (`.avsc` for Avro, `.jstruct.json` for JSON Structure).
|
|
1500
|
+
- `--schema-type`: (optional) Schema type: `avro` or `jstruct`. Auto-detected from file extension if omitted.
|
|
1501
|
+
- `--quiet`: (optional) Suppress output. Exit code 0 if all instances are valid, 1 if any are invalid.
|
|
1502
|
+
|
|
1503
|
+
Validation notes:
|
|
1504
|
+
|
|
1505
|
+
- Validates JSON instances against Avro schemas per the [Avrotize Schema specification](specs/avrotize-schema.md).
|
|
1506
|
+
- Supports all Avro primitive types: null, boolean, int, long, float, double, bytes, string.
|
|
1507
|
+
- Supports all Avro complex types: record, enum, array, map, fixed.
|
|
1508
|
+
- Supports logical types with both native and string encodings: decimal, uuid, date, time-millis, time-micros, timestamp-millis, timestamp-micros, duration.
|
|
1509
|
+
- Supports field `altnames` for JSON field name mapping.
|
|
1510
|
+
- Supports enum `altsymbols` for JSON symbol mapping.
|
|
1511
|
+
- For JSON Structure validation, requires the `json-structure` package.
|
|
1512
|
+
|
|
1513
|
+
Example:
|
|
1514
|
+
|
|
1515
|
+
```bash
|
|
1516
|
+
# Validate JSON file against Avro schema
|
|
1517
|
+
avrotize validate data.json --schema schema.avsc
|
|
1518
|
+
|
|
1519
|
+
# Validate multiple files
|
|
1520
|
+
avrotize validate file1.json file2.json --schema schema.avsc
|
|
1521
|
+
|
|
1522
|
+
# Validate JSONL file against JSON Structure schema
|
|
1523
|
+
avrotize validate events.jsonl --schema events.jstruct.json
|
|
1524
|
+
|
|
1525
|
+
# Quiet mode for CI/CD pipelines (exit code only)
|
|
1526
|
+
avrotize validate data.json --schema schema.avsc --quiet
|
|
1527
|
+
```
|
|
1528
|
+
|
|
1309
1529
|
### Convert JSON Structure schema to GraphQL schema
|
|
1310
1530
|
|
|
1311
1531
|
```bash
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
avrotize/__init__.py,sha256=t5h5wkHXr6M0mmHAB5rhjZ3Gxy9xutGTGIfojfao9rI,3820
|
|
2
2
|
avrotize/__main__.py,sha256=5pY8dYAURcOnFRvgb6fgaOIa_SOzPLIWbU8-ZTQ0jG4,88
|
|
3
|
-
avrotize/_version.py,sha256=
|
|
3
|
+
avrotize/_version.py,sha256=SyHtlUaPBniDxgdCpqbbMa2feaPUxZYTtD3GSvNnwNQ,704
|
|
4
4
|
avrotize/asn1toavro.py,sha256=QDNwfBfXMxSH-k487CA3CaGCGDzOLs4PpVbbENm5uF0,8386
|
|
5
|
-
avrotize/avrotize.py,sha256=
|
|
5
|
+
avrotize/avrotize.py,sha256=z5snX7ZxlXwL6XC2vmhymnHvhbqIW7Tz7ZW2jRWrMgY,5966
|
|
6
6
|
avrotize/avrotocpp.py,sha256=hRZV247_TDD7Sm6_8sFx-UH5SueLLx2Wg6TvAVUX0iE,25693
|
|
7
7
|
avrotize/avrotocsharp.py,sha256=jOx9ctuUSsdpNDEXnNNfFHOb8zWWHclUvYTF5rInOaM,72339
|
|
8
8
|
avrotize/avrotocsv.py,sha256=yqVbP4Ar8bZyEgOAmHuHAsQKfCVjIO5_pa5dtaKLZKE,4575
|
|
@@ -10,7 +10,7 @@ avrotize/avrotodatapackage.py,sha256=zSCphLvCYiBKRAUCdccsr-4JysH3PyAS6fSgwa65Tss
|
|
|
10
10
|
avrotize/avrotodb.py,sha256=IXqH5nHNgIErEIosxxSE5DIGzKJGk2FU2HIzmSt5eIs,47102
|
|
11
11
|
avrotize/avrotogo.py,sha256=9URV0SG8vvjadiNhXW12x_uRplO-JoHEqjIgViWWb4w,22545
|
|
12
12
|
avrotize/avrotographql.py,sha256=i6G7xWjH_Lsn_CLiM4BCPb8OyZuCCpsYjXwXNTRMwEE,7394
|
|
13
|
-
avrotize/avrotoiceberg.py,sha256=
|
|
13
|
+
avrotize/avrotoiceberg.py,sha256=DCzUqtUYyECIrwq9QRopgiON0QcsXIxI-r_RkPJpxnk,12960
|
|
14
14
|
avrotize/avrotojava.py,sha256=NZZ7mUFVzvp7HBsU0XPiCwl4GVXE1RtS86pfNFJsIq8,135427
|
|
15
15
|
avrotize/avrotojs.py,sha256=QjB6XjFnDrpZBZrrWqS0TN8fQfRXBfhHabfG73FOIo8,12249
|
|
16
16
|
avrotize/avrotojsons.py,sha256=I7EJz_IaCYLNzcycYXDOLAexHRHQfyb4yoWcxJ02vb0,22697
|
|
@@ -22,10 +22,11 @@ avrotize/avrotoparquet.py,sha256=qm5hfia5elW1Yn4KACG8bbudLAqQSwGk3fIkTvdT5Rg,908
|
|
|
22
22
|
avrotize/avrotoproto.py,sha256=STqbdGjVrgKrlKXt-6dZlekW_Oq0W0StRx80St1XqIc,22486
|
|
23
23
|
avrotize/avrotopython.py,sha256=7s73rKyaQDu5-uLnF4mK7Jat9F4jipQ-lLapPXg6aPI,34168
|
|
24
24
|
avrotize/avrotorust.py,sha256=HEcDirRBCbXQNNs_FmkT-sp1dWQgZ8A23qkQYUxVuXE,24255
|
|
25
|
-
avrotize/avrotots.py,sha256=
|
|
25
|
+
avrotize/avrotots.py,sha256=QBHcrJ90gCELFYL2msOrpsJA49l84JT-qPXw6dDnTIE,34404
|
|
26
26
|
avrotize/avrotoxsd.py,sha256=iGQq_8kC0kfKsqvqS6s_mO-kJ8N5G8vXOwqRI_DZUxc,17744
|
|
27
|
+
avrotize/avrovalidator.py,sha256=qwrjt7e97t53GK06iiY2KKv7DPnRf_2Uwv72ciom9s8,21369
|
|
27
28
|
avrotize/cddltostructure.py,sha256=MA2c-P3CIEAxEaBX-FF299gR55xcLEV3FrfTr2QfayM,74491
|
|
28
|
-
avrotize/commands.json,sha256=
|
|
29
|
+
avrotize/commands.json,sha256=pqHbLdA03fVtUaDCmNK55Xu-ugA776svY-x5GZQp4gg,111680
|
|
29
30
|
avrotize/common.py,sha256=enqNR1I9-SbW7fNJE3w7N2R87kiN6_9Oa7VB4b2AUBc,31913
|
|
30
31
|
avrotize/constants.py,sha256=LlgHrvT6RsRPrhFGRNHmFuIj3b1bSd45yC4rBCIGGVA,2753
|
|
31
32
|
avrotize/csvtoavro.py,sha256=TuIYm_Xv8gioEHl1YgWQKOYkFGGHfuwmK5RuEAEXbt8,4293
|
|
@@ -34,6 +35,7 @@ avrotize/dependency_resolver.py,sha256=LGOTutpobJ4kMjAwvs-l0Mt2tEoZFaXCazs-u38qn
|
|
|
34
35
|
avrotize/dependency_version.py,sha256=tvbpO2VstTSTmNA5jbzQl48u6jnIM7BHyASQrrgsRYU,16844
|
|
35
36
|
avrotize/jsonstoavro.py,sha256=ZzigsCjAxw_TflXCjTLKHTrPmkiZRZMpuaZICfT_r_I,120069
|
|
36
37
|
avrotize/jsonstostructure.py,sha256=AXqrh3gJsQveWmC9Pfj1rTcUY_4KaDP7L9ObBSQehF4,147687
|
|
38
|
+
avrotize/jsontoschema.py,sha256=bRLhBqN1tkKVFvmNWBtzMeWkNVj__dMspTxtzMwZoc4,4714
|
|
37
39
|
avrotize/jstructtoavro.py,sha256=sOq7Ru1b8_ZLCEsnBqx3gsMWd7dPAaYxoraAD0bz6rk,33891
|
|
38
40
|
avrotize/kstructtoavro.py,sha256=t97JY22n0uILK3WcvQu_Yp9ONvouJRLAC2bZ3rvZ1L0,2856
|
|
39
41
|
avrotize/kustotoavro.py,sha256=1oEk9mbqmP3N5-V7mBHSXpbSlYFzjJ7ajIDNJZxA1r8,21756
|
|
@@ -42,15 +44,17 @@ avrotize/parquettoavro.py,sha256=iAPrSYNkiH3fBKNVDfIgeXkQbAiopa252ULJrGgmBDI,553
|
|
|
42
44
|
avrotize/proto2parser.py,sha256=__9R3cqiUJXc_efPCZZcF7rt18kA7mfhmX0qm2v0eSw,19742
|
|
43
45
|
avrotize/proto3parser.py,sha256=MfE84c-oAWWuzYmKlEZ5g5LUF7YzZaASFh2trX3UCaw,15604
|
|
44
46
|
avrotize/prototoavro.py,sha256=hqXBGRxYojaEbEgoHZxXwMG4R1nWC7UMl_XNLWfqH38,17346
|
|
47
|
+
avrotize/schema_inference.py,sha256=puhn81PxhZiWldD7rZ2IfUQY7-hCw5pcMkVcEsP1FGY,31413
|
|
48
|
+
avrotize/sqltoavro.py,sha256=BpvsN8KJTp_0MLWl7JsbcQHvnA2hQgbwsJymtDKYXjo,48264
|
|
45
49
|
avrotize/structuretocddl.py,sha256=RK_dTJf0oAo6BIBM48NHRcWC96OtUjlgUC6HzXs5Lkk,21234
|
|
46
50
|
avrotize/structuretocpp.py,sha256=tBWOvyZPYQ1CHN6RgDnWlmzJ1giOyQ9SlHBHWvhPyiw,35898
|
|
47
51
|
avrotize/structuretocsharp.py,sha256=NnHzeJcOWDbZ_LnGV6AW9hXgP7rQW6Lld9utzxO8l-g,128208
|
|
48
52
|
avrotize/structuretocsv.py,sha256=w9cwXAnnakKaeTtXsLWWO8KwYnXUxyXvC7a-ZKs-E94,13851
|
|
49
53
|
avrotize/structuretodatapackage.py,sha256=NEHRt30KfVDWH1EQakvuMdRZTtfVXx8fsaYud0ofb2g,29768
|
|
50
|
-
avrotize/structuretodb.py,sha256=
|
|
54
|
+
avrotize/structuretodb.py,sha256=a1lEQliACdLaN5FWJSQBPEA-25aFTyDbmzs7xUr34bI,44934
|
|
51
55
|
avrotize/structuretogo.py,sha256=EtU7C5IqKzPrYaxKUQJMDMpk-rlc8eRwFzHe489jHp4,34100
|
|
52
56
|
avrotize/structuretographql.py,sha256=wcGXnrup5v5saRa1BhR6o-X8q8ujsQMVqrFHQTBPjww,20468
|
|
53
|
-
avrotize/structuretoiceberg.py,sha256=
|
|
57
|
+
avrotize/structuretoiceberg.py,sha256=MNk17eIbvhZ5p7s-goNr-DBjN6GcOEksSDLNnrpWbMQ,19354
|
|
54
58
|
avrotize/structuretojava.py,sha256=r6YfI-_HtdeGqotL5W31WR5sz-TE1VLdDegctpjeiC4,47886
|
|
55
59
|
avrotize/structuretojs.py,sha256=TJuhUGqn0F2omjPVcrPeFPHY2KynRceFg9gMGvEenxA,29608
|
|
56
60
|
avrotize/structuretojsons.py,sha256=PJrQBaf6yQHu5eFkePxbjPBEmL-fYfX2wj6OmH1jsWw,22495
|
|
@@ -61,6 +65,8 @@ avrotize/structuretopython.py,sha256=d9EZVDHq7r-x0ZYZIRYfCP6kub7MkEROuvzjTJfNVv0
|
|
|
61
65
|
avrotize/structuretorust.py,sha256=ChRmO7uzU-pMdDdS0Vtg-MVUaOaNhNUPwH-ZKKOHglU,35134
|
|
62
66
|
avrotize/structuretots.py,sha256=zsN4ssX8rU4izOqXIiQcE0LQJ5wouydvCPoTWwSKXj4,35289
|
|
63
67
|
avrotize/structuretoxsd.py,sha256=01VpasyWSMOx04sILHLP7H-WkhGdXAEGKohUUfgrNf0,32797
|
|
68
|
+
avrotize/validate.py,sha256=isaDJ4E3O8SKehOROKCMnwE1-vPhkrVU5E-DDeThwlQ,7747
|
|
69
|
+
avrotize/xmltoschema.py,sha256=I125hZtbzxzbE3BSYemYJBB8d-X9Zle0ooKVaiNu_bk,3726
|
|
64
70
|
avrotize/xsdtoavro.py,sha256=nQtNH_3pEZBp67oUCPqzhvItEExHTe-8obsIfNRXt8Y,19064
|
|
65
71
|
avrotize/avrotocpp/CMakeLists.txt.jinja,sha256=t2ADvvi3o20xfVrsRBaUvZlpVCDR4h6Szsf0GcuSkH0,3015
|
|
66
72
|
avrotize/avrotocpp/build.bat.jinja,sha256=Wc_QbzaW9Zz5wnUoHLg1agmpfv_2LtWhqnRoNa-j6Ig,96
|
|
@@ -93,7 +99,7 @@ avrotize/avrotopython/test_enum.jinja,sha256=PeVxzC-gV5bizVnRs58RPoUDbXGI696B-VO
|
|
|
93
99
|
avrotize/avrotorust/dataclass_enum.rs.jinja,sha256=g4aYp0a7Yk8jO6aIiTXU8Obm48Qv63zxbzGREsWBBK8,2102
|
|
94
100
|
avrotize/avrotorust/dataclass_struct.rs.jinja,sha256=_FAKZG9VSRhn5enJScH8Y3t5KJ0aXTRqzGbMTjO6-dk,8463
|
|
95
101
|
avrotize/avrotorust/dataclass_union.rs.jinja,sha256=Ye4zVqpHgFNMUuvULCqN1aCWqezbWHppF_N_0hU0W2U,4640
|
|
96
|
-
avrotize/avrotots/class_core.ts.jinja,sha256
|
|
102
|
+
avrotize/avrotots/class_core.ts.jinja,sha256=-IGYwhzITndhVUYOXZ8oElGMrf0G1Oq5Lkw3HJ7-5tI,5091
|
|
97
103
|
avrotize/avrotots/class_test.ts.jinja,sha256=zm3wXaSJ225y7knLj6I9zjrkPTEROLHYmBr8eDo7pCo,2752
|
|
98
104
|
avrotize/avrotots/enum_core.ts.jinja,sha256=6NN_nF2y9us2saM7Wag8IN73BdIF0vxhPxq83NQEnm8,1522
|
|
99
105
|
avrotize/avrotots/gitignore.jinja,sha256=iYEkEO7BDDeNPTG8RSGJruh0Ol7HdYG4otQHhYLvC1c,376
|
|
@@ -101,12 +107,12 @@ avrotize/avrotots/index.ts.jinja,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hS
|
|
|
101
107
|
avrotize/avrotots/package.json.jinja,sha256=3ZqMVLP1ALkyXcVVA5j8Kyi5obNLJBv7dlgwKzYZfa8,555
|
|
102
108
|
avrotize/avrotots/tsconfig.json.jinja,sha256=Job00bmIiHEWQjA-Ze2-OdoTzT-GjVbL36fYuqtXLNs,537
|
|
103
109
|
avrotize/dependencies/cpp/vcpkg/vcpkg.json,sha256=se5qnUVQ1Q6wN_DqgIioqKg_y7ouh9oly2iBAJJXkgw,414
|
|
104
|
-
avrotize/dependencies/cs/net90/dependencies.csproj,sha256=
|
|
110
|
+
avrotize/dependencies/cs/net90/dependencies.csproj,sha256=IrmfTD8mGk1zPsaaX2yyG5nW3TBDGYtqNPzKrK_yM6k,1239
|
|
105
111
|
avrotize/dependencies/go/go121/go.mod,sha256=ZHjoQNugEIJdkfwwLupG5x6KpUyDp8HxyVnJfHG3wuE,203
|
|
106
|
-
avrotize/dependencies/java/jdk21/pom.xml,sha256=
|
|
112
|
+
avrotize/dependencies/java/jdk21/pom.xml,sha256=YrfKmt8Ne5JVJNv6xEng84P2kOuKQ_tc7qeJkdcr3Bo,3179
|
|
107
113
|
avrotize/dependencies/python/py312/requirements.txt,sha256=zMr7YjvCg75g66b-HURTr-n_vjxSv0J7A97aydoyYLA,336
|
|
108
114
|
avrotize/dependencies/rust/stable/Cargo.toml,sha256=p2s1Q-dzyV7QUkJUkUW3mr4yDAdnLQc3C9f6E_iZZB0,506
|
|
109
|
-
avrotize/dependencies/typescript/node22/package.json,sha256=
|
|
115
|
+
avrotize/dependencies/typescript/node22/package.json,sha256=XEiq9nTES4aSHdLMOTp3lt9urzg3pOtQHOZ6gTOCW9s,386
|
|
110
116
|
avrotize/generic/generic.avsc,sha256=GwLLCGBdu6H3fwInIfrvGdebDWrbXEFgnsnFJNrCT4E,1272
|
|
111
117
|
avrotize/prototypes/any.avsc,sha256=8bNmD-5j43YSoNTOQ4VIFjuH93a46-LOApf-L-ha1P8,3390
|
|
112
118
|
avrotize/prototypes/api.avsc,sha256=5sFP8jEqDAUbcfZup-YurhKAwR6U3dee09CeFPAhexM,6521
|
|
@@ -164,8 +170,8 @@ avrotize/structuretots/index.ts.jinja,sha256=-R4R_En1N4W_BEN3z3bLts9Xi4KnBTDLrYM
|
|
|
164
170
|
avrotize/structuretots/package.json.jinja,sha256=OfJn4g68VhBP-yvJCdsWm_1RHx1kphsmdWpxu_Fst1E,819
|
|
165
171
|
avrotize/structuretots/test_class.ts.jinja,sha256=7tJ6hPo3A9zToTkwolVyXYhmZ_E4uI_OnnYsUUzUEdQ,1180
|
|
166
172
|
avrotize/structuretots/tsconfig.json.jinja,sha256=8Pl65JW8uOMEexxkteobo0ZEqsJBO31HegNRUrf8XGQ,515
|
|
167
|
-
avrotize-3.0.
|
|
168
|
-
avrotize-3.0.
|
|
169
|
-
avrotize-3.0.
|
|
170
|
-
avrotize-3.0.
|
|
171
|
-
avrotize-3.0.
|
|
173
|
+
avrotize-3.1.0.dist-info/entry_points.txt,sha256=m8J2TWiqbZh7SBQezc1CNrM_GVPWf01zOFcAKhzCC0U,51
|
|
174
|
+
avrotize-3.1.0.dist-info/licenses/LICENSE,sha256=xGtQGygTETTtDQJafZCUbpsed3GxO6grmqig-jGEuSk,11348
|
|
175
|
+
avrotize-3.1.0.dist-info/WHEEL,sha256=G2gURzTEtmeR8nrdXUJfNiB3VYVxigPQ-bEQujpNiNs,82
|
|
176
|
+
avrotize-3.1.0.dist-info/METADATA,sha256=7FZxNHL8FCLwayyJWEGRAiyd3FLAVLVx6Bjc0qKUkVo,93923
|
|
177
|
+
avrotize-3.1.0.dist-info/RECORD,,
|
|
File without changes
|
|
File without changes
|
|
File without changes
|