sqlglot 2.2.1__tar.gz → 3.1.0__tar.gz

{sqlglot-2.2.1/sqlglot.egg-info → sqlglot-3.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: sqlglot
-Version: 2.2.1
+Version: 3.1.0
 Summary: An easily customizable SQL parser and transpiler
 Home-page: https://github.com/tobymao/sqlglot
 Author: Toby Mao
@@ -19,7 +19,7 @@ License-File: LICENSE
 
 # SQLGlot
 
-SQLGlot is a no dependency Python SQL parser and transpiler. It can be used to format SQL or translate between different dialects like Presto, Spark, and Hive. It aims to read a wide variety of SQL inputs and output syntatically correct SQL in the targeted dialects.
+SQLGlot is a no dependency Python SQL parser, transpiler, and optimizer. It can be used to format SQL or translate between different dialects like Presto, Spark, and Hive. It aims to read a wide variety of SQL inputs and output syntatically correct SQL in the targeted dialects.
 
 It is currently the [fastest](#benchmarks) pure-Python SQL parser.
 
@@ -75,100 +75,45 @@ sqlglot.transpile(sql, write='spark', identify=True, pretty=True)[0]
 ```
 
 ```sql
-WITH baz AS (
-    SELECT
-      `a`,
-      `c`
-    FROM `foo`
-    WHERE
-      `a` = 1
+WITH `baz` AS (
+  SELECT
+    `a`,
+    `c`
+  FROM `foo`
+  WHERE
+    `a` = 1
 )
 SELECT
   `f`.`a`,
   `b`.`b`,
   `baz`.`c`,
-  CAST(`b`.`a` AS FLOAT) AS d
-FROM `foo` AS f
-JOIN `bar` AS b ON
-  `f`.`a` = `b`.`a`
-LEFT JOIN `baz` ON
-  `f`.`a` = `baz`.`a`
+  CAST(`b`.`a` AS FLOAT) AS `d`
+FROM `foo` AS `f`
+JOIN `bar` AS `b`
+  ON `f`.`a` = `b`.`a`
+LEFT JOIN `baz`
+  ON `f`.`a` = `baz`.`a`
 ```
 
-### Customization
-#### Custom Types
-A simple transform on types can be accomplished by providing a corresponding mapping:
-```python
-
-from sqlglot import *
-from sqlglot import expressions as exp
-
-transpile("SELECT CAST(a AS INT) FROM x", type_mapping={exp.DataType.Type.INT: "SPECIAL INT"})[0]
-```
-
-```sql
-SELECT CAST(a AS SPECIAL INT) FROM x
-```
+### Metadata
 
-More complicated transforms can be accomplished by using the Tokenizer, Parser, and Generator directly.
-#### Custom Functions
-In  this example, we want to parse a UDF SPECIAL_UDF and then output another version called SPECIAL_UDF_INVERSE with the arguments switched.
+You can explore SQL with expression helpers to do things like find columns and tables.
 
 ```python
-from sqlglot import *
-from sqlglot.expressions import Func
+from sqlglot import parse_one, exp
 
-class SpecialUdf(Func):
-    arg_types = {'a': True, 'b': True}
+# print all column references (a and b)
+for column in parse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Column):
+  print(column.alias_or_name)
 
-tokens = Tokenizer().tokenize("SELECT SPECIAL_UDF(a, b) FROM x")
-```
-Here is the output of the tokenizer:
+# find all projections in select statements (a and c)
+for select in parse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Select):
+  for projection in select.args["expressions"]:
+    print(projection.alias_or_name)
 
-```
-[
-    <Token token_type: TokenType.SELECT, text: SELECT, line: 0, col: 0>,
-    <Token token_type: TokenType.VAR, text: SPECIAL_UDF, line: 0, col: 7>,
-    <Token token_type: TokenType.L_PAREN, text: (, line: 0, col: 18>,
-    <Token token_type: TokenType.VAR, text: a, line: 0, col: 19>,
-    <Token token_type: TokenType.COMMA, text: ,, line: 0, col: 20>,
-    <Token token_type: TokenType.VAR, text: b, line: 0, col: 22>,
-    <Token token_type: TokenType.R_PAREN, text: ), line: 0, col: 23>,
-    <Token token_type: TokenType.FROM, text: FROM, line: 0, col: 25>,
-    <Token token_type: TokenType.VAR, text: x, line: 0, col: 30>,
-]
-
-```
-```python
-expression = Parser(functions={
-    **SpecialUdf.default_parser_mappings(),
-}).parse(tokens)[0]
-```
-
-The expression tree produced by the parser:
-
-```
-(SELECT distinct: False, expressions:
-  (SPECIALUDF a:
-    (COLUMN this:
-      (IDENTIFIER this: a, quoted: False)), b:
-    (COLUMN this:
-      (IDENTIFIER this: b, quoted: False))), from:
-  (FROM expressions:
-    (TABLE this:
-      (IDENTIFIER this: x, quoted: False))))
-```
-
-Finally generating the new SQL:
-
-```python
-Generator(transforms={
-    SpecialUdf: lambda self, e: f"SPECIAL_UDF_INVERSE({self.sql(e, 'b')}, {self.sql(e, 'a')})"
-}).generate(expression)
-```
-
-```sql
-SELECT SPECIAL_UDF_INVERSE(b, a) FROM x
+# find all tables (x, y, z)
+for table in parse_one("SELECT * FROM x JOIN y JOIN z").find_all(exp.Table):
+  print(table.name)
 ```
 
 ### Parser Errors
@@ -177,8 +122,8 @@ A syntax error will result in a parser error.
 transpile("SELECT foo( FROM bar")
 ```
 ```
-sqlglot.errors.ParseError: Expected )
-  SELECT foo( __FROM__ bar
+sqlglot.errors.ParseError: Expecting ). Line 1, Col: 13.
+  SELECT foo( FROM bar
 ```
 ### Unsupported Errors
 Presto APPROX_DISTINCT supports the accuracy argument which is not supported in Spark.
@@ -226,14 +171,13 @@ SELECT x FROM y, z
 There is also a way to recursively transform the parsed tree by applying a mapping function to each tree node:
 
 ```python
-import sqlglot
-import sqlglot.expressions as exp
+from sqlglot import exp, parse_one
 
-expression_tree = sqlglot.parse_one("SELECT a FROM x")
+expression_tree = parse_one("SELECT a FROM x")
 
 def transformer(node):
-    if isinstance(node, exp.Column) and node.text("this") == "a":
-        return sqlglot.parse_one("FUN(a)")
+    if isinstance(node, exp.Column) and node.name == "a":
+        return parse_one("FUN(a)")
     return node
 
 transformed_tree = expression_tree.transform(transformer)
@@ -244,6 +188,42 @@ Which outputs:
 SELECT FUN(a) FROM x
 ```
 
+### SQL Optimizer
+
+SQLGlot can rewrite queries into an "optimized" form. It performs a variety of [techniques](sqlglot/optimizer/optimizer.py) to create a new canonical AST. This AST can be used to standaradize queries or provide the foundations for implementing an actual engine.
+
+```python
+import sqlglot
+from sqlglot.optimizer import optimize
+
+>>>
+optimize(
+    sqlglot.parse_one("""
+    SELECT A OR (B OR (C AND D))
+    FROM x
+    WHERE Z = date '2021-01-01' + INTERVAL '1' month OR 1 = 0
+    """),
+    schema={"x": {"A": "INT", "B": "INT", "C": "INT", "D": "INT", "Z": "STRING"}}
+).sql(pretty=True)
+
+"""
+SELECT
+  (
+    "x"."A"
+    OR "x"."B"
+    OR "x"."C"
+  )
+  AND (
+    "x"."A"
+    OR "x"."B"
+    OR "x"."D"
+  ) AS "_col_0"
+FROM "x" AS "x"
+WHERE
+  "x"."Z" = CAST('2021-02-01' AS DATE)
+"""
+```
+
 ### SQL Annotations
 
 SQLGlot supports annotations in the sql expression. This is an experimental feature that is not part of any of the SQL standards but it can be useful when needing to annotate what a selected field is supposed to be. Below is an example:
@@ -255,6 +235,83 @@ SELECT
 FROM users
 ```
 
+### Customization
+#### Custom Types
+A simple transform on types can be accomplished by providing a corresponding mapping:
+```python
+
+from sqlglot import *
+
+transpile("SELECT CAST(a AS INT) FROM x", type_mapping={exp.DataType.Type.INT: "SPECIAL INT"})[0]
+```
+
+```sql
+SELECT CAST(a AS SPECIAL INT) FROM x
+```
+
+More complicated transforms can be accomplished by using the Tokenizer, Parser, and Generator directly.
+#### Custom Functions
+In  this example, we want to parse a UDF SPECIAL_UDF and then output another version called SPECIAL_UDF_INVERSE with the arguments switched.
+
+```python
+from sqlglot import *
+from sqlglot.expressions import Func
+
+class SpecialUdf(Func):
+    arg_types = {'a': True, 'b': True}
+
+tokens = Tokenizer().tokenize("SELECT SPECIAL_UDF(a, b) FROM x")
+```
+Here is the output of the tokenizer:
+
+```
+[
+    <Token token_type: TokenType.SELECT, text: SELECT, line: 0, col: 0>,
+    <Token token_type: TokenType.VAR, text: SPECIAL_UDF, line: 0, col: 7>,
+    <Token token_type: TokenType.L_PAREN, text: (, line: 0, col: 18>,
+    <Token token_type: TokenType.VAR, text: a, line: 0, col: 19>,
+    <Token token_type: TokenType.COMMA, text: ,, line: 0, col: 20>,
+    <Token token_type: TokenType.VAR, text: b, line: 0, col: 22>,
+    <Token token_type: TokenType.R_PAREN, text: ), line: 0, col: 23>,
+    <Token token_type: TokenType.FROM, text: FROM, line: 0, col: 25>,
+    <Token token_type: TokenType.VAR, text: x, line: 0, col: 30>,
+]
+
+```
+```python
+expression = Parser(functions={
+    **SpecialUdf.default_parser_mappings(),
+}).parse(tokens)[0]
+
+repr(expression)
+```
+
+The expression tree produced by the parser:
+
+```
+(SELECT distinct: False, expressions:
+  (SPECIALUDF a:
+    (COLUMN this:
+      (IDENTIFIER this: a, quoted: False)), b:
+    (COLUMN this:
+      (IDENTIFIER this: b, quoted: False))), from:
+  (FROM expressions:
+    (TABLE this:
+      (IDENTIFIER this: x, quoted: False))))
+```
+
+Finally generating the new SQL:
+
+```python
+Generator(transforms={
+    SpecialUdf: lambda self, e: f"SPECIAL_UDF_INVERSE({self.sql(e, 'b')}, {self.sql(e, 'a')})"
+}).generate(expression)
+```
+
+```sql
+SELECT SPECIAL_UDF_INVERSE(b, a) FROM x
+```
+
 ### Benchmarks
 
 [Benchmarks](benchmarks) run on Python 3.9.6 in seconds.
@@ -267,6 +324,19 @@ FROM users
 
 
 ## Run Tests and Lint
-```python -m unittest && python -m pylint sqlglot/ tests/```
+```
+pip install -r requirements.txt
+./format_code.sh
+./run_checks.sh
+```
+
+## Optional Dependencies
+SQLGlot uses [dateutil](https://github.com/dateutil/dateutil) to simplify literal timedelta expressions. The optimizer will not simplify expressions like
+
+```sql
+x + interval '1' month
+```
+
+if the module cannot be found.
 
 

{sqlglot-2.2.1 → sqlglot-3.1.0}/README.md

@@ -1,6 +1,6 @@
 # SQLGlot
 
-SQLGlot is a no dependency Python SQL parser and transpiler. It can be used to format SQL or translate between different dialects like Presto, Spark, and Hive. It aims to read a wide variety of SQL inputs and output syntatically correct SQL in the targeted dialects.
+SQLGlot is a no dependency Python SQL parser, transpiler, and optimizer. It can be used to format SQL or translate between different dialects like Presto, Spark, and Hive. It aims to read a wide variety of SQL inputs and output syntatically correct SQL in the targeted dialects.
 
 It is currently the [fastest](#benchmarks) pure-Python SQL parser.
 
@@ -56,100 +56,45 @@ sqlglot.transpile(sql, write='spark', identify=True, pretty=True)[0]
 ```
 
 ```sql
-WITH baz AS (
-    SELECT
-      `a`,
-      `c`
-    FROM `foo`
-    WHERE
-      `a` = 1
+WITH `baz` AS (
+  SELECT
+    `a`,
+    `c`
+  FROM `foo`
+  WHERE
+    `a` = 1
 )
 SELECT
   `f`.`a`,
   `b`.`b`,
   `baz`.`c`,
-  CAST(`b`.`a` AS FLOAT) AS d
-FROM `foo` AS f
-JOIN `bar` AS b ON
-  `f`.`a` = `b`.`a`
-LEFT JOIN `baz` ON
-  `f`.`a` = `baz`.`a`
+  CAST(`b`.`a` AS FLOAT) AS `d`
+FROM `foo` AS `f`
+JOIN `bar` AS `b`
+  ON `f`.`a` = `b`.`a`
+LEFT JOIN `baz`
+  ON `f`.`a` = `baz`.`a`
 ```
 
-### Customization
-#### Custom Types
-A simple transform on types can be accomplished by providing a corresponding mapping:
-```python
-
-from sqlglot import *
-from sqlglot import expressions as exp
-
-transpile("SELECT CAST(a AS INT) FROM x", type_mapping={exp.DataType.Type.INT: "SPECIAL INT"})[0]
-```
-
-```sql
-SELECT CAST(a AS SPECIAL INT) FROM x
-```
+### Metadata
 
-More complicated transforms can be accomplished by using the Tokenizer, Parser, and Generator directly.
-#### Custom Functions
-In  this example, we want to parse a UDF SPECIAL_UDF and then output another version called SPECIAL_UDF_INVERSE with the arguments switched.
+You can explore SQL with expression helpers to do things like find columns and tables.
 
 ```python
-from sqlglot import *
-from sqlglot.expressions import Func
+from sqlglot import parse_one, exp
 
-class SpecialUdf(Func):
-    arg_types = {'a': True, 'b': True}
+# print all column references (a and b)
+for column in parse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Column):
+  print(column.alias_or_name)
 
-tokens = Tokenizer().tokenize("SELECT SPECIAL_UDF(a, b) FROM x")
-```
-Here is the output of the tokenizer:
+# find all projections in select statements (a and c)
+for select in parse_one("SELECT a, b + 1 AS c FROM d").find_all(exp.Select):
+  for projection in select.args["expressions"]:
+    print(projection.alias_or_name)
 
-```
-[
-    <Token token_type: TokenType.SELECT, text: SELECT, line: 0, col: 0>,
-    <Token token_type: TokenType.VAR, text: SPECIAL_UDF, line: 0, col: 7>,
-    <Token token_type: TokenType.L_PAREN, text: (, line: 0, col: 18>,
-    <Token token_type: TokenType.VAR, text: a, line: 0, col: 19>,
-    <Token token_type: TokenType.COMMA, text: ,, line: 0, col: 20>,
-    <Token token_type: TokenType.VAR, text: b, line: 0, col: 22>,
-    <Token token_type: TokenType.R_PAREN, text: ), line: 0, col: 23>,
-    <Token token_type: TokenType.FROM, text: FROM, line: 0, col: 25>,
-    <Token token_type: TokenType.VAR, text: x, line: 0, col: 30>,
-]
-
-```
-```python
-expression = Parser(functions={
-    **SpecialUdf.default_parser_mappings(),
-}).parse(tokens)[0]
-```
-
-The expression tree produced by the parser:
-
-```
-(SELECT distinct: False, expressions:
-  (SPECIALUDF a:
-    (COLUMN this:
-      (IDENTIFIER this: a, quoted: False)), b:
-    (COLUMN this:
-      (IDENTIFIER this: b, quoted: False))), from:
-  (FROM expressions:
-    (TABLE this:
-      (IDENTIFIER this: x, quoted: False))))
-```
-
-Finally generating the new SQL:
-
-```python
-Generator(transforms={
-    SpecialUdf: lambda self, e: f"SPECIAL_UDF_INVERSE({self.sql(e, 'b')}, {self.sql(e, 'a')})"
-}).generate(expression)
-```
-
-```sql
-SELECT SPECIAL_UDF_INVERSE(b, a) FROM x
+# find all tables (x, y, z)
+for table in parse_one("SELECT * FROM x JOIN y JOIN z").find_all(exp.Table):
+  print(table.name)
 ```
 
 ### Parser Errors
@@ -158,8 +103,8 @@ A syntax error will result in a parser error.
 transpile("SELECT foo( FROM bar")
 ```
 ```
-sqlglot.errors.ParseError: Expected )
-  SELECT foo( __FROM__ bar
+sqlglot.errors.ParseError: Expecting ). Line 1, Col: 13.
+  SELECT foo( FROM bar
 ```
 ### Unsupported Errors
 Presto APPROX_DISTINCT supports the accuracy argument which is not supported in Spark.
@@ -207,14 +152,13 @@ SELECT x FROM y, z
 There is also a way to recursively transform the parsed tree by applying a mapping function to each tree node:
 
 ```python
-import sqlglot
-import sqlglot.expressions as exp
+from sqlglot import exp, parse_one
 
-expression_tree = sqlglot.parse_one("SELECT a FROM x")
+expression_tree = parse_one("SELECT a FROM x")
 
 def transformer(node):
-    if isinstance(node, exp.Column) and node.text("this") == "a":
-        return sqlglot.parse_one("FUN(a)")
+    if isinstance(node, exp.Column) and node.name == "a":
+        return parse_one("FUN(a)")
     return node
 
 transformed_tree = expression_tree.transform(transformer)
@@ -225,6 +169,42 @@ Which outputs:
 SELECT FUN(a) FROM x
 ```
 
+### SQL Optimizer
+
+SQLGlot can rewrite queries into an "optimized" form. It performs a variety of [techniques](sqlglot/optimizer/optimizer.py) to create a new canonical AST. This AST can be used to standaradize queries or provide the foundations for implementing an actual engine.
+
+```python
+import sqlglot
+from sqlglot.optimizer import optimize
+
+>>>
+optimize(
+    sqlglot.parse_one("""
+    SELECT A OR (B OR (C AND D))
+    FROM x
+    WHERE Z = date '2021-01-01' + INTERVAL '1' month OR 1 = 0
+    """),
+    schema={"x": {"A": "INT", "B": "INT", "C": "INT", "D": "INT", "Z": "STRING"}}
+).sql(pretty=True)
+
+"""
+SELECT
+  (
+    "x"."A"
+    OR "x"."B"
+    OR "x"."C"
+  )
+  AND (
+    "x"."A"
+    OR "x"."B"
+    OR "x"."D"
+  ) AS "_col_0"
+FROM "x" AS "x"
+WHERE
+  "x"."Z" = CAST('2021-02-01' AS DATE)
+"""
+```
+
 ### SQL Annotations
 
 SQLGlot supports annotations in the sql expression. This is an experimental feature that is not part of any of the SQL standards but it can be useful when needing to annotate what a selected field is supposed to be. Below is an example:
@@ -236,6 +216,83 @@ SELECT
 FROM users
 ```
 
+### Customization
+#### Custom Types
+A simple transform on types can be accomplished by providing a corresponding mapping:
+```python
+
+from sqlglot import *
+
+transpile("SELECT CAST(a AS INT) FROM x", type_mapping={exp.DataType.Type.INT: "SPECIAL INT"})[0]
+```
+
+```sql
+SELECT CAST(a AS SPECIAL INT) FROM x
+```
+
+More complicated transforms can be accomplished by using the Tokenizer, Parser, and Generator directly.
+#### Custom Functions
+In  this example, we want to parse a UDF SPECIAL_UDF and then output another version called SPECIAL_UDF_INVERSE with the arguments switched.
+
+```python
+from sqlglot import *
+from sqlglot.expressions import Func
+
+class SpecialUdf(Func):
+    arg_types = {'a': True, 'b': True}
+
+tokens = Tokenizer().tokenize("SELECT SPECIAL_UDF(a, b) FROM x")
+```
+Here is the output of the tokenizer:
+
+```
+[
+    <Token token_type: TokenType.SELECT, text: SELECT, line: 0, col: 0>,
+    <Token token_type: TokenType.VAR, text: SPECIAL_UDF, line: 0, col: 7>,
+    <Token token_type: TokenType.L_PAREN, text: (, line: 0, col: 18>,
+    <Token token_type: TokenType.VAR, text: a, line: 0, col: 19>,
+    <Token token_type: TokenType.COMMA, text: ,, line: 0, col: 20>,
+    <Token token_type: TokenType.VAR, text: b, line: 0, col: 22>,
+    <Token token_type: TokenType.R_PAREN, text: ), line: 0, col: 23>,
+    <Token token_type: TokenType.FROM, text: FROM, line: 0, col: 25>,
+    <Token token_type: TokenType.VAR, text: x, line: 0, col: 30>,
+]
+
+```
+```python
+expression = Parser(functions={
+    **SpecialUdf.default_parser_mappings(),
+}).parse(tokens)[0]
+
+repr(expression)
+```
+
+The expression tree produced by the parser:
+
+```
+(SELECT distinct: False, expressions:
+  (SPECIALUDF a:
+    (COLUMN this:
+      (IDENTIFIER this: a, quoted: False)), b:
+    (COLUMN this:
+      (IDENTIFIER this: b, quoted: False))), from:
+  (FROM expressions:
+    (TABLE this:
+      (IDENTIFIER this: x, quoted: False))))
+```
+
+Finally generating the new SQL:
+
+```python
+Generator(transforms={
+    SpecialUdf: lambda self, e: f"SPECIAL_UDF_INVERSE({self.sql(e, 'b')}, {self.sql(e, 'a')})"
+}).generate(expression)
+```
+
+```sql
+SELECT SPECIAL_UDF_INVERSE(b, a) FROM x
+```
+
 ### Benchmarks
 
 [Benchmarks](benchmarks) run on Python 3.9.6 in seconds.
@@ -248,4 +305,17 @@ FROM users
 
 
 ## Run Tests and Lint
-```python -m unittest && python -m pylint sqlglot/ tests/```
+```
+pip install -r requirements.txt
+./format_code.sh
+./run_checks.sh
+```
+
+## Optional Dependencies
+SQLGlot uses [dateutil](https://github.com/dateutil/dateutil) to simplify literal timedelta expressions. The optimizer will not simplify expressions like
+
+```sql
+x + interval '1' month
+```
+
+if the module cannot be found.

{sqlglot-2.2.1 → sqlglot-3.1.0}/setup.py

@@ -1,4 +1,4 @@
-from setuptools import setup
+from setuptools import setup, find_packages
 
 version = (
     open("sqlglot/__init__.py")
@@ -20,7 +20,7 @@ setup(
     author="Toby Mao",
     author_email="toby.mao@gmail.com",
     license="MIT",
-    packages=["sqlglot"],
+    packages=find_packages(include=["sqlglot", "sqlglot.*"]),
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Intended Audience :: Developers",