sqlframe 1.7.0__tar.gz → 1.8.0__tar.gz

{sqlframe-1.7.0 → sqlframe-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: sqlframe
-Version: 1.7.0
+Version: 1.8.0
 Summary: Turning PySpark Into a Universal DataFrame API
 Home-page: https://github.com/eakmanrq/sqlframe
 Author: Ryan Eakman

sqlframe-1.8.0/blogs/add_chatgpt_support.md

@@ -0,0 +1,101 @@
+# Adding ChatGPT to SQLFrame - What is the Right Amount of AI?
+
+<div align="center">
+  <img src="images/add_chatgpt_support/adding_ai_to_meal.jpeg" alt="Adding AI to Meal" width="800"/>
+</div>
+
+Like many people today, I often grapple with how much AI to integrate into my daily tasks. 
+In my experience, AI can be a valuable tool for brainstorming when writing, but over-reliance on it can lead to generic text devoid of personality. 
+While I might be overly verbose and fond of metaphors, these quirks add character and make communication more engaging. 
+Take "Marvin the Paranoid Robot" from "Hitchhiker's Guide to the Galaxy," for instance. 
+His melancholic outlook makes him relatable to the audience, even though his intelligence is 50,000 times greater than ours.
+
+<div align="center">
+  <img src="images/add_chatgpt_support/marvin_paranoid_robot.gif" alt="Sad Robot" width="800"/>
+</div>
+
+When it comes to coding, I strive to eliminate imperfections as much as possible. 
+However, AI's tendency to hallucinate makes even a "trust but verify" approach challenging. 
+More often than not, especially when working with new libraries or languages, I find myself in a "that just can't be right, so let's move on" mindset.
+
+<div align="center">
+  <img src="images/add_chatgpt_support/sunny_shake_head_no.gif" alt="Shake Head No" width="800"/>
+</div>
+
+When I first considered integrating AI into SQLFrame, it immediately struck me as a bad idea. 
+Correctness and consistency are crucial attributes of the product. 
+It's one thing to say this, but I've backed it up by writing around 1,800 tests to cover PySpark functionality and ensure reliability. 
+Until AI improves significantly in this area, SQLFrame will continue to rely on and enhance its understanding of PySpark, ensuring it's available where people need it most.
+
+I've received feedback that the SQL generated by SQLFrame can be hard to read, which is much appreciated feedback. 
+I realized that while my primary goal is to produce reliable SQL that runs consistently on my engine, readability is also crucial when sharing code with others. 
+Minor imperfections in the SQL are acceptable as long as they don't impede understanding. Large Language Models (LLMs) excel at understanding human communication and have been trained to interpret user intent. 
+Could SQLFrame leverage this capability? 
+Is it time for SQLFrame to join the AI hype train?
+
+<div align="center">
+  <img src="images/add_chatgpt_support/hype_train.gif" alt="All aboard the train" width="800"/>
+</div>
+
+Starting with [1.4.0](https://github.com/eakmanrq/sqlframe), you can now have SQLFrame enrich generated SQL using OpenAI. 
+Provide your API key and include `df.sql(openai_config={})` (to use defaults, [config options here](https://sqlframe.readthedocs.io/en/stable/configuration/#optimized)) when generating SQL to see what it looks like. 
+Let's take a look at the example from the README.
+
+```python
+from sqlframe.bigquery import BigQuerySession
+from sqlframe.bigquery import functions as F
+from sqlframe.bigquery import Window
+
+session = BigQuerySession()
+table_path = "bigquery-public-data.samples.natality"
+# Top 5 years with the greatest year-over-year % change in new families with single child
+df = (
+    session.table(table_path)
+    .where(F.col("ever_born") == 1)
+    .groupBy("year")
+    .agg(F.count("*").alias("num_single_child_families"))
+    .withColumn(
+        "last_year_num_single_child_families", 
+        F.lag(F.col("num_single_child_families"), 1).over(Window.orderBy("year"))
+    )
+    .withColumn(
+        "percent_change", 
+        (F.col("num_single_child_families") - F.col("last_year_num_single_child_families")) 
+        / F.col("last_year_num_single_child_families")
+    )
+    .orderBy(F.abs(F.col("percent_change")).desc())
+    .select(
+        F.col("year").alias("year"),
+        F.format_number("num_single_child_families", 0).alias("new families single child"),
+        F.format_number(F.col("percent_change") * 100, 2).alias("percent change"),
+    )
+    .limit(5)
+)
+```
+
+Next we will compare SQLGlot optimized, OpenAI replacing SQLGlot optimized CTE names, and then OpenAI doing a full rewrite.
+
+<div align="center">
+  <img src="images/add_chatgpt_support/sqlglot_optimized_code.png" alt="SQLGlot optimized code" width="800"/>
+</div>
+
+<div align="center">
+  <img src="images/add_chatgpt_support/openai_replacing_cte_names.png" alt="OpenAI Replacing CTE Names" width="800"/>
+</div>
+
+<div align="center">
+  <img src="images/add_chatgpt_support/openai_full_rewrite.png" alt="OpenAI Full Rewrite" width="800"/>
+</div>
+
+Overall, the results are promising and make me optimistic that SQLFrame can generate human-like SQL. 
+I believe the ideal approach would be to combine tuned SQLGlot optimization rules with OpenAI for tasks such as naming CTEs. 
+In the long term, it wouldn't surprise me if models are able to consistently convert accurate SQL into very human-friendly formats without needing any optimization assistance. 💪
+If you are new to SQLFrame, checkout the [repo](https://github.com/eakmanrq/sqlframe) or [announcement blog post](https://towardsdev.com/sqlframe-turning-pyspark-into-a-universal-dataframe-api-e06a1c678f35). 🚀 
+If you want to see how I added this integration, checkout [part 1](https://www.loom.com/share/3fab8493818b46f7bff03faf101d2bb6?sid=ffc6a134-f465-478a-a453-7c69b296dc77) or [part 2](https://www.loom.com/share/4055e86dafdd406283d7ba0e0656f0c9?sid=ff0e2af4-1f5a-4760-bf10-9feb89ef9451) where I added this feature ([part 2](https://www.loom.com/share/4055e86dafdd406283d7ba0e0656f0c9?sid=ff0e2af4-1f5a-4760-bf10-9feb89ef9451) is shorter and I think better). 🍿
+Final treat, I overrode the normal prompt with "turn the SQL into complete nonsense". The results:
+
+<div align="center">
+  <img src="images/add_chatgpt_support/nonsense_sql.png" alt="OpenAI Full Rewrite" width="800"/>
+</div>
+
+Nailed it.

{sqlframe-1.7.0 → sqlframe-1.8.0}/docs/spark.md

@@ -310,6 +310,7 @@ df.show(5)
 * [lpad](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.lpad.html)
 * [ltrim](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.ltrim.html)
 * [make_date](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.make_date.html)
+* [make_interval](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.make_interval.html)
 * [map_concat](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.map_concat.html)
 * [map_entries](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.map_entries.html)
 * [map_filter](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.map_filter.html)
@@ -356,6 +357,8 @@ df.show(5)
 * [row_number](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.row_number.html)
 * [rpad](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.rpad.html)
 * [rtrim](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.rtrim.html)
+* [schema_of_csv](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.schema_of_csv.html)
+* [schema_of_json](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.schema_of_json.html)
 * [sec](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.sec.html)
 * [second](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.second.html)
 * [sentences](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.sentences.html)
@@ -375,6 +378,7 @@ df.show(5)
 * [size](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.size.html)
 * [skewness](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.skewness.html)
 * [slice](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.slice.html)
+* [stack](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.stack.html)
 * [sort_array](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.sort_array.html)
 * [soundex](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.soundex.html)
 * [split](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.split.html)
@@ -405,6 +409,14 @@ df.show(5)
 * [translate](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.translate.html)
 * [trim](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.trim.html)
 * [trunc](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.trunc.html)
+* [try_add](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_add.html)
+* [try_avg](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_avg.html)
+* [try_divide](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_divide.html)
+* [try_multiply](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_multiply.html)
+* [try_subtract](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_subtract.html)
+* [try_sum](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_sum.html)
+* [try_to_binary](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_to_binary.html)
+* [try_to_number](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.try_to_number.html)
 * [typeof](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.typeof.html)
 * [unbase64](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.unbase64.html)
 * [unhex](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.unhex.html)

{sqlframe-1.7.0 → sqlframe-1.8.0}/setup.py

@@ -21,6 +21,7 @@ setup(
     install_requires=[
         "prettytable<3.11.0",
         "sqlglot>=24.0.0,<25.1",
+        "typing_extensions>=4.8,<5",
     ],
     extras_require={
         "bigquery": [
@@ -42,7 +43,6 @@ setup(
             "pre-commit>=3.5;python_version=='3.8'",
             "pre-commit>=3.7,<3.8;python_version>='3.9'",
             "ruff>=0.4.4,<0.5",
-            "typing_extensions>=4.11,<5",
             "types-psycopg2>=2.9,<3",
         ],
         "docs": [

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe/_version.py

@@ -12,5 +12,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '1.7.0'
-__version_tuple__ = version_tuple = (1, 7, 0)
+__version__ = version = '1.8.0'
+__version_tuple__ = version_tuple = (1, 8, 0)

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe/base/functions.py

@@ -151,9 +151,10 @@ def sumDistinct(col: ColumnOrName) -> Column:
 sum_distinct = sumDistinct
 
 
-@meta(unsupported_engines="*")
-def product(col: ColumnOrName) -> Column:
-    raise NotImplementedError("Product is not currently implemented")
+# Product does not have a SQL function available
+# @meta(unsupported_engines="*")
+# def product(col: ColumnOrName) -> Column:
+# raise NotImplementedError("Product is not currently implemented")
 
 
 @meta()
@@ -1430,6 +1431,8 @@ def to_json(col: ColumnOrName, options: t.Optional[t.Dict[str, str]] = None) ->
 
 @meta(unsupported_engines="*")
 def schema_of_json(col: ColumnOrName, options: t.Optional[t.Dict[str, str]] = None) -> Column:
+    if isinstance(col, str):
+        col = lit(col)
     if options is not None:
         options_col = create_map([lit(x) for x in _flatten(options.items())])
         return Column.invoke_anonymous_function(col, "SCHEMA_OF_JSON", options_col)
@@ -1438,6 +1441,8 @@ def schema_of_json(col: ColumnOrName, options: t.Optional[t.Dict[str, str]] = No
 
 @meta(unsupported_engines="*")
 def schema_of_csv(col: ColumnOrName, options: t.Optional[t.Dict[str, str]] = None) -> Column:
+    if isinstance(col, str):
+        col = lit(col)
     if options is not None:
         options_col = create_map([lit(x) for x in _flatten(options.items())])
         return Column.invoke_anonymous_function(col, "SCHEMA_OF_CSV", options_col)
@@ -1560,7 +1565,9 @@ def from_csv(
 ) -> Column:
     schema = schema if isinstance(schema, Column) else lit(schema)
     if options is not None:
-        option_cols = create_map([lit(x) for x in _flatten(options.items())])
+        option_cols = create_map(
+            [lit(str(x) if isinstance(x, bool) else x) for x in _flatten(options.items())]
+        )
         return Column.invoke_anonymous_function(col, "FROM_CSV", schema, option_cols)
     return Column.invoke_anonymous_function(col, "FROM_CSV", schema)
 
@@ -1667,6 +1674,79 @@ def nullif(col1: ColumnOrName, col2: ColumnOrName) -> Column:
     return Column.invoke_expression_over_column(col1, expression.Nullif, expression=col2)
 
 
+@meta(unsupported_engines="*")
+def stack(*cols: ColumnOrName) -> Column:
+    columns = [Column.ensure_col(x) for x in cols]
+    return Column.invoke_anonymous_function(
+        columns[0], "STACK", *columns[1:] if len(columns) > 1 else []
+    )
+
+
+@meta(unsupported_engines="*")
+def make_interval(
+    years: t.Optional[ColumnOrName] = None,
+    months: t.Optional[ColumnOrName] = None,
+    weeks: t.Optional[ColumnOrName] = None,
+    days: t.Optional[ColumnOrName] = None,
+    hours: t.Optional[ColumnOrName] = None,
+    mins: t.Optional[ColumnOrName] = None,
+    secs: t.Optional[ColumnOrName] = None,
+) -> Column:
+    values = [years, months, weeks, days, hours, mins, secs]
+    for value in reversed(values.copy()):
+        if value is not None:
+            break
+        values = values[:-1]
+    else:
+        raise ValueError("At least one value must be provided")
+    columns = [Column.ensure_col(x) if x is not None else lit(None) for x in values]
+    return Column.invoke_anonymous_function(columns[0], "MAKE_INTERVAL", *columns[1:])
+
+
+@meta(unsupported_engines="*")
+def try_add(left: ColumnOrName, right: ColumnOrName) -> Column:
+    return Column.invoke_anonymous_function(left, "TRY_ADD", right)
+
+
+@meta(unsupported_engines="*")
+def try_avg(col: ColumnOrName) -> Column:
+    return Column.invoke_anonymous_function(col, "TRY_AVG")
+
+
+@meta(unsupported_engines="*")
+def try_divide(left: ColumnOrName, right: ColumnOrName) -> Column:
+    return Column.invoke_anonymous_function(left, "TRY_DIVIDE", right)
+
+
+@meta(unsupported_engines="*")
+def try_multiply(left: ColumnOrName, right: ColumnOrName) -> Column:
+    return Column.invoke_anonymous_function(left, "TRY_MULTIPLY", right)
+
+
+@meta(unsupported_engines="*")
+def try_subtract(left: ColumnOrName, right: ColumnOrName) -> Column:
+    return Column.invoke_anonymous_function(left, "TRY_SUBTRACT", right)
+
+
+@meta(unsupported_engines="*")
+def try_sum(col: ColumnOrName) -> Column:
+    return Column.invoke_anonymous_function(col, "TRY_SUM")
+
+
+@meta(unsupported_engines="*")
+def try_to_binary(col: ColumnOrName, format: t.Optional[ColumnOrName] = None) -> Column:
+    if format is not None:
+        return Column.invoke_anonymous_function(col, "TRY_TO_BINARY", format)
+    return Column.invoke_anonymous_function(col, "TRY_TO_BINARY")
+
+
+@meta(unsupported_engines="*")
+def try_to_number(col: ColumnOrName, format: t.Optional[ColumnOrName] = None) -> Column:
+    if format is not None:
+        return Column.invoke_anonymous_function(col, "TRY_TO_NUMBER", format)
+    return Column.invoke_anonymous_function(col, "TRY_TO_NUMBER")
+
+
 @meta()
 def _lambda_quoted(value: str) -> t.Optional[bool]:
     return False if value == "_" else None

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe/base/session.py

@@ -569,6 +569,7 @@ class _BaseSession(t.Generic[CATALOG, READER, WRITER, DF, CONN]):
             self,
             key: t.Optional[str] = None,
             value: t.Optional[t.Any] = None,
+            conf: t.Optional[t.Any] = None,
             *,
             map: t.Optional[t.Dict[str, t.Any]] = None,
         ) -> Self:

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe/spark/catalog.py

@@ -519,7 +519,10 @@ class SparkCatalog(
                 )
                 for col in df.columns
             ]
-        return [Column(*x) for x in self._spark_catalog.listColumns(tableName, dbName)]
+        return [
+            Column(**{name: x._asdict()[name] for name in Column._fields})
+            for x in self._spark_catalog.listColumns(tableName, dbName)
+        ]
 
     def listFunctions(
         self, dbName: t.Optional[str] = None, pattern: t.Optional[str] = None

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe/spark/functions.py

@@ -8,9 +8,7 @@ globals().update(
     {
         name: func
         for name, func in inspect.getmembers(module, inspect.isfunction)
-        if hasattr(func, "unsupported_engines")
-        and "spark" not in func.unsupported_engines
-        and "*" not in func.unsupported_engines
+        if hasattr(func, "unsupported_engines") and "spark" not in func.unsupported_engines
     }
 )
 

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe/spark/functions.pyi

@@ -132,6 +132,7 @@ from sqlframe.base.functions import (
     lpad as lpad,
     ltrim as ltrim,
     make_date as make_date,
+    make_interval as make_interval,
     map_concat as map_concat,
     map_entries as map_entries,
     map_filter as map_filter,
@@ -177,6 +178,8 @@ from sqlframe.base.functions import (
     row_number as row_number,
     rpad as rpad,
     rtrim as rtrim,
+    schema_of_csv as schema_of_csv,
+    schema_of_json as schema_of_json,
     sec as sec,
     second as second,
     sentences as sentences,
@@ -200,6 +203,7 @@ from sqlframe.base.functions import (
     soundex as soundex,
     split as split,
     sqrt as sqrt,
+    stack as stack,
     stddev as stddev,
     stddev_pop as stddev_pop,
     stddev_samp as stddev_samp,
@@ -225,6 +229,14 @@ from sqlframe.base.functions import (
     translate as translate,
     trim as trim,
     trunc as trunc,
+    try_add as try_add,
+    try_avg as try_avg,
+    try_divide as try_divide,
+    try_multiply as try_multiply,
+    try_subtract as try_subtract,
+    try_sum as try_sum,
+    try_to_binary as try_to_binary,
+    try_to_number as try_to_number,
     typeof as typeof,
     unbase64 as unbase64,
     unhex as unhex,

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: sqlframe
-Version: 1.7.0
+Version: 1.8.0
 Summary: Turning PySpark Into a Universal DataFrame API
 Home-page: https://github.com/eakmanrq/sqlframe
 Author: Ryan Eakman

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe.egg-info/SOURCES.txt

@@ -12,10 +12,19 @@ setup.py
 .github/CODEOWNERS
 .github/workflows/main.workflow.yaml
 .github/workflows/publish.workflow.yaml
+blogs/add_chatgpt_support.md
 blogs/sqlframe_universal_dataframe_api.md
 blogs/images/but_wait_theres_more.gif
 blogs/images/cake.gif
 blogs/images/you_get_pyspark_api.gif
+blogs/images/add_chatgpt_support/adding_ai_to_meal.jpeg
+blogs/images/add_chatgpt_support/hype_train.gif
+blogs/images/add_chatgpt_support/marvin_paranoid_robot.gif
+blogs/images/add_chatgpt_support/nonsense_sql.png
+blogs/images/add_chatgpt_support/openai_full_rewrite.png
+blogs/images/add_chatgpt_support/openai_replacing_cte_names.png
+blogs/images/add_chatgpt_support/sqlglot_optimized_code.png
+blogs/images/add_chatgpt_support/sunny_shake_head_no.gif
 docs/bigquery.md
 docs/configuration.md
 docs/duckdb.md

{sqlframe-1.7.0 → sqlframe-1.8.0}/sqlframe.egg-info/requires.txt

@@ -1,5 +1,6 @@
 prettytable<3.11.0
 sqlglot<25.1,>=24.0.0
+typing_extensions<5,>=4.8
 
 [bigquery]
 google-cloud-bigquery-storage<3,>=2
@@ -19,7 +20,6 @@ pytest-xdist<3.7,>=3.6
 pytest<8.3,>=8.2.0
 ruff<0.5,>=0.4.4
 types-psycopg2<3,>=2.9
-typing_extensions<5,>=4.11
 
 [dev:python_version == "3.8"]
 pre-commit>=3.5

{sqlframe-1.7.0 → sqlframe-1.8.0}/tests/integration/engines/bigquery/test_bigquery_session.py

@@ -16,5 +16,5 @@ def test_session_from_config():
     conn.cursor().execute("CREATE SCHEMA IF NOT EXISTS db1")
     conn.cursor().execute("CREATE TABLE IF NOT EXISTS db1.test_table (cola INT, colb STRING)")
     session = BigQuerySession.builder.config("default_dataset", "sqlframe.db1").getOrCreate()
-    columns = session.catalog.get_columns("test_table")
+    columns = session.catalog.get_columns("db1.test_table")
     assert columns == {"`cola`": exp.DataType.build("BIGINT"), "`colb`": exp.DataType.build("TEXT")}