jupyter-duckdb 0.9.2.2.dev202401171015__tar.gz → 1.4.106__tar.gz

{jupyter-duckdb-0.9.2.2.dev202401171015/src/jupyter_duckdb.egg-info → jupyter_duckdb-1.4.106}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: jupyter-duckdb
-Version: 0.9.2.2.dev202401171015
+Version: 1.4.106
 Summary: a basic wrapper kernel for DuckDB
 Home-page: https://github.com/erictroebs/jupyter-duckdb
 Author: Eric Tröbs
@@ -12,9 +12,19 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 Requires-Dist: jupyter
-Requires-Dist: graphviz==0.20.1
+Requires-Dist: graphviz==0.21
 Requires-Dist: checkmarkandcross
-Requires-Dist: duckdb==0.9.2
+Requires-Dist: duckdb==1.4.1
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
 
 # DuckDB Kernel for Jupyter
 
@@ -22,10 +32,6 @@ This is a simple DuckDB wrapper kernel which accepts SQL as input, executes it
 using a previously loaded DuckDB instance and formats the output as a table.
 There are some magic commands that make teaching easier with this kernel.
 
-## Quick Start
-
-[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/git/https%3A%2F%2Fdbgit.prakinf.tu-ilmenau.de%2Fertr8623%2Fjupyter-duckdb.git/master)
-
 ## Table of Contents
 
 - [Setup](#setup)
@@ -39,6 +45,7 @@ There are some magic commands that make teaching easier with this kernel.
     - [Ship Tests With Your Notebook](#ship-tests-with-your-notebooks)
     - [Relational Algebra](#relational-algebra)
     - [Domain Calculus](#domain-calculus)
+    - [Automated Parser Selection](#automated-parser-selection)
 
 ## Setup
 
@@ -75,6 +82,12 @@ Execute the following command to pull and run a prepared image.
 docker run -p 8888:8888 troebs/jupyter-duckdb
 ```
 
+There is also a second image. It contains an additional instance of PostgreSQL:
+
+```bash
+docker run -p 8888:8888 troebs/jupyter-duckdb:postgresql
+```
+
 This image can also be used with JupyterHub and the
 [DockerSpawner / SwarmSpawner](https://github.com/jupyterhub/dockerspawner)
 and probably with the
@@ -106,28 +119,36 @@ another DuckDB file or a file with SQL statements. In the first case the
 included tables will be copied to the new database, while in the second case the
 SQL statements are just executed. We find this feature very useful to work in a
 temporary copy of the data and therefore be able to restart at any time. The
-last optional parameter `WITH_TESTS` is described in
-detail [below](#ship-tests-with-your-notebooks).
+optional parameter `NAME` may be used to name a connection and reference it
+later by using the magic command `USE`.
 
 ```
 %CREATE data.duckdb OF my_statements.sql
 ```
 
 `LOAD` on the other hand loads an existing database and returns an error if it
-does not exist. (That is why `OF` cannot be used with `LOAD`! `WITH_TESTS` on
-the other hand is available also with this magic command.)
+does not exist. (That is why `OF` cannot be used with `LOAD`! `NAME` on the
+other hand is available also with this magic command.)
 
 ```
 %LOAD data.duckdb
 ```
 
-Only one database can be open at any time. If a new database is created or
-loaded, the current one is closed first and saved to disk if necessary.
+Multiple databases can be open at any time. If a new database with the same
+name is created or loaded, the current one is closed first and saved to disk
+if necessary.
 
 Please note that `:memory:` is also a valid file path for DuckDB. The data is
 then stored exclusively in the main memory. In combination with `CREATE`
 and `OF` this makes it possible to work on a temporary copy in memory.
 
+Although the name suggests otherwise, the kernel can also be used with other
+databases:
+- **SQLite** is automatically used as a fallback if the DuckDB dependency is
+  missing.
+- To connect to a **PostgreSQL** instance, you need to specify a database URI
+  starting with `(postgresql|postgres|pgsql|psql|pg)://`.
+
 ### Schema Diagrams
 
 The magic command `SCHEMA` can be used to create a simple schema diagram of the
@@ -143,6 +164,10 @@ representation requires more space, but can improve readability.
 %SCHEMA TD
 ```
 
+The optional argument `ONLY`, followed by one or more table names separated by a
+comma, can be used to display only the named tables and all those connected with
+a foreign key.
+
 Graphviz (`dot` in PATH) is required to render schema diagrams.
 
 ### Number of Rows
@@ -190,10 +215,10 @@ FROM bar
 
 ### Ship Tests With Your Notebooks
 
-Simple tests can be loaded together with the database with the help of
-the `WITH_TESTS` parameter. These tests are stored as a JSON file. Each test is
-assigned a unique name, a result set and whether the test should check the order
-of the result. A very simple test file looks like the following JSON object:
+Simple tests can be loaded from json files with the help of magic command
+`LOAD_TESTS`. These tests are stored as a JSON file. Each test is assigned a
+unique name, a result set and whether the test should check the order of the
+result. A very simple test file looks like the following JSON object:
 
 ```json
 {
@@ -224,6 +249,11 @@ UNION
 SELECT 1, 'Name 1'
 ```
 
+By default, failed tests will display an explanation, but the notebook will
+continue to run. Set the `DUCKDB_TESTS_RAISE_EXCEPTION` environment variable to
+`true` to raise an exception when a test fails. This can be useful for automated
+testing in CI environments.
+
 Disclaimer: The integrated testing is work-in-progress and thus subject to
 potentially incompatible changes and enhancements.
 
@@ -234,7 +264,7 @@ magic command `RA` activates the relational algebra mode for a single cell:
 
 ```
 %RA
-π a, b (σ c = 1 (R))
+π [a, b] (σ [c = 1] (R))
 ```
 
 The supported operations are:
@@ -247,6 +277,12 @@ The supported operations are:
 - Difference `\`
 - Natural Join `⋈`
 - Cross Product `×`
+- Division `÷`
+
+The optional flag `ANALYZE` can be used to add an execution diagram to the
+output.
+
+You can also add comments to queries using `--` or `/* */`, just like in SQL.
 
 The Dockerfile also installs the Jupyter Lab plugin
 [jupyter-ra-extension](https://pypi.org/project/jupyter-ra-extension/). It adds
@@ -262,3 +298,12 @@ magic command `DC` activates the domain calculus mode for a single cell:
 %DC
 { a, b | R(a, b, c) ∧ c = 1 }
 ```
+
+### Automated Parser Selection
+
+`%ALL_RA` or `%ALL_DC` enables the corresponding parser for all subsequently
+executed cells.
+
+If the magic command `%AUTO_PARSER` is added to a cell, a parser is
+automatically selected. If `%GUESS_PARSER` is executed, the parser is
+automatically selected for all subsequent cells.

jupyter-duckdb-0.9.2.2.dev202401171015/PKG-INFO → jupyter_duckdb-1.4.106/README.md

@@ -1,31 +1,9 @@
-Metadata-Version: 2.1
-Name: jupyter-duckdb
-Version: 0.9.2.2.dev202401171015
-Summary: a basic wrapper kernel for DuckDB
-Home-page: https://github.com/erictroebs/jupyter-duckdb
-Author: Eric Tröbs
-Author-email: eric.troebs@tu-ilmenau.de
-Project-URL: Bug Tracker, https://github.com/erictroebs/jupyter-duckdb/issues
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-Requires-Dist: jupyter
-Requires-Dist: graphviz==0.20.1
-Requires-Dist: checkmarkandcross
-Requires-Dist: duckdb==0.9.2
-
 # DuckDB Kernel for Jupyter
 
 This is a simple DuckDB wrapper kernel which accepts SQL as input, executes it
 using a previously loaded DuckDB instance and formats the output as a table.
 There are some magic commands that make teaching easier with this kernel.
 
-## Quick Start
-
-[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/git/https%3A%2F%2Fdbgit.prakinf.tu-ilmenau.de%2Fertr8623%2Fjupyter-duckdb.git/master)
-
 ## Table of Contents
 
 - [Setup](#setup)
@@ -39,6 +17,7 @@ There are some magic commands that make teaching easier with this kernel.
     - [Ship Tests With Your Notebook](#ship-tests-with-your-notebooks)
     - [Relational Algebra](#relational-algebra)
     - [Domain Calculus](#domain-calculus)
+    - [Automated Parser Selection](#automated-parser-selection)
 
 ## Setup
 
@@ -75,6 +54,12 @@ Execute the following command to pull and run a prepared image.
 docker run -p 8888:8888 troebs/jupyter-duckdb
 ```
 
+There is also a second image. It contains an additional instance of PostgreSQL:
+
+```bash
+docker run -p 8888:8888 troebs/jupyter-duckdb:postgresql
+```
+
 This image can also be used with JupyterHub and the
 [DockerSpawner / SwarmSpawner](https://github.com/jupyterhub/dockerspawner)
 and probably with the
@@ -106,28 +91,36 @@ another DuckDB file or a file with SQL statements. In the first case the
 included tables will be copied to the new database, while in the second case the
 SQL statements are just executed. We find this feature very useful to work in a
 temporary copy of the data and therefore be able to restart at any time. The
-last optional parameter `WITH_TESTS` is described in
-detail [below](#ship-tests-with-your-notebooks).
+optional parameter `NAME` may be used to name a connection and reference it
+later by using the magic command `USE`.
 
 ```
 %CREATE data.duckdb OF my_statements.sql
 ```
 
 `LOAD` on the other hand loads an existing database and returns an error if it
-does not exist. (That is why `OF` cannot be used with `LOAD`! `WITH_TESTS` on
-the other hand is available also with this magic command.)
+does not exist. (That is why `OF` cannot be used with `LOAD`! `NAME` on the
+other hand is available also with this magic command.)
 
 ```
 %LOAD data.duckdb
 ```
 
-Only one database can be open at any time. If a new database is created or
-loaded, the current one is closed first and saved to disk if necessary.
+Multiple databases can be open at any time. If a new database with the same
+name is created or loaded, the current one is closed first and saved to disk
+if necessary.
 
 Please note that `:memory:` is also a valid file path for DuckDB. The data is
 then stored exclusively in the main memory. In combination with `CREATE`
 and `OF` this makes it possible to work on a temporary copy in memory.
 
+Although the name suggests otherwise, the kernel can also be used with other
+databases:
+- **SQLite** is automatically used as a fallback if the DuckDB dependency is
+  missing.
+- To connect to a **PostgreSQL** instance, you need to specify a database URI
+  starting with `(postgresql|postgres|pgsql|psql|pg)://`.
+
 ### Schema Diagrams
 
 The magic command `SCHEMA` can be used to create a simple schema diagram of the
@@ -143,6 +136,10 @@ representation requires more space, but can improve readability.
 %SCHEMA TD
 ```
 
+The optional argument `ONLY`, followed by one or more table names separated by a
+comma, can be used to display only the named tables and all those connected with
+a foreign key.
+
 Graphviz (`dot` in PATH) is required to render schema diagrams.
 
 ### Number of Rows
@@ -190,10 +187,10 @@ FROM bar
 
 ### Ship Tests With Your Notebooks
 
-Simple tests can be loaded together with the database with the help of
-the `WITH_TESTS` parameter. These tests are stored as a JSON file. Each test is
-assigned a unique name, a result set and whether the test should check the order
-of the result. A very simple test file looks like the following JSON object:
+Simple tests can be loaded from json files with the help of magic command
+`LOAD_TESTS`. These tests are stored as a JSON file. Each test is assigned a
+unique name, a result set and whether the test should check the order of the
+result. A very simple test file looks like the following JSON object:
 
 ```json
 {
@@ -224,6 +221,11 @@ UNION
 SELECT 1, 'Name 1'
 ```
 
+By default, failed tests will display an explanation, but the notebook will
+continue to run. Set the `DUCKDB_TESTS_RAISE_EXCEPTION` environment variable to
+`true` to raise an exception when a test fails. This can be useful for automated
+testing in CI environments.
+
 Disclaimer: The integrated testing is work-in-progress and thus subject to
 potentially incompatible changes and enhancements.
 
@@ -234,7 +236,7 @@ magic command `RA` activates the relational algebra mode for a single cell:
 
 ```
 %RA
-π a, b (σ c = 1 (R))
+π [a, b] (σ [c = 1] (R))
 ```
 
 The supported operations are:
@@ -247,6 +249,12 @@ The supported operations are:
 - Difference `\`
 - Natural Join `⋈`
 - Cross Product `×`
+- Division `÷`
+
+The optional flag `ANALYZE` can be used to add an execution diagram to the
+output.
+
+You can also add comments to queries using `--` or `/* */`, just like in SQL.
 
 The Dockerfile also installs the Jupyter Lab plugin
 [jupyter-ra-extension](https://pypi.org/project/jupyter-ra-extension/). It adds
@@ -262,3 +270,12 @@ magic command `DC` activates the domain calculus mode for a single cell:
 %DC
 { a, b | R(a, b, c) ∧ c = 1 }
 ```
+
+### Automated Parser Selection
+
+`%ALL_RA` or `%ALL_DC` enables the corresponding parser for all subsequently
+executed cells.
+
+If the magic command `%AUTO_PARSER` is added to a cell, a parser is
+automatically selected. If `%GUESS_PARSER` is executed, the parser is
+automatically selected for all subsequent cells.

jupyter_duckdb-1.4.106/setup.py

@@ -0,0 +1,54 @@
+import os
+
+# configuration
+PACKAGE_VERSION = '1.4.106'
+DUCKDB_VERSION = '1.4.1'
+
+DEPENDENCIES = [
+    'jupyter',
+    'graphviz==0.21',
+    'checkmarkandcross'
+]
+
+if os.getenv('SQLITE') != '1' and os.getenv('DUCKDB') != '0':
+    DEPENDENCIES += [f'duckdb=={DUCKDB_VERSION}']
+
+# main setup
+if __name__ == '__main__':
+    from setuptools import setup, find_namespace_packages
+
+    # load README.md as long_description
+    with open('README.md', 'r', encoding='utf-8') as file:
+        long_description = file.read()
+
+    # main setup call
+    setup(
+        name='jupyter-duckdb',
+        version=PACKAGE_VERSION,
+        python_requires='>=3.10',
+        install_requires=DEPENDENCIES,
+        author='Eric Tröbs',
+        author_email='eric.troebs@tu-ilmenau.de',
+        description='a basic wrapper kernel for DuckDB',
+        long_description=long_description,
+        long_description_content_type='text/markdown',
+        url='https://github.com/erictroebs/jupyter-duckdb',
+        project_urls={
+            'Bug Tracker': 'https://github.com/erictroebs/jupyter-duckdb/issues',
+        },
+        classifiers=[
+            'Programming Language :: Python :: 3',
+            'License :: OSI Approved :: MIT License',
+            'Operating System :: OS Independent',
+        ],
+        package_dir={'': 'src'},
+        packages=find_namespace_packages(where='src'),
+        include_package_data=True,
+        package_data={
+            'duckdb_kernel': [
+                'kernel.json',
+                'visualization/lib/*.css',
+                'visualization/lib/*.js',
+            ]
+        }
+    )

{jupyter-duckdb-0.9.2.2.dev202401171015 → jupyter_duckdb-1.4.106}/src/duckdb_kernel/db/Column.py

@@ -4,10 +4,11 @@ from .Table import Table
 
 
 class Column:
-    def __init__(self, table: Table, name: str, data_type: str):
+    def __init__(self, table: Table, name: str, data_type: str, null: bool):
         self.table: Table = table
         self.name: str = name
         self.data_type: str = data_type
+        self.null: bool = null
 
     def __hash__(self):
         return self.name.__hash__()

{jupyter-duckdb-0.9.2.2.dev202401171015 → jupyter_duckdb-1.4.106}/src/duckdb_kernel/db/Connection.py

@@ -1,6 +1,6 @@
 from typing import Dict, List, Tuple, Any
 
-from . import Table
+from .Table import Table
 
 
 class Connection:
@@ -10,6 +10,20 @@ class Connection:
     def close(self):
         pass
 
+    def copy(self) -> 'Connection':
+        raise NotImplementedError
+
+    @staticmethod
+    def plain_explain() -> bool:
+        return False
+
+    @staticmethod
+    def multiple_statements_per_query() -> bool:
+        return True
+
+    def __str__(self) -> str:
+        raise NotImplementedError
+
     def execute(self, query: str) -> Tuple[List[str], List[List[Any]]]:
         raise NotImplementedError
 

{jupyter-duckdb-0.9.2.2.dev202401171015 → jupyter_duckdb-1.4.106}/src/duckdb_kernel/db/Table.py

@@ -3,7 +3,7 @@ from typing import List, Optional
 
 from . import Column
 from . import ForeignKey
-from .Constraint import Constraint
+from . import Constraint
 
 
 class Table:
@@ -14,11 +14,19 @@ class Table:
         self.unique_keys: List[Constraint] = []
         self.foreign_keys: List[ForeignKey] = []
 
+    @staticmethod
+    def normalize_name(name: str) -> str:
+        return name.lower()
+
     @property
     def id(self) -> str:
         name = re.sub(r'[^A-Za-z]', '_', self.name)
         return f'table_{name}'
 
+    @property
+    def normalized_name(self) -> str:
+        return self.normalize_name(self.name)
+
     def get_column(self, name: str) -> "Column":
         for column in self.columns:
             if column.name == name:

{jupyter-duckdb-0.9.2.2.dev202401171015 → jupyter_duckdb-1.4.106}/src/duckdb_kernel/db/__init__.py

@@ -1,12 +1,6 @@
-from .DatabaseError import DatabaseError
 from .Column import Column
+from .Connection import Connection
 from .Constraint import Constraint
+from .DatabaseError import DatabaseError
 from .ForeignKey import ForeignKey
 from .Table import Table
-
-try:
-    from .duckdb import Connection
-    SQLITE_MODE = False
-except ImportError:
-    from .sqlite import Connection
-    SQLITE_MODE = True

jupyter_duckdb-1.4.106/src/duckdb_kernel/db/error/EmptyResultError.py

@@ -0,0 +1,5 @@
+from ..DatabaseError import DatabaseError
+
+
+class EmptyResultError(DatabaseError):
+    pass

jupyter_duckdb-1.4.106/src/duckdb_kernel/db/error/__init__.py

@@ -0,0 +1 @@
+from .EmptyResultError import EmptyResultError

{jupyter-duckdb-0.9.2.2.dev202401171015/src/duckdb_kernel/db → jupyter_duckdb-1.4.106/src/duckdb_kernel/db/implementation}/duckdb/Connection.py

@@ -2,17 +2,29 @@ from typing import Dict, List, Tuple, Any
 
 import duckdb
 
-from .. import DatabaseError, Column, Constraint, ForeignKey, Table
-from ..Connection import Connection as Base
+from ... import DatabaseError, Column, Constraint, ForeignKey, Table
+from ...Connection import Connection as Base
+from ...error import EmptyResultError
 
 
 class Connection(Base):
     def __init__(self, path: str):
+        self.path: str = path
         self.con: duckdb.DuckDBPyConnection = duckdb.connect(path, read_only=False)
 
     def close(self):
         self.con.close()
 
+    def copy(self) -> 'Connection':
+        return Connection(self.path)
+
+    @staticmethod
+    def plain_explain() -> bool:
+        return True
+
+    def __str__(self) -> str:
+        return f'DuckDB: {self.path}'
+
     def execute(self, query: str) -> Tuple[List[str], List[List[Any]]]:
         with self.con.cursor() as cursor:
             try:
@@ -28,7 +40,10 @@ class Connection(Base):
                     raise e
 
             # get rows
-            rows = cursor.fetchall()
+            try:
+                rows = cursor.fetchall()
+            except duckdb.InvalidInputException as e:
+                raise EmptyResultError(str(e))
 
             # get columns
             if cursor.description is None:
@@ -40,7 +55,7 @@ class Connection(Base):
 
     def analyze(self) -> Dict[str, Table]:
         tables: Dict[str, Table] = {}
-        constraints: Dict[int, Constraint] = {}
+        constraints: Dict[Tuple, Constraint] = {}
 
         # Get table names first. In the columns table we can not filter
         # for base tables and some of the tables might not be contained
@@ -51,21 +66,23 @@ class Connection(Base):
                     WHERE table_type == 'BASE TABLE'
                 ''').fetchall():
             table = Table(table_name)
-            tables[table_name] = table
+            tables[table.normalized_name] = table
 
         # Get column names and data types for each table.
-        for table_name, column_name, data_type in self.con.execute('''
+        for table_name, column_name, data_type, is_nullable in self.con.execute('''
                     SELECT
                         table_name,
                         column_name,
-                        data_type
+                        data_type,
+                        is_nullable
                     FROM information_schema.columns
                     ORDER BY ordinal_position ASC
                 ''').fetchall():
-            if table_name in tables:
-                table = tables[table_name]
+            normalized_table_name = Table.normalize_name(table_name)
+            if normalized_table_name in tables:
+                table = tables[normalized_table_name]
 
-                column = Column(table, column_name, data_type)
+                column = Column(table, column_name, data_type, is_nullable == 'YES')
                 table.columns.append(column)
 
         # Find primary keys.
@@ -79,10 +96,12 @@ class Connection(Base):
                     ORDER BY constraint_index ASC
                 ''').fetchall():
             # get table
-            if table_name not in tables:
+            normalized_table_name = Table.normalize_name(table_name)
+
+            if normalized_table_name not in tables:
                 raise AssertionError(f'unknown table {table_name} for constraint {constraint_index}')
 
-            table = tables[table_name]
+            table = tables[normalized_table_name]
 
             # store constraint
             if constraint_index in constraints:
@@ -93,7 +112,7 @@ class Connection(Base):
                 table,
                 tuple(table.get_column(c) for c in constraint_columns)
             )
-            constraints[constraint_index] = constraint
+            constraints[(normalized_table_name, *constraint_columns)] = constraint
 
             # store key
             if table.primary_key is not None:
@@ -112,10 +131,12 @@ class Connection(Base):
                     ORDER BY constraint_index ASC
                 ''').fetchall():
             # get table
-            if table_name not in tables:
+            normalized_table_name = Table.normalize_name(table_name)
+
+            if normalized_table_name not in tables:
                 raise AssertionError(f'unknown table {table_name} for constraint {constraint_index}')
 
-            table = tables[table_name]
+            table = tables[normalized_table_name]
 
             # store constraint
             if constraint_index in constraints:
@@ -126,32 +147,37 @@ class Connection(Base):
                 table,
                 tuple(table.get_column(c) for c in constraint_columns)
             )
-            constraints[constraint_index] = constraint
+            constraints[(normalized_table_name, *constraint_columns)] = constraint
 
             # store key
             table.unique_keys.append(constraint)
 
         # Find foreign keys.
-        for table_name, constraint_index, constraint_columns in self.con.execute('''
+        for table_name, constraint_index, constraint_columns, referenced_table, referenced_column_names, in self.con.execute('''
                     SELECT
                         table_name,
                         constraint_index,
-                        constraint_column_names
+                        constraint_column_names,
+                        referenced_table,
+                        referenced_column_names
                     FROM duckdb_constraints()
                     WHERE constraint_type = 'FOREIGN KEY'
                     ORDER BY constraint_index ASC
                 ''').fetchall():
             # get table
-            if table_name not in tables:
+            normalized_table_name = Table.normalize_name(table_name)
+
+            if normalized_table_name not in tables:
                 raise AssertionError(f'unknown table {table_name} for constraint {constraint_index}')
 
-            table = tables[table_name]
+            table = tables[normalized_table_name]
 
             # lookup constraint
-            if constraint_index not in constraints:
-                raise AssertionError(f'constraint with index {constraint_index} not discovered previously')
+            constraint_key = (Table.normalize_name(referenced_table), *referenced_column_names)
+            if constraint_key not in constraints:
+                raise AssertionError(f'constraint with key {constraint_key} not discovered previously')
 
-            constraint = constraints[constraint_index]
+            constraint = constraints[constraint_key]
 
             # store key
             key = ForeignKey(tuple(table.get_column(c) for c in constraint_columns), constraint)