fullscript.spellbook 2.1.2__tar.gz → 2.1.4__tar.gz

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/.gitlab-ci.yml

@@ -1,15 +1,11 @@
 include:
   - component: $CI_SERVER_FQDN/devops/ci-cd/build-python/template@0.0.2
-  - component: $CI_SERVER_FQDN/devops/ci-cd/publish-python-package/template@initial-impl
+  - component: $CI_SERVER_FQDN/devops/ci-cd/publish-python-package/template@0.1.1
     inputs:
-      public_python_package_image_tag: 87fa387a0fe221b8a630f7c0e78fbbaf6c23e9a1
       artifacts_job_name: build_spellbook
-  - component: $CI_SERVER_FQDN/devops/ci-cd/publish-python-package/template@initial-impl
+  - component: $CI_SERVER_FQDN/devops/ci-cd/publish-python-package/template@0.1.1
     inputs:
-      public_python_package_image_tag: 87fa387a0fe221b8a630f7c0e78fbbaf6c23e9a1
       upload_target: pypi
-      pypi_rules_config:
-        - when: always
       artifacts_job_name: build_spellbook
   - project: devops/ci-cd/workflow-skip-duplicate-pipelines
     file: template.yaml

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.2
 Name: fullscript.spellbook
-Version: 2.1.2
+Version: 2.1.4
 Summary: A simple package to make your life easier.
 Author-email: Thomas Edwards <thomas.edwards@fullscript.com>, Gui Chagas <gui.chagas@fullscript.com>, David King <david.king@fullscript.com>, Meinhard Mare <meinhard.mare@fullscript.com>, Karsten Melchers <karsten.melchers@fullscript.com>
 License: MIT
-Project-URL: homepage, https://git.fullscript.io/data/spellbook
+Project-URL: homepage, https://github.com/Fullscript/spellbook
 Requires-Python: <3.12,>=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -136,6 +136,7 @@ Spellbook is built to simplify your workflows and supercharge your data explorat
 - [Features](#features)
 - [Installation](#installation)
 - [Configuration](#configuration)
+- [Documentation](#documentation)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -190,6 +191,14 @@ google_accounts:
     service_account: "your_base64_encoded_service_account_credentials_here"  # Add your Base64-encoded service account JSON here
 ```
 
+## Documentation
+For detailed documentation on how to use Spellbook, please refer to the [Documentation](documentation) folder in the repository.  
+**Functionality:**
+- [Database Magic](documentation/db_magic.md)
+- [Drive Magic](documentation/drive_magic.md)
+- [Sheet Magic](documentation/sheet_magic.md)
+- [Stat Magic](documentation/stat_magic.md)
+
 ## Contributing
 
 We welcome contributions! Please follow these steps:

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/README.md

@@ -17,6 +17,7 @@ Spellbook is built to simplify your workflows and supercharge your data explorat
 - [Features](#features)
 - [Installation](#installation)
 - [Configuration](#configuration)
+- [Documentation](#documentation)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -71,6 +72,14 @@ google_accounts:
     service_account: "your_base64_encoded_service_account_credentials_here"  # Add your Base64-encoded service account JSON here
 ```
 
+## Documentation
+For detailed documentation on how to use Spellbook, please refer to the [Documentation](documentation) folder in the repository.  
+**Functionality:**
+- [Database Magic](documentation/db_magic.md)
+- [Drive Magic](documentation/drive_magic.md)
+- [Sheet Magic](documentation/sheet_magic.md)
+- [Stat Magic](documentation/stat_magic.md)
+
 ## Contributing
 
 We welcome contributions! Please follow these steps:

fullscript_spellbook-2.1.4/documentation/db_magic.md

@@ -0,0 +1,98 @@
+A magical wrapper to perform work with databases.
+
+
+##### Key Features
+* **Unified Interface**: Abstracts the complexities of connecting and interacting with Snowflake and MySQL databases.
+* **Seamless Data Handling**: Fetch, manipulate, and write data with Pandas DataFrames.
+* **Flexibility**: Supports secure credentials management via configuration files and environment variables.
+
+```python
+from spellbook import db_magic
+```
+
+### Methods Overview
+
+
+##### get_data(db_name, query)
+
+Fetches data from a specified database and returns it as a Pandas DataFrame. This method abstracts the connection and retrieval process for both Snowflake and MySQL databases, allowing you to focus on the SQL query.
+
+Parameters:
+* `db_name` (str): The database name defined in the configuration file.
+* `query` (str): The SQL query to execute.
+
+Returns:
+* A Pandas DataFrame containing the result of the query.
+
+```python
+# Fetch data from Snowflake
+# Option 1:
+query = "SELECT * FROM my_table LIMIT 10;"
+df = db_magic.get_data("snowflake_db", query)
+
+# Option 2:
+df = db_magic.get_data("snowflake_db", 'my_sql_query.sql')
+
+# Option 3:
+df = db_magic.get_data("snowflake_db", 'user/project/my_query.sql')
+
+
+# Fetch data from MySQL
+query = "SELECT * FROM users WHERE is_active = 1;"
+df = db_magic.get_data("mysql_db", query)
+
+# Display results
+print(df.head())
+```
+
+##### write_data(db_name, schema, df, table_name, overwrite=True, chunk_size=1000, **kwargs)
+
+Writes a Pandas DataFrame to a table in Snowflake. It supports creating tables automatically and overwriting existing tables.
+
+Parameters:
+* `db_name` (str): The database name defined in the configuration file.
+* `schema` (str): The schema where the table is located.
+* `df` (Pandas DataFrame): The data to write.
+* `table_name` (str): The name of the destination table.
+* `overwrite` (bool): Whether to overwrite the table if it exists (default: True).
+* `chunk_size` (int): Number of rows to write per chunk (default: 1000).
+
+```python
+import pandas as pd
+
+# Create a sample DataFrame
+data = {"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]}
+df = pd.DataFrame(data)
+
+# Write data to Snowflake
+db_magic.write_data("snowflake_db", "my_schema", df, "my_table", overwrite=True)
+```
+
+##### run_sql(db_name, query)
+
+Executes a SQL query directly on a specified database without returning results. Ideal for operations like creating tables, updating rows, or running stored procedures.
+
+Parameters:
+* `db_name` (str): The database name defined in the configuration file.
+* `query` (str): The SQL query to execute.
+
+Returns:
+* `None`
+
+```python
+
+# Execute SQL in Snowflake
+create_table_query = """
+CREATE TABLE my_table (
+    id INT,
+    name STRING
+);
+"""
+db_magic.run_sql("snowflake_db", create_table_query)
+
+# Execute SQL in MySQL
+update_query = "UPDATE users SET is_active = 1 WHERE last_login > '2024-01-01';"
+db_magic.run_sql("mysql_db", update_query)
+
+print("SQL executed successfully!")
+```

fullscript_spellbook-2.1.4/documentation/drive_magic.md

@@ -0,0 +1,77 @@
+## Different Authentication Methods
+
+This package can use 2 different authentication methods:
+
+- Service Account: The same method we currently use in `sheet_magic`
+- OAuth: Method where the user has to allow the script to use their credentials
+
+Both methods have their pros and cons:
+
+- Service Account:
+  - It's the easiest to use as one probably already has the setup ready.
+  - It can only access folders shared with the service account email (similarly as it can only access sheets we share with it)
+    - Because we're not managers of the data team's shared folders, we cannot share those folders with this account. This is a road block for now.
+- OAuth:
+  - The user has to grant access constantly (every time the scope changes or the authentication token expires)
+  - This method can access any folder shared with you as the script will be using your identity (this includes shared folders where you're not the owner)
+
+## OAuth Setup
+
+Step-by-step:
+
+  1 - Access this link and follow the [OAuth client ID credentials](https://developers.google.com/workspace/guides/create-credentials#oauth-client-id) process
+  
+  2 - Probably you'll be asked to setup the OAuth Consent Screen. At the second page of this part ("Scopes"), be sure to add all Google Drive API scopes.
+
+  3 - In the "Client ID for Web Application" page, add `http://localhost:3000/` as an "Authorized redirect URIs"
+
+  4 - From the same page, download the client secret as a JSON file.
+
+  5 - Now the process is similar to what is done for `'GS_SERVICE_ACCOUNT'`. Copy the contents of the JSON file.
+
+  6 - Go to the terminal and use `pbpaste | base64 | pbcopy` to paste, encode, and copy the contents again.
+
+  7 - Store the encoded contents as an envvar called `'GS_CLIENT_SECRETS'`
+
+## Initiate
+
+```python
+from spellbook import drive_magic as dm
+```
+
+## Usage examples
+
+##### Getting service
+```python
+scopes = ['https://www.googleapis.com/auth/drive.metadata.readonly',
+          'https://www.googleapis.com/auth/drive.file',
+          'https://www.googleapis.com/auth/drive.readonly',
+          'https://www.googleapis.com/auth/drive',
+          'https://www.googleapis.com/auth/drive.file',
+          'https://www.googleapis.com/auth/drive.appdata']
+
+# service = dm.get_service(scopes=scopes, auth_type='service_account')
+service = dm.get_service(scopes=scopes, auth_type='oauth')
+del scopes
+```
+
+##### Listing contents of a folder
+```python
+folder_url = 'https://drive.google.com/drive/u/0/folders/1zgsK7EkEcYsvmiQ0InT04Bg5yBdG6b-D'
+df_files = dm.list_folder_contents(service, folder_url)
+```
+
+##### Uploading a file
+```python
+dm.upload_file(service, folder_url, 'samples/01_sample_pdf.pdf')
+```
+
+##### Downloading a file
+```python
+temp_file_id = '1JAYgOUztUNYrx5qJispiEM-B2oXdLQzO'
+dm.download_file(service, temp_file_id, '02_sample_picture_v2.jpeg')
+```
+##### Deleting / thrashing a file
+```python
+dm.delete_file(service, temp_file_id)
+```

fullscript_spellbook-2.1.4/documentation/sheet_magic.md

@@ -0,0 +1,138 @@
+### Key Features
+
+1. **Sheet Management**:
+  * Create, delete, or clear sheets with ease.
+  * Access sheets by name or index.
+2. **Data Handling**:
+  * Read and write Pandas DataFrames.
+  * Target specific ranges or clear entire sheets.
+3. **Flexibility**:
+  * Automatically create sheets if they don’t exist.
+  * Handle trailing empty cells in row counting.
+4. **Seamless Google Sheets Integration**:
+  * Authenticate with Google Service Account credentials.
+  * Perform operations with minimal setup.
+
+# Setup
+
+## Create a Google Service account
+To use this feature, you must create a Google service account. Instructions on how to do this are available on the Google [IAM documentation](https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console)
+
+## Saving Your `GS_SERVICE_ACCOUNT` Environment Variable
+How to save your 'GS_SERVICE_ACCOUNT' env-var to use this code
+
+To use Google Service Account functionality with this code, you need to save your Google Service Account JSON as an encrypted string in an environment variable. Follow these steps:
+1.Convert the JSON File Content to a Base64-Encoded String
+  * Copy the content of your service account JSON file to your clipboard.
+  * Open your terminal and run the following command to encode it:
+  ```bash
+   pbpaste | base64 | pbcopy
+  ```
+  * This command encrypts the JSON content using Base64 and places the encoded string back into your clipboard.
+ 
+2.Save the Encoded String as an Environment Variable
+  * Add the encoded string as an environment variable.
+  * Use a name like GS_SERVICE_ACCOUNT for consistency.
+
+For example, add the following line to your environment file (e.g., .env):
+```bash
+GS_SERVICE_ACCOUNT=your_base64_encoded_string
+```
+3.Why Base64 Encoding?
+  * Base64 ensures the JSON content is securely represented as a single string, making it compatible with environment variables.
+
+Once set up, your code will securely retrieve the service account details directly from the environment variable.
+
+## Initialize
+
+To begin using sheet_magic, initialize the handler with the URL of the Google Sheet you want to work with.
+
+```python
+from spellbook import sheet_magic
+
+# Replace with your actual Google Sheet URL
+sheet_url = 'https://docs.google.com/spreadsheets/d/your_sheet_id/edit'
+working_sheet = sheet_magic.GSheetsHandler(gsheet_url=sheet_url, service_account_name='my_google_service_account')
+```
+
+<div style="border-left: 6px solid #f39c12; padding: 10px; background-color: #fcf8e3;">
+
+⚠️ **Important Notice:** In order to use sheet_magic, you need to add the Google service account to the sheet you are working on
+
+</div>
+
+### Usage Examples
+
+1. Create a New Sheet
+
+Add a new sheet to your Google Sheets document.
+```python
+# Create a new sheet named 'test_sheet'
+working_sheet.create_sheet(worksheet='test_sheet')
+```
+
+2. Delete an Existing Sheet
+
+Delete a sheet by its name or index.
+```python
+# Delete a sheet by its name
+working_sheet.delete_sheet(worksheet='test_sheet')
+
+# Delete a sheet by its index
+working_sheet.delete_sheet(worksheet=1)
+```
+
+3. Write Data to a Sheet
+
+Write a Pandas DataFrame to a specified sheet. You can specify a starting position, and the sheet will be resized or extended as needed.
+
+```python
+import pandas as pd
+
+# Sample DataFrame
+data = {'Name': ['Alice', 'Bob'], 'Age': [25, 30]}
+df_test = pd.DataFrame(data)
+
+# Write DataFrame to a sheet by name
+working_sheet.write_to_sheet(input_dataframe=df_test, worksheet='data_sheet')
+
+# Write DataFrame to a sheet by index
+working_sheet.write_to_sheet(input_dataframe=df_test, worksheet=2)
+```
+
+4. Count Rows in a Sheet
+
+Count the number of rows in a specified column, with an option to include trailing empty cells.
+```python
+# Count rows in column 1 of the sheet 'data_sheet'
+row_count = working_sheet.count_rows(worksheet='data_sheet')
+
+# Count rows, including empty trailing cells
+row_count_with_empty = working_sheet.count_rows(worksheet='data_sheet', include_tailing_empty=True)
+
+print(f"Rows in column 1: {row_count}")
+```
+
+5. Clear a Range
+
+Clear the contents of a specific range or the entire sheet.
+```python
+# Clear the entire sheet
+working_sheet.clear_range(worksheet='data_sheet')
+
+# Clear a specific range (e.g., B2:D4)
+working_sheet.clear_range(worksheet='data_sheet', clear_range_str='B2:D4')
+```
+
+6. Read Data from a Sheet
+
+Read data from a sheet into a Pandas DataFrame. Optionally, specify a target range.
+```python
+# Read all data from the sheet 'data_sheet'
+df_read = working_sheet.read_sheet(worksheet='data_sheet')
+
+# Read a specific range (e.g., B1:C4) from the sheet
+df_read_partial = working_sheet.read_sheet(worksheet='data_sheet', target_range_str='B1:C4')
+
+print(df_read.head())
+```

fullscript_spellbook-2.1.4/documentation/snowflake_key_pair_setup.md

@@ -0,0 +1,35 @@
+A step-by-step guide to establishing a connection to Snowflake using Snowpark for Python with key-pair authentication.
+
+### 1. Generate Key Pair:
+* **Private Key**: Create an unencrypted private key using OpenSSL:
+```bash
+openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
+```
+> Note: If you want a passphrase for additional security, remove the `-nocrypt` flag.
+
+* **Public Key**: Derive the public key from the private key:
+```bash
+openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
+```
+
+### 2. Assign Public Key to Snowflake User (will require admin access):
+* In Snowflake, associate the generated public key with your user account:
+```sql
+ALTER USER <username> SET rsa_public_key='<contents_of_rsa_key.pub>';
+```
+
+* Replace `<username>` with your actual Snowflake username and `<contents_of_rsa_key.pub>` with the content of your public key file.
+
+
+### 3. Base64 Encode the Private Key:
+Since private keys are typically multi-line strings, encoding them in Base64 ensures they can be stored as single-line environment variables.
+```bash
+openssl base64 -in rsa_key.p8 -out rsa_key_base64.txt
+```
+This command reads your rsa_key.p8 file and outputs a Base64-encoded version to rsa_key_base64.txt.
+
+### 4. Set the Environment Variable:
+Once you have the Base64-encoded key, set it as an environment variable.
+```bash
+export PRIVATE_KEY_BASE64='your_base64_encoded_key_here'
+```

fullscript_spellbook-2.1.4/documentation/stats_magic.md

@@ -0,0 +1,154 @@
+The stats_magic module provides a suite of statistical tests and tools for analyzing datasets. It simplifies the process of conducting normality tests, variance analysis, ANOVA, and various other statistical methods. The module supports multi-group comparisons as well as two-group analyses, making it ideal for exploratory data analysis and hypothesis testing.
+
+### Key Features
+
+1. **Comprehensive Statistical Tests:**
+  * Covers normality, variance, mean, and median comparisons.
+  * Includes both parametric and non-parametric methods.
+2. **Multi-Group Analysis:**
+  * Tools like ANOVA, Tukey’s, and Dunn’s tests for multi-level datasets.
+3. **Two-Group Analysis:**
+  * Student’s t-test, Welch’s t-test, and Mann-Whitney U test for pairwise comparisons.
+4. **Detailed Outputs:**
+  * Returns results as Pandas DataFrames for easy analysis.
+  * Supports verbose mode for detailed explanations of results.
+
+# Setup
+
+## Import the Module
+To begin using stats_magic, import it into your Python environment.
+```python
+from spellbook import stats_magic
+```
+
+### Multi-Level Tests
+
+These tests are designed for datasets with more than two groups, allowing comparisons across multiple categories.
+
+1. **Shapiro-Wilk Normality Test**
+
+Tests if the data in each group follows a normal distribution.
+```python
+shapiro_df = stats_magic.shapiro_wilk_normality_test(
+    df=df_penguins,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    verbose=True
+)
+```
+Key Parameters:
+* `group_column`: The column defining groups (e.g., species).
+* `values_column`: The column containing numeric values to test.
+* `alpha_level`: Significance level (default 0.05).
+* `verbose`: Print detailed output (default False).
+
+2. **Levene’s Variance Test**
+
+Tests if the variances across groups are equal.
+```python
+levene_df = stats_magic.levene_variance_test(
+    df=df_penguins,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    center_method='mean',
+    verbose=True
+)
+```
+Key Parameters:
+* `center_method`: Method to calculate central tendency (mean, median, or trimmed).
+
+3. **ANOVA**
+
+Performs an analysis of variance to test if means across groups are significantly different.
+```python
+anova_df = stats_magic.anova(
+    df=df_penguins,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    verbose=True
+)
+```
+
+4. **Tukey’s Test**
+
+Conducts pairwise comparisons between group means.
+```python
+tukey_test_df = stats_magic.tukey_test(
+    df=df_penguins,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    verbose=True
+)
+```
+
+5. **Mood’s Median Test**
+
+Tests if the medians across groups are significantly different.
+```python
+mood_median_test_df = stats_magic.mood_median_test(
+    df=df_penguins,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    verbose=True
+)
+```
+
+6. **Dunn’s Test**
+
+Performs pairwise comparisons between groups, adjusting p-values to account for multiple tests.
+```python
+dunn_test_df = stats_magic.dunn_test(
+    df=df_penguins,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    verbose=True
+)
+```
+
+### Two-Level Tests
+
+These tests are designed for datasets with exactly two groups, allowing direct comparisons between them.
+
+**Setup: Create a Two-Group Dataset**
+Filter the dataset to include only two groups.
+
+```python
+two_group_df = df_penguins[
+    (df_penguins['species'] == 'Adelie') | (df_penguins['species'] == 'Gentoo')
+]
+```
+
+1. **Student’s t-test / Welch’s t-test**
+
+Tests if the means of two groups are significantly different.
+```python
+t_welch_test_df = stats_magic.t_welch_test(
+    df=two_group_df,
+    group_column='species',
+    values_column='flipper_length_mm',
+    equal_variances=True,
+    alpha_level=0.05,
+    verbose=True
+)
+```
+Key Parameters:
+* `equal_variances`: Set to False to use Welch’s t-test for unequal variances.
+
+2. **Mann-Whitney U Test**
+
+Tests if the distributions of two groups are significantly different.
+```python
+mann_whitney_u_test_df = stats_magic.mann_whitney_u_test(
+    df=two_group_df,
+    group_column='species',
+    values_column='flipper_length_mm',
+    alpha_level=0.05,
+    verbose=True
+)
+```

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/fullscript.spellbook.egg-info/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.2
 Name: fullscript.spellbook
-Version: 2.1.2
+Version: 2.1.4
 Summary: A simple package to make your life easier.
 Author-email: Thomas Edwards <thomas.edwards@fullscript.com>, Gui Chagas <gui.chagas@fullscript.com>, David King <david.king@fullscript.com>, Meinhard Mare <meinhard.mare@fullscript.com>, Karsten Melchers <karsten.melchers@fullscript.com>
 License: MIT
-Project-URL: homepage, https://git.fullscript.io/data/spellbook
+Project-URL: homepage, https://github.com/Fullscript/spellbook
 Requires-Python: <3.12,>=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -136,6 +136,7 @@ Spellbook is built to simplify your workflows and supercharge your data explorat
 - [Features](#features)
 - [Installation](#installation)
 - [Configuration](#configuration)
+- [Documentation](#documentation)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -190,6 +191,14 @@ google_accounts:
     service_account: "your_base64_encoded_service_account_credentials_here"  # Add your Base64-encoded service account JSON here
 ```
 
+## Documentation
+For detailed documentation on how to use Spellbook, please refer to the [Documentation](documentation) folder in the repository.  
+**Functionality:**
+- [Database Magic](documentation/db_magic.md)
+- [Drive Magic](documentation/drive_magic.md)
+- [Sheet Magic](documentation/sheet_magic.md)
+- [Stat Magic](documentation/stat_magic.md)
+
 ## Contributing
 
 We welcome contributions! Please follow these steps:

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/fullscript.spellbook.egg-info/SOURCES.txt

@@ -6,7 +6,13 @@ LICENSE
 MANIFEST.in
 README.md
 pyproject.toml
+requirements.txt
 spellbook_config.yaml
+documentation/db_magic.md
+documentation/drive_magic.md
+documentation/sheet_magic.md
+documentation/snowflake_key_pair_setup.md
+documentation/stats_magic.md
 fullscript.spellbook.egg-info/PKG-INFO
 fullscript.spellbook.egg-info/SOURCES.txt
 fullscript.spellbook.egg-info/dependency_links.txt

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/pyproject.toml

@@ -16,7 +16,7 @@ authors = [
     {name = "Meinhard Mare", email = "meinhard.mare@fullscript.com"},
     {name = "Karsten Melchers", email = "karsten.melchers@fullscript.com"}
 ]
-urls = {homepage = "https://git.fullscript.io/data/spellbook"}
+urls = {homepage = "https://github.com/Fullscript/spellbook"}
 dependencies = [
     "appnope==0.1.4",
     "asn1crypto==1.5.1",

fullscript_spellbook-2.1.4/requirements.txt

@@ -0,0 +1,252 @@
+#
+# This file is autogenerated by pip-compile with Python 3.11
+# by the following command:
+#
+#    pip-compile
+#
+appnope==0.1.4
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+asn1crypto==1.5.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+asttokens==2.4.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   stack-data
+certifi==2024.8.30
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   requests
+    #   snowflake-connector-python
+cffi==1.17.1
+    # via
+    #   cryptography
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+charset-normalizer==3.4.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   requests
+    #   snowflake-connector-python
+cloudpickle==2.2.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-snowpark-python
+comm==0.2.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+cryptography==43.0.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   pyopenssl
+    #   snowflake-connector-python
+debugpy==1.8.8
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+decorator==5.1.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+executing==2.1.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   stack-data
+filelock==3.16.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+idna==3.10
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   requests
+    #   snowflake-connector-python
+ipykernel==6.29.5
+    # via fullscript.spellbook (pyproject.toml)
+ipython==8.29.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+jedi==0.19.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+jupyter-client==8.6.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+jupyter-core==5.7.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+    #   jupyter-client
+matplotlib-inline==0.1.7
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+    #   ipython
+nest-asyncio==1.6.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+numpy==2.1.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   pandas
+packaging==24.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+    #   snowflake-connector-python
+pandas==2.2.3
+    # via fullscript.spellbook (pyproject.toml)
+parso==0.8.4
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   jedi
+pexpect==4.9.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+platformdirs==4.3.6
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   jupyter-core
+    #   snowflake-connector-python
+prompt-toolkit==3.0.48
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+protobuf==5.28.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-snowpark-python
+psutil==6.1.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+ptyprocess==0.7.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   pexpect
+pure-eval==0.2.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   stack-data
+pyarrow==18.0.0
+    # via fullscript.spellbook (pyproject.toml)
+pycparser==2.22
+    # via
+    #   cffi
+    #   fullscript.spellbook (pyproject.toml)
+pygments==2.18.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+pyjwt==2.9.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+pyopenssl==24.2.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+python-dateutil==2.9.0.post0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   jupyter-client
+    #   pandas
+pytz==2024.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   pandas
+    #   snowflake-connector-python
+pyyaml==6.0.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-snowpark-python
+pyzmq==26.2.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+    #   jupyter-client
+requests==2.32.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+six==1.16.0
+    # via
+    #   asttokens
+    #   fullscript.spellbook (pyproject.toml)
+    #   python-dateutil
+snowflake-connector-python==3.12.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-snowpark-python
+    #   snowflake-sqlalchemy
+snowflake-snowpark-python==1.25.0
+    # via fullscript.spellbook (pyproject.toml)
+snowflake-sqlalchemy==1.7.0
+    # via fullscript.spellbook (pyproject.toml)
+sortedcontainers==2.4.0
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+sqlalchemy==2.0.36
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-sqlalchemy
+stack-data==0.6.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+tomlkit==0.13.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-connector-python
+tornado==6.4.1
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+    #   jupyter-client
+traitlets==5.14.3
+    # via
+    #   comm
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipykernel
+    #   ipython
+    #   jupyter-client
+    #   jupyter-core
+    #   matplotlib-inline
+typing-extensions==4.12.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   ipython
+    #   snowflake-connector-python
+    #   snowflake-snowpark-python
+    #   sqlalchemy
+tzdata==2024.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   pandas
+tzlocal==5.2
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   snowflake-snowpark-python
+urllib3==2.2.3
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   requests
+wcwidth==0.2.13
+    # via
+    #   fullscript.spellbook (pyproject.toml)
+    #   prompt-toolkit
+wheel==0.45.1
+    # via snowflake-snowpark-python
+
+# The following packages are considered to be unsafe in a requirements file:
+# setuptools

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/spellbook/db_magic.py

@@ -5,7 +5,7 @@ import pandas as pd
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
 import base64
-from spellbook.utils import load_and_parse_config, get_wizard
+from spellbook.utils import load_and_parse_config, get_wizard, load_sql_file
 
 
 # Needs a very specific pip install
@@ -108,7 +108,9 @@ def read_mysql(db_name, query):
     if con is None:
         return pd.DataFrame()
     try:
-        df = pd.read_sql(query, con)
+        df = pd.read_sql(
+            load_sql_file(query),
+            con)
         df.columns = df.columns.str.lower()
         return df
     except Exception as e:
@@ -129,7 +131,9 @@ def execute_mysql(query, dbname):
         # Establish a connection and run the query
         con = mysql_connect(dbname)
         mycursor = con.connection.cursor()
-        mycursor.execute(query)
+        mycursor.execute(
+            load_sql_file(query)
+        )
     except Exception as e:
         # Handle errors
         print('An error was encountered executing mysql, ', e)
@@ -151,7 +155,9 @@ def execute_snowflake(db_name, query):
             return
 
         # Execute query
-        session.sql(query).collect()
+        session.sql(
+            load_sql_file(query)
+        ).collect()
         print("Query executed successfully.")
 
     except Exception as e:
@@ -177,7 +183,9 @@ def read_snowflake(db_name, query):
             return pd.DataFrame()
 
         # Execute query and fetch data
-        df = session.sql(query).to_pandas()
+        df = (session.sql(
+                load_sql_file(query)
+        ).to_pandas())
         df.columns = df.columns.str.lower()  # Normalize column names
         return df
 
@@ -238,10 +246,16 @@ def get_data(db_name, query, **kwargs):
     # Call the correct database read function
     if db_config['type'] == 'snowflake':
         print(get_wizard())
-        df = read_snowflake(db_name, query)
+        df = read_snowflake(
+            db_name,
+            load_sql_file(query)
+        )
     elif db_config['type'] == 'mysql':
         print(get_wizard())
-        df = read_mysql(db_name, query)
+        df = read_mysql(
+            db_name,
+            load_sql_file(query)
+        )
     else:
         print('Valid database types are, snowflake and mysql')
     return df
@@ -258,9 +272,15 @@ def run_sql(db_name, query, **kwargs):
     # Call the correct database run function
     if db_config['type'] == 'snowflake':
         print(get_wizard())
-        execute_snowflake(db_name, query)
+        execute_snowflake(
+            db_name,
+            load_sql_file(query)
+        )
     elif db_config['type'] == 'mysql':
         print(get_wizard())
-        execute_mysql(db_name, query)
+        execute_mysql(
+            db_name,
+            load_sql_file(query)
+        )
     else:
         print('Valid database types are, snowflake and mysql')

{fullscript_spellbook-2.1.2 → fullscript_spellbook-2.1.4}/spellbook/utils.py

@@ -90,4 +90,34 @@ def get_wizard():
         print('An error was encountered loading the wizard:', e)
         wizard = None
 
-    return wizard
+    return wizard
+
+
+def load_sql_file(query_file):
+    """
+    Load the contents of a SQL file or return the SQL string directly.
+
+    :param query_file: str, path to the SQL file or raw SQL query string
+    :return: str, SQL query
+    :raises ValueError: if the file path is invalid or not a .sql file
+    """
+    if isinstance(query_file, str) and os.path.exists(query_file):
+        # Ensure it's a .sql file
+        if not query_file.lower().endswith('.sql'):
+            raise ValueError(f"The file {query_file} is not a valid SQL file.")
+
+        # Use a context manager to read the file safely
+        try:
+            with open(query_file, 'r', encoding='utf-8') as file:
+                query_str = file.read()
+        except Exception as e:
+            raise ValueError(f"Error reading the file {query_file}: {e}")
+    else:
+        # Assume it's a raw SQL query string
+        query_str = query_file
+
+    # Check for empty SQL content
+    if not query_str.strip():
+        raise ValueError("The SQL query is empty. Please provide valid SQL content.")
+
+    return query_str