sqlshell 0.2.3__tar.gz → 0.3.0__tar.gz

{sqlshell-0.2.3 → sqlshell-0.3.0}/.gitignore

@@ -30,6 +30,9 @@ tests/data/*.delta
 tests/data/*.duckdb
 tests/data/*.sqls
 
+# Sample data files - exclude CSV files but keep other sample files
+sample_data/*.csv
+
 # IDE
 .idea/
 .vscode/
@@ -61,11 +64,8 @@ ENV/
 .venv
 
 # Project specific
-*.db
-*.xlsx
-*.xls
-*.csv
-sample_sales_data.xlsx
+# Don't ignore sample_data/ since it contains example files
+# But ignore any dynamic database files
 pool.db
 
 # Logs
@@ -86,3 +86,12 @@ tests/
 *.sqls
 sqlshell-0.2.0/
 TODO
+
+# Temporary files and directories
+temp/
+tmp/
+*.tmp
+
+# Backup files
+*.bak
+*.backup

{sqlshell-0.2.3/sqlshell.egg-info → sqlshell-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sqlshell
-Version: 0.2.3
+Version: 0.3.0
 Summary: A powerful SQL shell with GUI interface for data analysis
 Author: SQLShell Team
 License-Expression: MIT
@@ -25,21 +25,22 @@ Requires-Dist: fastparquet>=2023.10.1
 Requires-Dist: xlrd>=2.0.1
 Requires-Dist: deltalake
 Requires-Dist: Pillow>=10.0.0
-Requires-Dist: shap
 Requires-Dist: xgboost
 Requires-Dist: scikit-learn
 Requires-Dist: matplotlib>=3.10.0
 Requires-Dist: scipy>=1.15.0
+Requires-Dist: seaborn>=0.13.0
+Requires-Dist: nltk>=3.8.1
 
 # SQLShell
 
 <div align="center">
 
-<img src="https://github.com/oyvinrog/SQLShell/raw/main/sqlshell_logo.png" alt="SQLShell Logo" width="180" height="auto">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/sqlshell_logo.png" alt="SQLShell Logo" width="180" height="auto">
 
 **A powerful SQL shell with GUI interface for data analysis**
 
-<img src="https://github.com/oyvinrog/SQLShell/raw/main/sqlshell_demo.png" alt="SQLShell Interface" width="80%" height="auto">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/sqlshell_demo.png" alt="SQLShell Interface" width="80%" height="auto">
 
 </div>
 
@@ -53,9 +54,18 @@ Requires-Dist: scipy>=1.15.0
 - **Table Preview** - Quick view of imported data tables
 - **Test Data Generation** - Built-in sample data for testing and learning
 - **Multiple Views** - Support for multiple concurrent table views
-- **Productivity Tools** - Streamlined workflow with keyboard shortcuts (e.g., Ctrl+Enter for query execution)
+- **Productivity Tools** - Streamlined workflow with F5/F9 shortcuts and Ctrl+Enter for query execution
 - **Explain Column** - Analyze relationships between data columns directly from query results
 
+## ⚡ F5/F9 Quick Execution
+
+SQLShell includes powerful keyboard shortcuts for efficient SQL execution:
+
+- **F5**: Execute all SQL statements in the editor sequentially
+- **F9**: Execute only the current SQL statement (where your cursor is positioned)
+
+This allows for rapid testing and development - place your cursor in any statement and press F9 to execute just that query, or press F5 to run everything.
+
 ## 📦 Installation
 
 ### Using pip (Recommended)
@@ -166,7 +176,7 @@ FROM test_v;
 SQLShell provides powerful table profiling tools to help you understand your data. These tools are accessible from the left-hand side table menu via right-click on any table:
 
 <div align="center">
-<img src="https://github.com/oyvinrog/SQLShell/raw/main/column_profiler.png" alt="Column Profiler" width="80%" height="auto">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/column_profiler.png" alt="Column Profiler" width="80%" height="auto">
 </div>
 
 ### Table Profiling Options
@@ -261,7 +271,50 @@ The Column Profiler now offers in-depth multivariate analysis to explore relatio
    - Identify patterns and outliers in multivariate relationships
    - Make better decisions about feature selection for analysis and modeling
 
-This feature is particularly useful for exploratory data analysis, helping you understand your data structure and relationships on the fly.
+This feature is particularly useful for data scientists and analysts who need to understand variable relationships quickly without writing complex correlation queries.
+
+### One-hot encoding
+
+If you are working with text (i.e. job description or job title to analyze salary), you would want to 
+do 'one-hot encoding'. 
+
+1. **How to Access**:
+   - Right-click on any column header in the query results table
+   - Select "Encode text" from the context menu
+
+<div align="center">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/column_encoding.png" alt="Column Profiler" width="80%" height="auto">
+</div>
+
+2. **How It Works**:
+   - SQLShell tokenizes the text into meaningful words and phrases
+   - Each unique token becomes a new binary feature (1 if present, 0 if absent)
+   - The system applies intelligent filtering to remove common words with low information value
+   - Results appear as a new query with encoded columns automatically added
+
+<!-- Screenshot 2: Encoding process/dialog showing options -->
+
+3. **Applications**:
+   - Analyze how specific keywords in job descriptions correlate with salary levels
+   - Identify which terms in product descriptions drive higher sales
+   - Extract features from unstructured text for further analysis
+   - Prepare text data for statistical modeling and machine learning
+
+4. **Using the Encoded Data**:
+   - After encoding, SQLShell presents a visualization showing top correlations
+   - Sort encoded features by correlation strength to identify key terms
+   - Use encoded columns in subsequent queries for deeper analysis
+   - Join encoded results with other tables for cross-dataset insights
+
+<!-- Screenshot 3: Results showing correlation between job descriptions and salary -->
+
+5. **Benefits**:
+   - Transform unstructured text into structured, analyzable data
+   - Discover hidden patterns between text content and numerical outcomes
+   - Identify specific terms that have the strongest relationship with target variables
+   - Perform advanced text analysis without specialized NLP knowledge
+
+This feature is particularly powerful for HR analytics, marketing text analysis, and any scenario where you need to extract insights from unstructured text data.
 
 ## 📋 Requirements
 
@@ -279,3 +332,69 @@ This feature is particularly useful for exploratory data analysis, helping you u
 ## 📄 License
 
 This project is licensed under the MIT License - see the LICENSE file for details.
+
+## 📁 Project Structure
+
+```
+SQLShell/
+├── sqlshell/                   # Main package
+│   ├── __init__.py
+│   ├── main.py                 # Main application entry point
+│   ├── execution_handler.py    # F5/F9 SQL execution functionality
+│   ├── editor_integration.py   # Editor integration utilities
+│   ├── query_tab.py           # Query tab implementation
+│   ├── splash_screen.py       # Application splash screen
+│   └── styles.py              # UI styling
+├── tests/                     # Test files
+│   ├── f5_f9_functionality/   # F5/F9 functionality tests and demos
+│   │   ├── README.md          # Documentation for F5/F9 tests
+│   │   ├── test_execution_handler.py  # Comprehensive test suite
+│   │   └── demo_f5_f9.py      # Interactive demo
+│   └── test_query_executor.py # Other test files
+├── docs/                      # Documentation
+│   ├── F5_F9_FUNCTIONALITY.md # Detailed F5/F9 documentation
+│   └── IMPLEMENTATION_SUMMARY.md # Implementation details
+├── assets/                    # Assets and resources
+│   └── images/               # Images and screenshots
+│       ├── sqlshell_logo.png
+│       ├── sqlshell_demo.png
+│       ├── column_profiler.png
+│       └── column_encoding.png
+├── sample_data/              # Sample data files
+│   ├── test_*.csv           # Test CSV files
+│   ├── california_housing_data.parquet
+│   └── pool.db              # Sample database
+├── main.py                  # Application launcher
+├── run.py                   # Alternative launcher
+├── README.md               # This file
+├── requirements.txt        # Python dependencies
+├── pyproject.toml         # Project configuration
+└── MANIFEST.in            # Package manifest
+```
+
+## 🧪 Testing
+
+The project includes comprehensive tests for the F5/F9 functionality:
+
+```bash
+# Run the interactive test suite
+cd tests/f5_f9_functionality
+python test_execution_handler.py
+
+# Try the interactive demo
+python demo_f5_f9.py
+```
+
+For complete documentation on F5/F9 functionality, see `docs/F5_F9_FUNCTIONALITY.md`.
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Add tests if applicable
+5. Submit a pull request
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

{sqlshell-0.2.3 → sqlshell-0.3.0}/README.md

@@ -2,11 +2,11 @@
 
 <div align="center">
 
-<img src="https://github.com/oyvinrog/SQLShell/raw/main/sqlshell_logo.png" alt="SQLShell Logo" width="180" height="auto">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/sqlshell_logo.png" alt="SQLShell Logo" width="180" height="auto">
 
 **A powerful SQL shell with GUI interface for data analysis**
 
-<img src="https://github.com/oyvinrog/SQLShell/raw/main/sqlshell_demo.png" alt="SQLShell Interface" width="80%" height="auto">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/sqlshell_demo.png" alt="SQLShell Interface" width="80%" height="auto">
 
 </div>
 
@@ -20,9 +20,18 @@
 - **Table Preview** - Quick view of imported data tables
 - **Test Data Generation** - Built-in sample data for testing and learning
 - **Multiple Views** - Support for multiple concurrent table views
-- **Productivity Tools** - Streamlined workflow with keyboard shortcuts (e.g., Ctrl+Enter for query execution)
+- **Productivity Tools** - Streamlined workflow with F5/F9 shortcuts and Ctrl+Enter for query execution
 - **Explain Column** - Analyze relationships between data columns directly from query results
 
+## ⚡ F5/F9 Quick Execution
+
+SQLShell includes powerful keyboard shortcuts for efficient SQL execution:
+
+- **F5**: Execute all SQL statements in the editor sequentially
+- **F9**: Execute only the current SQL statement (where your cursor is positioned)
+
+This allows for rapid testing and development - place your cursor in any statement and press F9 to execute just that query, or press F5 to run everything.
+
 ## 📦 Installation
 
 ### Using pip (Recommended)
@@ -133,7 +142,7 @@ FROM test_v;
 SQLShell provides powerful table profiling tools to help you understand your data. These tools are accessible from the left-hand side table menu via right-click on any table:
 
 <div align="center">
-<img src="https://github.com/oyvinrog/SQLShell/raw/main/column_profiler.png" alt="Column Profiler" width="80%" height="auto">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/column_profiler.png" alt="Column Profiler" width="80%" height="auto">
 </div>
 
 ### Table Profiling Options
@@ -228,7 +237,50 @@ The Column Profiler now offers in-depth multivariate analysis to explore relatio
    - Identify patterns and outliers in multivariate relationships
    - Make better decisions about feature selection for analysis and modeling
 
-This feature is particularly useful for exploratory data analysis, helping you understand your data structure and relationships on the fly.
+This feature is particularly useful for data scientists and analysts who need to understand variable relationships quickly without writing complex correlation queries.
+
+### One-hot encoding
+
+If you are working with text (i.e. job description or job title to analyze salary), you would want to 
+do 'one-hot encoding'. 
+
+1. **How to Access**:
+   - Right-click on any column header in the query results table
+   - Select "Encode text" from the context menu
+
+<div align="center">
+<img src="https://raw.githubusercontent.com/oyvinrog/SQLShell/main/assets/images/column_encoding.png" alt="Column Profiler" width="80%" height="auto">
+</div>
+
+2. **How It Works**:
+   - SQLShell tokenizes the text into meaningful words and phrases
+   - Each unique token becomes a new binary feature (1 if present, 0 if absent)
+   - The system applies intelligent filtering to remove common words with low information value
+   - Results appear as a new query with encoded columns automatically added
+
+<!-- Screenshot 2: Encoding process/dialog showing options -->
+
+3. **Applications**:
+   - Analyze how specific keywords in job descriptions correlate with salary levels
+   - Identify which terms in product descriptions drive higher sales
+   - Extract features from unstructured text for further analysis
+   - Prepare text data for statistical modeling and machine learning
+
+4. **Using the Encoded Data**:
+   - After encoding, SQLShell presents a visualization showing top correlations
+   - Sort encoded features by correlation strength to identify key terms
+   - Use encoded columns in subsequent queries for deeper analysis
+   - Join encoded results with other tables for cross-dataset insights
+
+<!-- Screenshot 3: Results showing correlation between job descriptions and salary -->
+
+5. **Benefits**:
+   - Transform unstructured text into structured, analyzable data
+   - Discover hidden patterns between text content and numerical outcomes
+   - Identify specific terms that have the strongest relationship with target variables
+   - Perform advanced text analysis without specialized NLP knowledge
+
+This feature is particularly powerful for HR analytics, marketing text analysis, and any scenario where you need to extract insights from unstructured text data.
 
 ## 📋 Requirements
 
@@ -246,3 +298,69 @@ This feature is particularly useful for exploratory data analysis, helping you u
 ## 📄 License
 
 This project is licensed under the MIT License - see the LICENSE file for details.
+
+## 📁 Project Structure
+
+```
+SQLShell/
+├── sqlshell/                   # Main package
+│   ├── __init__.py
+│   ├── main.py                 # Main application entry point
+│   ├── execution_handler.py    # F5/F9 SQL execution functionality
+│   ├── editor_integration.py   # Editor integration utilities
+│   ├── query_tab.py           # Query tab implementation
+│   ├── splash_screen.py       # Application splash screen
+│   └── styles.py              # UI styling
+├── tests/                     # Test files
+│   ├── f5_f9_functionality/   # F5/F9 functionality tests and demos
+│   │   ├── README.md          # Documentation for F5/F9 tests
+│   │   ├── test_execution_handler.py  # Comprehensive test suite
+│   │   └── demo_f5_f9.py      # Interactive demo
+│   └── test_query_executor.py # Other test files
+├── docs/                      # Documentation
+│   ├── F5_F9_FUNCTIONALITY.md # Detailed F5/F9 documentation
+│   └── IMPLEMENTATION_SUMMARY.md # Implementation details
+├── assets/                    # Assets and resources
+│   └── images/               # Images and screenshots
+│       ├── sqlshell_logo.png
+│       ├── sqlshell_demo.png
+│       ├── column_profiler.png
+│       └── column_encoding.png
+├── sample_data/              # Sample data files
+│   ├── test_*.csv           # Test CSV files
+│   ├── california_housing_data.parquet
+│   └── pool.db              # Sample database
+├── main.py                  # Application launcher
+├── run.py                   # Alternative launcher
+├── README.md               # This file
+├── requirements.txt        # Python dependencies
+├── pyproject.toml         # Project configuration
+└── MANIFEST.in            # Package manifest
+```
+
+## 🧪 Testing
+
+The project includes comprehensive tests for the F5/F9 functionality:
+
+```bash
+# Run the interactive test suite
+cd tests/f5_f9_functionality
+python test_execution_handler.py
+
+# Try the interactive demo
+python demo_f5_f9.py
+```
+
+For complete documentation on F5/F9 functionality, see `docs/F5_F9_FUNCTIONALITY.md`.
+
+## 🤝 Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Add tests if applicable
+5. Submit a pull request
+
+## 📄 License
+
+This project is licensed under the MIT License - see the LICENSE file for details.

sqlshell-0.3.0/docs/F5_F9_FUNCTIONALITY.md

@@ -0,0 +1,265 @@
+# F5/F9 SQL Execution Functionality
+
+This document describes the new modular F5/F9 execution functionality added to SQLShell, which allows you to execute SQL statements individually or all at once.
+
+## Overview
+
+The F5/F9 functionality provides a convenient way to execute SQL statements in your editor:
+
+- **F5**: Execute all SQL statements in the editor
+- **F9**: Execute the current SQL statement (the statement containing the cursor)
+- **Ctrl+Enter**: Execute the entire editor content (existing behavior)
+
+## Features
+
+### 🚀 Key Features
+
+1. **Smart Statement Parsing**: Automatically detects individual SQL statements separated by semicolons
+2. **Comment Handling**: Properly handles SQL comments (`--` and `/* */`) and string literals
+3. **String Literal Support**: Correctly handles semicolons inside string literals
+4. **Cursor-Based Execution**: F9 executes only the statement where your cursor is positioned
+5. **Visual Feedback**: Clear status messages showing which statement was executed
+6. **Non-Destructive**: Doesn't modify your original query text
+
+### 🎯 Use Cases
+
+- **Development**: Test individual queries while developing complex multi-statement scripts
+- **Debugging**: Execute specific statements to isolate issues
+- **Data Exploration**: Run different analytical queries sequentially
+- **Migration Scripts**: Execute database migration statements one by one
+
+## How to Use
+
+### Basic Usage
+
+1. **Write Multiple SQL Statements**: Separate each statement with a semicolon (`;`)
+   ```sql
+   SELECT * FROM employees WHERE department = 'Engineering';
+   
+   SELECT * FROM projects WHERE status = 'Active';
+   
+   SELECT COUNT(*) FROM assignments;
+   ```
+
+2. **Execute All Statements (F5)**:
+   - Press `F5` or click the "F5 - Execute All" button
+   - All statements will be executed sequentially
+   - Results from the last statement will be displayed
+
+3. **Execute Current Statement (F9)**:
+   - Place your cursor anywhere within a SQL statement
+   - Press `F9` or click the "F9 - Execute Current" button
+   - Only that statement will be executed
+
+### Advanced Examples
+
+#### Example 1: Data Analysis Workflow
+```sql
+-- Step 1: Get basic employee stats
+SELECT department, COUNT(*) as emp_count, AVG(salary) as avg_salary
+FROM employees 
+GROUP BY department;
+
+-- Step 2: Find high performers
+SELECT name, department, salary
+FROM employees 
+WHERE salary > (SELECT AVG(salary) FROM employees)
+ORDER BY salary DESC;
+
+-- Step 3: Project workload analysis
+SELECT p.project_name, COUNT(a.employee_id) as team_size
+FROM projects p
+LEFT JOIN assignments a ON p.project_id = a.project_id
+GROUP BY p.project_name
+ORDER BY team_size DESC;
+```
+
+#### Example 2: Database Setup Script
+```sql
+-- Create temporary table
+CREATE TEMPORARY TABLE temp_analysis AS
+SELECT department, AVG(salary) as avg_salary
+FROM employees
+GROUP BY department;
+
+-- Analyze the data
+SELECT * FROM temp_analysis WHERE avg_salary > 65000;
+
+-- Clean up
+DROP TABLE IF EXISTS temp_analysis;
+```
+
+## Technical Implementation
+
+### Architecture
+
+The F5/F9 functionality is implemented using a modular architecture:
+
+```
+sqlshell/
+├── execution_handler.py      # Core parsing and execution logic
+├── editor_integration.py     # Integration with SQLEditor
+└── query_tab.py             # UI integration
+```
+
+### Key Components
+
+1. **SQLExecutionHandler**: Parses SQL text and handles execution
+2. **ExecutionKeyHandler**: Manages F5/F9 key press events
+3. **EditorExecutionIntegration**: Integrates with existing SQLEditor
+
+### Statement Parsing Features
+
+- **Comment Removal**: Safely removes `--` and `/* */` comments while preserving string literals
+- **String Handling**: Properly handles single and double quotes, including escaped quotes
+- **Semicolon Detection**: Smart detection of statement-ending semicolons
+- **Position Tracking**: Tracks start and end positions of each statement
+
+## Testing
+
+### Test Files
+
+- **`test_execution_handler.py`**: Comprehensive testing UI for F5/F9 functionality
+- **`demo_f5_f9.py`**: Demo script with sample data
+
+### Running Tests
+
+1. **Basic Tests** (command line):
+   ```bash
+   python test_execution_handler.py --test
+   ```
+
+2. **Interactive Test UI**:
+   ```bash
+   python test_execution_handler.py
+   ```
+
+3. **Full Demo with Sample Data**:
+   ```bash
+   python demo_f5_f9.py
+   ```
+
+## Integration Guide
+
+### For Developers
+
+To integrate F5/F9 functionality into an existing SQLEditor:
+
+```python
+from sqlshell.editor_integration import integrate_execution_functionality
+
+# Integrate with your editor
+integration = integrate_execution_functionality(
+    your_editor, 
+    execute_callback=your_execute_function
+)
+
+# The editor now supports F5/F9 automatically
+```
+
+### Custom Execution Callback
+
+```python
+def my_execute_callback(query_text):
+    """Custom function to execute a SQL query."""
+    # Your database execution logic here
+    result = database.execute(query_text)
+    # Handle result display
+    display_results(result)
+
+# Set the callback
+integration.set_execute_callback(my_execute_callback)
+```
+
+## User Interface
+
+### New UI Elements
+
+1. **F5 - Execute All Button**: Executes all statements sequentially
+2. **F9 - Execute Current Button**: Executes the statement at cursor position
+3. **Enhanced Help Text**: Updated instructions showing F5/F9 usage
+
+### Status Bar Messages
+
+- `Statement executed: [query preview] | Time: 0.15s | Rows: 42`
+- `Executed 4 statement(s)` (for F5)
+- Clear indication of which functionality was used
+
+## Keyboard Shortcuts
+
+| Shortcut | Action | Description |
+|----------|--------|-------------|
+| `F5` | Execute All | Run all SQL statements in the editor |
+| `F9` | Execute Current | Run the statement containing the cursor |
+| `Ctrl+Enter` | Execute Query | Run entire editor content (existing) |
+
+## Error Handling
+
+### Robust Error Management
+
+- **Empty Statements**: Gracefully skips empty statements
+- **Syntax Errors**: Shows clear error messages with statement context
+- **Connection Issues**: Handles database connection problems
+- **Parsing Errors**: Provides fallback behavior for complex SQL
+
+### Error Messages
+
+- Clear indication of which statement failed
+- Preservation of original query text
+- Non-blocking error handling (other statements can still execute)
+
+## Best Practices
+
+### Writing Multi-Statement SQL
+
+1. **Use Clear Separation**: Always end statements with semicolons
+2. **Add Comments**: Document what each statement does
+3. **Test Incrementally**: Use F9 to test each statement individually
+4. **Handle Dependencies**: Be aware of statement execution order for F5
+
+### Example Best Practice
+```sql
+-- Step 1: Data preparation
+CREATE TEMPORARY TABLE employee_stats AS
+SELECT department, COUNT(*) as count, AVG(salary) as avg_sal
+FROM employees 
+GROUP BY department;
+
+-- Step 2: Analysis (depends on Step 1)
+SELECT * FROM employee_stats 
+WHERE avg_sal > 70000
+ORDER BY count DESC;
+
+-- Step 3: Cleanup (always include cleanup)
+DROP TABLE IF EXISTS employee_stats;
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **F5/F9 Not Working**: Ensure the editor has focus and the integration is properly loaded
+2. **Wrong Statement Executed**: Check cursor position; F9 executes the statement containing the cursor
+3. **Parsing Issues**: Complex string literals with semicolons may need careful quoting
+
+### Debug Information
+
+Enable debug mode by running:
+```bash
+python test_execution_handler.py
+```
+
+This provides detailed parsing information and execution logs.
+
+## Future Enhancements
+
+### Planned Features
+
+- **Statement Highlighting**: Visual indication of which statement will be executed
+- **Execution History**: Track and replay previously executed statements
+- **Batch Execution Options**: More granular control over statement execution
+- **Performance Metrics**: Detailed timing and resource usage for each statement
+
+### Feedback and Contributions
+
+The F5/F9 functionality is designed to be modular and extensible. Contributions and feedback are welcome!