crushdataai 1.2.1 → 1.2.7

package/assets/antigravity/data-analyst.md

@@ -25,12 +25,16 @@ Before writing any code, ask the user:
 2. **Data Context**: Which tables contain the data? What time range?
 3. **Metric Definitions**: How does YOUR company define the key metrics? Any filters?
 
+4. **Script Organization**: All analysis scripts must be saved in an `analysis/` folder.
+5. **Python Environment**: Check for `venv` or `.venv`. If missing, run `python3 -m venv venv`. Install/Run inside venv.
+6. **Reports**: Save all validation/profiling outputs to `reports/` folder. Create if missing.
+
 ### 2. Secure Data Access
 - **Check Connections**: Run `npx crushdataai connections` first.
 - **Missing Data?**: If the data source is not listed (e.g. on Desktop/Database), **INSTRUCT** the user to run:
   `npx crushdataai connect`
-- **Get Code**: Use `npx crushdataai snippet <name>` to access data.
-- **Security**: **DO NOT** ask for credentials or manual file moves.
+- **Get Code**: **ALWAYS** use `npx crushdataai snippet <name>` to get loading code.
+- **Security**: **DO NOT** ask user to copy/move files to `data/`. Treat connected data as read-only.
 
 ---
 
@@ -67,9 +71,21 @@ print(f"Date range: {df['date'].min()} to {df['date'].max()}")
 print(f"Missing values:\n{df.isnull().sum()}")
 ```
 
-Ask: "I found X rows, Y users, dates from A to B. Does this match expectation?"
+72: Ask: "I found X rows, Y users, dates from A to B. Does this match expectation?"
+73: 
+74: ---
+75: 
+76: ## Step 3b: Data Cleaning & Transformation (ETL)
+77: 
+78: Based on profiling findings, perform necessary cleaning BEFORE analysis:
+79: - **Handle Missing Values**: Impute or drop based on logic.
+80: - **Remove Duplicates**: Check primary keys.
+81: - **Fix Data Types**: Ensure dates are datetime objects, numbers are numeric.
+82: - **Feature Engineering**: Create calculated fields needed for analysis.
+- **Script Location**: Save all cleaning/ETL scripts in `etl/` folder.
+83: 
+84: ---
 
----
 
 ## Step 4: Execute with Validation
 

package/assets/claude/SKILL.md

@@ -29,15 +29,27 @@ When user requests data analysis work (analyze, query, dashboard, metrics, EDA,
    - How does YOUR company define the key metrics?
    - Any filters to apply? (exclude test users, internal accounts?)
    - What timezone should I use for dates?
-   - What timezone should I use for dates?
+
+4. **Script Organization**
+    - Save all analysis scripts in an `analysis/` folder.
+    - Create this folder if it does not exist.
+   
+5. **Python Environment**
+   - Check for `venv` or `.venv`. If missing, `python3 -m venv venv`.
+   - Install dependencies in venv (`venv/bin/pip install`).
+   - Run scripts using venv python (`venv/bin/python`).
+
+6. **Reports**
+   - Save all profiling, validation, and findings to `reports/` folder.
+   - Create this folder if it does not exist.
 ```
 
 ### 1b. Secure Data Access
 - **Check Connections**: Run `npx crushdataai connections` first.
 - **Missing Data?**: If the data source is not listed (e.g. on Desktop/Database), **INSTRUCT** the user to run:
   `npx crushdataai connect`
-- **Get Code**: Use `npx crushdataai snippet <name>` to access data.
-- **Security**: **DO NOT** ask for credentials or manual file moves.
+- **Get Code**: **ALWAYS** use `npx crushdataai snippet <name>` to get loading code.
+- **Security**: **DO NOT** ask user to copy/move files to `data/`. Treat connected data as read-only.
 
 ### Step 2: Search Relevant Domains
 
@@ -97,6 +109,15 @@ FROM table;
 **Report findings to user before proceeding:**
 > "I found X rows, Y unique users, date range from A to B. Does this match your expectation?"
 
+### Step 3b: Data Cleaning & Transformation (ETL)
+
+**Address data quality issues found in Step 3:**
+1. **Cleaning**: Handle missing values, remove invalid duplicates, fix types.
+2. **Transformation**: Standardize categories, parse dates, normalize text.
+3. **Feature Engineering**: Create calculated columns needed for metrics.
+
+*Tip: Save cleaning scripts to `etl/` folder. Save processed data to `data/processed/`.*
+
 ### Step 4: Execute Analysis with Validation
 
 **Before JOINs:**

package/assets/copilot/data-analyst.prompt.md

@@ -18,13 +18,16 @@ Before coding, ask:
 - What business question should this answer?
 - Which tables contain the data?
 - How does YOUR company define key metrics?
+- **Script Folder**: Save scripts in `analysis/`. Create folder if needed.
+- **Python Env**: Check for `venv`. If missing, create `venv`. Always run inside venv.
+- **Reports**: Save all outputs to `reports/` folder (profiling, validation).
 
 ### 2. Secure Data Access
 - **Check Connections**: Run `npx crushdataai connections` first.
 - **Missing Data?**: If the data source is not listed (e.g. on Desktop/Database), **INSTRUCT** the user to run:
   `npx crushdataai connect`
-- **Get Code**: Use `npx crushdataai snippet <name>` to access data.
-- **Security**: **DO NOT** ask for credentials or manual file moves.
+- **Get Code**: **ALWAYS** use `npx crushdataai snippet <name>` to get loading code.
+- **Security**: **DO NOT** ask user to copy/move files to `data/`. Treat connected data as read-only.
 
 ### 3. Search Knowledge Base
 ```bash
@@ -40,6 +43,12 @@ print(f"Shape: {df.shape}, Dates: {df['date'].min()} to {df['date'].max()}")
 ```
 Report and confirm before proceeding.
 
+### 3b. Data Cleaning & Transformation (ETL)
+- Clean: Missing, duplicates, types
+- Transform: Feature engineering
+- Save: Scripts in `etl/`
+- Verify: Re-check shape
+
 ### 4. Validate
 - Verify JOINs
 - Check totals

package/assets/cursor/data-analyst.md

@@ -15,13 +15,16 @@ Before coding, ask:
 - What business question should this answer?
 - Which tables contain the data?
 - How does YOUR company define the key metrics?
+- **Script Folder**: Save scripts in `analysis/`. Create folder if needed.
+- **Python Env**: Check for `venv`. If missing, create `venv`. Always run inside venv.
+- **Reports**: Save all outputs to `reports/` folder (profiling, validation).
 
 ### 2. Secure Data Access
 - **Check Connections**: Run `npx crushdataai connections` first.
 - **Missing Data?**: If the data source is not listed (e.g. on Desktop/Database), **INSTRUCT** the user to run:
   `npx crushdataai connect`
-- **Get Code**: Use `npx crushdataai snippet <name>` to access data.
-- **Security**: **DO NOT** ask for credentials or manual file moves.
+- **Get Code**: **ALWAYS** use `npx crushdataai snippet <name>` to get loading code.
+- **Security**: **DO NOT** ask user to copy/move files to `data/`. Treat connected data as read-only.
 
 ### 2. Search Knowledge Base
 ```bash
@@ -38,6 +41,12 @@ print(f"Shape: {df.shape}, Date range: {df['date'].min()} to {df['date'].max()}"
 ```
 Report findings and ask user for confirmation.
 
+### 3b. Data Cleaning & Transformation (ETL)
+- **Clean**: Missing values, types, duplicates.
+- **Transform**: Standardize formats, create calculated fields.
+- **Save**: Scripts go to `etl/` folder.
+- **Verify**: Check data again after cleaning.
+
 ### 4. Validate Before Delivery
 - Check JOINs don't multiply rows unexpectedly
 - Verify totals seem reasonable

package/assets/kiro/data-analyst.md

@@ -14,13 +14,16 @@ Before writing code, gather:
 - Data tables/sources
 - Company-specific metric definitions
 - Time range and filters
+- **Script Folder**: Save scripts in `analysis/`. Create folder if needed.
+- **Python Env**: Check for `venv`. If missing, create `venv`. Always run inside venv.
+- **Reports**: Save all outputs to `reports/` folder (profiling, validation).
 
 ### 2. Secure Data Access
 - **Check Connections**: Run `npx crushdataai connections` first.
 - **Missing Data?**: If the data source is not listed (e.g. on Desktop/Database), **INSTRUCT** the user to run:
   `npx crushdataai connect`
-- **Get Code**: Use `npx crushdataai snippet <name>` to access data.
-- **Security**: **DO NOT** ask for credentials or manual file moves.
+- **Get Code**: **ALWAYS** use `npx crushdataai snippet <name>` to get loading code.
+- **Security**: **DO NOT** ask user to copy/move files to `data/`. Treat connected data as read-only.
 
 ### 3. Search Before Implementing
 ```bash
@@ -37,6 +40,12 @@ Run and report:
 
 Ask: "Does this match your expectation?"
 
+### 3b. Data Cleaning & Transformation (ETL)
+- Handle missing/duplicates
+- Fix types & formats
+- Create calculated fields
+- **Save**: Scripts go to `etl/` folder
+
 ### 4. Validate Before Delivery
 - Sanity check totals
 - Compare to benchmarks

package/assets/shared/workflows.csv

@@ -1,9 +1,10 @@
 Workflow Name,Step Number,Step Name,Description,Questions to Ask,Tools/Commands,Outputs,Common Mistakes
 Exploratory Data Analysis,1,Define Objectives,Understand what insights are needed,"What business questions should this EDA answer? Who is the primary audience for these findings?","None - conversation with stakeholder","Clear list of questions to answer","Starting analysis without clear goals"
-Exploratory Data Analysis,2,Data Profiling,Understand data structure shape and types,"How many rows do you expect? What date range should I focus on?","df.info(), df.describe(), df.isnull().sum(), df.dtypes","Data profile report with shape types and missing values","Skipping profiling and diving straight into analysis"
-Exploratory Data Analysis,3,Univariate Analysis,Analyze individual columns distributions,"Are there any columns I should focus on specifically?","df.hist(), df.value_counts(), df.describe()","Histograms and value distributions for key columns","Not checking for outliers or unexpected values"
-Exploratory Data Analysis,4,Bivariate Analysis,Relationships between variables,"Which relationships are most important to understand?","df.corr(), scatter plots, grouped statistics","Correlation matrix and scatter plots showing relationships","Missing important correlations by not testing all pairs"
-Exploratory Data Analysis,5,Document Findings,Summarize insights,"What format do you prefer for the findings summary?","Markdown report generation","Summary report with key insights and recommendations","Not prioritizing findings by business impact"
+Exploratory Data Analysis,2,Data Profiling,Understand data structure shape and types,"How many rows do you expect? What date range should I focus on?","df.info(), df.describe(), df.isnull().sum(), df.dtypes","Save profiling attributes to reports/profiling_report.md","Skipping profiling and diving straight into analysis"
+Exploratory Data Analysis,3,Data Cleaning,Handle missing/duplicates and fix types,"Are there known quality issues? Should we impute or drop?","df.fillna(), df.drop_duplicates(), df.astype(); Save script to etl/ folder","Cleaned dataset ready for analysis","Skipping cleaning leads to wrong distributions"
+Exploratory Data Analysis,4,Univariate Analysis,Analyze individual columns distributions,"Are there any columns I should focus on specifically?","df.hist(), df.value_counts(), df.describe()","Histograms and value distributions for key columns","Not checking for outliers or unexpected values"
+Exploratory Data Analysis,5,Bivariate Analysis,Relationships between variables,"Which relationships are most important to understand?","df.corr(), scatter plots, grouped statistics","Correlation matrix and scatter plots showing relationships","Missing important correlations by not testing all pairs"
+Exploratory Data Analysis,6,Document Findings,Summarize insights,"What format do you prefer for the findings summary?","Markdown report generation","Save summary report to reports/insights.md","Not prioritizing findings by business impact"
 Dashboard Creation,1,Define Audience,Who will use the dashboard and for what purpose,"Is this for executives (high-level KPIs) or analysts (detailed breakdowns)? How often will they view it?","None - conversation","Clear audience definition and use case","Building for wrong audience (too detailed for execs or too simple for analysts)"
 Dashboard Creation,2,Identify KPIs,What metrics matter most to track,"What are your top 5-7 metrics? Do you have targets for each?","Search industry metrics database","Prioritized list of KPIs with targets","Too many metrics (7+ KPIs causes cognitive overload)"
 Dashboard Creation,3,Data Preparation,Get data into usable format,"Which tables contain this data? What granularity (daily/weekly)?","SQL queries, pandas transformations","Clean aggregated data ready for visualization","Not validating data before visualization"
@@ -16,19 +17,22 @@ A/B Test Analysis,4,Statistical Analysis,Calculate significance and effect size,
 A/B Test Analysis,5,Interpret Results,What does this mean for business,"Should we roll out, iterate, or abandon based on results?","Business impact calculation","Actionable recommendation with expected impact","Declaring winner without considering practical significance"
 Cohort Analysis,1,Define Cohort,How to group users for analysis,"Should I cohort by signup date, first purchase, or another event?","None - conversation","Cohort definition documented","Using wrong cohort definition for the question"
 Cohort Analysis,2,Define Metric,What to measure over time,"Should I track retention, revenue, or activity? Over what time periods?","None - conversation","Metric and time periods defined","Measuring wrong metric for the business question"
-Cohort Analysis,3,Build Cohort Table,SQL for cohort pivot table,"Is there a specific date range to analyze?","SQL with window functions, pivot tables","Cohort table with periods as columns","Off-by-one errors in period calculations"
-Cohort Analysis,4,Visualize,Create retention heatmap,"Any specific cohorts to highlight?","Heatmap visualization","Color-coded retention heatmap","Using colors that don't show progression clearly"
-Cohort Analysis,5,Insights,Identify patterns and explain why,"Which cohorts performed best/worst?","Comparative analysis","Insights report with recommended actions","Not investigating WHY cohorts differ"
+Cohort Analysis,3,Data QC & Prep,Clean data and handle nulls before grouping,"Are user IDs unique? Any null dates?","check_duplicates(), handled nulls; Save script to etl/ folder","Clean cohort/event dataset","Including null dates in cohort groups"
+Cohort Analysis,4,Build Cohort Table,SQL for cohort pivot table,"Is there a specific date range to analyze?","SQL with window functions, pivot tables","Cohort table with periods as columns","Off-by-one errors in period calculations"
+Cohort Analysis,5,Visualize,Create retention heatmap,"Any specific cohorts to highlight?","Heatmap visualization","Color-coded retention heatmap","Using colors that don't show progression clearly"
+Cohort Analysis,6,Insights,Identify patterns and explain why,"Which cohorts performed best/worst?","Comparative analysis","Insights report with recommended actions","Not investigating WHY cohorts differ"
 Funnel Analysis,1,Define Steps,What are the funnel stages in order,"What is the first step? What is the final conversion event?","None - conversation","Ordered list of funnel steps","Missing steps or having steps out of order"
-Funnel Analysis,2,Count Users,How many users at each step,"What time window should I use for the funnel?","SQL to count distinct users per step","User counts at each stage","Counting sessions instead of unique users"
-Funnel Analysis,3,Calculate Drop-off,Where are users leaving,"Are there any known issues at specific steps?","Conversion rate between steps","Drop-off rates between each step","Comparing non-sequential steps"
-Funnel Analysis,4,Visualize,Create funnel chart,"Prefer horizontal bars or funnel shape?","Funnel or horizontal bar visualization","Funnel visualization","Not labeling percentages clearly"
-Funnel Analysis,5,Recommendations,How to improve conversion,"What lever do you have to improve each step?","Analysis of biggest opportunities","Prioritized list of improvement suggestions","Focusing on small improvements instead of biggest drop-offs"
+Funnel Analysis,2,Data QC & Prep,Ensure event data is clean,"Are event names standardized? Any missing sessions?","Mapping event names, removing bots; Save script to etl/ folder","Clean event stream","Counting duplicate events"
+Funnel Analysis,3,Count Users,How many users at each step,"What time window should I use for the funnel?","SQL to count distinct users per step","User counts at each stage","Counting sessions instead of unique users"
+Funnel Analysis,4,Calculate Drop-off,Where are users leaving,"Are there any known issues at specific steps?","Conversion rate between steps","Drop-off rates between each step","Comparing non-sequential steps"
+Funnel Analysis,5,Visualize,Create funnel chart,"Prefer horizontal bars or funnel shape?","Funnel or horizontal bar visualization","Funnel visualization","Not labeling percentages clearly"
+Funnel Analysis,6,Recommendations,How to improve conversion,"What lever do you have to improve each step?","Analysis of biggest opportunities","Prioritized list of improvement suggestions","Focusing on small improvements instead of biggest drop-offs"
 Time Series Analysis,1,Define Metric,What to analyze over time,"Daily revenue, weekly users, or monthly orders? How far back?","None - conversation","Metric and time range defined","Wrong granularity (too granular hides trends or too aggregated misses patterns)"
-Time Series Analysis,2,Aggregate,Group by time period,"Any specific date filters? Exclude weekends?","SQL with DATE_TRUNC, GROUP BY","Aggregated time series data","Timezone issues in date grouping"
-Time Series Analysis,3,Decompose,Identify trend seasonality residual,"Is there known seasonality (weekly/monthly/yearly)?","Seasonal decomposition, moving averages","Decomposed components visualization","Ignoring seasonality when comparing periods"
-Time Series Analysis,4,Compare Periods,YoY MoM WoW comparisons,"Which comparison periods matter most?","LAG functions, period-over-period calculations","Comparison table with growth rates","Comparing incomplete periods"
-Time Series Analysis,5,Forecast (optional),Predict future values,"Do you need forecasting? What horizon?","Simple forecasting models","Forecast with confidence intervals","Overfitting on historical data"
+Time Series Analysis,2,Data QC & Prep,Handle gaps and outliers in time series,"Missing dates? Extreme outliers?","Resampling, outlier detection; Save script to etl/ folder","Clean time series","Interpolating missing data incorrectly"
+Time Series Analysis,3,Aggregate,Group by time period,"Any specific date filters? Exclude weekends?","SQL with DATE_TRUNC, GROUP BY","Aggregated time series data","Timezone issues in date grouping"
+Time Series Analysis,4,Decompose,Identify trend seasonality residual,"Is there known seasonality (weekly/monthly/yearly)?","Seasonal decomposition, moving averages","Decomposed components visualization","Ignoring seasonality when comparing periods"
+Time Series Analysis,5,Compare Periods,YoY MoM WoW comparisons,"Which comparison periods matter most?","LAG functions, period-over-period calculations","Comparison table with growth rates","Comparing incomplete periods"
+Time Series Analysis,6,Forecast (optional),Predict future values,"Do you need forecasting? What horizon?","Simple forecasting models","Forecast with confidence intervals","Overfitting on historical data"
 Customer Segmentation,1,Define Variables,What to segment on,"RFM, behavior, or demographics? What actions should differ by segment?","None - conversation","Segmentation variables defined","Choosing variables that don't drive different actions"
 Customer Segmentation,2,Feature Engineering,Calculate segment variables,"What time window for calculating features?","SQL or Python for RFM or other features","Feature table ready for segmentation","Using raw values instead of normalized scores"
 Customer Segmentation,3,Clustering,Group similar customers,"How many segments should we create?","K-means or rule-based segmentation","Cluster assignments","Too many or too few segments"

package/assets/windsurf/data-analyst.md

@@ -12,13 +12,16 @@ Ask before coding:
 - Business question this analysis should answer
 - Which tables/databases contain the data
 - Company-specific metric definitions
+- **Script Folder**: Save scripts in `analysis/`. Create folder if needed.
+- **Python Env**: Check for `venv`. If missing, create `venv`. Always run inside venv.
+- **Reports**: Save all outputs to `reports/` folder (profiling, validation).
 
 ### 2. Secure Data Access
 - **Check Connections**: Run `npx crushdataai connections` first.
 - **Missing Data?**: If the data source is not listed (e.g. on Desktop/Database), **INSTRUCT** the user to run:
   `npx crushdataai connect`
-- **Get Code**: Use `npx crushdataai snippet <name>` to access data.
-- **Security**: **DO NOT** ask for credentials or manual file moves.
+- **Get Code**: **ALWAYS** use `npx crushdataai snippet <name>` to get loading code.
+- **Security**: **DO NOT** ask user to copy/move files to `data/`. Treat connected data as read-only.
 
 ### 3. Search Knowledge
 ```bash
@@ -35,6 +38,12 @@ Run profiling before any analysis:
 print(f"Shape: {df.shape}, Dates: {df['date'].min()} to {df['date'].max()}")
 ```
 
+### 3b. Data Cleaning & Transformation (ETL)
+- Handle missing values/duplicates
+- Fix data types
+- Create calculated fields
+- **Save**: Scripts go to `etl/` folder
+
 ### 4. Validate
 - Verify JOINs don't multiply rows
 - Check totals are reasonable

package/dist/connectors/shopify/index.d.ts

@@ -4,6 +4,7 @@ export declare class ShopifyConnector implements Connector {
     type: string;
     test(connection: Connection): Promise<boolean>;
     getTables(connection: Connection): Promise<Table[]>;
+    private fetchTotalCount;
     getData(connection: Connection, tableName: string, page: number, limit: number): Promise<TableData>;
     getSnippet(connection: Connection, lang: string): string;
 }

package/dist/connectors/shopify/index.js

@@ -6,23 +6,168 @@ class ShopifyConnector {
         this.type = 'shopify';
     }
     async test(connection) {
+        console.log(`[Shopify] Testing connection for ${connection.name} (Store: ${connection.store})`);
         if (!connection.store || !connection.apiKey) {
-            throw new Error('Store URL and API Key are required');
+            throw new Error('Store URL and Admin API Access Token are required');
+        }
+        // Basic validation for Store URL
+        if (connection.store.includes('_') && !connection.store.includes('.')) {
+            throw new Error(`Invalid Store URL: "${connection.store}". It looks like you might have pasted an API key/secret here. The Store URL should be something like "your-shop.myshopify.com".`);
+        }
+        const storeUrl = connection.store.replace(/\/$/, '');
+        const url = `${storeUrl.startsWith('http') ? '' : 'https://'}${storeUrl}/admin/api/2024-04/shop.json`;
+        try {
+            const response = await fetch(url, {
+                headers: {
+                    'X-Shopify-Access-Token': connection.apiKey,
+                    'Content-Type': 'application/json'
+                }
+            });
+            if (!response.ok) {
+                const errorData = await response.json();
+                console.error(`[Shopify] Test failed:`, errorData);
+                throw new Error(errorData.errors || `Shopify API error: ${response.statusText}`);
+            }
+            console.log(`[Shopify] Connection test successful for ${connection.name}`);
+            return true;
+        }
+        catch (error) {
+            console.error(`[Shopify] Connection test error:`, error.message);
+            throw new Error(`Connection failed: ${error.message}`);
         }
-        return true;
     }
     async getTables(connection) {
-        return [];
+        console.log(`[Shopify] getTables called for ${connection.name}`);
+        const tables = ['orders', 'products', 'customers'];
+        const result = [];
+        for (const table of tables) {
+            try {
+                const count = await this.fetchTotalCount(connection, table);
+                result.push({ name: table, type: 'shopify', rowCount: count });
+            }
+            catch (e) {
+                console.warn(`[Shopify] Could not fetch count for ${table}:`, e);
+                result.push({ name: table, type: 'shopify' });
+            }
+        }
+        return result;
+    }
+    async fetchTotalCount(connection, tableName) {
+        if (!connection.store || !connection.apiKey)
+            return null;
+        const storeUrl = connection.store.replace(/\/$/, '');
+        const url = `${storeUrl.startsWith('http') ? '' : 'https://'}${storeUrl}/admin/api/2024-04/${tableName}/count.json`;
+        try {
+            const response = await fetch(url, {
+                headers: {
+                    'X-Shopify-Access-Token': connection.apiKey,
+                    'Content-Type': 'application/json'
+                }
+            });
+            if (!response.ok)
+                return null;
+            const data = await response.json();
+            return data.count;
+        }
+        catch {
+            return null;
+        }
     }
     async getData(connection, tableName, page, limit) {
-        return {
-            columns: [],
-            rows: [],
-            pagination: { page, limit, totalRows: 0, totalPages: 0, startIdx: 0, endIdx: 0 }
-        };
+        if (!connection.store || !connection.apiKey) {
+            throw new Error('Store URL and Admin API Access Token are required');
+        }
+        const storeUrl = connection.store.replace(/\/$/, '');
+        let url = `${storeUrl.startsWith('http') ? '' : 'https://'}${storeUrl}/admin/api/2024-04/${tableName}.json?limit=${limit}`;
+        // Add status=any for orders to fetch more than just 'open' orders
+        if (tableName === 'orders') {
+            url += '&status=any';
+        }
+        console.log(`[Shopify] Fetching ${tableName} from: ${url}`);
+        try {
+            const response = await fetch(url, {
+                headers: {
+                    'X-Shopify-Access-Token': connection.apiKey,
+                    'Content-Type': 'application/json'
+                }
+            });
+            if (!response.ok) {
+                const errorData = await response.json();
+                console.error(`[Shopify] Error response:`, errorData);
+                throw new Error(errorData.errors || `Shopify API error: ${response.statusText}`);
+            }
+            const data = await response.json();
+            const rawRows = data[tableName] || [];
+            console.log(`[Shopify] Successfully fetched ${rawRows.length} rows for ${tableName}`);
+            // Transform Shopify nested JSON into flat rows for the UI
+            const rows = rawRows.map((item) => {
+                const flat = {};
+                for (const key in item) {
+                    if (typeof item[key] === 'object' && item[key] !== null) {
+                        flat[key] = JSON.stringify(item[key]);
+                    }
+                    else {
+                        flat[key] = item[key];
+                    }
+                }
+                return flat;
+            });
+            const columns = rows.length > 0 ? Object.keys(rows[0]) : [];
+            const totalRows = (await this.fetchTotalCount(connection, tableName)) || rows.length;
+            const totalPages = Math.ceil(totalRows / limit) || 1;
+            return {
+                columns,
+                rows,
+                pagination: {
+                    page,
+                    limit,
+                    totalRows,
+                    totalPages,
+                    startIdx: (page - 1) * limit + 1,
+                    endIdx: (page - 1) * limit + rows.length
+                }
+            };
+        }
+        catch (error) {
+            console.error(`[Shopify] Fetch error:`, error.message);
+            throw new Error(`Failed to fetch data: ${error.message}`);
+        }
     }
     getSnippet(connection, lang) {
-        return `# Shopify snippet generation not implemented yet`;
+        const storeUrl = connection.store?.replace(/\/$/, '') || 'your-shop.myshopify.com';
+        const apiKey = connection.apiKey || 'shpat_...';
+        if (lang === 'python') {
+            return `import requests
+import pandas as pd
+
+# Connection: ${connection.name}
+# Type: shopify
+shop_url = "${storeUrl}"
+access_token = "${apiKey}"
+api_version = "2024-04"
+
+def fetch_shopify_data(endpoint):
+    url = f"{'https://' if not shop_url.startswith('http') else ''}{shop_url}/admin/api/{api_version}/{endpoint}.json"
+    headers = {
+        "X-Shopify-Access-Token": access_token,
+        "Content-Type": "application/json"
+    }
+    
+    response = requests.get(url, headers=headers)
+    response.raise_for_status()
+    return response.json()
+
+try:
+    # Example: Fetching orders
+    data = fetch_shopify_data("orders")
+    df = pd.DataFrame(data["orders"])
+    print(f"Successfully loaded {len(df)} orders from ${connection.name}")
+    print(df.head())
+except Exception as e:
+    print(f"Error loading Shopify data: {e}")
+`;
+        }
+        return `# Language ${lang} not supported for Shopify connector yet.`;
     }
 }
 exports.ShopifyConnector = ShopifyConnector;

package/dist/server.js

@@ -67,7 +67,7 @@ app.use(express_1.default.static(uiPath));
 app.use('/api/connections', connections_1.default);
 // API: Health check
 app.get('/api/health', (_req, res) => {
-    res.json({ status: 'ok', version: '2.0.0' });
+    res.json({ status: 'ok', version: '1.2.6' });
 });
 // Serve index.html for root
 app.get('/', (_req, res) => {

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "crushdataai",
-    "version": "1.2.1",
+    "version": "1.2.7",
     "description": "CLI to install CrushData AI data analyst skill for AI coding assistants",
     "main": "dist/index.js",
     "bin": {
@@ -48,4 +48,4 @@
         "type": "git",
         "url": "git+https://github.com/SankaiAI/crushdataai-agent.git"
     }
-}
+}

package/ui/index.html

@@ -54,13 +54,6 @@
       </svg>
       <h3>No data source selected</h3>
       <p>Select a connection from the sidebar or add a new one</p>
-      <button class="btn" id="add-first-btn">
-        <svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
-          <line x1="12" y1="5" x2="12" y2="19" />
-          <line x1="5" y1="12" x2="19" y2="12" />
-        </svg>
-        Add Data Source
-      </button>
     </div>
 
     <!-- Connection Details (shown when selected) -->

package/ui/main.js

@@ -258,7 +258,6 @@ function closeModal() {
 }
 
 document.getElementById('add-connection-btn').addEventListener('click', openModal);
-document.getElementById('add-first-btn').addEventListener('click', openModal);
 document.getElementById('modal-close').addEventListener('click', closeModal);
 document.getElementById('modal-overlay').addEventListener('click', (e) => {
     if (e.target.id === 'modal-overlay') closeModal();

package/ui/styles.css

@@ -1,15 +1,31 @@
 :root {
-  --primary: #8B5CF6;
-  --primary-dark: #7C3AED;
+  /* Crushdataai brand palette - matching landing page */
+  --primary: #7C3AED;
+  --primary-dark: #6D28D9;
   --primary-light: #A78BFA;
-  --bg: #F8FAFC;
-  --sidebar-bg: #FFFFFF;
+  --primary-glow: rgba(124, 58, 237, 0.12);
+
+  /* Refined backgrounds */
+  --bg: #FAFAFB;
   --card: #FFFFFF;
-  --border: #E2E8F0;
-  --border-light: #F1F5F9;
-  --text: #1E293B;
-  --text-muted: #64748B;
-  --text-light: #94A3B8;
+  --border: #E5E7EB;
+  --border-light: #F3F4F6;
+
+  /* Brand-matched dark sidebar */
+  --sidebar-bg: linear-gradient(180deg, #1F1F3D 0%, #171728 100%);
+  --sidebar-surface: rgba(255, 255, 255, 0.05);
+  --sidebar-hover: rgba(255, 255, 255, 0.08);
+  --sidebar-active: rgba(124, 58, 237, 0.25);
+  --sidebar-text: #F8FAFC;
+  --sidebar-text-muted: #A1A1AA;
+  --sidebar-border: rgba(255, 255, 255, 0.08);
+
+  /* Text colors */
+  --text: #18181B;
+  --text-muted: #71717A;
+  --text-light: #A1A1AA;
+
+  /* Status colors */
   --success: #10B981;
   --error: #EF4444;
   --warning: #F59E0B;
@@ -33,18 +49,19 @@ body {
 .sidebar {
   width: 280px;
   background: var(--sidebar-bg);
-  border-right: 1px solid var(--border);
+  border-right: 1px solid var(--sidebar-border);
   display: flex;
   flex-direction: column;
   height: 100vh;
   position: fixed;
   left: 0;
   top: 0;
+  box-shadow: 4px 0 32px rgba(0, 0, 0, 0.15);
 }
 
 .sidebar-header {
   padding: 1.25rem;
-  border-bottom: 1px solid var(--border);
+  border-bottom: 1px solid var(--sidebar-border);
 }
 
 .logo {
@@ -61,6 +78,7 @@ body {
   display: flex;
   align-items: center;
   justify-content: center;
+  box-shadow: 0 4px 12px rgba(124, 58, 237, 0.35);
 }
 
 .logo-icon svg {
@@ -72,10 +90,11 @@ body {
 .logo-text {
   font-size: 1.1rem;
   font-weight: 700;
+  color: var(--sidebar-text);
 }
 
 .logo-text span {
-  background: linear-gradient(135deg, var(--primary), #EC4899);
+  background: linear-gradient(135deg, var(--primary-light), #F472B6);
   -webkit-background-clip: text;
   background-clip: text;
   -webkit-text-fill-color: transparent;
@@ -91,8 +110,8 @@ body {
   font-size: 0.7rem;
   font-weight: 600;
   text-transform: uppercase;
-  letter-spacing: 0.05em;
-  color: var(--text-light);
+  letter-spacing: 0.08em;
+  color: var(--sidebar-text-muted);
   margin-bottom: 0.75rem;
   padding: 0 0.5rem;
 }
@@ -108,24 +127,25 @@ body {
   padding: 0.625rem 0.75rem;
   border-radius: 8px;
   cursor: pointer;
-  transition: all 0.15s;
+  transition: all 0.2s ease;
   margin-bottom: 0.25rem;
+  color: var(--sidebar-text);
 }
 
 .connection-item:hover {
-  background: var(--border-light);
+  background: var(--sidebar-hover);
 }
 
 .connection-item.active {
-  background: rgba(139, 92, 246, 0.1);
-  color: var(--primary);
+  background: var(--sidebar-active);
+  color: var(--primary-light);
 }
 
 .connection-item .icon {
   font-size: 1.25rem;
   width: 32px;
   height: 32px;
-  background: var(--border-light);
+  background: var(--sidebar-hover);
   border-radius: 6px;
   display: flex;
   align-items: center;
@@ -133,7 +153,7 @@ body {
 }
 
 .connection-item.active .icon {
-  background: rgba(139, 92, 246, 0.15);
+  background: var(--primary-glow);
 }
 
 .connection-item .info {
@@ -151,7 +171,7 @@ body {
 
 .connection-item .type {
   font-size: 0.75rem;
-  color: var(--text-muted);
+  color: var(--sidebar-text-muted);
 }
 
 .connection-item .status {
@@ -168,19 +188,19 @@ body {
   width: 100%;
   padding: 0.625rem 0.75rem;
   background: transparent;
-  border: 1px dashed var(--border);
+  border: 1px dashed var(--sidebar-text-muted);
   border-radius: 8px;
-  color: var(--text-muted);
+  color: var(--sidebar-text-muted);
   font-size: 0.875rem;
   cursor: pointer;
-  transition: all 0.15s;
+  transition: all 0.2s ease;
   margin-top: 0.5rem;
 }
 
 .add-connection-btn:hover {
-  border-color: var(--primary);
-  color: var(--primary);
-  background: rgba(139, 92, 246, 0.05);
+  border-color: var(--primary-light);
+  color: var(--primary-light);
+  background: var(--primary-glow);
 }
 
 /* Main Content */
@@ -196,6 +216,15 @@ body {
   /* Prevents flex item from overflowing */
 }
 
+/* Centered empty state wrapper */
+.main>.empty-state {
+  flex: 1;
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+  justify-content: center;
+}
+
 .main-header {
   display: flex;
   justify-content: space-between;
@@ -325,14 +354,15 @@ body {
 }
 
 .empty-state svg {
-  width: 64px;
-  height: 64px;
-  margin-bottom: 1rem;
-  opacity: 0.4;
+  width: 80px;
+  height: 80px;
+  margin-bottom: 1.5rem;
+  opacity: 0.3;
+  stroke: var(--primary);
 }
 
 .empty-state h3 {
-  font-size: 1rem;
+  font-size: 1.25rem;
   font-weight: 600;
   color: var(--text);
   margin-bottom: 0.5rem;
@@ -341,6 +371,7 @@ body {
 .empty-state p {
   font-size: 0.875rem;
   margin-bottom: 1.5rem;
+  max-width: 320px;
 }
 
 /* Buttons */
@@ -354,15 +385,22 @@ body {
   font-weight: 500;
   font-family: inherit;
   cursor: pointer;
-  transition: all 0.2s;
+  transition: all 0.2s ease;
   display: inline-flex;
   align-items: center;
   gap: 0.5rem;
+  box-shadow: 0 2px 8px rgba(124, 58, 237, 0.25);
 }
 
 .btn:hover {
-  transform: translateY(-1px);
-  box-shadow: 0 4px 12px rgba(139, 92, 246, 0.3);
+  transform: translateY(-2px);
+  box-shadow: 0 4px 16px rgba(124, 58, 237, 0.35);
+}
+
+/* Empty state button - smaller */
+.empty-state .btn {
+  padding: 0.5rem 1rem !important;
+  font-size: 0.8125rem !important;
 }
 
 .btn-secondary {