opencode-knowledge-work 0.0.0-stage → 0.1.0

package/content/data/skills/data-data-visualization/SKILL.md

@@ -0,0 +1,305 @@
+---
+name: data-visualization
+description: Create effective data visualizations with Python (matplotlib, seaborn, plotly). Use when building charts, choosing the right chart type for a dataset, creating publication-quality figures, or applying design principles like accessibility and color theory.
+user-invocable: false
+---
+
+# Data Visualization Skill
+
+Chart selection guidance, Python visualization code patterns, design principles, and accessibility considerations for creating effective data visualizations.
+
+## Chart Selection Guide
+
+### Choose by Data Relationship
+
+| What You're Showing | Best Chart | Alternatives |
+|---|---|---|
+| **Trend over time** | Line chart | Area chart (if showing cumulative or composition) |
+| **Comparison across categories** | Vertical bar chart | Horizontal bar (many categories), lollipop chart |
+| **Ranking** | Horizontal bar chart | Dot plot, slope chart (comparing two periods) |
+| **Part-to-whole composition** | Stacked bar chart | Treemap (hierarchical), waffle chart |
+| **Composition over time** | Stacked area chart | 100% stacked bar (for proportion focus) |
+| **Distribution** | Histogram | Box plot (comparing groups), violin plot, strip plot |
+| **Correlation (2 variables)** | Scatter plot | Bubble chart (add 3rd variable as size) |
+| **Correlation (many variables)** | Heatmap (correlation matrix) | Pair plot |
+| **Geographic patterns** | Choropleth map | Bubble map, hex map |
+| **Flow / process** | Sankey diagram | Funnel chart (sequential stages) |
+| **Relationship network** | Network graph | Chord diagram |
+| **Performance vs. target** | Bullet chart | Gauge (single KPI only) |
+| **Multiple KPIs at once** | Small multiples | Dashboard with separate charts |
+
+### When NOT to Use Certain Charts
+
+- **Pie charts**: Avoid unless <6 categories and exact proportions matter less than rough comparison. Humans are bad at comparing angles. Use bar charts instead.
+- **3D charts**: Never. They distort perception and add no information.
+- **Dual-axis charts**: Use cautiously. They can mislead by implying correlation. Clearly label both axes if used.
+- **Stacked bar (many categories)**: Hard to compare middle segments. Use small multiples or grouped bars instead.
+- **Donut charts**: Slightly better than pie charts but same fundamental issues. Use for single KPI display at most.
+
+## Python Visualization Code Patterns
+
+### Setup and Style
+
+```python
+import matplotlib.pyplot as plt
+import matplotlib.ticker as mticker
+import seaborn as sns
+import pandas as pd
+import numpy as np
+
+# Professional style setup
+plt.style.use('seaborn-v0_8-whitegrid')
+plt.rcParams.update({
+    'figure.figsize': (10, 6),
+    'figure.dpi': 150,
+    'font.size': 11,
+    'axes.titlesize': 14,
+    'axes.titleweight': 'bold',
+    'axes.labelsize': 11,
+    'xtick.labelsize': 10,
+    'ytick.labelsize': 10,
+    'legend.fontsize': 10,
+    'figure.titlesize': 16,
+})
+
+# Colorblind-friendly palettes
+PALETTE_CATEGORICAL = ['#4C72B0', '#DD8452', '#55A868', '#C44E52', '#8172B3', '#937860']
+PALETTE_SEQUENTIAL = 'YlOrRd'
+PALETTE_DIVERGING = 'RdBu_r'
+```
+
+### Line Chart (Time Series)
+
+```python
+fig, ax = plt.subplots(figsize=(10, 6))
+
+for label, group in df.groupby('category'):
+    ax.plot(group['date'], group['value'], label=label, linewidth=2)
+
+ax.set_title('Metric Trend by Category', fontweight='bold')
+ax.set_xlabel('Date')
+ax.set_ylabel('Value')
+ax.legend(loc='upper left', frameon=True)
+ax.spines['top'].set_visible(False)
+ax.spines['right'].set_visible(False)
+
+# Format dates on x-axis
+fig.autofmt_xdate()
+
+plt.tight_layout()
+plt.savefig('trend_chart.png', dpi=150, bbox_inches='tight')
+```
+
+### Bar Chart (Comparison)
+
+```python
+fig, ax = plt.subplots(figsize=(10, 6))
+
+# Sort by value for easy reading
+df_sorted = df.sort_values('metric', ascending=True)
+
+bars = ax.barh(df_sorted['category'], df_sorted['metric'], color=PALETTE_CATEGORICAL[0])
+
+# Add value labels
+for bar in bars:
+    width = bar.get_width()
+    ax.text(width + 0.5, bar.get_y() + bar.get_height()/2,
+            f'{width:,.0f}', ha='left', va='center', fontsize=10)
+
+ax.set_title('Metric by Category (Ranked)', fontweight='bold')
+ax.set_xlabel('Metric Value')
+ax.spines['top'].set_visible(False)
+ax.spines['right'].set_visible(False)
+
+plt.tight_layout()
+plt.savefig('bar_chart.png', dpi=150, bbox_inches='tight')
+```
+
+### Histogram (Distribution)
+
+```python
+fig, ax = plt.subplots(figsize=(10, 6))
+
+ax.hist(df['value'], bins=30, color=PALETTE_CATEGORICAL[0], edgecolor='white', alpha=0.8)
+
+# Add mean and median lines
+mean_val = df['value'].mean()
+median_val = df['value'].median()
+ax.axvline(mean_val, color='red', linestyle='--', linewidth=1.5, label=f'Mean: {mean_val:,.1f}')
+ax.axvline(median_val, color='green', linestyle='--', linewidth=1.5, label=f'Median: {median_val:,.1f}')
+
+ax.set_title('Distribution of Values', fontweight='bold')
+ax.set_xlabel('Value')
+ax.set_ylabel('Frequency')
+ax.legend()
+ax.spines['top'].set_visible(False)
+ax.spines['right'].set_visible(False)
+
+plt.tight_layout()
+plt.savefig('histogram.png', dpi=150, bbox_inches='tight')
+```
+
+### Heatmap
+
+```python
+fig, ax = plt.subplots(figsize=(10, 8))
+
+# Pivot data for heatmap format
+pivot = df.pivot_table(index='row_dim', columns='col_dim', values='metric', aggfunc='sum')
+
+sns.heatmap(pivot, annot=True, fmt=',.0f', cmap='YlOrRd',
+            linewidths=0.5, ax=ax, cbar_kws={'label': 'Metric Value'})
+
+ax.set_title('Metric by Row Dimension and Column Dimension', fontweight='bold')
+ax.set_xlabel('Column Dimension')
+ax.set_ylabel('Row Dimension')
+
+plt.tight_layout()
+plt.savefig('heatmap.png', dpi=150, bbox_inches='tight')
+```
+
+### Small Multiples
+
+```python
+categories = df['category'].unique()
+n_cats = len(categories)
+n_cols = min(3, n_cats)
+n_rows = (n_cats + n_cols - 1) // n_cols
+
+fig, axes = plt.subplots(n_rows, n_cols, figsize=(5*n_cols, 4*n_rows), sharex=True, sharey=True)
+axes = axes.flatten() if n_cats > 1 else [axes]
+
+for i, cat in enumerate(categories):
+    ax = axes[i]
+    subset = df[df['category'] == cat]
+    ax.plot(subset['date'], subset['value'], color=PALETTE_CATEGORICAL[i % len(PALETTE_CATEGORICAL)])
+    ax.set_title(cat, fontsize=12)
+    ax.spines['top'].set_visible(False)
+    ax.spines['right'].set_visible(False)
+
+# Hide empty subplots
+for j in range(i+1, len(axes)):
+    axes[j].set_visible(False)
+
+fig.suptitle('Trends by Category', fontsize=14, fontweight='bold', y=1.02)
+plt.tight_layout()
+plt.savefig('small_multiples.png', dpi=150, bbox_inches='tight')
+```
+
+### Number Formatting Helpers
+
+```python
+def format_number(val, format_type='number'):
+    """Format numbers for chart labels."""
+    if format_type == 'currency':
+        if abs(val) >= 1e9:
+            return f'${val/1e9:.1f}B'
+        elif abs(val) >= 1e6:
+            return f'${val/1e6:.1f}M'
+        elif abs(val) >= 1e3:
+            return f'${val/1e3:.1f}K'
+        else:
+            return f'${val:,.0f}'
+    elif format_type == 'percent':
+        return f'{val:.1f}%'
+    elif format_type == 'number':
+        if abs(val) >= 1e9:
+            return f'{val/1e9:.1f}B'
+        elif abs(val) >= 1e6:
+            return f'{val/1e6:.1f}M'
+        elif abs(val) >= 1e3:
+            return f'{val/1e3:.1f}K'
+        else:
+            return f'{val:,.0f}'
+    return str(val)
+
+# Usage with axis formatter
+ax.yaxis.set_major_formatter(mticker.FuncFormatter(lambda x, p: format_number(x, 'currency')))
+```
+
+### Interactive Charts with Plotly
+
+```python
+import plotly.express as px
+import plotly.graph_objects as go
+
+# Simple interactive line chart
+fig = px.line(df, x='date', y='value', color='category',
+              title='Interactive Metric Trend',
+              labels={'value': 'Metric Value', 'date': 'Date'})
+fig.update_layout(hovermode='x unified')
+fig.write_html('interactive_chart.html')
+fig.show()
+
+# Interactive scatter with hover data
+fig = px.scatter(df, x='metric_a', y='metric_b', color='category',
+                 size='size_metric', hover_data=['name', 'detail_field'],
+                 title='Correlation Analysis')
+fig.show()
+```
+
+## Design Principles
+
+### Color
+
+- **Use color purposefully**: Color should encode data, not decorate
+- **Highlight the story**: Use a bright accent color for the key insight; grey everything else
+- **Sequential data**: Use a single-hue gradient (light to dark) for ordered values
+- **Diverging data**: Use a two-hue gradient with neutral midpoint for data with a meaningful center
+- **Categorical data**: Use distinct hues, maximum 6-8 before it gets confusing
+- **Avoid red/green only**: 8% of men are red-green colorblind. Use blue/orange as primary pair
+
+### Typography
+
+- **Title states the insight**: "Revenue grew 23% YoY" beats "Revenue by Month"
+- **Subtitle adds context**: Date range, filters applied, data source
+- **Axis labels are readable**: Never rotated 90 degrees if avoidable. Shorten or wrap instead
+- **Data labels add precision**: Use on key points, not every single bar
+- **Annotation highlights**: Call out specific points with text annotations
+
+### Layout
+
+- **Reduce chart junk**: Remove gridlines, borders, backgrounds that don't carry information
+- **Sort meaningfully**: Categories sorted by value (not alphabetically) unless there's a natural order (months, stages)
+- **Appropriate aspect ratio**: Time series wider than tall (3:1 to 2:1); comparisons can be squarer
+- **White space is good**: Don't cram charts together. Give each visualization room to breathe
+
+### Accuracy
+
+- **Bar charts start at zero**: Always. A bar from 95 to 100 exaggerates a 5% difference
+- **Line charts can have non-zero baselines**: When the range of variation is meaningful
+- **Consistent scales across panels**: When comparing multiple charts, use the same axis range
+- **Show uncertainty**: Error bars, confidence intervals, or ranges when data is uncertain
+- **Label your axes**: Never make the reader guess what the numbers mean
+
+## Accessibility Considerations
+
+### Color Blindness
+
+- Never rely on color alone to distinguish data series
+- Add pattern fills, different line styles (solid, dashed, dotted), or direct labels
+- Test with a colorblind simulator (e.g., Coblis, Sim Daltonism)
+- Use the colorblind-friendly palette: `sns.color_palette("colorblind")`
+
+### Screen Readers
+
+- Include alt text describing the chart's key finding
+- Provide a data table alternative alongside the visualization
+- Use semantic titles and labels
+
+### General Accessibility
+
+- Sufficient contrast between data elements and background
+- Text size minimum 10pt for labels, 12pt for titles
+- Avoid conveying information only through spatial position (add labels)
+- Consider printing: does the chart work in black and white?
+
+### Accessibility Checklist
+
+Before sharing a visualization:
+- [ ] Chart works without color (patterns, labels, or line styles differentiate series)
+- [ ] Text is readable at standard zoom level
+- [ ] Title describes the insight, not just the data
+- [ ] Axes are labeled with units
+- [ ] Legend is clear and positioned without obscuring data
+- [ ] Data source and date range are noted

package/content/data/skills/data-explore-data/SKILL.md

@@ -0,0 +1,325 @@
+---
+name: explore-data
+description: Profile and explore a dataset to understand its shape, quality, and patterns. Use when encountering a new table or file, checking null rates and column distributions, spotting data quality issues like duplicates or suspicious values, or deciding which dimensions and metrics to analyze.
+argument-hint: "<table or file>"
+---
+
+# /explore-data - Profile and Explore a Dataset
+
+> If you see unfamiliar placeholders or need to check which tools are connected, see [CONNECTORS.md](../../CONNECTORS.md).
+
+Generate a comprehensive data profile for a table or uploaded file. Understand its shape, quality, and patterns before diving into analysis.
+
+## Usage
+
+```
+/explore-data <table_name or file>
+```
+
+## Workflow
+
+### 1. Access the Data
+
+**If a data warehouse MCP server is connected:**
+
+1. Resolve the table name (handle schema prefixes, suggest matches if ambiguous)
+2. Query table metadata: column names, types, descriptions if available
+3. Run profiling queries against the live data
+
+**If a file is provided (CSV, Excel, Parquet, JSON):**
+
+1. Read the file and load into a working dataset
+2. Infer column types from the data
+
+**If neither:**
+
+1. Ask the user to provide a table name (with their warehouse connected) or upload a file
+2. If they describe a table schema, provide guidance on what profiling queries to run
+
+### 2. Understand Structure
+
+Before analyzing any data, understand its structure:
+
+**Table-level questions:**
+- How many rows and columns?
+- What is the grain (one row per what)?
+- What is the primary key? Is it unique?
+- When was the data last updated?
+- How far back does the data go?
+
+**Column classification** — categorize each column as one of:
+- **Identifier**: Unique keys, foreign keys, entity IDs
+- **Dimension**: Categorical attributes for grouping/filtering (status, type, region, category)
+- **Metric**: Quantitative values for measurement (revenue, count, duration, score)
+- **Temporal**: Dates and timestamps (created_at, updated_at, event_date)
+- **Text**: Free-form text fields (description, notes, name)
+- **Boolean**: True/false flags
+- **Structural**: JSON, arrays, nested structures
+
+### 3. Generate Data Profile
+
+Run the following profiling checks:
+
+**Table-level metrics:**
+- Total row count
+- Column count and types breakdown
+- Approximate table size (if available from metadata)
+- Date range coverage (min/max of date columns)
+
+**All columns:**
+- Null count and null rate
+- Distinct count and cardinality ratio (distinct / total)
+- Most common values (top 5-10 with frequencies)
+- Least common values (bottom 5 to spot anomalies)
+
+**Numeric columns (metrics):**
+```
+min, max, mean, median (p50)
+standard deviation
+percentiles: p1, p5, p25, p75, p95, p99
+zero count
+negative count (if unexpected)
+```
+
+**String columns (dimensions, text):**
+```
+min length, max length, avg length
+empty string count
+pattern analysis (do values follow a format?)
+case consistency (all upper, all lower, mixed?)
+leading/trailing whitespace count
+```
+
+**Date/timestamp columns:**
+```
+min date, max date
+null dates
+future dates (if unexpected)
+distribution by month/week
+gaps in time series
+```
+
+**Boolean columns:**
+```
+true count, false count, null count
+true rate
+```
+
+**Present the profile as a clean summary table**, grouped by column type (dimensions, metrics, dates, IDs).
+
+### 4. Identify Data Quality Issues
+
+Apply the quality assessment framework below. Flag potential problems:
+
+- **High null rates**: Columns with >5% nulls (warn), >20% nulls (alert)
+- **Low cardinality surprises**: Columns that should be high-cardinality but aren't (e.g., a "user_id" with only 50 distinct values)
+- **High cardinality surprises**: Columns that should be categorical but have too many distinct values
+- **Suspicious values**: Negative amounts where only positive expected, future dates in historical data, obviously placeholder values (e.g., "N/A", "TBD", "test", "999999")
+- **Duplicate detection**: Check if there's a natural key and whether it has duplicates
+- **Distribution skew**: Extremely skewed numeric distributions that could affect averages
+- **Encoding issues**: Mixed case in categorical fields, trailing whitespace, inconsistent formats
+
+### 5. Discover Relationships and Patterns
+
+After profiling individual columns:
+
+- **Foreign key candidates**: ID columns that might link to other tables
+- **Hierarchies**: Columns that form natural drill-down paths (country > state > city)
+- **Correlations**: Numeric columns that move together
+- **Derived columns**: Columns that appear to be computed from others
+- **Redundant columns**: Columns with identical or near-identical information
+
+### 6. Suggest Interesting Dimensions and Metrics
+
+Based on the column profile, recommend:
+
+- **Best dimension columns** for slicing data (categorical columns with reasonable cardinality, 3-50 values)
+- **Key metric columns** for measurement (numeric columns with meaningful distributions)
+- **Time columns** suitable for trend analysis
+- **Natural groupings** or hierarchies apparent in the data
+- **Potential join keys** linking to other tables (ID columns, foreign keys)
+
+### 7. Recommend Follow-Up Analyses
+
+Suggest 3-5 specific analyses the user could run next:
+
+- "Trend analysis on [metric] by [time_column] grouped by [dimension]"
+- "Distribution deep-dive on [skewed_column] to understand outliers"
+- "Data quality investigation on [problematic_column]"
+- "Correlation analysis between [metric_a] and [metric_b]"
+- "Cohort analysis using [date_column] and [status_column]"
+
+## Output Format
+
+```
+## Data Profile: [table_name]
+
+### Overview
+- Rows: 2,340,891
+- Columns: 23 (8 dimensions, 6 metrics, 4 dates, 5 IDs)
+- Date range: 2021-03-15 to 2024-01-22
+
+### Column Details
+[summary table]
+
+### Data Quality Issues
+[flagged issues with severity]
+
+### Recommended Explorations
+[numbered list of suggested follow-up analyses]
+```
+
+---
+
+## Quality Assessment Framework
+
+### Completeness Score
+
+Rate each column:
+- **Complete** (>99% non-null): Green
+- **Mostly complete** (95-99%): Yellow -- investigate the nulls
+- **Incomplete** (80-95%): Orange -- understand why and whether it matters
+- **Sparse** (<80%): Red -- may not be usable without imputation
+
+### Consistency Checks
+
+Look for:
+- **Value format inconsistency**: Same concept represented differently ("USA", "US", "United States", "us")
+- **Type inconsistency**: Numbers stored as strings, dates in various formats
+- **Referential integrity**: Foreign keys that don't match any parent record
+- **Business rule violations**: Negative quantities, end dates before start dates, percentages > 100
+- **Cross-column consistency**: Status = "completed" but completed_at is null
+
+### Accuracy Indicators
+
+Red flags that suggest accuracy issues:
+- **Placeholder values**: 0, -1, 999999, "N/A", "TBD", "test", "xxx"
+- **Default values**: Suspiciously high frequency of a single value
+- **Stale data**: Updated_at shows no recent changes in an active system
+- **Impossible values**: Ages > 150, dates in the far future, negative durations
+- **Round number bias**: All values ending in 0 or 5 (suggests estimation, not measurement)
+
+### Timeliness Assessment
+
+- When was the table last updated?
+- What is the expected update frequency?
+- Is there a lag between event time and load time?
+- Are there gaps in the time series?
+
+## Pattern Discovery Techniques
+
+### Distribution Analysis
+
+For numeric columns, characterize the distribution:
+- **Normal**: Mean and median are close, bell-shaped
+- **Skewed right**: Long tail of high values (common for revenue, session duration)
+- **Skewed left**: Long tail of low values (less common)
+- **Bimodal**: Two peaks (suggests two distinct populations)
+- **Power law**: Few very large values, many small ones (common for user activity)
+- **Uniform**: Roughly equal frequency across range (often synthetic or random)
+
+### Temporal Patterns
+
+For time series data, look for:
+- **Trend**: Sustained upward or downward movement
+- **Seasonality**: Repeating patterns (weekly, monthly, quarterly, annual)
+- **Day-of-week effects**: Weekday vs. weekend differences
+- **Holiday effects**: Drops or spikes around known holidays
+- **Change points**: Sudden shifts in level or trend
+- **Anomalies**: Individual data points that break the pattern
+
+### Segmentation Discovery
+
+Identify natural segments by:
+- Finding categorical columns with 3-20 distinct values
+- Comparing metric distributions across segment values
+- Looking for segments with significantly different behavior
+- Testing whether segments are homogeneous or contain sub-segments
+
+### Correlation Exploration
+
+Between numeric columns:
+- Compute correlation matrix for all metric pairs
+- Flag strong correlations (|r| > 0.7) for investigation
+- Note: Correlation does not imply causation -- flag this explicitly
+- Check for non-linear relationships (e.g., quadratic, logarithmic)
+
+## Schema Understanding and Documentation
+
+### Schema Documentation Template
+
+When documenting a dataset for team use:
+
+```markdown
+## Table: [schema.table_name]
+
+**Description**: [What this table represents]
+**Grain**: [One row per...]
+**Primary Key**: [column(s)]
+**Row Count**: [approximate, with date]
+**Update Frequency**: [real-time / hourly / daily / weekly]
+**Owner**: [team or person responsible]
+
+### Key Columns
+
+| Column | Type | Description | Example Values | Notes |
+|--------|------|-------------|----------------|-------|
+| user_id | STRING | Unique user identifier | "usr_abc123" | FK to users.id |
+| event_type | STRING | Type of event | "click", "view", "purchase" | 15 distinct values |
+| revenue | DECIMAL | Transaction revenue in USD | 29.99, 149.00 | Null for non-purchase events |
+| created_at | TIMESTAMP | When the event occurred | 2024-01-15 14:23:01 | Partitioned on this column |
+
+### Relationships
+- Joins to `users` on `user_id`
+- Joins to `products` on `product_id`
+- Parent of `event_details` (1:many on event_id)
+
+### Known Issues
+- [List any known data quality issues]
+- [Note any gotchas for analysts]
+
+### Common Query Patterns
+- [Typical use cases for this table]
+```
+
+### Schema Exploration Queries
+
+When connected to a data warehouse, use these patterns to discover schema:
+
+```sql
+-- List all tables in a schema (PostgreSQL)
+SELECT table_name, table_type
+FROM information_schema.tables
+WHERE table_schema = 'public'
+ORDER BY table_name;
+
+-- Column details (PostgreSQL)
+SELECT column_name, data_type, is_nullable, column_default
+FROM information_schema.columns
+WHERE table_name = 'my_table'
+ORDER BY ordinal_position;
+
+-- Table sizes (PostgreSQL)
+SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
+FROM pg_catalog.pg_statio_user_tables
+ORDER BY pg_total_relation_size(relid) DESC;
+
+-- Row counts for all tables (general pattern)
+-- Run per-table: SELECT COUNT(*) FROM table_name
+```
+
+### Lineage and Dependencies
+
+When exploring an unfamiliar data environment:
+
+1. Start with the "output" tables (what reports or dashboards consume)
+2. Trace upstream: What tables feed into them?
+3. Identify raw/staging/mart layers
+4. Map the transformation chain from raw data to analytical tables
+5. Note where data is enriched, filtered, or aggregated
+
+## Tips
+
+- For very large tables (100M+ rows), profiling queries use sampling by default -- mention if you need exact counts
+- If exploring a new dataset for the first time, this command gives you the lay of the land before writing specific queries
+- The quality flags are heuristic -- not every flag is a real problem, but each is worth a quick look