aigroup-workflow 2.1.2 → 2.2.0

package/skills/python/pandas-pro/references/dataframe-operations.md

@@ -0,0 +1,420 @@
+# DataFrame Operations
+
+---
+
+## Overview
+
+DataFrame operations form the foundation of pandas work. This reference covers indexing, selection, filtering, and sorting with pandas 2.0+ best practices.
+
+---
+
+## Indexing and Selection
+
+### Label-Based Selection with `.loc[]`
+
+Use `.loc[]` for label-based indexing. Always preferred over chained indexing.
+
+```python
+import pandas as pd
+import numpy as np
+
+# Sample DataFrame
+df = pd.DataFrame({
+    'name': ['Alice', 'Bob', 'Charlie', 'Diana'],
+    'age': [25, 30, 35, 28],
+    'salary': [50000, 60000, 70000, 55000],
+    'department': ['Engineering', 'Sales', 'Engineering', 'Marketing']
+}, index=['a', 'b', 'c', 'd'])
+
+# Single value
+value = df.loc['a', 'name']  # 'Alice'
+
+# Single row (returns Series)
+row = df.loc['a']
+
+# Multiple rows
+rows = df.loc[['a', 'c']]
+
+# Row and column slices (inclusive on both ends)
+subset = df.loc['a':'c', 'name':'salary']
+
+# Boolean indexing with .loc
+adults = df.loc[df['age'] >= 30]
+
+# Boolean indexing with column selection
+adults_names = df.loc[df['age'] >= 30, 'name']
+
+# Multiple conditions
+engineering_seniors = df.loc[
+    (df['department'] == 'Engineering') & (df['age'] >= 30),
+    ['name', 'salary']
+]
+```
+
+### Position-Based Selection with `.iloc[]`
+
+Use `.iloc[]` for integer position-based indexing.
+
+```python
+# Single value by position
+value = df.iloc[0, 0]  # First row, first column
+
+# Single row by position
+first_row = df.iloc[0]
+
+# Slice rows (exclusive end, like Python)
+first_three = df.iloc[:3]
+
+# Specific rows and columns by position
+subset = df.iloc[[0, 2], [0, 2]]  # Rows 0,2 and columns 0,2
+
+# Range selection
+block = df.iloc[1:3, 0:2]  # Rows 1-2, columns 0-1
+```
+
+### When to Use `.loc[]` vs `.iloc[]`
+
+| Scenario | Use | Example |
+|----------|-----|---------|
+| Known column names | `.loc[]` | `df.loc[:, 'name']` |
+| Filter by condition | `.loc[]` | `df.loc[df['age'] > 25]` |
+| First/last N rows | `.iloc[]` | `df.iloc[:5]` or `df.iloc[-5:]` |
+| Specific row positions | `.iloc[]` | `df.iloc[[0, 5, 10]]` |
+| Unknown column order | `.iloc[]` | `df.iloc[:, 0]` |
+
+---
+
+## Filtering DataFrames
+
+### Boolean Masks
+
+```python
+# Single condition
+mask = df['age'] > 25
+filtered = df[mask]
+
+# Multiple conditions (use parentheses!)
+mask = (df['age'] > 25) & (df['salary'] < 65000)
+filtered = df[mask]
+
+# OR conditions
+mask = (df['department'] == 'Engineering') | (df['department'] == 'Sales')
+filtered = df[mask]
+
+# NOT condition
+mask = ~(df['department'] == 'Marketing')
+filtered = df[mask]
+```
+
+### Using `.query()` for Readable Filters
+
+```python
+# Simple query - more readable for complex conditions
+result = df.query('age > 25 and salary < 65000')
+
+# Using variables with @
+min_age = 25
+result = df.query('age > @min_age')
+
+# String comparisons
+result = df.query('department == "Engineering"')
+
+# In-list filtering
+depts = ['Engineering', 'Sales']
+result = df.query('department in @depts')
+
+# Complex expressions
+result = df.query('(age > 25) and (department != "Marketing")')
+```
+
+### Using `.isin()` for Multiple Values
+
+```python
+# Filter by multiple values
+departments = ['Engineering', 'Sales']
+filtered = df[df['department'].isin(departments)]
+
+# Negation
+filtered = df[~df['department'].isin(departments)]
+
+# Multiple columns
+conditions = {
+    'department': ['Engineering', 'Sales'],
+    'age': [25, 30, 35]
+}
+# Filter where department is in list AND age is in list
+mask = df['department'].isin(conditions['department']) & df['age'].isin(conditions['age'])
+```
+
+### String Filtering with `.str` Accessor
+
+```python
+df = pd.DataFrame({
+    'email': ['alice@example.com', 'bob@test.org', 'charlie@example.com'],
+    'name': ['Alice Smith', 'Bob Jones', 'Charlie Brown']
+})
+
+# Contains
+mask = df['email'].str.contains('example')
+
+# Starts/ends with
+mask = df['email'].str.endswith('.com')
+mask = df['name'].str.startswith('A')
+
+# Regex matching
+mask = df['email'].str.match(r'^[a-z]+@example\.com$')
+
+# Case-insensitive
+mask = df['name'].str.lower().str.contains('alice')
+# Or with case parameter
+mask = df['name'].str.contains('alice', case=False)
+
+# Handle NaN in string columns
+mask = df['email'].str.contains('example', na=False)
+```
+
+---
+
+## Sorting
+
+### Basic Sorting
+
+```python
+# Sort by single column (ascending)
+sorted_df = df.sort_values('age')
+
+# Sort descending
+sorted_df = df.sort_values('age', ascending=False)
+
+# Sort by multiple columns
+sorted_df = df.sort_values(['department', 'salary'], ascending=[True, False])
+
+# Sort by index
+sorted_df = df.sort_index()
+sorted_df = df.sort_index(ascending=False)
+```
+
+### Advanced Sorting
+
+```python
+# Sort with NaN handling
+df_with_nan = pd.DataFrame({
+    'name': ['Alice', 'Bob', 'Charlie'],
+    'score': [85.0, np.nan, 90.0]
+})
+
+# NaN at end (default)
+sorted_df = df_with_nan.sort_values('score', na_position='last')
+
+# NaN at beginning
+sorted_df = df_with_nan.sort_values('score', na_position='first')
+
+# Custom sort order using Categorical
+order = ['Marketing', 'Sales', 'Engineering']
+df['department'] = pd.Categorical(df['department'], categories=order, ordered=True)
+sorted_df = df.sort_values('department')
+
+# Sort by computed values without adding column
+sorted_df = df.iloc[df['name'].str.len().argsort()]
+```
+
+### In-Place Sorting
+
+```python
+# Modify DataFrame in place
+df.sort_values('age', inplace=True)
+
+# Reset index after sorting
+df.sort_values('age', inplace=True)
+df.reset_index(drop=True, inplace=True)
+
+# Or chain
+df = df.sort_values('age').reset_index(drop=True)
+```
+
+---
+
+## Column Operations
+
+### Adding and Modifying Columns
+
+```python
+# Add new column
+df['bonus'] = df['salary'] * 0.1
+
+# Conditional column with np.where
+df['seniority'] = np.where(df['age'] >= 30, 'Senior', 'Junior')
+
+# Multiple conditions with np.select
+conditions = [
+    df['age'] < 25,
+    df['age'] < 35,
+    df['age'] >= 35
+]
+choices = ['Junior', 'Mid', 'Senior']
+df['level'] = np.select(conditions, choices, default='Unknown')
+
+# Using .assign() for method chaining (returns new DataFrame)
+df_new = df.assign(
+    bonus=lambda x: x['salary'] * 0.1,
+    total_comp=lambda x: x['salary'] + x['salary'] * 0.1
+)
+```
+
+### Renaming Columns
+
+```python
+# Rename specific columns
+df = df.rename(columns={'name': 'full_name', 'age': 'years'})
+
+# Rename all columns with function
+df.columns = df.columns.str.lower().str.replace(' ', '_')
+
+# Using rename with function
+df = df.rename(columns=str.upper)
+```
+
+### Dropping Columns
+
+```python
+# Drop single column
+df = df.drop('bonus', axis=1)
+# Or
+df = df.drop(columns=['bonus'])
+
+# Drop multiple columns
+df = df.drop(columns=['bonus', 'level'])
+
+# Drop columns by condition
+cols_to_drop = [col for col in df.columns if col.startswith('temp_')]
+df = df.drop(columns=cols_to_drop)
+```
+
+### Reordering Columns
+
+```python
+# Explicit order
+new_order = ['name', 'department', 'age', 'salary']
+df = df[new_order]
+
+# Move specific column to front
+cols = ['salary'] + [c for c in df.columns if c != 'salary']
+df = df[cols]
+
+# Using .reindex()
+df = df.reindex(columns=['name', 'age', 'salary', 'department'])
+```
+
+---
+
+## Index Operations
+
+### Setting and Resetting Index
+
+```python
+# Set column as index
+df = df.set_index('name')
+
+# Reset index back to column
+df = df.reset_index()
+
+# Drop index completely
+df = df.reset_index(drop=True)
+
+# Set multiple columns as index (MultiIndex)
+df = df.set_index(['department', 'name'])
+```
+
+### Working with MultiIndex
+
+```python
+# Create MultiIndex DataFrame
+df = pd.DataFrame({
+    'department': ['Eng', 'Eng', 'Sales', 'Sales'],
+    'team': ['Backend', 'Frontend', 'East', 'West'],
+    'headcount': [10, 8, 15, 12]
+}).set_index(['department', 'team'])
+
+# Select from MultiIndex
+df.loc['Eng']  # All Eng rows
+df.loc[('Eng', 'Backend')]  # Specific row
+
+# Cross-section with .xs()
+df.xs('Backend', level='team')  # All Backend teams
+
+# Reset specific level
+df.reset_index(level='team')
+```
+
+---
+
+## Copying DataFrames
+
+### When to Use `.copy()`
+
+```python
+# ALWAYS copy when modifying a subset
+subset = df[df['age'] > 25].copy()
+subset['new_col'] = 100  # Safe, no SettingWithCopyWarning
+
+# Without copy - may raise warning or fail silently
+# BAD:
+# subset = df[df['age'] > 25]
+# subset['new_col'] = 100  # SettingWithCopyWarning!
+
+# Deep copy (default) - copies data
+df_copy = df.copy()  # or df.copy(deep=True)
+
+# Shallow copy - shares data, only copies structure
+df_shallow = df.copy(deep=False)
+```
+
+---
+
+## Best Practices Summary
+
+1. **Use `.loc[]` and `.iloc[]`** - Never use chained indexing
+2. **Parenthesize conditions** - `(cond1) & (cond2)` not `cond1 & cond2`
+3. **Use `.query()` for readability** - Especially with complex filters
+4. **Copy before modifying subsets** - Always use `.copy()`
+5. **Use vectorized operations** - Avoid row iteration for filtering
+6. **Handle NaN explicitly** - Use `na=False` in string operations
+7. **Prefer method chaining** - Use `.assign()` for column creation
+
+---
+
+## Anti-Patterns to Avoid
+
+```python
+# BAD: Chained indexing
+df['A']['B'] = value  # May not work, raises warning
+
+# GOOD: Use .loc
+df.loc[:, ('A', 'B')] = value
+# Or for row selection then assignment:
+df.loc[df['A'] > 0, 'B'] = value
+
+# BAD: Iterating for filtering
+result = []
+for idx, row in df.iterrows():
+    if row['age'] > 25:
+        result.append(row)
+
+# GOOD: Boolean indexing
+result = df[df['age'] > 25]
+
+# BAD: Multiple separate assignments
+df = df[df['age'] > 25]
+df = df[df['salary'] > 50000]
+
+# GOOD: Combined filter
+df = df[(df['age'] > 25) & (df['salary'] > 50000)]
+```
+
+---
+
+## Related References
+
+- `data-cleaning.md` - After selection, clean the data
+- `aggregation-groupby.md` - Group and aggregate filtered data
+- `performance-optimization.md` - Optimize filtering on large datasets