@synsci/cli-darwin-x64-baseline 1.1.76 → 1.1.78

package/bin/skills/scikit-learn/references/unsupervised_learning.md

@@ -0,0 +1,505 @@
+# Unsupervised Learning Reference
+
+## Overview
+
+Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.
+
+## Clustering
+
+### K-Means
+
+**KMeans (`sklearn.cluster.KMeans`)**
+- Partition-based clustering into K clusters
+- Key parameters:
+  - `n_clusters`: Number of clusters to form
+  - `init`: Initialization method ('k-means++', 'random')
+  - `n_init`: Number of initializations (default=10)
+  - `max_iter`: Maximum iterations
+- Use when: Know number of clusters, spherical cluster shapes
+- Fast and scalable
+- Example:
+```python
+from sklearn.cluster import KMeans
+
+model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
+labels = model.fit_predict(X)
+centers = model.cluster_centers_
+
+# Inertia (sum of squared distances to nearest center)
+print(f"Inertia: {model.inertia_}")
+```
+
+**MiniBatchKMeans**
+- Faster K-Means using mini-batches
+- Use when: Large datasets, need faster training
+- Slightly less accurate than K-Means
+- Example:
+```python
+from sklearn.cluster import MiniBatchKMeans
+
+model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
+labels = model.fit_predict(X)
+```
+
+### Density-Based Clustering
+
+**DBSCAN (`sklearn.cluster.DBSCAN`)**
+- Density-Based Spatial Clustering
+- Key parameters:
+  - `eps`: Maximum distance between two samples to be neighbors
+  - `min_samples`: Minimum samples in neighborhood to form core point
+  - `metric`: Distance metric
+- Use when: Arbitrary cluster shapes, presence of noise/outliers
+- Automatically determines number of clusters
+- Labels noise points as -1
+- Example:
+```python
+from sklearn.cluster import DBSCAN
+
+model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
+labels = model.fit_predict(X)
+
+# Number of clusters (excluding noise)
+n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
+n_noise = list(labels).count(-1)
+print(f"Clusters: {n_clusters}, Noise points: {n_noise}")
+```
+
+**HDBSCAN (`sklearn.cluster.HDBSCAN`)**
+- Hierarchical DBSCAN with adaptive epsilon
+- More robust than DBSCAN
+- Key parameter: `min_cluster_size`
+- Use when: Varying density clusters
+- Example:
+```python
+from sklearn.cluster import HDBSCAN
+
+model = HDBSCAN(min_cluster_size=10, min_samples=5)
+labels = model.fit_predict(X)
+```
+
+**OPTICS (`sklearn.cluster.OPTICS`)**
+- Ordering points to identify clustering structure
+- Similar to DBSCAN but doesn't require eps parameter
+- Key parameters: `min_samples`, `max_eps`
+- Use when: Varying density, exploratory analysis
+- Example:
+```python
+from sklearn.cluster import OPTICS
+
+model = OPTICS(min_samples=5, max_eps=0.5)
+labels = model.fit_predict(X)
+```
+
+### Hierarchical Clustering
+
+**AgglomerativeClustering**
+- Bottom-up hierarchical clustering
+- Key parameters:
+  - `n_clusters`: Number of clusters (or use `distance_threshold`)
+  - `linkage`: 'ward', 'complete', 'average', 'single'
+  - `metric`: Distance metric
+- Use when: Need dendrogram, hierarchical structure important
+- Example:
+```python
+from sklearn.cluster import AgglomerativeClustering
+
+model = AgglomerativeClustering(n_clusters=3, linkage='ward')
+labels = model.fit_predict(X)
+
+# Create dendrogram using scipy
+from scipy.cluster.hierarchy import dendrogram, linkage
+Z = linkage(X, method='ward')
+dendrogram(Z)
+```
+
+### Other Clustering Methods
+
+**MeanShift**
+- Finds clusters by shifting points toward mode of density
+- Automatically determines number of clusters
+- Key parameter: `bandwidth`
+- Use when: Don't know number of clusters, arbitrary shapes
+- Example:
+```python
+from sklearn.cluster import MeanShift, estimate_bandwidth
+
+# Estimate bandwidth
+bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
+model = MeanShift(bandwidth=bandwidth)
+labels = model.fit_predict(X)
+```
+
+**SpectralClustering**
+- Uses graph-based approach with eigenvalues
+- Key parameters: `n_clusters`, `affinity` ('rbf', 'nearest_neighbors')
+- Use when: Non-convex clusters, graph structure
+- Example:
+```python
+from sklearn.cluster import SpectralClustering
+
+model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
+labels = model.fit_predict(X)
+```
+
+**AffinityPropagation**
+- Finds exemplars by message passing
+- Automatically determines number of clusters
+- Key parameters: `damping`, `preference`
+- Use when: Don't know number of clusters
+- Example:
+```python
+from sklearn.cluster import AffinityPropagation
+
+model = AffinityPropagation(damping=0.9, random_state=42)
+labels = model.fit_predict(X)
+n_clusters = len(model.cluster_centers_indices_)
+```
+
+**BIRCH**
+- Balanced Iterative Reducing and Clustering using Hierarchies
+- Memory efficient for large datasets
+- Key parameters: `n_clusters`, `threshold`, `branching_factor`
+- Use when: Very large datasets
+- Example:
+```python
+from sklearn.cluster import Birch
+
+model = Birch(n_clusters=3, threshold=0.5)
+labels = model.fit_predict(X)
+```
+
+### Clustering Evaluation
+
+**Metrics when ground truth is known:**
+```python
+from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
+from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score
+
+# Compare predicted labels with true labels
+ari = adjusted_rand_score(y_true, y_pred)
+nmi = normalized_mutual_info_score(y_true, y_pred)
+ami = adjusted_mutual_info_score(y_true, y_pred)
+fmi = fowlkes_mallows_score(y_true, y_pred)
+```
+
+**Metrics without ground truth:**
+```python
+from sklearn.metrics import silhouette_score, calinski_harabasz_score
+from sklearn.metrics import davies_bouldin_score
+
+# Silhouette: [-1, 1], higher is better
+silhouette = silhouette_score(X, labels)
+
+# Calinski-Harabasz: higher is better
+ch_score = calinski_harabasz_score(X, labels)
+
+# Davies-Bouldin: lower is better
+db_score = davies_bouldin_score(X, labels)
+```
+
+**Elbow method for K-Means:**
+```python
+from sklearn.cluster import KMeans
+import matplotlib.pyplot as plt
+
+inertias = []
+K_range = range(2, 11)
+for k in K_range:
+    model = KMeans(n_clusters=k, random_state=42)
+    model.fit(X)
+    inertias.append(model.inertia_)
+
+plt.plot(K_range, inertias, 'bo-')
+plt.xlabel('Number of clusters')
+plt.ylabel('Inertia')
+plt.title('Elbow Method')
+```
+
+## Dimensionality Reduction
+
+### Principal Component Analysis (PCA)
+
+**PCA (`sklearn.decomposition.PCA`)**
+- Linear dimensionality reduction using SVD
+- Key parameters:
+  - `n_components`: Number of components (int or float for explained variance)
+  - `whiten`: Whiten components to unit variance
+- Use when: Linear relationships, want to explain variance
+- Example:
+```python
+from sklearn.decomposition import PCA
+
+# Keep components explaining 95% variance
+pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
+print(f"Original dimensions: {X.shape[1]}")
+print(f"Reduced dimensions: {X_reduced.shape[1]}")
+print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
+print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")
+
+# Or specify exact number of components
+pca = PCA(n_components=2)
+X_2d = pca.fit_transform(X)
+```
+
+**IncrementalPCA**
+- PCA for large datasets that don't fit in memory
+- Processes data in batches
+- Key parameter: `n_components`, `batch_size`
+- Example:
+```python
+from sklearn.decomposition import IncrementalPCA
+
+pca = IncrementalPCA(n_components=50, batch_size=100)
+X_reduced = pca.fit_transform(X)
+```
+
+**KernelPCA**
+- Non-linear dimensionality reduction using kernels
+- Key parameters: `n_components`, `kernel` ('linear', 'poly', 'rbf', 'sigmoid')
+- Use when: Non-linear relationships
+- Example:
+```python
+from sklearn.decomposition import KernelPCA
+
+pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
+X_reduced = pca.fit_transform(X)
+```
+
+### Manifold Learning
+
+**t-SNE (`sklearn.manifold.TSNE`)**
+- t-distributed Stochastic Neighbor Embedding
+- Excellent for 2D/3D visualization
+- Key parameters:
+  - `n_components`: Usually 2 or 3
+  - `perplexity`: Balance between local and global structure (5-50)
+  - `learning_rate`: Usually 10-1000
+  - `n_iter`: Number of iterations (min 250)
+- Use when: Visualizing high-dimensional data
+- Note: Slow on large datasets, no transform() method
+- Example:
+```python
+from sklearn.manifold import TSNE
+
+tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, n_iter=1000, random_state=42)
+X_embedded = tsne.fit_transform(X)
+
+# Visualize
+import matplotlib.pyplot as plt
+plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
+plt.title('t-SNE visualization')
+```
+
+**UMAP (not in scikit-learn, but compatible)**
+- Uniform Manifold Approximation and Projection
+- Faster than t-SNE, preserves global structure better
+- Install: `uv pip install umap-learn`
+- Example:
+```python
+from umap import UMAP
+
+reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
+X_embedded = reducer.fit_transform(X)
+```
+
+**Isomap**
+- Isometric Mapping
+- Preserves geodesic distances
+- Key parameters: `n_components`, `n_neighbors`
+- Use when: Non-linear manifolds
+- Example:
+```python
+from sklearn.manifold import Isomap
+
+isomap = Isomap(n_components=2, n_neighbors=5)
+X_embedded = isomap.fit_transform(X)
+```
+
+**Locally Linear Embedding (LLE)**
+- Preserves local neighborhood structure
+- Key parameters: `n_components`, `n_neighbors`
+- Example:
+```python
+from sklearn.manifold import LocallyLinearEmbedding
+
+lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
+X_embedded = lle.fit_transform(X)
+```
+
+**MDS (Multidimensional Scaling)**
+- Preserves pairwise distances
+- Key parameter: `n_components`, `metric` (True/False)
+- Example:
+```python
+from sklearn.manifold import MDS
+
+mds = MDS(n_components=2, metric=True, random_state=42)
+X_embedded = mds.fit_transform(X)
+```
+
+### Matrix Factorization
+
+**NMF (Non-negative Matrix Factorization)**
+- Factorizes into non-negative matrices
+- Key parameters: `n_components`, `init` ('nndsvd', 'random')
+- Use when: Data is non-negative (images, text)
+- Interpretable components
+- Example:
+```python
+from sklearn.decomposition import NMF
+
+nmf = NMF(n_components=10, init='nndsvd', random_state=42)
+W = nmf.fit_transform(X)  # Document-topic matrix
+H = nmf.components_  # Topic-word matrix
+```
+
+**TruncatedSVD**
+- SVD for sparse matrices
+- Similar to PCA but works with sparse data
+- Use when: Text data, sparse matrices
+- Example:
+```python
+from sklearn.decomposition import TruncatedSVD
+
+svd = TruncatedSVD(n_components=100, random_state=42)
+X_reduced = svd.fit_transform(X_sparse)
+print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")
+```
+
+**FastICA**
+- Independent Component Analysis
+- Separates multivariate signal into independent components
+- Key parameter: `n_components`
+- Use when: Signal separation (e.g., audio, EEG)
+- Example:
+```python
+from sklearn.decomposition import FastICA
+
+ica = FastICA(n_components=10, random_state=42)
+S = ica.fit_transform(X)  # Independent sources
+A = ica.mixing_  # Mixing matrix
+```
+
+**LatentDirichletAllocation (LDA)**
+- Topic modeling for text data
+- Key parameters: `n_components` (number of topics), `learning_method` ('batch', 'online')
+- Use when: Topic modeling, document clustering
+- Example:
+```python
+from sklearn.decomposition import LatentDirichletAllocation
+
+lda = LatentDirichletAllocation(n_components=10, random_state=42)
+doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution
+
+# Get top words for each topic
+feature_names = vectorizer.get_feature_names_out()
+for topic_idx, topic in enumerate(lda.components_):
+    top_words = [feature_names[i] for i in topic.argsort()[-10:]]
+    print(f"Topic {topic_idx}: {', '.join(top_words)}")
+```
+
+## Outlier and Novelty Detection
+
+### Outlier Detection
+
+**IsolationForest**
+- Isolates anomalies using random trees
+- Key parameters:
+  - `contamination`: Expected proportion of outliers
+  - `n_estimators`: Number of trees
+- Use when: High-dimensional data, efficiency important
+- Example:
+```python
+from sklearn.ensemble import IsolationForest
+
+model = IsolationForest(contamination=0.1, random_state=42)
+predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers
+```
+
+**LocalOutlierFactor**
+- Measures local density deviation
+- Key parameters: `n_neighbors`, `contamination`
+- Use when: Varying density regions
+- Example:
+```python
+from sklearn.neighbors import LocalOutlierFactor
+
+lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
+predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
+outlier_scores = lof.negative_outlier_factor_
+```
+
+**One-Class SVM**
+- Learns decision boundary around normal data
+- Key parameters: `nu` (upper bound on outliers), `kernel`, `gamma`
+- Use when: Small training set of normal data
+- Example:
+```python
+from sklearn.svm import OneClassSVM
+
+model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
+model.fit(X_train)
+predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers
+```
+
+**EllipticEnvelope**
+- Assumes Gaussian distribution
+- Key parameter: `contamination`
+- Use when: Data is Gaussian-distributed
+- Example:
+```python
+from sklearn.covariance import EllipticEnvelope
+
+model = EllipticEnvelope(contamination=0.1, random_state=42)
+predictions = model.fit_predict(X)
+```
+
+## Gaussian Mixture Models
+
+**GaussianMixture**
+- Probabilistic clustering with mixture of Gaussians
+- Key parameters:
+  - `n_components`: Number of mixture components
+  - `covariance_type`: 'full', 'tied', 'diag', 'spherical'
+- Use when: Soft clustering, need probability estimates
+- Example:
+```python
+from sklearn.mixture import GaussianMixture
+
+gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
+gmm.fit(X)
+
+# Predict cluster labels
+labels = gmm.predict(X)
+
+# Get probability of each cluster
+probabilities = gmm.predict_proba(X)
+
+# Information criteria for model selection
+print(f"BIC: {gmm.bic(X)}")  # Lower is better
+print(f"AIC: {gmm.aic(X)}")  # Lower is better
+```
+
+## Choosing the Right Method
+
+### Clustering:
+- **Know K, spherical clusters**: K-Means
+- **Arbitrary shapes, noise**: DBSCAN, HDBSCAN
+- **Hierarchical structure**: AgglomerativeClustering
+- **Very large data**: MiniBatchKMeans, BIRCH
+- **Probabilistic**: GaussianMixture
+
+### Dimensionality Reduction:
+- **Linear, variance explanation**: PCA
+- **Non-linear, visualization**: t-SNE, UMAP
+- **Non-negative data**: NMF
+- **Sparse data**: TruncatedSVD
+- **Topic modeling**: LatentDirichletAllocation
+
+### Outlier Detection:
+- **High-dimensional**: IsolationForest
+- **Varying density**: LocalOutlierFactor
+- **Gaussian data**: EllipticEnvelope