@synsci/cli-darwin-x64 1.1.97 → 1.1.99

package/bin/skills/scikit-learn/references/unsupervised_learning.md

@@ -1,505 +0,0 @@
-# Unsupervised Learning Reference
-
-## Overview
-
-Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.
-
-## Clustering
-
-### K-Means
-
-**KMeans (`sklearn.cluster.KMeans`)**
-- Partition-based clustering into K clusters
-- Key parameters:
-  - `n_clusters`: Number of clusters to form
-  - `init`: Initialization method ('k-means++', 'random')
-  - `n_init`: Number of initializations (default=10)
-  - `max_iter`: Maximum iterations
-- Use when: Know number of clusters, spherical cluster shapes
-- Fast and scalable
-- Example:
-```python
-from sklearn.cluster import KMeans
-
-model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
-labels = model.fit_predict(X)
-centers = model.cluster_centers_
-
-# Inertia (sum of squared distances to nearest center)
-print(f"Inertia: {model.inertia_}")
-```
-
-**MiniBatchKMeans**
-- Faster K-Means using mini-batches
-- Use when: Large datasets, need faster training
-- Slightly less accurate than K-Means
-- Example:
-```python
-from sklearn.cluster import MiniBatchKMeans
-
-model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
-labels = model.fit_predict(X)
-```
-
-### Density-Based Clustering
-
-**DBSCAN (`sklearn.cluster.DBSCAN`)**
-- Density-Based Spatial Clustering
-- Key parameters:
-  - `eps`: Maximum distance between two samples to be neighbors
-  - `min_samples`: Minimum samples in neighborhood to form core point
-  - `metric`: Distance metric
-- Use when: Arbitrary cluster shapes, presence of noise/outliers
-- Automatically determines number of clusters
-- Labels noise points as -1
-- Example:
-```python
-from sklearn.cluster import DBSCAN
-
-model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
-labels = model.fit_predict(X)
-
-# Number of clusters (excluding noise)
-n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
-n_noise = list(labels).count(-1)
-print(f"Clusters: {n_clusters}, Noise points: {n_noise}")
-```
-
-**HDBSCAN (`sklearn.cluster.HDBSCAN`)**
-- Hierarchical DBSCAN with adaptive epsilon
-- More robust than DBSCAN
-- Key parameter: `min_cluster_size`
-- Use when: Varying density clusters
-- Example:
-```python
-from sklearn.cluster import HDBSCAN
-
-model = HDBSCAN(min_cluster_size=10, min_samples=5)
-labels = model.fit_predict(X)
-```
-
-**OPTICS (`sklearn.cluster.OPTICS`)**
-- Ordering points to identify clustering structure
-- Similar to DBSCAN but doesn't require eps parameter
-- Key parameters: `min_samples`, `max_eps`
-- Use when: Varying density, exploratory analysis
-- Example:
-```python
-from sklearn.cluster import OPTICS
-
-model = OPTICS(min_samples=5, max_eps=0.5)
-labels = model.fit_predict(X)
-```
-
-### Hierarchical Clustering
-
-**AgglomerativeClustering**
-- Bottom-up hierarchical clustering
-- Key parameters:
-  - `n_clusters`: Number of clusters (or use `distance_threshold`)
-  - `linkage`: 'ward', 'complete', 'average', 'single'
-  - `metric`: Distance metric
-- Use when: Need dendrogram, hierarchical structure important
-- Example:
-```python
-from sklearn.cluster import AgglomerativeClustering
-
-model = AgglomerativeClustering(n_clusters=3, linkage='ward')
-labels = model.fit_predict(X)
-
-# Create dendrogram using scipy
-from scipy.cluster.hierarchy import dendrogram, linkage
-Z = linkage(X, method='ward')
-dendrogram(Z)
-```
-
-### Other Clustering Methods
-
-**MeanShift**
-- Finds clusters by shifting points toward mode of density
-- Automatically determines number of clusters
-- Key parameter: `bandwidth`
-- Use when: Don't know number of clusters, arbitrary shapes
-- Example:
-```python
-from sklearn.cluster import MeanShift, estimate_bandwidth
-
-# Estimate bandwidth
-bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
-model = MeanShift(bandwidth=bandwidth)
-labels = model.fit_predict(X)
-```
-
-**SpectralClustering**
-- Uses graph-based approach with eigenvalues
-- Key parameters: `n_clusters`, `affinity` ('rbf', 'nearest_neighbors')
-- Use when: Non-convex clusters, graph structure
-- Example:
-```python
-from sklearn.cluster import SpectralClustering
-
-model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
-labels = model.fit_predict(X)
-```
-
-**AffinityPropagation**
-- Finds exemplars by message passing
-- Automatically determines number of clusters
-- Key parameters: `damping`, `preference`
-- Use when: Don't know number of clusters
-- Example:
-```python
-from sklearn.cluster import AffinityPropagation
-
-model = AffinityPropagation(damping=0.9, random_state=42)
-labels = model.fit_predict(X)
-n_clusters = len(model.cluster_centers_indices_)
-```
-
-**BIRCH**
-- Balanced Iterative Reducing and Clustering using Hierarchies
-- Memory efficient for large datasets
-- Key parameters: `n_clusters`, `threshold`, `branching_factor`
-- Use when: Very large datasets
-- Example:
-```python
-from sklearn.cluster import Birch
-
-model = Birch(n_clusters=3, threshold=0.5)
-labels = model.fit_predict(X)
-```
-
-### Clustering Evaluation
-
-**Metrics when ground truth is known:**
-```python
-from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
-from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score
-
-# Compare predicted labels with true labels
-ari = adjusted_rand_score(y_true, y_pred)
-nmi = normalized_mutual_info_score(y_true, y_pred)
-ami = adjusted_mutual_info_score(y_true, y_pred)
-fmi = fowlkes_mallows_score(y_true, y_pred)
-```
-
-**Metrics without ground truth:**
-```python
-from sklearn.metrics import silhouette_score, calinski_harabasz_score
-from sklearn.metrics import davies_bouldin_score
-
-# Silhouette: [-1, 1], higher is better
-silhouette = silhouette_score(X, labels)
-
-# Calinski-Harabasz: higher is better
-ch_score = calinski_harabasz_score(X, labels)
-
-# Davies-Bouldin: lower is better
-db_score = davies_bouldin_score(X, labels)
-```
-
-**Elbow method for K-Means:**
-```python
-from sklearn.cluster import KMeans
-import matplotlib.pyplot as plt
-
-inertias = []
-K_range = range(2, 11)
-for k in K_range:
-    model = KMeans(n_clusters=k, random_state=42)
-    model.fit(X)
-    inertias.append(model.inertia_)
-
-plt.plot(K_range, inertias, 'bo-')
-plt.xlabel('Number of clusters')
-plt.ylabel('Inertia')
-plt.title('Elbow Method')
-```
-
-## Dimensionality Reduction
-
-### Principal Component Analysis (PCA)
-
-**PCA (`sklearn.decomposition.PCA`)**
-- Linear dimensionality reduction using SVD
-- Key parameters:
-  - `n_components`: Number of components (int or float for explained variance)
-  - `whiten`: Whiten components to unit variance
-- Use when: Linear relationships, want to explain variance
-- Example:
-```python
-from sklearn.decomposition import PCA
-
-# Keep components explaining 95% variance
-pca = PCA(n_components=0.95)
-X_reduced = pca.fit_transform(X)
-
-print(f"Original dimensions: {X.shape[1]}")
-print(f"Reduced dimensions: {X_reduced.shape[1]}")
-print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
-print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")
-
-# Or specify exact number of components
-pca = PCA(n_components=2)
-X_2d = pca.fit_transform(X)
-```
-
-**IncrementalPCA**
-- PCA for large datasets that don't fit in memory
-- Processes data in batches
-- Key parameter: `n_components`, `batch_size`
-- Example:
-```python
-from sklearn.decomposition import IncrementalPCA
-
-pca = IncrementalPCA(n_components=50, batch_size=100)
-X_reduced = pca.fit_transform(X)
-```
-
-**KernelPCA**
-- Non-linear dimensionality reduction using kernels
-- Key parameters: `n_components`, `kernel` ('linear', 'poly', 'rbf', 'sigmoid')
-- Use when: Non-linear relationships
-- Example:
-```python
-from sklearn.decomposition import KernelPCA
-
-pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
-X_reduced = pca.fit_transform(X)
-```
-
-### Manifold Learning
-
-**t-SNE (`sklearn.manifold.TSNE`)**
-- t-distributed Stochastic Neighbor Embedding
-- Excellent for 2D/3D visualization
-- Key parameters:
-  - `n_components`: Usually 2 or 3
-  - `perplexity`: Balance between local and global structure (5-50)
-  - `learning_rate`: Usually 10-1000
-  - `n_iter`: Number of iterations (min 250)
-- Use when: Visualizing high-dimensional data
-- Note: Slow on large datasets, no transform() method
-- Example:
-```python
-from sklearn.manifold import TSNE
-
-tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, n_iter=1000, random_state=42)
-X_embedded = tsne.fit_transform(X)
-
-# Visualize
-import matplotlib.pyplot as plt
-plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
-plt.title('t-SNE visualization')
-```
-
-**UMAP (not in scikit-learn, but compatible)**
-- Uniform Manifold Approximation and Projection
-- Faster than t-SNE, preserves global structure better
-- Install: `uv pip install umap-learn`
-- Example:
-```python
-from umap import UMAP
-
-reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
-X_embedded = reducer.fit_transform(X)
-```
-
-**Isomap**
-- Isometric Mapping
-- Preserves geodesic distances
-- Key parameters: `n_components`, `n_neighbors`
-- Use when: Non-linear manifolds
-- Example:
-```python
-from sklearn.manifold import Isomap
-
-isomap = Isomap(n_components=2, n_neighbors=5)
-X_embedded = isomap.fit_transform(X)
-```
-
-**Locally Linear Embedding (LLE)**
-- Preserves local neighborhood structure
-- Key parameters: `n_components`, `n_neighbors`
-- Example:
-```python
-from sklearn.manifold import LocallyLinearEmbedding
-
-lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
-X_embedded = lle.fit_transform(X)
-```
-
-**MDS (Multidimensional Scaling)**
-- Preserves pairwise distances
-- Key parameter: `n_components`, `metric` (True/False)
-- Example:
-```python
-from sklearn.manifold import MDS
-
-mds = MDS(n_components=2, metric=True, random_state=42)
-X_embedded = mds.fit_transform(X)
-```
-
-### Matrix Factorization
-
-**NMF (Non-negative Matrix Factorization)**
-- Factorizes into non-negative matrices
-- Key parameters: `n_components`, `init` ('nndsvd', 'random')
-- Use when: Data is non-negative (images, text)
-- Interpretable components
-- Example:
-```python
-from sklearn.decomposition import NMF
-
-nmf = NMF(n_components=10, init='nndsvd', random_state=42)
-W = nmf.fit_transform(X)  # Document-topic matrix
-H = nmf.components_  # Topic-word matrix
-```
-
-**TruncatedSVD**
-- SVD for sparse matrices
-- Similar to PCA but works with sparse data
-- Use when: Text data, sparse matrices
-- Example:
-```python
-from sklearn.decomposition import TruncatedSVD
-
-svd = TruncatedSVD(n_components=100, random_state=42)
-X_reduced = svd.fit_transform(X_sparse)
-print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")
-```
-
-**FastICA**
-- Independent Component Analysis
-- Separates multivariate signal into independent components
-- Key parameter: `n_components`
-- Use when: Signal separation (e.g., audio, EEG)
-- Example:
-```python
-from sklearn.decomposition import FastICA
-
-ica = FastICA(n_components=10, random_state=42)
-S = ica.fit_transform(X)  # Independent sources
-A = ica.mixing_  # Mixing matrix
-```
-
-**LatentDirichletAllocation (LDA)**
-- Topic modeling for text data
-- Key parameters: `n_components` (number of topics), `learning_method` ('batch', 'online')
-- Use when: Topic modeling, document clustering
-- Example:
-```python
-from sklearn.decomposition import LatentDirichletAllocation
-
-lda = LatentDirichletAllocation(n_components=10, random_state=42)
-doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution
-
-# Get top words for each topic
-feature_names = vectorizer.get_feature_names_out()
-for topic_idx, topic in enumerate(lda.components_):
-    top_words = [feature_names[i] for i in topic.argsort()[-10:]]
-    print(f"Topic {topic_idx}: {', '.join(top_words)}")
-```
-
-## Outlier and Novelty Detection
-
-### Outlier Detection
-
-**IsolationForest**
-- Isolates anomalies using random trees
-- Key parameters:
-  - `contamination`: Expected proportion of outliers
-  - `n_estimators`: Number of trees
-- Use when: High-dimensional data, efficiency important
-- Example:
-```python
-from sklearn.ensemble import IsolationForest
-
-model = IsolationForest(contamination=0.1, random_state=42)
-predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers
-```
-
-**LocalOutlierFactor**
-- Measures local density deviation
-- Key parameters: `n_neighbors`, `contamination`
-- Use when: Varying density regions
-- Example:
-```python
-from sklearn.neighbors import LocalOutlierFactor
-
-lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
-predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
-outlier_scores = lof.negative_outlier_factor_
-```
-
-**One-Class SVM**
-- Learns decision boundary around normal data
-- Key parameters: `nu` (upper bound on outliers), `kernel`, `gamma`
-- Use when: Small training set of normal data
-- Example:
-```python
-from sklearn.svm import OneClassSVM
-
-model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
-model.fit(X_train)
-predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers
-```
-
-**EllipticEnvelope**
-- Assumes Gaussian distribution
-- Key parameter: `contamination`
-- Use when: Data is Gaussian-distributed
-- Example:
-```python
-from sklearn.covariance import EllipticEnvelope
-
-model = EllipticEnvelope(contamination=0.1, random_state=42)
-predictions = model.fit_predict(X)
-```
-
-## Gaussian Mixture Models
-
-**GaussianMixture**
-- Probabilistic clustering with mixture of Gaussians
-- Key parameters:
-  - `n_components`: Number of mixture components
-  - `covariance_type`: 'full', 'tied', 'diag', 'spherical'
-- Use when: Soft clustering, need probability estimates
-- Example:
-```python
-from sklearn.mixture import GaussianMixture
-
-gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
-gmm.fit(X)
-
-# Predict cluster labels
-labels = gmm.predict(X)
-
-# Get probability of each cluster
-probabilities = gmm.predict_proba(X)
-
-# Information criteria for model selection
-print(f"BIC: {gmm.bic(X)}")  # Lower is better
-print(f"AIC: {gmm.aic(X)}")  # Lower is better
-```
-
-## Choosing the Right Method
-
-### Clustering:
-- **Know K, spherical clusters**: K-Means
-- **Arbitrary shapes, noise**: DBSCAN, HDBSCAN
-- **Hierarchical structure**: AgglomerativeClustering
-- **Very large data**: MiniBatchKMeans, BIRCH
-- **Probabilistic**: GaussianMixture
-
-### Dimensionality Reduction:
-- **Linear, variance explanation**: PCA
-- **Non-linear, visualization**: t-SNE, UMAP
-- **Non-negative data**: NMF
-- **Sparse data**: TruncatedSVD
-- **Topic modeling**: LatentDirichletAllocation
-
-### Outlier Detection:
-- **High-dimensional**: IsolationForest
-- **Varying density**: LocalOutlierFactor
-- **Gaussian data**: EllipticEnvelope