Unsupervised Learning Reference

Targets scikit-learn 1.9.1. Snippets with caller-supplied data/columns are illustrative; fit all learned preprocessing inside the training folds when estimating predictive performance.

Overview

Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.

Clustering

K-Means

**KMeans (sklearn.cluster.KMeans)**

• Partition-based clustering into K clusters
• Key parameters:
• n_clusters: Number of clusters to form
• init: Initialization method ('k-means++', 'random')
• n_init: Number of initializations (default='auto'; one run with k-means++, ten with random init). Set an integer explicitly for repeatable search effort
• max_iter: Maximum iterations
• Use when: Know number of clusters, spherical cluster shapes
• Fast and scalable
• Example:

from sklearn.cluster import KMeans

model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
labels = model.fit_predict(X)
centers = model.cluster_centers_

# Inertia (sum of squared distances to nearest center)
print(f"Inertia: {model.inertia_}")

MiniBatchKMeans

• Faster K-Means using mini-batches
• Use when: Large datasets, need faster training
• Slightly less accurate than K-Means
• Example:

from sklearn.cluster import MiniBatchKMeans

model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
labels = model.fit_predict(X)

Density-Based Clustering

**DBSCAN (sklearn.cluster.DBSCAN)**

• Density-Based Spatial Clustering
• Key parameters:
• eps: Maximum distance between two samples to be neighbors
• min_samples: Minimum samples in neighborhood to form core point
• metric: Distance metric
• Use when: Arbitrary cluster shapes, presence of noise/outliers
• Automatically determines number of clusters
• Labels noise points as -1
• Example:

from sklearn.cluster import DBSCAN

model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
labels = model.fit_predict(X)

# Number of clusters (excluding noise)
n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Clusters: {n_clusters}, Noise points: {n_noise}")

**HDBSCAN (sklearn.cluster.HDBSCAN)**

• Hierarchical DBSCAN with adaptive epsilon (added in scikit-learn 1.3)
• Handles varying density; performance and semantics still depend on data and parameters
• scikit-learn includes the point itself in min_samples; the contrib hdbscan package does not. Do not transfer that parameter unchanged between implementations
• Key parameter: min_cluster_size
• Use when: Varying density clusters
• Example:

from sklearn.cluster import HDBSCAN

model = HDBSCAN(min_cluster_size=10, min_samples=5)
labels = model.fit_predict(X)

**OPTICS (sklearn.cluster.OPTICS)**

• Ordering points to identify clustering structure
• Similar to DBSCAN but doesn't require eps parameter
• Key parameters: min_samples, max_eps
• Use when: Varying density, exploratory analysis
• Example:

from sklearn.cluster import OPTICS

model = OPTICS(min_samples=5, max_eps=0.5)
labels = model.fit_predict(X)

Hierarchical Clustering

AgglomerativeClustering

• Bottom-up hierarchical clustering
• Key parameters:
• n_clusters: Number of clusters (or use distance_threshold)
• linkage: 'ward', 'complete', 'average', 'single'
• metric: Distance metric
• Use when: Need dendrogram, hierarchical structure important
• Example:

from sklearn.cluster import AgglomerativeClustering

model = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = model.fit_predict(X)

# Create dendrogram using scipy
from scipy.cluster.hierarchy import dendrogram, linkage
Z = linkage(X, method='ward')
dendrogram(Z)

Other Clustering Methods

MeanShift

• Finds clusters by shifting points toward mode of density
• Automatically determines number of clusters
• Key parameter: bandwidth
• Use when: Don't know number of clusters, arbitrary shapes
• Example:

from sklearn.cluster import MeanShift, estimate_bandwidth

# Estimate bandwidth
bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
model = MeanShift(bandwidth=bandwidth)
labels = model.fit_predict(X)

SpectralClustering

• Uses graph-based approach with eigenvalues
• Key parameters: n_clusters, affinity ('rbf', 'nearest_neighbors')
• Use when: Non-convex clusters, graph structure
• Example:

from sklearn.cluster import SpectralClustering

model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
labels = model.fit_predict(X)

AffinityPropagation

• Finds exemplars by message passing
• Automatically determines number of clusters
• Key parameters: damping, preference
• Use when: Don't know number of clusters
• Example:

from sklearn.cluster import AffinityPropagation

model = AffinityPropagation(damping=0.9, random_state=42)
labels = model.fit_predict(X)
n_clusters = len(model.cluster_centers_indices_)

BIRCH

• Balanced Iterative Reducing and Clustering using Hierarchies
• Memory efficient for large datasets
• Key parameters: n_clusters, threshold, branching_factor
• Use when: Very large datasets
• Example:

from sklearn.cluster import Birch

model = Birch(n_clusters=3, threshold=0.5)
labels = model.fit_predict(X)

Clustering Evaluation

Metrics when ground truth is known:

from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score

# Compare predicted labels with true labels
ari = adjusted_rand_score(y_true, y_pred)
nmi = normalized_mutual_info_score(y_true, y_pred)
ami = adjusted_mutual_info_score(y_true, y_pred)
fmi = fowlkes_mallows_score(y_true, y_pred)

Metrics without ground truth: These require 2 <= n_labels < n_samples. Noise label -1 is otherwise treated as an ordinary cluster: explicitly decide whether to exclude noise and report retained coverage. Scores on different subsets are not directly comparable, and these Euclidean indices favor compact/convex clusters.

from sklearn.metrics import silhouette_score, calinski_harabasz_score
from sklearn.metrics import davies_bouldin_score

# Silhouette: [-1, 1], higher is better
silhouette = silhouette_score(X, labels)

# Calinski-Harabasz: higher is better
ch_score = calinski_harabasz_score(X, labels)

# Davies-Bouldin: lower is better
db_score = davies_bouldin_score(X, labels)

Elbow method for K-Means:

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

inertias = []
K_range = range(2, 11)
for k in K_range:
    model = KMeans(n_clusters=k, random_state=42)
    model.fit(X)
    inertias.append(model.inertia_)

plt.plot(K_range, inertias, 'bo-')
plt.xlabel('Number of clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')

Dimensionality Reduction

Principal Component Analysis (PCA)

**PCA (sklearn.decomposition.PCA)**

• Linear dimensionality reduction using SVD
• Key parameters:
• n_components: Number of components (int or float for explained variance)
• whiten: Whiten components to unit variance
• Use when: Linear relationships, want to explain variance
• Example:

from sklearn.decomposition import PCA

# Keep components explaining 95% variance
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)

print(f"Original dimensions: {X.shape[1]}")
print(f"Reduced dimensions: {X_reduced.shape[1]}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")

# Or specify exact number of components
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X)

IncrementalPCA

• PCA for large datasets that don't fit in memory
• Processes data in batches
• Key parameter: n_components, batch_size
• Example:

from sklearn.decomposition import IncrementalPCA

pca = IncrementalPCA(n_components=50, batch_size=100)
X_reduced = pca.fit_transform(X)

KernelPCA

• Non-linear dimensionality reduction using kernels
• Key parameters: n_components, kernel ('linear', 'poly', 'rbf', 'sigmoid')
• Use when: Non-linear relationships
• Example:

from sklearn.decomposition import KernelPCA

pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
X_reduced = pca.fit_transform(X)

Manifold Learning

**t-SNE (sklearn.manifold.TSNE)**

• t-distributed Stochastic Neighbor Embedding
• Excellent for 2D/3D visualization
• Key parameters:
• n_components: Usually 2 or 3
• perplexity: Balance between local and global structure (5-50)
• learning_rate: Usually 10-1000
• max_iter: Number of iterations (minimum 250; renamed from n_iter in 1.5)
• Use when: Visualizing high-dimensional data
• perplexity must be less than the sample count. No transform() method; refitting on new data changes the embedding
• Example:

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, max_iter=1000, random_state=42)
X_embedded = tsne.fit_transform(X)

# Visualize
import matplotlib.pyplot as plt
plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
plt.title('t-SNE visualization')

UMAP (not in scikit-learn, but compatible)

• Uniform Manifold Approximation and Projection
• Runtime and apparent geometry depend on dataset and settings; neither UMAP nor t-SNE reliably preserves global distances or proves clusters
• Install: uv pip install umap-learn
• Example:

from umap import UMAP

reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
X_embedded = reducer.fit_transform(X)

Isomap

• Isometric Mapping
• Preserves geodesic distances
• Key parameters: n_components, n_neighbors
• Use when: Non-linear manifolds
• Example:

from sklearn.manifold import Isomap

isomap = Isomap(n_components=2, n_neighbors=5)
X_embedded = isomap.fit_transform(X)

Locally Linear Embedding (LLE)

• Preserves local neighborhood structure
• Key parameters: n_components, n_neighbors
• Example:

from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
X_embedded = lle.fit_transform(X)

MDS (Multidimensional Scaling)

• Preserves pairwise distances
• Key parameters: n_components, metric, metric_params, init (e.g. 'classical_mds' in 1.8+)
• Note: dissimilarity is deprecated; the former metric boolean was renamed to metric_mds
• Example:

from sklearn.manifold import MDS

mds = MDS(n_components=2, metric='euclidean', init='classical_mds', random_state=42)
X_embedded = mds.fit_transform(X)

**ClassicalMDS (sklearn.manifold.ClassicalMDS)**

• Classical (Torgerson) MDS via eigendecomposition of the double-centered distance matrix
• Added in scikit-learn 1.8; faster initialization path for MDS
• Example:

from sklearn.manifold import ClassicalMDS

cmds = ClassicalMDS(n_components=2)
X_embedded = cmds.fit_transform(X)

Matrix Factorization

NMF (Non-negative Matrix Factorization)

• Factorizes into non-negative matrices
• Key parameters: n_components, init ('nndsvd', 'random')
• Use when: Data is non-negative (images, text)
• Interpretable components
• Example:

from sklearn.decomposition import NMF

nmf = NMF(n_components=10, init='nndsvd', random_state=42)
W = nmf.fit_transform(X)  # Document-topic matrix
H = nmf.components_  # Topic-word matrix

TruncatedSVD

• SVD for sparse matrices
• Unlike PCA, does not center input, preserving sparsity; components need not match centered PCA
• Use when: Text data, sparse matrices
• Example:

from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=100, random_state=42)
X_reduced = svd.fit_transform(X_sparse)
print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")

FastICA

• Independent Component Analysis
• Separates multivariate signal into independent components
• Key parameter: n_components
• Use when: Signal separation (e.g., audio, EEG)
• Example:

from sklearn.decomposition import FastICA

ica = FastICA(n_components=10, random_state=42)
S = ica.fit_transform(X)  # Independent sources
A = ica.mixing_  # Mixing matrix

LatentDirichletAllocation (LDA)

• Topic modeling for text data
• Key parameters: n_components (number of topics), learning_method ('batch', 'online')
• Use when: Topic modeling, document clustering
• Example:

from sklearn.decomposition import LatentDirichletAllocation

lda = LatentDirichletAllocation(n_components=10, random_state=42)
doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution

# Get top words for each topic
feature_names = vectorizer.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[-10:]]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

Outlier and Novelty Detection

Outlier Detection

IsolationForest

• Isolates anomalies using random trees
• Key parameters:
• contamination: Expected proportion of outliers
• n_estimators: Number of trees
• Use when: High-dimensional data, efficiency important
• Example:

from sklearn.ensemble import IsolationForest

model = IsolationForest(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers

LocalOutlierFactor

• Measures local density deviation
• Key parameters: n_neighbors, contamination
• Use when: Varying density regions
• Example:

from sklearn.neighbors import LocalOutlierFactor

lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
outlier_scores = lof.negative_outlier_factor_

For novelty detection use LocalOutlierFactor(novelty=True).fit(X_train) and call predict/score_samples only on unseen observations. Do not use novelty predictions to reproduce training fit_predict labels.

One-Class SVM

• Learns decision boundary around normal data
• nu bounds the fraction of training margin errors and lower-bounds the support-vector fraction; it is not an estimated test-set outlier prevalence. Other parameters: kernel, gamma
• Use when: Small training set of normal data
• Example:

from sklearn.svm import OneClassSVM

model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
model.fit(X_train)
predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers

EllipticEnvelope

• Assumes Gaussian distribution
• Key parameter: contamination
• Use when: Data is Gaussian-distributed
• Example:

from sklearn.covariance import EllipticEnvelope

model = EllipticEnvelope(contamination=0.1, random_state=42)
predictions = model.fit_predict(X)

Gaussian Mixture Models

GaussianMixture

• Probabilistic clustering with mixture of Gaussians
• Key parameters:
• n_components: Number of mixture components
• covariance_type: 'full', 'tied', 'diag', 'spherical'
• Use when: Soft clustering, need probability estimates
• Example:

from sklearn.mixture import GaussianMixture

gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)

# Predict cluster labels
labels = gmm.predict(X)

# Get probability of each cluster
probabilities = gmm.predict_proba(X)

# Information criteria for model selection
print(f"BIC: {gmm.bic(X)}")  # Lower is better
print(f"AIC: {gmm.aic(X)}")  # Lower is better

Choosing the Right Method

Clustering:

• Know K, spherical clusters: K-Means
• Arbitrary shapes, noise: DBSCAN, HDBSCAN
• Hierarchical structure: AgglomerativeClustering
• Very large data: MiniBatchKMeans, BIRCH
• Probabilistic: GaussianMixture

Dimensionality Reduction:

• Linear, variance explanation: PCA
• Non-linear, visualization: t-SNE, UMAP
• Non-negative data: NMF
• Sparse data: TruncatedSVD
• Topic modeling: LatentDirichletAllocation

Outlier Detection:

• High-dimensional: IsolationForest
• Varying density: LocalOutlierFactor
• Gaussian data: EllipticEnvelope

Upstream references

• https://scikit-learn.org/stable/modules/clustering.html
• https://scikit-learn.org/stable/modules/generated/sklearn.cluster.HDBSCAN.html
• https://scikit-learn.org/stable/modules/generated/sklearn.manifold.ClassicalMDS.html
• https://scikit-learn.org/stable/modules/outlier_detection.html
• https://umap-learn.readthedocs.io/en/latest/parameters.html