Dimensionality Reduction: PCA, NMF, and Sparse Data | Blog

More features isn’t always better. A dataset with hundreds of columns often has a much smaller “true” structure hiding inside it, plus a bunch of noise that just slows everything down and can actively hurt models like regression and classification.

Dimensionality reduction is how you find that smaller structure and work with it directly, instead of dragging around every feature you started with.

What Dimensionality Reduction Actually Does

It finds the patterns that show up repeatedly in your data and uses them to represent the dataset in a more compact form. Fewer dimensions, same important structure, faster computation.

The side benefit: stripping out noisy or uninformative features tends to help downstream tasks like regression and classification, not just speed.

Intrinsic Dimension: How Many Dimensions Do You Actually Need?

The intrinsic dimension of a dataset is the number of features it actually takes to describe it well, which is often a lot smaller than the number of columns you started with.

A dataset can have 200 features and an intrinsic dimension of 5. The other 195 are redundant, noisy, or just weakly informative. Dimensionality reduction is the tool for finding and collapsing down to that smaller true structure.

Principal Component Analysis (PCA)

PCA finds the directions in your data where it varies the most. Those directions are called principal components, and after fitting a PCA model, you can pull them out through .components_.

Decorrelation

Part of what PCA does along the way: it rotates the data to line up with the coordinate axes and shifts it to have a mean of zero. The result is a set of features that are no longer linearly correlated with each other.

Linear correlation itself is usually measured with Pearson correlation, which runs from -1 to 1. Zero means no linear relationship.

Using PCA to Estimate Intrinsic Dimension

PCA transforms your original features into new ones (the principal components), ordered by how much variance each one captures. High-variance components carry more of the dataset’s real information. Low-variance ones mostly carry noise.

So you can estimate intrinsic dimension by looking at how many components actually carry meaningful variance, and treating the rest as noise. There’s no universal correct number here. It depends on how much information you’re willing to trade away for a smaller representation.

Choosing How Many Components to Keep

Plot the explained variance of each component. Usually there’s a clear point where additional components stop adding much. Keep enough components to retain most of the variance, and drop the ones that barely move the needle.

TF-IDF and Sparse Matrices

Dimensionality reduction shows up a lot in text data too.

Build a word-frequency matrix: rows are documents, columns are words. Most cells are zero, since any single document only uses a small fraction of all possible words. That’s what makes it a sparse matrix.

TF-IDF is the standard way to weight this kind of data. Quick recap since I covered this in more depth in [my search and information retrieval post] (link it here): TF (term frequency) measures how often a word shows up in a document, IDF (inverse document frequency) down-weights words that are common across every document (like “the”). Words that are frequent in one document but rare everywhere else score highest.

PCA and Sparse Data: The Catch

Scikit-learn’s PCA doesn’t work with sparse matrices. That’s a real limitation for TF-IDF or word-frequency data, which is sparse by nature.

The fix is TruncatedSVD. It does a similar job to PCA, but it’s built to handle sparse matrices, which makes it the practical choice for text data.

Non-Negative Matrix Factorization (NMF)

NMF is another dimensionality reduction technique, but the philosophy is different. Where PCA chases maximum variance, NMF represents data as combinations of parts, and it tends to be a lot easier to interpret as a result.

Some examples of what “parts” means in practice:

Documents as combinations of common topics.
Images as combinations of common visual patterns.

A hard requirement: every feature value has to be non-negative. NMF simply doesn’t work if your data has negative values.

Other properties worth knowing:

You have to specify the number of components up front.
NMF works with both NumPy arrays and sparse matrices like csr_matrix.
Both the components and the feature values it produces stay non-negative.

Reconstruction

Combine NMF’s components and features back together, and you get an approximate reconstruction of the original data. That’s matrix factorization in a nutshell: the original dataset represented through a smaller set of non-negative building blocks.

PCA vs. NMF

	PCA	NMF
Goal	Maximum variance	Non-negative, part-based representation
Component values	Can be positive or negative	Always non-negative
Interpretability	Often harder to interpret	Generally easier to interpret
Common uses	Compression, decorrelation, visualization	Topic modeling, pattern discovery

Short version: PCA optimizes for variance and compression. NMF optimizes for interpretability and representing data as parts.

Using NMF to Build Recommendation Systems

Here’s where NMF gets genuinely practical: recommendations.

Take a word-frequency matrix of documents, run NMF on it, and documents covering similar topics end up with similar NMF feature values. Once you’ve got that, you can compare those feature representations directly to find similar items.

Cosine similarity is the usual tool for that comparison. Quick recap: it measures the angle between two vectors rather than the raw distance between them. A smaller angle means higher similarity. (I go into this in more depth in [my search and information retrieval post] as well, in the KNN section.)

This pattern generalizes past documents:

Recommending similar articles.
Recommending similar musical artists.
Finding similar users or products.

Same underlying idea every time: items with similar latent feature representations tend to actually be similar.

Choosing Between PCA and NMF

PCA fits when you want maximum variance, general-purpose dimensionality reduction, decorrelation, or compression.

NMF fits when your data is non-negative and you want a representation you can actually explain in terms of parts, like topics in documents or patterns in images.

The real question isn’t “which is better.” It’s “what do you need the reduced representation to actually tell you.”

FAQ

What is dimensionality reduction?

A way to represent a dataset using fewer dimensions while preserving the patterns that actually matter in the data.

What is PCA?

A dimensionality reduction technique that finds the directions, called principal components, along which data varies the most.

What is intrinsic dimension?

The number of features actually needed to describe a dataset’s underlying structure, often much smaller than the number of features it started with.

What is explained variance in PCA?

A measure of how much of a dataset’s variance, and therefore information, each principal component captures.

Why can’t PCA be used directly on sparse matrices?

Scikit-learn’s PCA implementation doesn’t support sparse matrices. TruncatedSVD does the same kind of job but works with sparse data, which makes it the right choice for TF-IDF or word-frequency matrices.

What is NMF?

A dimensionality reduction technique that represents data as combinations of non-negative parts, generally more interpretable than PCA but requiring all feature values to be non-negative.

What’s the real difference between PCA and NMF?

PCA optimizes for capturing maximum variance. NMF optimizes for a non-negative, part-based representation that’s usually easier to interpret.

How is NMF used in recommendation systems?

It represents items (documents, articles, products) as combinations of latent features. Items with similar feature representations, measured with cosine similarity, get treated as similar for recommendation purposes.