Machine Learning Fundamentals: Model Evaluation, Hyperparameter Tuning, Missing Data, and Scaling

Training a model is the easy part. The stuff that actually separates a working model from a broken one: picking the right metric, tuning it properly, handling missing data, and scaling features. Get any of these wrong and a model can look great on paper while doing nothing useful.

Why Accuracy Lies to You

Accuracy is correct predictions divided by total predictions. Simple, intuitive, and often misleading.

Here’s the classic trap: fraud detection. Fraudulent transactions are a tiny percentage of all transactions. A model that predicts “legitimate” every single time, never once flagging fraud, will still post a very high accuracy score. It’s also completely useless.

That’s why accuracy falls apart on imbalanced datasets. You need metrics that actually account for the imbalance:

• Precision
• Recall
• F1 score

In all three, whatever you’re trying to detect (fraud, cancer, malware) is the positive class.

Precision

Also called positive predictive value (PPV). High precision means a low false positive rate. Back to fraud: high precision means you’re not flagging a lot of legitimate transactions as fraud by mistake.

Recall

Also called sensitivity. High recall means a low false negative rate. In the fraud example, high recall means you’re actually catching most of the real fraud, not letting it slip through.

F1 Score

The harmonic mean of precision and recall. One number that accounts for both, instead of forcing you to eyeball two separate ones.

One note on reading classification_report output: the support column is the number of actual instances of each class in your true labels, not a performance metric itself, just context for interpreting the others.

Which Metric Actually Matters for Your Problem

It depends entirely on which kind of mistake costs you more.

Problem	Prioritize	Why
Detecting cancer	Recall	Missing a real case (false negative) is the costly mistake
Detecting malware	Recall or F1	Same logic, missed threats are expensive
Identifying high-value sales leads for a limited-capacity team	Precision	Wasted effort on bad leads is the costly mistake
Detecting fraudulent transactions	Recall	Missing real fraud usually costs more than a false alarm

The rule that ties it together: use precision when false positives are the expensive mistake, recall when false negatives are.

ROC Curve and AUC

The ROC (Receiver Operating Characteristic) curve shows how the true positive rate and false positive rate change as you shift the classification threshold. False positive rate on the x-axis, true positive rate on the y-axis. A curve that sits above the diagonal “random guessing” line means your model is beating chance. ROC AUC is just the area under that curve, a single number summarizing overall performance across every threshold.

Hyperparameter Tuning

Hyperparameters are the settings you choose before fitting a model, as opposed to the parameters the model learns from the data itself.

Tuning them means:

1. Try different hyperparameter values.
2. Fit the model separately for each combination.
3. Evaluate how each one performs.
4. Keep the best-performing set.

Cross-Validation Keeps You Honest

Tune hyperparameters directly against your test set, and you’ll overfit to that test set without realizing it. Cross-validation is the fix:

• Split your data first.
• Cross-validate on the training set only.
• Keep the test set untouched during tuning.
• Use the test set once, at the end, for the final evaluation.

That keeps the test set meaningful. It’s your one honest read on how the model performs on data it’s never influenced in any way.

GridSearchCV vs. RandomizedSearchCV

GridSearchCV tries every combination of the hyperparameter values you specify. Thorough, but the number of model fits explodes fast as you add more hyperparameters or more possible values per hyperparameter.

from sklearn.model_selection import GridSearchCV

RandomizedSearchCV samples random combinations instead of testing everything. Faster, and often gets you nearly as good a result without the combinatorial blowup.

Handling Missing Values

Real datasets have gaps. You’ve got two main options.

Option 1: drop the rows.

music_df = music_df.dropna()

Before doing this, check how much you’d actually be losing:

print(music_df.isna().sum().sort_values())

Dropping rows is the simplest fix, but it’s not always the right one, especially if missing values are common or concentrated in a way that would bias what’s left.

Option 2: impute.

Imputation means filling in a reasonable guess instead of throwing the row away:

• Mean or median for numerical columns.
• Mode (most common value) for categorical columns.

SimpleImputer in scikit-learn handles this:

from sklearn.impute import SimpleImputer

One rule that’s easy to miss: split your data into train and test before imputing. Imputing on the full dataset first leaks information from the test set into training, and your evaluation numbers stop being trustworthy.

Putting Imputation in a Pipeline

A pipeline chains preprocessing steps and a model together into a single workflow. Every step except the last has to be a transformer, which makes an imputer a natural fit as a step before your actual classifier.

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

imputer = SimpleImputer()

knn = KNeighborsClassifier(n_neighbors=3)

steps = [
    ("imputer", imputer),
    ("knn", knn)
]

pipeline = Pipeline(steps)

pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_test)

print(confusion_matrix(y_test, y_pred))

Bundling preprocessing and modeling into one pipeline keeps the whole workflow consistent, and makes it much harder to accidentally leak data between steps.

Centering and Scaling

Run .describe() on a dataset and you’ll get a quick overview: range, mean, standard deviation, per column. Worth doing early, because features on very different scales can quietly dominate a model’s decisions even when they shouldn’t.

Two common fixes:

Standardization: subtract the mean, divide by the standard deviation. Resulting features are centered at zero with a variance of 1.

Min-max scaling: subtract the minimum, divide by the range. Resulting features run from 0 to 1 (you can also normalize to a -1 to +1 range if that fits your use case better).

Models like KNN, linear regression (including Ridge and Lasso), logistic regression, and neural networks are all sensitive to feature scale. Scale your data before comparing these models, or the comparison won’t mean much.

Evaluating and Comparing Models

A few things matter beyond the raw performance number.

Dataset size. Small dataset, simpler model. Some models need a lot of data to actually perform well, and won’t have it.

Interpretability. Some models you can explain in plain terms to a stakeholder. Others are closer to a black box. Matters a lot depending on who needs to trust the decision.

Flexibility. Models with fewer assumptions about the data can fit more complex patterns, but that flexibility can come at the cost of interpretability or the need for more data.

Metrics by Problem Type

Regression:

• RMSE
• R-squared

Classification:

• Accuracy
• Confusion matrix
• Precision
• Recall
• F1 score
• ROC AUC

A solid workflow: train several models, compare them on the appropriate metrics first, and only then move into hyperparameter tuning on whichever ones look promising. Tuning a bad model choice is wasted effort.

Key Takeaways

• Accuracy can be misleading on imbalanced datasets.
• Precision and recall measure different kinds of mistakes. Pick based on which mistake actually costs you more.
• F1 score gives you both in one number.
• ROC curves show how threshold choice affects performance.
• Hyperparameter tuning finds better model configurations, but only means something if paired with proper cross-validation.
• Keep the test set untouched until final evaluation.
• Missing values can be dropped or imputed. Split before you impute.
• Pipelines keep preprocessing and modeling bundled together, and reduce the risk of data leakage.
• Feature scaling matters for models like KNN, linear/logistic regression, and neural networks.
• Model selection is more than a leaderboard: dataset size, interpretability, and flexibility all matter too.

FAQ

Why isn’t accuracy a good metric for imbalanced datasets?

A model can score high on accuracy just by predicting the majority class every time, while completely failing at the thing you actually care about, like catching a rare fraud case.

What’s the difference between precision and recall?

Precision measures how many of your positive predictions were actually correct (low false positive rate). Recall measures how many of the actual positives you caught (low false negative rate).

When should I prioritize recall over precision?

When missing a real positive case is more costly than a false alarm, like missing a cancer diagnosis or a case of fraud.

What is ROC AUC?

The area under the ROC curve, a single score summarizing how well a model distinguishes between classes across every possible classification threshold.

What’s the difference between GridSearchCV and RandomizedSearchCV?

GridSearchCV tests every combination of specified hyperparameter values. RandomizedSearchCV tests a random sample of combinations, trading some thoroughness for speed.

Should I drop or impute missing values?

It depends on how much data you’d lose and whether the missing values are meaningful. Imputation (filling in a reasonable estimate) is often preferable to dropping rows outright, but always split your data before imputing to avoid data leakage.

Why does feature scaling matter?

Models like KNN, linear/logistic regression, and neural networks can let features with larger numeric ranges dominate the result, even when those features aren’t actually more important. Scaling puts features on equal footing.