Deep Learning Basics with PyTorch: Neural Networks, Training, and Optimization | Blog

A neural network doesn’t know anything when you create it. Every weight starts random. Everything it “learns” comes from a loop: make a prediction, measure how wrong it was, adjust, repeat. This post walks through that loop in PyTorch, piece by piece.

The Basic Structure

Every neural network has three kinds of layers:

Input layer: where your data enters.
Hidden layers: where the actual learning happens.
Output layer: where predictions come out.

One thing worth knowing up front: deep learning models generally need a lot more data than traditional machine learning models to actually perform well.

Tensors: The Basic Unit in PyTorch

A tensor is PyTorch’s version of an array or matrix. Everything, your data, your weights, your gradients, gets stored and processed as tensors.

A couple of rules that trip people up early:

Addition and subtraction only work if both tensors have the same shape.
Element-wise multiplication uses . Matrix multiplication uses @. Mixing these up gives you very wrong numbers, not an error, which is worse.

Building the Network

A fully connected layer means every neuron connects to every neuron in the previous layer. In PyTorch, you build these with nn.Linear.

Two numbers you need to get right:

Input dimension: how many features you’re feeding in.
Output dimension: how many classes or values you’re predicting.

Every linear layer has two learnable pieces:

Weights: how much each input feature matters.
Bias: a baseline output, independent of the inputs.

One useful fact for sanity-checking your architecture: a neural network with zero hidden layers is basically just a linear model. All the “deep” in deep learning comes from what you stack in between.

Hidden Layers and Model Capacity

Hidden layers sit between input and output. In PyTorch, you stack them with nn.Sequential, and the input dimension of each layer has to match the output dimension of the one before it.

Add more hidden layers, and you add more learnable parameters. The total parameter count is called model capacity.

More capacity means the model can represent more complex patterns. It’s not free, though:

Too much capacity, and the model can overfit, memorizing the training data instead of learning general patterns.
Too little capacity, and it underfits, failing to pick up on patterns that are actually there.

Quick tool: .numel() gives you the total number of elements in a tensor, useful for checking how big your layers actually are.

Activation Functions

Here’s the part that makes “deep” learning actually work: activation functions introduce non-linearity.

Without them, stacking ten linear layers is mathematically no different from having one. Linear functions composed with linear functions are still linear. Activation functions are what let a network learn relationships that aren’t just straight lines.

The output right before an activation function gets applied is called the pre-activation output, worth knowing if you’re ever debugging layer by layer.

Sigmoid

Used for binary classification. Squashes any input into a value between 0 and 1.

nn.Sigmoid()

Softmax

Used for multi-class classification. Converts a set of outputs into probabilities that sum to 1 across all classes.

nn.Softmax(dim=-1)

Worth noting: a network with linear layers followed by sigmoid is functionally similar to logistic regression. Not a coincidence, deep learning genuinely generalizes a lot of “classical” methods.

The Forward Pass

Data goes in, flows through each layer in order, and predictions come out the other end. That’s the forward pass. It happens the same way during both training and inference (actually using the model).

What happens at the end depends on the task:

Binary classification: final output through sigmoid, giving you a probability between 0 and 1.
Multi-class classification: final output through softmax, giving you a probability per class.

Loss Functions

The loss function is how you measure “how wrong was that prediction.” Training exists to minimize this number.

For classification, Cross Entropy Loss is the standard choice. It compares your predicted class scores against the true labels.

One-Hot Encoding

Labels in classification tasks are often represented as one-hot vectors, mostly zeros, with a single 1 marking the correct class.

Say the true class is 1 out of 3 possible classes:

[0, 1, 0]

Lower loss means better predictions. That’s the whole scoreboard.

Backpropagation and Gradient Descent

Weights and biases start random. Training is the process of nudging them toward values that actually predict something.

Backpropagation

Backpropagation computes gradients, the derivatives that tell you how much each parameter is contributing to the loss, and in which direction to move it.

Access your model’s parameters directly with:

model.parameters()

Gradient Descent

Gradient descent takes those gradients and updates each parameter in whatever direction reduces the loss.

How big each update step is comes down to the learning rate:

Too large, and training can become unstable, overshooting good values entirely.
Too small, and training crawls, technically working but taking forever to get anywhere.

Optimizers

You don’t manually update every weight and bias. Optimizers handle that, using the model’s parameters, the computed gradients, and the learning rate.

The common ones in PyTorch:

SGD: basic gradient descent, straightforward and well understood.
Adam: adapts the learning rate as training goes, and it’s the default most people reach for.
RMSprop: another adaptive option, less common than Adam but still used.

All available through:

torch.optim

Why ReLU Wins in Hidden Layers

Sigmoid and softmax have a real problem for hidden layers: their outputs are squashed into a narrow range (0 to 1), and their gradients get extremely small for inputs far from zero.

That matters because of how backpropagation works. Each layer’s gradient update depends on gradients flowing back from later layers. If those gradients are tiny, the earlier layers barely update at all. That’s the vanishing gradient problem, and it can effectively stall learning in the earlier parts of a deep network.

This is why sigmoid and softmax are generally reserved for the output layer, not stacked through hidden layers.

ReLU

ReLU (Rectified Linear Unit) is the standard default for hidden layers instead.

The formula:

f(x) = max(0, x)

Positive input, passes through unchanged. Negative input, becomes zero. Simple, cheap to compute, and it avoids squashing gradients the way sigmoid does, which is exactly why it holds up better across deep networks.

Leaky ReLU

ReLU has its own weak spot: any negative input produces exactly zero output, always. If a neuron ends up consistently receiving negative inputs, it can get stuck outputting zero forever and effectively stop learning. This is known as the dying ReLU problem.

Leaky ReLU fixes it with a small tweak. Positive inputs behave exactly like regular ReLU. Negative inputs, instead of flattening to zero, get multiplied by a small coefficient (PyTorch’s default is usually 0.01).

That keeps the gradient non-zero even for negative inputs, so neurons can keep learning instead of going permanently silent. It’s become common across a lot of modern architectures for exactly this reason.

Stochastic Gradient Descent, in Practice

Training a neural network is, at its core, an optimization problem: minimize the loss by adjusting parameters. SGD is one of the standard algorithms for doing that.

Two hyperparameters matter most here:

Learning rate: same tradeoff as before, too small is slow, too large is unstable.

Momentum: helps the optimizer build up speed in a consistent direction, smoothing out oscillations and helping avoid getting stuck in a mediocre local minimum.

A useful intuition: as training approaches a good minimum, gradients naturally shrink, which naturally shrinks the update steps too. That’s the model making finer and finer adjustments as it converges, not you needing to intervene manually.

A common starting point: learning rate 0.001, momentum 0.95. Not universal, but a reasonable default to start tuning from.

Putting Together a Training Loop

The full process, start to finish:

Create the model.
Choose a loss function.
Prepare the dataset.
Define an optimizer.
Run the training loop.

Inside each iteration of that loop:

Forward pass, generate predictions.
Calculate the loss.
Backpropagation, compute gradients.
Optimizer step, update the parameters.

Repeat across multiple epochs until the model’s actually learned something useful.

The whole cycle, visually:

Input
  ↓
Forward Pass
  ↓
Predictions
  ↓
Loss
  ↓
Backpropagation
  ↓
Gradients
  ↓
Optimizer
  ↓
Updated Parameters
  ↓
Repeat

Regression vs. Classification: What Changes at the Output

The final layer is the one place where your task type really changes the architecture.

Regression: final layer outputs a continuous value directly, no activation squashing it.
Classification: final layer typically runs through sigmoid (binary) or softmax (multi-class) to produce probabilities.

How It All Connects

Every piece in this post is one link in the same chain:

The network makes a prediction through a forward pass.
The loss function scores how wrong that prediction was.
Backpropagation figures out how much each parameter contributed to that wrongness.
The optimizer uses that information to nudge every parameter in a better direction.
Repeat, for as many epochs as it takes.

That’s the entire objective of training, in one sentence: adjust parameters until the loss goes down and the model actually generalizes to data it hasn’t seen.

FAQ

What is a tensor in PyTorch?

PyTorch’s core data structure, similar to an array or matrix, used to store and process everything from input data to model weights.

What is model capacity?

The total number of learnable parameters in a model. Higher capacity means more complex patterns can be learned, but also more risk of overfitting.

Why are activation functions needed?

They introduce non-linearity. Without them, stacking multiple linear layers is mathematically no different from having a single linear layer.

What is a forward pass?

Passing input data through a neural network, layer by layer, to produce a prediction. Used during both training and inference.

What is backpropagation?

The process of computing gradients that show how much each model parameter contributed to the loss, used to guide parameter updates.

What does the learning rate control?

How large each parameter update step is during optimization. Too high and training gets unstable, too low and training crawls.

Why is ReLU commonly used in hidden layers?

It avoids the vanishing gradient problem that sigmoid and softmax run into, making it more effective for training deep networks.

What is the dying ReLU problem?

When a neuron consistently receives negative inputs, ReLU always outputs zero for it, and the neuron can effectively stop learning entirely.

What is Leaky ReLU?

A variation of ReLU that allows a small, non-zero output for negative inputs instead of flattening them to zero, which keeps gradients alive and helps avoid the dying ReLU problem.

What is an optimizer?

The component that automatically updates model parameters using gradients during training. SGD, Adam, and RMSprop are the common options in PyTorch.