Perceptual Intelligence Lab

PyTorch Introduction II

  1. Objective Functions
  2. Optimizers
  3. Activation Functions
  4. Multilayer Perceptron

Objective Functions

Jupyter Notebook

Regression Objective Functions

Consider $\mathbf{y}\in \mathbb{R}^m$ to be the actual values we want to predict, and $\mathbf{\hat{y}} \in \mathbb{R}^m$ is the prediction made by our model. More specifically, $\mathbf{\hat{y}}=f(\mathbf{X}, \mathbf{w})$, where $f$ is the prediction model. We can use several objective functions to evaluate our model. In this section, we will discuss the most commonly used ones: MeanAbsoluteError, MeanSquaredError and HuberError in the regression setting.

Mean Absolute Error (L1 Loss)

Equation:

$L(\mathbf{y}, \mathbf{\hat{y}})=\frac{1}{m}\sum\limits_{i=1}^m|y_i - \hat{y}_i|$

Derivative:

$\frac{\partial L(\mathbf{y}, \mathbf{\hat{y}})}{\partial\mathbf{\hat{y}}}=[d_i]=\begin{cases}\frac{1}{m};&\text{if } \hat{y}_i > y_i\-\frac{1}{m}&\text{otherwise}\end{cases}$

Properties:

  • Less sensitive to samples with large residual between prediction and actual value.

import torch.nn as nn

l1_loss_fn = nn.L1Loss()
loss_l1 = l1_loss_fn(y_pred, y_true)

Mean Squared Error (L2 Loss)

Equation:

$L(\mathbf{y}, \mathbf{\hat{y}})=\frac{1}{m}\sum\limits_{i=1}^m(y_i - \hat{y}_i)^2$

Derivative:

$\frac{\partial L(\mathbf{y}, \mathbf{\hat{y}})}{\partial\mathbf{\hat{y}}}=-\frac{2}{m}(\mathbf{y}-\mathbf{\hat{y}})$,non-differentiable at $y_i = \hat{y}_i$

Properties:

  • More sensitive to samples with large residual between prediction and actual value.

import torch.nn as nn

l2_loss_fn = nn.MSELoss()
loss_l2 = l2_loss_fn(y_pred, y_true)

Huber Loss

Equation:

$L(\mathbf{y}, \mathbf{\hat{y}})=\frac{1}{m}\sum\limits_{i=1}^m\begin{cases}\frac{1}{2}(y_i-\hat{y}_i)^2;;;&\text{if};|y_i-\hat{y}_i|\leq\delta\\delta(|y_i-\hat{y}_i|-\frac{1}{2}\delta)&\text{otherwise}\end{cases}$

Derivative:

$\frac{\partial L(\mathbf{y}, \mathbf{\hat{y}})}{\partial \mathbf{\hat{y}}}=[d_i]=\begin{cases}\hat{y}_i-y_i;;;&\text{if};|y_i-\hat{y}_i|\leq\delta\-\delta&\text{if};|y_i-\hat{y}_i|>\delta\text{ and }|y_i-\hat{y}_i|>0\\delta&\text{otherwise}\end{cases}$

Properties:

  • Hyperparameter $\delta$ can be used to control how many penalties should be given to samples with large residual.

import torch.nn as nn

huber_loss_fn = nn.HuberLoss(delta=1.0) 
loss = huber_loss_fn(y_pred, y_true)

Classification Loss

Cross Entropy

Consider $y_{ij}=1$ denotes the sample $i$ when it belongs to class $j$, $y_i=0$ denotes the sample when it does not belong to class $j$, $\hat{y}_{ij}$ denotes the prediction of the probability to assign sample $i$ to class $j$

Equation:

$H(\mathbf{y}, \mathbf{\hat{y}})=-\frac{1}{m}\sum\limits_{i=1}^m\sum\limits_{j=1}^Ky_{ij}\ln\hat{y}_{ij}$

Derivative:

$\frac{\partial H(\mathbf{y}, \mathbf{\hat{y}})}{\partial\mathbf{\hat{y}}}=[d_{ij}]=-\frac{y_{ij}}{\hat{y}_{ij}}$

Properties:

  • The inferred probability $\mathbf{\hat{y}}$ is more accurate compare to hinge loss.
  • Sigmoid or softmax activation functions are preferred to used in the output layer.

import torch
import torch.nn as nn

# batch_size = 3,num_classes = 5
input_logits = torch.randn(3, 5, requires_grad=True)  
target_labels = torch.tensor([1, 4, 0], dtype=torch.long)

criterion = nn.CrossEntropyLoss()  # softmax in it
loss = criterion(input_logits, target_labels)
print(f"Logits:\n{input_logits}")
print(f"\nTarget Labels: {target_labels}")
print(f"\nCross Entropy Loss: {loss.item()}")
Logits:
tensor([[ 0.1658, -2.7232, -1.8102,  1.6303,  1.1059],
        [-0.1948, -1.3010,  0.5102, -1.4028, -0.2423],
        [-0.0086,  0.1855,  0.1415,  1.5857, -0.4522]], requires_grad=True)

Target Labels: tensor([1, 4, 0])

Cross Entropy Loss: 2.914790391921997

Hinge Loss (Crammer and Singer)

Consider $y_{ij}=1$ denotes the sample $i$ when it belongs to class $j$, $y_i=0$ denotes the sample when it does not belong to class $j$, $\hat{y}_{ij}$ denotes the prediction of the distance to assign sample $i$ to class $j$, the positive value suggests higher confidence while the negative value suggests lower confidence. The prediction value can be an unbounded continuous value.

Equation:

$\text{neg}i =\text{max}{j}((1-y_{ij})\hat{y}_{ij})$

$\text{pos}=\sum\limits_{j=1}^my_{ij}\hat{y}_{ij}$

$L(\mathbf{y}, \mathbf{\hat{y}})=\frac{1}{m}\sum\limits_{i=1}^m\text{max}(0, 1 + \text{neg}_i - \text{pos})$

Derivative:

$\frac{\partial L(\mathbf{y}, \mathbf{\hat{y}})}{\partial \mathbf{\hat{y}}}=[d_{ij}]=\begin{cases}1-2y_{ij};&\text{if }{(1-y_{ij})\hat{y}{ij}}=\text{neg}i\text{ and neg}i-\text{pos}>-1\-y{ij};&\text{if }{(1-y{ij})\hat{y}{ij}}\ne\text{neg}_i\text{ and neg}_i-\text{pos}>-1\0&\text{otherwise}\end{cases}$

Properties:

  • Data point far away from the decision boundary do not contribute to the loss function.
  • Linear or hyperbolic tangent activation functions are preferred to used in the output layer.
  • Need additional methods (e.g. Platt scaling) to estimate probability for each class.

import torch
import torch.nn as nn

criterion_multimargin = nn.MultiMarginLoss(margin=1.0)
input_logits = torch.randn(3, 5, requires_grad=True)
target_labels = torch.tensor([1, 4, 0], dtype=torch.long)

loss_multimargin = criterion_multimargin(input_logits, target_labels)
print(f"Logits:\n{input_logits}")
print(f"\nTarget Labels: {target_labels}")
print(f"MultiMargin Loss: {loss_multimargin.item():.4f}")
Logits:
tensor([[-0.6638, -0.5470,  0.2552,  0.1364,  0.8997],
        [-0.4054, -1.0357, -0.8514,  1.1676, -1.8394],
        [-0.6159, -0.2628, -1.0372,  0.1841, -0.4444]], requires_grad=True)

Target Labels: tensor([1, 4, 0])
MultiMargin Loss: 1.4634

Optimizers

Jupyter Notebook

Adaptive Gradient Descent (AdaGrad)

Equation:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\sum\limits_{i=1}^t\mathbf{g}^{(t)T}\mathbf{g}^{(t)}+\varepsilon}}\mathbf{g}^{(t)}$

$\mathbf{g}^{(t)}=∇_\mathbf{w}L(\mathbf{w}^{(t)})$

Parameters:

  • $\varepsilon$ is a constant that make sure the denominator is not zero (default in PyTorch: $1e-8$).
  • $\eta$ is the learning rate (default in PyTorch: $0.001$)

Properties:

  • The magnitude of the update will always be smaller in later iterations.
  • The update become very inefficient in later iterations.
  • Vulnerable to local minimum.
  • Converge even with large learning rate.

API:

opt = torch.optim.Adagrad(params = [w], lr=learning_rate)
opt.zero_grad()
loss.backward()
opt.step()

RMSprop

Equation:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{G^{(t)}} + \varepsilon}\mathbf{g}^{(t)}$

$\mathbf{g}^{(t)}=\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$

$G^{(t)}=\rho G^{(t-1)}+(1-\rho)\mathbf{g}^{(t)T}\mathbf{g}^{(t)}$

Parameters:

  • $\varepsilon$ is a constant that make sure the denominator is not zero (default in Tensorflow/Keras: $1e-7$).
  • $\eta$ is the learning rate (default in Tensorflow/Keras: $0.001$)
  • $\rho$ is the parameter that control how much the model should consider the computed gradient (sum of magnitude) in previous iteration. (default in Tensorflow/Keras $0.9$)

Properties:

  • Vulnerable to local minimum.
  • Compare to AdaGrad, RMSProp use a weighted average between previous magnitude of gradient and the magnitude of gradient in current iteration. Keeping the normalization on the magnitude of gradient, but allows more efficient optimization in later iterations.

API:

opt = torch.optim.RMSprop(params, lr=learning_rate)
opt.zero_grad()
loss.backward()
opt.step()

Momentum

Equation:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\mathbf{v}^{(t)}$

$\mathbf{v}^{(t)}=\gamma\mathbf{v}^{(t-1)}+\eta\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$

Parameters:

  • $\eta$ is the learning rate (default in Tensorflow/Keras: $0.01$)
  • $\gamma$ is the momentum coefficient controlling how much the model should consider the computed gradient before.

Properties:

  • Implemented in SGD and RMSProp in Tensorflow and Keras.
  • Generalize well to most of the application.
  • Consider both magnitude and direction of previous gradient when update the parameters.
  • Might not converge with large learning rate.
  • Less vulnerable to local minimum.

API:

opt = torch.optim.SGD(params, lr=learning_rate, momentum=momentum)
opt.zero_grad()
loss.backward()
opt.step()

Nesterov Accelerate Gradient (NAG)

Equation:

$\tilde{\mathbf{w}}^{(t)} = \mathbf{w}^{(t)} + \gamma \mathbf{v}^{(t-1)} = \mathbf{w}^{(t)} + \gamma(\mathbf{w}^{(t)} - \mathbf{w}^{(t-1)})$

$\mathbf{v}^{(t)} = \gamma \mathbf{v}^{(t-1)} + \eta \nabla_\mathbf{w} L(\tilde{\mathbf{w}}^{(t)})$

$\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \mathbf{v}^{(t)}$

Parameters:

  • $\eta$ is the learning rate (default in Tensorflow/Keras: $0.01$)
  • $\gamma$ is the momentum coefficient controlling how much the model should consider the computed gradient before.

Properties:

  • Implemented in SGD in Tensorflow and Keras.
  • Generalize well to most of the application.
  • Consider both magnitude and direction of previous gradient when update the parameters.
  • NAG can be seen as approximating the second-order correction of the gradient, by taking into account the gradient at the lookahead position $\mathbf{w}^{(t)} + \gamma(\mathbf{w}^{(t)} - \mathbf{w}^{(t-1)})$.Therefore, it usually converges faster than the standard Momentum method.
  • Might not converge with large learning rate.
  • Less vulnerable to local minimum.

API:

opt = torch.optim.SGD(params, lr=learning_rate, momentum=momentum, nesterov=True)
opt.zero_grad()
loss.backward()
opt.step()

Adam

Equation:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\hat{v}^{(t)} + \varepsilon}}\mathbf{\hat{m}}^{(t)}$

$\mathbf{g}^{(t)}=\nabla_\mathbf{w}L(\mathbf{w}^{(t)})$

$\mathbf{m}^{(t)}=\beta_1\mathbf{m}^{(t-1)}+(1-\beta_1)\mathbf{g}^{(t)}$

$\mathbf{\hat{m}}^{(t)}=\frac{\mathbf{m^{(t)}}}{1-\beta_1^t}$

$G^{(t)}=\mathbf{g}^{(t)T}\mathbf{g}^{(t)}$

$v^{(t)}=\beta_2v^{(t-1)}+(1-\beta_2)G^{(t)}$

$\hat{v}^{(t)}=\frac{v^{(t)}}{1-\beta_2^t}$

Parameters:

  • $\eta$ is the learning rate (default in Tensorflow/Keras: $0.01$)
  • $\varepsilon$ is a constant that make sure the denominator is not zero (default in Tensorflow/Keras: $1e-7$).
  • $\beta_1$ is the parameter that control how much the model should consider the (mainly) the direction of gradient before. (default in Tensorflow/Keras $0.9$)
  • $\beta_2$ is the parameter that control how much the model should consider the magnitude of gradient before. (default in Tensorflow/Keras $0.999$)
  • Since $m_t$ (the first moment estimate) and $v_t$ (the second moment estimate) are typically initialized as zero vectors, they tend to be biased towards zero in the early stages of training (when $t$ is small), leading to an underestimation of the true moments.$\hat{m}_t$ and $\hat{v}_t$ are the bias-corrected first and second moments. As $t$ (the time step) increases, the denominator of the correction factor approaches 1, and consequently, the effect of the correction diminishes.

Properties:

  • Consider both magnitude and direction of previous gradient when update the parameters.
  • Generalize well to most of the application.
  • Using the exponentially moving average to adaptively adjust the learning rate.
  • Converge with slightly larger large learning rate.
  • Less vunerable to local minimum.

API:

opt = torch.optim.Adam(params, lr=learning_rate)
opt.zero_grad()
loss.backward()
opt.step()

Adam with Weight Decay (AdamW)

When training the neural network, we often add $L2$-penalty to the loss function in order to prevent overfitting:

$L(y, f)=(\mathbf{y}-f(\mathbf{X}, \mathbf{w}))^2+\frac{\lambda}{2}\mathbf{w}^T\mathbf{w}=L’(\mathbf{y}, f(\mathbf{X}, \mathbf{w}))+\frac{\lambda}{2}\mathbf{w}^T\mathbf{w}$

However, if we introduce $L2$-penalty in combination with Adam optimizer:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\frac{\eta}{\sqrt{\hat{y}^{(t)}+\varepsilon}}\frac{1}{1-\beta_1}(\beta_1\mathbf{m}^{(t-1)}+(1-\beta_1)(\mathbf{g}^{(t)}+\lambda\mathbf{w}^{(t)}))$

If $\hat{y}^{(t)}$ is large (suggests that the magnitude of the gradient is large), the $L2$-penalty actually becomes smaller. In other words, adding $L2$-penalty only penalize the weight with smaller changes, and therefore the effect of using $L2$-penalty with Adam have limited effect. To address this, Ilya Loshchilov and Frank Hutter proposed a new method to decouples the choice of weight decay factor from the setting of the learning rate. To be more specific, the authors proposed:

$\mathbf{w}^{(t+1)}=\mathbf{w}^{(t)}-\eta(\frac{1}{\sqrt{\hat{v}^{(t)} + \varepsilon}}\mathbf{\hat{m}}^{(t)}+\lambda\mathbf{w}^{(t)})$

API:

opt = torch.optim.AdamW(params, lr=learning_rate, weight_decay=weight_decay)
opt.zero_grad()
loss.backward()
opt.step()

Activation Functions

Identity Function

Equation

$$ f(x) = x $$

Derivative

$$ \frac{\partial f(x)}{\partial x} = 1 $$

Properties

  • Mathematically does not affect the network at all.
  • Practically, in order to unify the design structure of the neural network (as linear transformation followed by activation function), most of the deep learning framework implemented identify function as one of the activation functions.
import torch
import torch.nn as nn

X = torch.linspace(-10., 10., 21)
identity = nn.Identity()
identity(X)

Sigmoid

Equation:

$\sigma(x)=\frac{1}{1+\exp(-x)}$

Derivative:

$\frac{\partial\sigma(x)}{\partial x}=\sigma(x)(1-\sigma(x))$

Properties:

  • Neural network with only one hidden layer of sigmoid function is equivalent to logistic regression.
  • One very useful property of sigmoid function is that the range is $0$ to $1$ (bounded range); therefore, it is commonly used when we want to model probability (e.g. output layer of binary classification problem).
  • In binary classification problem, sigmoid function is a special case of softmax function.
  • In multi-class classification, sigmoid is preferable over softmax when we assume a sample can belong to multiple class since sigmoid function treat the probability of each class independently.
  • Could lead to gradient vanishing problem (which can be addressed using batch normalization).
import torch
import torch.nn as nn

X = torch.linspace(-10., 10., 21)
sigmoid = nn.Sigmoid()
output = sigmoid(X)
print(output)

Hyperbolic Tangent

Equation:

$\tanh(x)=\frac{\exp(x)-\exp(-x)}{\exp(x)+\exp(-x)}$

Derivative

$\frac{\partial(\tanh(x))}{\partial x}=\frac{1}{\cosh^2(x)}$

Properties:

  • Commonly used in the gate of long short-term memory (LSTM) and gate recurrent unit (GRU).
import torch
import torch.nn as nn

X = torch.linspace(-10., 10., 21)
tanh = nn.Tanh()
output = tanh(X)
print(output)

Rectified Linear Unit (ReLU)

Equation

$\text{ReLU}(x)=\max(0, x)$

Derivative

$\frac{\partial(\text{ReLU}(x))}{\partial x}=\begin{cases}1;&\text{if } x > 0\0;&\text{otherwise}\end{cases}$

Properties:

  • The most commonly used activation function in the hidden layer of neural networks.
  • The output of the activation function is a sparse (have many zeros) tensor. Therefore, it requires less computational time and memory to reach convergence.
  • The sparsity could potentially prevent overfitting.
  • Computational efficient to compute the derivative since it does not require exponential computation (about 1.5x faster than sigmoid and 2x faster than hyperbolic tangent).
  • However, the sparsity might also lead to dying ReLU.
  • Might cause gradient exploding problem (will discuss in recurrent neural networks) when the gradient of the linear combination before applying ReLU is large.
  • Alleviate gradient vanishing problem.
import torch
import torch.nn as nn

X = torch.linspace(-10., 10., 21)
relu = nn.ReLU()
output = relu(X)
print(output)

Leaky ReLU

Equation

$f(x)=\begin{cases} x;;;;;;;;;&\text{if }x > 0\ ax &\text{otherwise}\end{cases}$

Derivative

$\frac{\partial f(x)}{\partial x}=\begin{cases}1;&\text{if } x > 0\a;&\text{otherwise}\end{cases}$

Parameters

  • $a$ is a hyperparameter controlling the slope of negative values (default in Keras: $0.2$)

Properties:

  • Computationally more efficient than ELU.
  • Might cause gradient exploding problem when the gradient of the linear combination before applying Leaky ReLU is large.
  • Alleviate gradient vanishing problem.
  • Alleviate the dying ReLU problem.
import torch
import torch.nn as nn

X = torch.linspace(-10., 10., 21)
leaky_relu = nn.LeakyReLU(negative_slope=0.2)
output = leaky_relu(X)
print(output)

Exponential Linear Unit (ELU)

Equation: $f(x)=\begin{cases}x;;;;;;;;;&\text{if }x > 0\a(\exp(x)-1) &\text{otherwise}\end{cases}$

Derivative: $\frac{\partial f(x)}{\partial x}=\begin{cases}1;&\text{if }x > 0\a\exp(x)&\text{otherwise}\end{cases}$

Parameters:

  • $a$ is a coefficient of exponential transformation on negative values (default in Tensorflow: $1.0$)

Properties:

  • $f(x)\to-a$ when $x \to -\infty$
  • Converge faster than Leaky ReLU.
  • Might cause gradient exploding problem when the gradient of the linear combination before applying ELU is large.
  • Alleviate gradient vanishing problem.
  • Alleviate the dying ReLU problem.
import torch
import torch.nn as nn

X = torch.linspace(-10., 10., 21)
elu = nn.ELU(alpha=1.0)
output = elu(X)
print(output)

Softmax

Equation:

$\mathbf{\hat{y}}=\text{softmax}(\mathbf{x})=\frac{\exp(\mathbf{x})}{\sum\limits_{i=1}^{n} \exp(x_i)}=\begin{bmatrix} \frac{\exp(x_1)}{\sum\limits_{i=1}^{n} \exp(x_i)} \ \frac{\exp(x_2)}{\sum\limits_{i=1}^{n} \exp(x_i)} \ \vdots\ \frac{\exp(x_n)}{\sum\limits_{i=1}^{n} \exp(x_i)} \ \end{bmatrix}$

Derivative:

$\frac{\partial\mathbf{\hat{y}}}{\partial\mathbf{x}}=\begin{bmatrix} \frac{\partial\hat{y}_1}{\partial x_1} & \frac{\partial\hat{y}_1}{\partial x_2} & \cdots & \frac{\partial\hat{y}_1}{\partial x_n} \ \frac{\partial\hat{y}_2}{\partial x_1} & \frac{\partial\hat{y}_2}{\partial x_2} & \cdots & \frac{\partial\hat{y}_2}{\partial x_n} \ \vdots & \vdots & \ddots & \vdots \ \frac{\partial\hat{y}_n}{\partial x_1} & \frac{\partial\hat{y}_n}{\partial x_2} & \cdots & \frac{\partial\hat{y}_n}{\partial x_n}\end{bmatrix} = \begin{bmatrix} \hat{y}_1(1-\hat{y}_1) & \hat{y}_1(0-\hat{y}_2) & \cdots & \hat{y}_1(0-\hat{y}_n) \ \hat{y}_2(0-\hat{y}_1) & \hat{y}_2(1-\hat{y}_2) & \cdots & \hat{y}_2(0-\hat{y}_n) \ \vdots & \vdots & \ddots & \vdots \ \hat{y}_n(0-\hat{y}_1) & \hat{y}_n(0-\hat{y}_2) & \cdots & \hat{y}_n(1-\hat{y}_n) \end{bmatrix} = \mathbf{1}\hat{y}^T\odot(\mathbf{I}-\hat{y}\mathbf{1}^T)$

Properties:

  • Note that the input for softmax should be a vector instead of a scalar.
  • Similar to sigmoid function, the range of softmax function is $0$ to $1$ (bounded range); therefore, it is also commonly used when we want to model probability.
  • In multi-class classification, softmax is preferable over sigmoid when we assume that a sample can only belong to one class.
import torch
import torch.nn.functional as F

X = torch.linspace(-1., 1., 5)
X1, X2 = torch.meshgrid(X, X, indexing='ij')
X1 = X1.reshape(-1, 1)
X2 = X2.reshape(-1, 1)
X_concat = torch.cat([X1, X2], dim=1)
output = F.softmax(X_concat, dim=1)
print(output)

Multilayer Perceptron

Jupyter Notebook

Forward Propagation

The multilayer perceptron consists of several layers of operations. The output of each neuron in the layer is the linear combination of the input tensor followed by an activation function. For instance, in $Layer^1$, the first neuron $\mathbf{a}^1_1$ is:

$\mathbf{a}^1_1 = f^1(\mathbf{X^0w^1_1}+b^1)$

The output of every neuron in $Layer^1$ will then be stacked into a new Tensor:

$\mathbf{A}^1=\begin{bmatrix} \mathbf{a}^1_1 & \mathbf{a}^1_2 & \cdots &　\mathbf{a}^1_{n1}\end{bmatrix}=\begin{bmatrix} a_{11}^1 & a_{21}^1 & \cdots & a_{n_11}^1\ a_{12}^1 & a_{22}^1 & \cdots & a_{n_12}^1\ \vdots & \vdots & \ddots & \vdots\ a_{1m}^1 & a_{2m}^1 & \cdots & a_{n_1m}^1\\end{bmatrix}$

where $n_1$ is the number of neuron in the $Layer^1$.

We can denote the output of each layer with $a_{ij}^l$, where $l$ is the index of layer, $i$ is the index of neuron in $Layer^l$, $j$ is the index of the sample. The output of $Layer^{l}$ will then be fed to the next layer $Layer^{l+1}$ as the input. This process continues until the last layer $Layer^{k}$ of the network, generating the prediction $\mathbf{\hat{Y}}=\mathbf{A}^k$. Note that the number of neuron in $Layer^k$ has to be the same as the shape of $\mathbf{Y}$ (number of classes in classification, or number of targets in regression). Note that we apply different initialization for the weight in the different neuron, and therefore the neurons in the same layer generate slightly different output (will discuss in Weight Initialization).

Back Propagation

The goal to train this multilayer perceptron network is to find the optimal weight $\mathbf{W}^*$ in every layer that optimizes the objective function between the output of the network $\mathbf{\hat{Y}}$ and the ground truth target $\mathbf{Y}$. To be more specific, we first need to compute the gradient of the objective function with respect to the output of the network $\mathbf{\hat{Y}}$:

$\nabla_\mathbf{\hat{Y}}L(\mathbf{Y}, \mathbf{\hat{Y}})$

The Jacobian matrix of all the output from every neuron in $Layer^k$ with respect to the linear combination $\mathbf{X}^k$:

$J_{f^k}(\mathbf{\mathbf{x}})=\frac{\partial\mathbf{a}^k_i}{\partial\mathbf{x}_i^k}$

And the Jacobian matrix of the output of linear combination with respect to the weight $\mathbf{w}$:

$J_{\mathbf{A}^{k-1}w^k_i}(\mathbf{a_i^{k-1}})=\frac{\partial\mathbf{x}_i^k}{\partial\mathbf{w}_i^k}=\mathbf{a}_i^{k-1}$

where $k$ denotes the layer in the multilayer perceptron network. The gradient of the objective function with respect to the weight in the last layer $\mathbf{w}^k$ is:

$\nabla_\mathbf{w_1^k}L(\mathbf{Y}, \mathbf{\hat{Y}})=(\frac{\partial\mathbf{x}_i^{k}}{\partial\mathbf{w}_i^k})^T(\frac{\partial\mathbf{a}_i^k}{\partial\mathbf{x}i^k})^T\nabla\mathbf{\hat{Y}}L(\mathbf{Y}, \mathbf{\hat{Y}})$

The process continues from the output layer to the input layer; therefore, this process is called back propagation. The gradient of the objective function with respect to the weight $\mathbf{w}_i$ (contribute to neuron $i$) in layer $l$ can be written as:

$\nabla_{w_i^l}L(\mathbf{Y}, \mathbf{\hat{Y}})=(\frac{\partial\mathbf{x}_i^l}{\partial\mathbf{w}i^l})^T(\frac{\partial\mathbf{a}i^l}{\partial\mathbf{x}i^l})^T\sum\limits{j=1}^{n{l+1}}\nabla\mathbf{w_j^{l+1}}L(\mathbf{Y}, \mathbf{\hat{Y}})$

After computing the gradient, we can apply gradient descent in combination with optimizers to optimize the weight in each layer

PyTorch Implementation

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import random_split, DataLoader
import torchvision
import torchvision.transforms as transforms

transform = transforms.ToTensor()
mnist_train = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)

train_size = int(0.75 * len(mnist_train))
val_size = len(mnist_train) - train_size
train_set, val_set = random_split(mnist_train, [train_size, val_size])

train_loader = DataLoader(train_set, batch_size=64, shuffle=True)
val_loader = DataLoader(val_set, batch_size=64)

class MLP(nn.Module):
    def __init__(self, encoding_dim=128):
        super().__init__()
        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28*28, encoding_dim),
            nn.ReLU(),
            nn.Linear(encoding_dim, 10)
        )
    def forward(self, x):
        return self.network(x)

model = MLP()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())

num_epochs = 10
train_loss, val_loss, train_acc, val_acc = [], [], [], []

for epoch in range(num_epochs):
    model.train()
    correct = total = running_loss = 0
    for x, y in train_loader:
        optimizer.zero_grad()
        out = model(x)
        loss = criterion(out, y)
        loss.backward()
        optimizer.step()
        running_loss += loss.item() * x.size(0)
        correct += (out.argmax(1) == y).sum().item()
        total += y.size(0)
    train_loss.append(running_loss / total)
    train_acc.append(correct / total)

    model.eval()
    correct = total = val_running_loss = 0
    with torch.no_grad():
        for x, y in val_loader:
            out = model(x)
            loss = criterion(out, y)
            val_running_loss += loss.item() * x.size(0)
            correct += (out.argmax(1) == y).sum().item()
            total += y.size(0)
    val_loss.append(val_running_loss / total)
    val_acc.append(correct / total)

    print(f"Epoch {epoch+1}: Train Acc {train_acc[-1]:.4f}, Val Acc {val_acc[-1]:.4f}")

Related articles

PyTorch Introduction I

1. Download Anaconda or Miniconda. 2. After installation, open a terminal. 3. Create a dedicated virtual environment. 4. Install PyTorch according to your system and CUDA version. A tensor is the m…

Training