Batch normalization

Batch normalization

Batch normalization (BN) is a method used to make training of artificial neural networks faster and more stable through normalization of the layers’ inputs by re-centering and re-scaling1. It was proposed by Sergey Ioffe and Christian Szegedy in 2015.

Mechanism:

  • It normalizes the activations of each layer for every mini-batch independently.
  • It adds two trainable parameters per layer, which scale and shift the normalized output.
  • It introduces some noise to the output of each layer.

Pros:

  • It helps improve the performance of machine learning models by making training faster and more stable.
  • It helps reduce overfitting.
  • It helps reduce the dependence on initialization.
  • Works well with other optimization methods (SGD w/ momentum, RMSprop, Adam)
  • Allows for higher learning rates

Cons:

  • It can be computationally expensive.
  • It can cause problems when used with small batch sizes.
  • Is not good for RNNs / LSTMs.
  • Have to use a different calculation between training and testing.

Examples:

  • In a convolutional neural network (CNN), batch normalization can be applied after each convolutional layer.
  • In a deep neural network (DNN), batch normalization can be applied after each fully connected layer.

Code using numpy:

def batch_norm(Z, gamma, beta, epsilon):
    """
    @Z is a numpy.ndarray of shape (m, n) that should be normalized
        @m is the number of data points
        @n is the number of features in Z
    @gamma is a numpy.ndarray of shape (1, n)
    containing the scales used for batch normalization
    @beta is a numpy.ndarray of shape (1, n)
    containing the offsets used for batch normalization
    @epsilon is a small number used to avoid division by zero
    Returns: the normalized Z matrix
    """
    # Calculate the mean and variance of Z
    mean = np.mean(Z, axis=0)
    var = np.var(Z, axis=0)

    # Normalize Z
    Z_norm = (Z - mean) / np.sqrt(var + epsilon)

    # Scale and shift the normalized Z using gamma and beta
    Z_norm_scaled_shifted = gamma * Z_norm + beta

    return Z_norm_scaled_shifted

Code using tensorflow:

def create_batch_norm_layer(prev, n, activation):
    """
    @prev is the activated output of the previous layer
    @n is the number of nodes in the layer to be created
    @activation is the activation function that
    should be used on the output of the layer
    Returns: a tensor of the activated output for the layer
    """
    # Initialize the base layer with 'Dense' function from tensorflow
    # The 'prev' input is passed through a dense layer with n nodes
    k_init = tf.contrib.layers.variance_scaling_initializer(mode="FAN_AVG")
    layer = tf.layers.Dense(units=n,
                            kernel_initializer=k_init,
                            use_bias=False)(prev)

    # Initialize trainable parameters 'gamma' and 'beta' as vectors of 1
    # and 0 respectively, with shape (1, n)
    gamma = tf.Variable(initial_value=tf.ones(shape=(1, n)), name="gamma")
    beta = tf.Variable(initial_value=tf.zeros(shape=(1, n)), name="beta")

    # Calculate the batch mean and variance of the previous layer
    # using the tf.nn.moments() function
    mean, variance = tf.nn.moments(layer, axes=[0])

    # Create a batch normalization layer using the
    # tf.nn.batch_normalization() function
    # with gamma, beta, mean, variance and epsilon=1e-8 as arguments
    # Apply the activation function to
    # the output of the batch normalization layer
    # The final result is a tensor of the activated output for the layer
    layer_norm = tf.nn.batch_normalization(layer,
                                           mean,
                                           variance,
                                           beta,
                                           gamma,
                                           1e-8)
    output = activation(layer_norm)
    return output


Comments

Post a Comment

Popular posts from this blog

Mini-Batch Gradient Descent

Image
Mini-Batch Gradient Descent Mini-Batch Gradient Descent is a variation of the Gradient Descent algorithm that is commonly used in deep learning. It is also known as Stochastic Gradient Descent (SGD), although technically, SGD refers to the version of the algorithm that uses a batch size of 1. To implement Mini-Batch Gradient Descent, we first randomly shuffle the training data, and then divide it into batches of a fixed size. We then loop over the batches and perform the following steps for each batch: Compute the gradients of the loss function with respect to the model parameters using the current batch of data. Update the model parameters using the computed gradients and the learning rate. Repeat until the entire dataset has been processed a fixed number of times (epochs). Mechanism: It updates the model in small batches instead of one large batch. It reduces the variance of the parameter updates, which can lead to more stable convergence. It can make use of highly optimized matrix o
Read more

Saddle points, ravines and local optimum

Saddle points, ravines and local optimum: In the context of optimization, a saddle point is a critical point of a function where the slopes (gradients) of the function in different directions are zero, but some directions are minima and others are maxima. This means that a gradient descent algorithm may get stuck at the saddle point and fail to converge to the global optimum. A ravine is a narrow valley in the optimization landscape where the gradients are steep and the optimization algorithm can converge quickly. However, ravines can also be problematic because they can lead to oscillations and slow convergence when the optimization algorithm overshoots the minimum. A local optimum is a point in the optimization landscape where the function has the lowest value in a local neighborhood, but there may be other points that have lower values in other parts of the landscape. Optimization algorithms like gradient descent can converge to local optima instead of the global optimum, which can
Read more

Gradient descent with momentum

Image
 Gradient descent with momentum Gradient descent with momentum is an optimization algorithm that is used to update the weights of a neural network during training. It is an extension of the standard gradient descent algorithm that adds a momentum term to the update rule. The momentum term helps accelerate gradients vectors in the right directions, thus leading to faster convergence. The basic idea of Gradient Descent with momentum is to calculate the exponentially weighted average of past gradients and use this average to update the weights instead of using only the current gradient. This helps smooth out fluctuations in the gradient and helps prevent oscillations in the optimization process. How works? v_t = beta * v_t-1 + alpha * Grad(w) w = w - v_t Pros: It also works much faster than the algorithm Standard Gradient Descent. Less oscillation around ravines and local optima Cons: It requires tuning of an additional hyperparameter (momentum) which can be time-consuming. Can cause mode
Read more