Download this notebook

Neural Network Framework - Exercise: Fully Connected Network

Table of Contents

Introduction

For a better understanding of neural networks, you will start to implement a framework on your own. The given notebook explains some core functions and concepts of the framework, so all of you have the same starting point. Our previous exercises were self-contained and not very modular. You are going to change that. Let us begin with a fully connected network on the now well-known MNIST dataset. The Pipeline will be:

  • Define a model architecture
  • Construct a neural network based on the architecture
  • Define an evaluation criteria
  • Optimize the model (training)

Read the whole notebook carefully to understand how the pipeline works even if there is no specific implementation work required from you.

Requirements

Knowledge

TODO

Python-Modules

# third party
import numpy as np
from deep_teaching_commons.data.fundamentals.mnist import Mnist

Data

We load the MNIST dataset. Have a look at the data structure that is necessary to use feed data into the framework. A batch is a 4d tensor with: (image_i, channel, width, height).

# create mnist loader from deep_teaching_commons
mnist_loader = Mnist(data_dir='data/MNIST')

# load all data, labels are not one-hot-encoded, images are flatten and pixel squashed between [0,1]
train_images, train_labels, test_images, test_labels = mnist_loader.get_all_data(flatten=False, one_hot_enc=False, normalized=True)
print(train_images.shape, train_labels.shape)

# reshape to match general framework architecture 
train_images, test_images = train_images.reshape(60000, 1, 28, 28), test_images.reshape(10000, 1, 28, 28)            
print(train_images.shape, train_labels.shape)

# shuffle training data
shuffle_index = np.random.permutation(60000)
train_images, train_labels = train_images[shuffle_index], train_labels[shuffle_index]

Towards a Neural Network Framework

To create custom models you have to be able to define layers and activation functions in a modular way. Layers and activation functions are therefore modelled as objects. Each object that you want to use has to implement a forward and a backward method that is used later by the NeuralNetwork class. Additionally the self.params attribute is mandatory to meet the specification of the NeuralNetwork class. It is used to store all learnable parameters that you need for the optimization algorithm. Implemented that way you can use the objects as building blocks and stack them up to create a custom model. Be aware of using an activation function after each layer except the last one, cause the softmax function is applied by default during loss calculation of the network output.

After completing this notebook you can move the implemented functions to the script files for further development. The framework consists of the following files:

  • layer.py
  • activation_func.py
  • network.py
  • cost_func
  • optimizer.py
  • utils.py

After processing the notebook, it certainly becomes clear which functionality belongs into which file.

Exercise: Define Layers

The first layers added to the framework are a flatten and a fully-connected layer, which we need to build an architecture for the corresponding fully connected network — sidenote sometimes, depending on the framework, the term dense layer is used instead of fully-connected.

All kind of neural network layers and regularization techniques that can be inserted as layers into a architecture will be implemented in the file layer.py later.

Task:

Implement the methods FullyConnected.forward and FullyConnected.backward.

class Flatten(object):
    ''' Flatten layer used to reshape inputs into vector representation
    
    Layer should be used in the forward pass before a dense layer to 
    transform a given tensor into a vector. 
    '''
    def __init__(self):
        self.params = []

    def forward(self, X):
        ''' Reshapes a n-dim representation into a vector 
            by preserving the number of input rows.
        
        Examples:
            [10000,[1,28,28]] -> [10000,784]
        '''
        self.X_shape = X.shape
        self.out_shape = (self.X_shape[0], -1)    
        out = X.reshape(-1).reshape(self.out_shape)
        return out

    def backward(self, dout):
        ''' Restore dimensions before flattening operation
        '''
        out = dout.reshape(self.X_shape)
        return out, []

class FullyConnected(object):
    ''' Fully connected layer implemtenting linear function hypothesis 
        in the forward pass and its derivation in the backward pass.
    '''
    def __init__(self, in_size, out_size):
        ''' initialize all learning parameters in the layer
        
        Weights will be initialized with modified Xavier initialization.
        Biases will be initialized with zero. 
        '''
        self.W = np.random.randn(in_size, out_size) * np.sqrt(2. / in_size)
        self.b = np.zeros((1, out_size))
        self.params = [self.W, self.b]

    def forward(self, X):
        ''' Linear combiationn of images, weights and bias terms
            
        Args:
            X: Matrix of images (flatten represenation)
    
        Returns:
            out: Sum of X*W+b  
        '''
        self.X = X
        ############################################
        #                   TODO                   #    
        ############################################
        raise NotImplementedError("Your task...")
        # out = 
        ############################################
        #             END OF YOUR CODE             #
        ############################################
        return out

    def backward(self, dout):
        ''' Restore dimensions before flattening operation
            
        Args:
            dout: Derivation of the local out
    
        Returns:
            dX : Derivation with respect to X
            dW : Derivation with respect to W
            db : Derivation with respect to b
        '''
        ############################################
        #                   TODO                   #    
        ############################################
        raise NotImplementedError("Your task...")
        #dX =
        #dW =
        ############################################
        #             END OF YOUR CODE             #
        ############################################        
        db = np.sum(dout, axis=0)
        return dX, [dW, db]

Testing

Once you've connected many types of layers in a network and you notice an error in your training, it can be difficult to track down which layer exactly has a buggy implementation. Since you're implementing each layer in a modular fashion you can also test them individually. So, it's a good practice to write tests for each of your layers at this point already.

There are properties you know should hold true about the input and output of your layer. In the FullyConnected layer, you may want to test:

  • In the forward pass: which shape should the return value have?
  • In the backward pass: which shape should the derivatives dX, dW and db have?
  • In the backward pass: which shape do you expect from the argument dout?

Exercise: Define Activation Function

First, remember that activation functions are non-linearities added to your architecture. As an example the classic ReLU function is implemented here:

$ f ( x ) = \left\{ \begin{array} { l l } { x } & { \text { if } x > 0 } \\ { 0 } & { \text { otherwise } } \end{array} \right. $

The ReLU function matches the current weight initialization in the fully-connected layer. Note that may have to be changed if you implement other activation functions.

Actually, the activation function belongs into the layer.py, for the sake of clarity, however, the functions are put into a separate file activation_func.py.

Task:

Implement the ReLU class.

class ReLU(object):
    ''' Implements activation function rectified linear unit (ReLU) 
    
    ReLU activation function is defined as the positive part of 
    its argument. 
    (paper: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.165.6419&rep=rep1&type=pdf)
    '''
    def __init__(self):
        self.params = []

    def forward(self, X):
        ''' In the forward pass return the identity for x > 0
        
        Safe input for backprop and forward all values that are above 0.
        '''
        self.X = X
        ############################################
        #                   TODO                   #    
        ############################################
        raise NotImplementedError("Your task...")
        #return        
        ############################################
        #             END OF YOUR CODE             #
        ############################################

    def backward(self, dout):
        ''' Derivative of ReLU
        
        Retruns:
            dX: for all x \elem X <= 0 in forward pass 
                return 0 else x
            []: no gradients on ReLU operation
        '''
        dX = dout.copy()
        ############################################
        #                   TODO                   #    
        ############################################
        raise NotImplementedError("Your task...")
        #dX =
        ############################################
        #             END OF YOUR CODE             #
        ############################################
        return dX, []

NeuralNetwork Class

A NeuralNetwork object connects all layers and activation functions of a model architecture using the forward and backward methods of the containing objects. Calling forward on the NeuralNetwork object will pass a given input through the whole computational graph. The backward function calculates the gradients via backpropagation.

It further creates a global list of all parameters in the network during initialization, which is used later in optimization process.

A predict function implements a foward pass with the application of a given score function at the end of the calculation. At the momennt it is suited for the softmax function, taking only the max argument.

class NeuralNetwork(object):
    ''' Creates a neural network from a given layer architecture 
    
    This class is suited for fully-connected network and
    convolutional neural network architectures. It connects 
    the layers and passes the data from one end to another.
    '''
    def __init__(self, layers, score_func=None):
        ''' Setup a global parameter list and initilize a
            score function that is used for predictions.
        
        Args:
            layer: neural network architecture based on layer and activation function objects
            score_func: function that is used as classifier on the output
        '''
        self.layers = layers
        self.params = []
        for layer in self.layers:
            self.params.append(layer.params)
        self.score_func = score_func

    def forward(self, X):
        ''' Pass input X through all layers in the network 
        '''
        for layer in self.layers:
            X = layer.forward(X)
        return X

    def backward(self, dout):
        grads = []
        ''' Backprop through the network and keep a list of the gradients
            from each layer.
        '''
        for layer in reversed(self.layers):
            dout, grad = layer.backward(dout)
            grads.append(grad)
        return grads

    def predict(self, X):
        ''' Run a forward pass and use the score function to classify 
            the output.
        '''
        X = self.forward(X)
        return np.argmax(self.score_func(X), axis=1)

Exercise: Define a Cost Function

Implementations of different cost functions should be placed into cost_func.py. A cost function object defines the criteria your network is evaluating during the optimization process. Further the class contains score functions that can be used as classification criteria for predictions using a certain model. So it is necessary to provide a cost function object to a optimization algorithm for the training process.

Task:

Implement the softmax method.

class CostCriteria(object):
    ''' Implements different types of loss and score functions for neural networks
    
    Todo:
        - Implement init that defines score and loss function 
    '''
    def softmax(X):
        ''' Numeric stable calculation of softmax
        '''
        ############################################
        #                   TODO                   #    
        ############################################
        raise NotImplementedError("Your task...")
        #return =
        ############################################
        #             END OF YOUR CODE             #
        ############################################        
    
    def cross_entropy_softmax(X, y):
        ''' Computes loss and prepares dout for backprop 

        https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/
        '''
        m = y.shape[0]
        p = CostCriteria.softmax(X)
        log_likelihood = -np.log(p[range(m), y])
        loss = np.sum(log_likelihood) / m
        dout = p.copy()
        dout[range(m), y] -= 1
        return loss, dout

Testing

Softmax turns each row (each sample) into a probability distribution over the output classes. So you may want to test

  • the shape of the return value should contain the same number of samples
  • if each row is a valid probability distribution. So all values should of the return value should be [0..1] and each row should sum to 1.

Optimization with SGD

The file optimizer.py contains implementations of optimization algorithms. Your optimizer needs your custom network, data and loss function and some additional hyperparameter as arguments to optimize your model. 

class Optimizer(object):   
    def get_minibatches(X, y, batch_size):
        ''' Decomposes data set into small subsets (batches)
        '''
        m = X.shape[0]
        batches = []
        for i in range(0, m, batch_size):
            X_batch = X[i:i + batch_size, :, :, :]
            y_batch = y[i:i + batch_size, ]
            batches.append((X_batch, y_batch))
        return batches    

    def sgd(network, X_train, y_train, cost_function, batch_size=32, epoch=100, learning_rate=0.001, X_test=None, y_test=None, verbose=None):
        ''' Optimize a given network with stochastical gradient descent
        
        Args:
            X_train: trainings data
            y_train: trainings label (ground truth)
            cost_function: cost function
            batch_size: size of a single batch
            epoch: amount of epochs
            learning_rate: the rate which is going to be multiplied with the gradient
            X_test: trainings data if you want to test your model in each epcoh
            y_test: trainings labels
            verbose: if set it prints out training accuracy and test accuracy
        Returns:
            Model with optimized parameters
        '''
        minibatches = Optimizer.get_minibatches(X_train, y_train, batch_size)
        for i in range(epoch):
            loss = 0
            if verbose:
                print('Epoch',i)
            for X_mini, y_mini in minibatches:
                # calculate loss and derivation of the last layer
                loss, dout = cost_function(network.forward(X_mini), y_mini)
                # Do not train in epoch 0, so we now performance b4 training
                if i > 0:
                    # calculate gradients via backpropagation
                    grads = network.backward(dout)
                    # run vanilla sgd update for all learnable parameters in self.params
                    for param, grad in zip(network.params, reversed(grads)):
                        for i in range(len(grad)):
                            param[i] += - learning_rate * grad[i]
            if verbose:
                train_acc = np.mean(y_train == network.predict(X_train))
                test_acc = np.mean(y_test == network.predict(X_test))                                
                print("Loss = {0} :: Training = {1} :: Test = {2}".format(loss, train_acc, test_acc))
        return network

Put it All Together

Now you have to put all parts together to create and train a fully connected neural network. First, you have to define an individual network architecture by flattening the input and stacking fully-connected layer with activation functions, e.g.:

Input -> Flatten -> Dense -> Activation -> Dense -> Activation -> Dense -> Activation -> Dense

You have to initialize all objects you need to build your custom architecture and put them into a list afterward. Your architecture is then given to a NeuralNetwork object that handles the inter-layer communication during the forward and backward pass. It will also set the evaluation criteria at the end of the network, because of that you end your architecture with a fully-connected layer. The pipeline above is implemented in the following cell.

Finally, you can train the model with an optimization algorithm and a cost function, here stochastic gradient descent and cross-entropy with softmax. That kind of pipeline is similar to the one you would create with a more sophisticated framework like Tensorflow or PyTorch.

# design a three hidden layer architecture with Dense-Layer
# and ReLU as activation function
def fcn_mnist():
    flat = Flatten()
    hidden_01 = FullyConnected(784, 500)
    relu_01 = ReLU()
    hidden_02 = FullyConnected(500, 200)
    relu_02 = ReLU()
    hidden_03 = FullyConnected(200, 100)
    relu_03 = ReLU()
    ouput = FullyConnected(100, 10)
    return [flat, hidden_01, relu_01, hidden_02, relu_02, hidden_03, relu_03, ouput]

# create a neural network on specified architecture with softmax as score function
fcn = NeuralNetwork(fcn_mnist(), score_func=CostCriteria.softmax)

# optimize the network and a softmax loss
fcn = Optimizer.sgd(fcn, train_images, train_labels, CostCriteria.cross_entropy_softmax, batch_size=64, epoch=10, learning_rate=0.01, X_test=test_images, y_test=test_labels, verbose=True)

Exercise: Experiment with the Framework

Here is your last exercise for this notebook:

Now you have a basic idea of how to build a fully-connected neural network with the framework. The next steps are straight forward. Download all script files of the framework from Moodle or the exercise-repository and move the implemented functions into the correct script files:

  • layer.py
  • activation_func.py
  • cost_func

After chose a dataset you like and create a data loader in the script file utils.py. Load your data, build a neural network and try to build a good classifier. Have fun!

Licenses

Notebook License (CC-BY-SA 4.0)

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

Neural Networks - Exercise: Simple MNIST Network
by Benjamin Voigt
is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at https://gitlab.com/deep.TEACHING.

Code License (MIT)

The following license only applies to code cells of the notebook.

Copyright 2018 Benjamin Voigt

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.