The simple example of Theano and Lasagne super power

I mentioned in my initial post “Deep Learning Frameworks Overview” that my choice of Deep Learning library is (at least for now)  Theano and Lasagne combination. However, I did not use, in all my post, some most important word: the experiment. So, let’s assume that you have some idea and want to test it quickly. For example, what if we add to a standard CNN (omitted maxpooling for a clarity):

some extra “convolutional branch”, that is concatenated with last but one layer:

This experiment is really easy to do in (based on Theano) Lasagne. I have just added a build_modified_cnn  method to a mnist example (bolded text refers to my “convolutional branch”, rest is the same as a standard build_cnn method):

def build_modified_cnn(input_var=None):
    l_in = lasagne.layers.InputLayer(shape=(None, 1, 28, 28),
        input_var=input_var)

    l_conv1 = lasagne.layers.Conv2DLayer(
        l_in, num_filters=32, filter_size=(5, 5),
        nonlinearity=lasagne.nonlinearities.rectify,
        W=lasagne.init.GlorotUniform())

    l_conv1A = lasagne.layers.Conv2DLayer(
        l_in, num_filters=32, filter_size=(10, 10),
        nonlinearity=lasagne.nonlinearities.rectify,
        W=lasagne.init.GlorotUniform())

    l_maxpool1A = lasagne.layers.MaxPool2DLayer(l_conv1A, 
        pool_size=(5, 5))

    l_dense1A = lasagne.layers.FlattenLayer(l_maxpool1A)

    l_maxpool1 = lasagne.layers.MaxPool2DLayer(l_conv1, 
        pool_size=(2, 2))

    l_conv2 = lasagne.layers.Conv2DLayer(
        l_maxpool1, num_filters=32, filter_size=(5, 5),
        nonlinearity=lasagne.nonlinearities.rectify)
    
    l_maxpool2 = lasagne.layers.MaxPool2DLayer(l_conv2, 
        pool_size=(2, 2))

    l_dense1 = lasagne.layers.DenseLayer(
        lasagne.layers.dropout(l_maxpool2, p=.5),
        num_units=256,
        nonlinearity=lasagne.nonlinearities.rectify)

    l_concat = lasagne.layers.ConcatLayer([l_dense1, 
        l_dense1A])

    l_dense2 = lasagne.layers.DenseLayer(
        lasagne.layers.dropout(l_concat, p=.5),
        num_units=10,
        nonlinearity=lasagne.nonlinearities.softmax)

    return l_dense2

We see here some standard layers such as Conv2DLayer and MaxPool2DLayer and less standard however self-explanatory ones: FlattenLayer and ConcatLayer. Some results of the test on mnist dataset:

Type	Epoch	Accuracy	Time
Standard CNN	1	56.75 %	6.767s
Standard CNN	30	96.76 %	6.923s
Modified CNN	1	72.51 %	9.552s
Modified CNN	30	96.03 %	9.592s

Not this time. My modification made a significant progress, if we look only at first epoch. However, in general, it learns slower than a standard one. But to see it, I needed only a few minutes of coding – and this is a true power of Theano and Theano-based libs!

 

Advertisement