Deep Learning Workshop at codecentric AG in Solingen

Big Data – a buzz word you can find everywhere these days, from nerdy blogs to scientific research papers and even in the news. But how does Big Data Analysis work, exactly? In order to find that out, I attended the workshop on “Deep Learning with Keras and TensorFlow”.

On a stormy Thursday afternoon, we arrived at the modern and light-flooded codecentric AG headquarters. There, we met performance expert Dieter Dirkes and Data Scientist Dr. Shirin Glander. In the following two days, Shirin gave us a hands-on introduction into the secrets of Deep Learning and helped us to program our first Neural Net. After a short round of introduction of the participants, it became clear that many different areas and domains are interested in Deep Learning: geologists want to classify (satellite) images, energy providers want to analyse time-series, insurers want to predict numbers and I – a humanities major – want to classify text. And codecentric employees were also interested in getting to know the possibilities of Deep Learning, so that a third of the participants were employees from the company itself.

The second day began with a tasty Latte Macchiato and a nice “Good Morning” in the big kitchen of codecentric headquarters. The friendly room, with real moss on the only wall that’s not glass, emanated a very creative air and made the start of the day much easier. For this day, Shirin had a Neuronal Net prepared, that we could use to apply and try out what we’ve learned. We spent a lot of time exploring the effects that changing different parameters had on the performance of the Neural Net. During the practical part, many questions came up, that Shirin answered and discussed with us. We also talked about our own projects and ways to advance them with what we learned in the workshop.

Deep Learning with Artificial Neural Nets

Artificial Neural Nets (ANN) can be used to analyse highly complex data. Deep Learning is a subdomain of machine learning, which is again a subdomain of Artificial Intelligence. ANNs are modelled after the human brain, which has been adapted to learn efficiently and transfer knowledge to new scenarios. Researchers tried to mimic the brains ability to learn by building ANN, which are able to learn from training data and transfer this knowledge to new data.

Already back in 1958, Frank Rosenblatt [1] presented his concept of ANNs, the so called perceptron. The perceptron works with a numeric input and uses weighted connections in a node or neuron; together with an activation function, it calculates the result.

With increasing computational power and increasing amounts of data, multi-layer perceptrons and deep neural nets became more and more prevalent. Alongside the boom in machine learning, more and more deep learning frameworks, libraries and programming interfaces were developed. Today, there is a plethora of easy-to-use and open-source materials for deep learning. The bar to get your feet wet with this complex topic is as low as never before; you need almost no deep mathematical knowledge any more in order to build neural nets.

TensorFlow and Keras

TensorFlow and Keras are such deep learning APIs. TensorFlow has been developed by Google Brain and has been written in Python and C++. Right now, it is one of the most often used open-source libraries for machine learning. Keras builds on top of it and works specifically with neural nets. Same as TensorFlow, it is open-source and has been written in Python. There are several other libraries that can be used as backend with Keras, e.g. Theano and CNTK. TensorFlow is based on the concept of data flow graphs, meaning that every neural net can be seen as a form of mathematical operation in nodes and multi-dimensional data objects, the so called tensors, as edges. With TensorFlow we can use the integrated TensorBoard. TensorBoard is a collection of visualisation tools, that makes it easy to analyse processes, potential errors and optimize neural nets.

Image classification with Neural Nets

Most of the research into neural nets has so far involved image classification. The most famous data set for this task is MNIST [2], which contains 28 x 28 pixel images of hand-written images and their respective labels. The MNIST dataset is optimal for getting started with image classification and neural nets. Classification with neural nets falls into the category of supervised learning because it uses examples to train models. The hand-written digits of the MNIST dataset can be used to learn a classification task, namely which pixel arrangement corresponds to which digit.

The neural net will be fed input data from the training set, which it will pass on to the next layer of neurons. The initial weights were picked at random and every neuron in a layer will calculate an output based on the incoming data and the weights. The output will then be passed on again to following layers. In the final layer, a softmax function calculated the probabilities, with which it sorts the input image to the classes. The prediction of this input will be the class with the highest probability. Because we know the correct class for every image in our training set, we can calculate the difference between prediction and correct class. This is called cross-entropy. A cross-entropy of 0 would mean that the neural net predicted the correct class with 100% certainty. After every cycle of training, we calculate an error rate, the so called loss, which represents the cross-entropy of the entire training set.

During training, we aim to minimize the loss function. In order to achieve this, the weights of the neurons need to be optimized, e.g. by using backpropagation with stochastic gradient descent. The direction in which to change the weights is found with a gradient; we want to “descend” this gradient to find the smallest error rate. The step size with which we traverse the gradient landscape is defined by the learning rate. We can start with big steps to prevent getting stuck in local minima, but with advanced training we want to reduce the learning rate so that we can – ideally – reach the global minimum. With a loss function of 0, we would have achieved the perfect result.

The final evaluation of our neural net’s performance is measured on independent test data. We want to avoid that our net learned the training set too specifically and isn’t able to generalize any more. There are many different parameters that can be adjusted when training a neural net to find the optimal composition for our classification tasks. We try out different combinations and observe the effect they have on the result.

I want to thank Shirin for a great workshop! You made the complex topic of deep learning and neural nets much more accessible to us.

Maria Hartmann

—

Workshop description: https://www.codecentric.de/schulung/deep-learning-mit-keras-und-tensorflow/

[1] Rosenblatt, Frank (1985). The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Web: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.335.-3398&rep=rep1&type=pdf

[2] LeCun, Yann. The MNIST Database of Handwritten Digits. Web: http://yann.lecun.com/-exdb/mnist/

—

Guest author Maria Hartmann studies Digital Humanities at the University of Trier. Through her love of linguistics she came to the classification of semi-structures texts with neural nets – which she now works on for her Master’s thesis. TensorFlow and Keras help her in her studies.