Getting Schooled on Deep Learning

This image was produced by a neural network called the Deep Dream Generator

Despite their great usefulness, deep learning neural nets have been unexplained in terms of how they do what they do.

New discoveries are being made, however, that shed light inside the black box of these networks:

New Theory Cracks Open the Black Box of Deep Neural Networks

Oct 2017, WIRED

In one case, the researchers used small networks that could be trained to label input data with a 1 or 0 (think “dog” or “no dog”) and gave their 282 neural connections random initial strengths. They then tracked what happened as the networks engaged in deep learning with 3,000 sample input data sets.

-WIRED

This is an example of an experiment using a network of only couple hundred nodes. All the nodes have been tagged and watched to see how they evolve over iterations of the network. In other words the network is given a task, let's say to recognize face in a picture, and it attempts that task over and over.

The special thing about neural networks is that they learn how to do this task better with every iteration. When our brains do this we call it trial-and-error. We learn how to do things by trying over and over again, and hopefully we get better. These networks try over and over and eventually get better at what they're supposed to be doing.

What's happening while they try has been unknown, or we could say that it still is unknown. But experiments like this are helping us to learn more.

This is a good moment to recall that olfaction, or to be more specific – the olfactory bulb – is a biological neural network. The olfactory bulb is the brain of the nose; it is the nexus at which molecules in the air are translated into electrical signals that the brain can recognize as a smell.

The brain is a very large neural network. The olfactory bulb, on the other hand, is a very good model of this neural network phenomenon. It can be teased apart and separated from the rest of the brain very easily because it functions as its own brain.

Before we had a brain – the human part, the cortex, the one that helps to read this text – we had only a nose and the nose-brain. It can be said, in the spirit of metaphor of course, that the olfactory system was the first cortex, it was built on top of the limbic system, and it still connects to the limbic system in its own separate way.

Smell works differently than all the other senses; it has a direct line to the limbic system, which is the command center of the brain. It has a direct line to the White House, in other words...

So as we discover what neural networks are really doing in their hyperconnected webs, let us remember that the olfactory system did it first.

Post Script

Deep Dream wiki