Deep learning is impacting everything from healthcare to transportation to manufacturing, and more. Companies are turning to deep learning to solve hard problems, like speech recognition, object recognition, and machine translation.
One of the most impressive achievements this year was AlphaGo beating the best Go player in the world. With the victory, Go joins checkers, chess, othello, and Jeopardy as games machines have defeated human at.
While beating someone at a board game might not seem useful on the surface, this is a huge deal. Before the victory, Go was written off as a candidate for a competent AI. Due in part to the amount of human intuition necessary to play the game. The victory makes an entire class of problems once considered intractable ripe for solving.
While it might seem like this technology is still years away, we are beginning to see commercial use. Such is the case with self-driving cars. Companies like Google, Tesla, and Uber are already testing autonomous cars on the streets.
What Is Deep Learning?
The best way to think of this relationship is to visualize them as concentric circles:
At the outer most ring you have artificial intelligence (using computers to reason). One layer inside of that is machine learning. With artificial neural networks and deep learning at the center.
Broadly speaking, deep learning is a more approachable name for an artificial neural network. The “deep” in deep learning refers to the depth of the network. An artificial neural network can be very shallow.
Neural networks are inspired by the structure of the cerebral cortex. At the basic level is the perceptron, the mathematical representation of a biological neuron. Like in the cerebral cortex, there can be several layers of interconnected perceptrons.
The first layer is the input layer. Each node in this layer takes an input, and then passes its output as the input to each node in the next layer. There are generally no connections between nodes in the same layer and the last layer produces the outputs.
We call the middle part the hidden layer. These neurons have no connection to the outside (e.g. input or output) and are only activated by nodes in the previous layer.
Think of deep learning as the technique for learning in neural networks that utilizes multiple layers of abstraction to solve pattern recognition problems. In the 1980s, most neural networks were a single layer due to the cost of computation and availability of data.
Machine learning is considered a branch or approach of Artificial intelligence, whereas deep learning is a specialized type of machine learning.
Machine learning involves computer intelligence that doesn’t know the answers up front. Instead, the program will run against training data, verify the success of its attempts, and modify its approach accordingly. Machine learning typical requires a sophisticated education, spanning software engineering and computer science to statistical methods and linear algebra.
In supervised learning, a machine learning algorithm uses a labeled dataset to infer the desired outcome. This takes a lot of data and time, since the data needs to be labeled by hand. Supervised learning is great for classification and regression problems.
For example, let’s say that we were running a company and want to determine the effect of bonuses on employee retention. If we had historical data – i.e. employee bonus amount and tenure – we could use supervised machine learning.
With unsupervised learning, there aren’t any predefined or corresponding answers. The goal is to figure out the hidden patterns in the data. It’s usually used for clustering and associative tasks, like grouping customers by behavior. Amazon’s “customers who also bought…” recommendations are a type of associative task.
While supervised learning can be useful, we often have to resort to unsupervised learning. Deep learning has proven to be an effective unsupervised learning technique.
Why is Deep Learning Important?
Computers have long had techniques for recognizing features inside of images. The results weren’t always great. Computer vision has been a main beneficiary of deep learning. Computer vision using deep learning now rivals humans on many image recognition tasks.
Facebook has had great success with identifying faces in photographs by using deep learning. It’s not just a marginal improvement, but a game changer: “Asked whether two unfamiliar photos of faces show the same person, a human being will get it right 97.53 percent of the time. New software developed by researchers at Facebook can score 97.25 percent on the same challenge, regardless of variations in lighting or whether the person in the picture is directly facing the camera.”
Speech recognition is a another area that’s felt deep learning’s impact. Spoken languages are so vast and ambiguous. Baidu – one of the leading search engines of China – has developed a voice recognition system that is faster and more accurate than humans at producing text on a mobile phone. In both English and Mandarin.
What is particularly fascinating, is that generalizing the two languages didn’t require much additional design effort: “Historically, people viewed Chinese and English as two vastly different languages, and so there was a need to design very different features,” Andrew Ng says, chief scientist at Baidu. “The learning algorithms are now so general that you can just learn.”
Google is now using deep learning to manage the energy at the company’s data centers. They’ve cut their energy needs for cooling by 40%. That translates to about a 15% improvement in power usage efficiency for the company and hundreds of millions of dollars in savings.
Deep Learning Microservices
Here’s a quick overview of some deep learning use cases and microservices.
Illustration Tagger. An implementation of Illustration2Vec, this microservice can tag an image with the safe, questionable, or explicit rating, the copyright, and general category tag to understand what’s in the image. DeepFilter is a style transfer service for applying artistic filters to images.
The age classifier uses face detection to determine the age of a person in a photo. The Places 365 Classifier uses a pre-trained CNN and based on Places: An Image Database for Deep Scene Understanding B. Zhou, et al., 2016 to identify particular locations in images, such as a courtyard, drugstore, hotel room, glacier, mountain, etc. Lastly, there is InceptionNet, a direct implementation of Google’s InceptionNet using TensorFlow. It takes an image (such as a car), and returns the top 5 classes the model predicts are relevant to the image.
Open Source Deep Learning Frameworks
Deep learnings is made accessible by a number of open source projects. Some of the most popular technologies include, but are not limited to, Deeplearning4j (DL4j), Theano, Torch, TensorFlow, and Caffe. The deciding factors on which one to use are the tech stack they target, and if they are low-level, academic, or application focused. Here’s an overview of each:
- Integrates with Hadoop and Spark
- Very popular in Academia
- Fairly low level
- Interfaced with via Python and Numpy
- Lua based
- In house versions used by Facebook and Twitter
- Contains pretrained models
- Google written successor to Theano
- Interfaced with via Python and Numpy
- Highly parallel
- Can be somewhat slow for certain problem sets
- Not general purpose. Focuses on machine-vision problems
- Implemented in C++ and is very fast
- Not easily extensible
- Has a Python interface
Deep Learning Online Courses
Google and Udacity have collaborated on a free online deep learning course, part of Udacity’s Machine Learning Engineer Nanodegree. This program is geared towards experienced software developers, who want to develop a specialty in machine learning, and some of its subspecialties like machine learning.
Another option is the very popular Andrew Ng course on machine learning, hosted by Coursera and Stanford.
- Machine Learning – Stanford by Andrew Ng in Coursera (2010-2014)
- Machine Learning – Caltech by Yaser Abu-Mostafa (2012-2014)
- Machine Learning – Carnegie Mellon by Tom Mitchell (Spring 2011)
- Neural Networks for Machine Learning by Geoffrey Hinton in Coursera (2012)
- Neural networks class by Hugo Larochelle from Université de Sherbrooke (2013)
Deep Learning Books
While many deep learning courses require a rigorous educational background to get started, this isn’t the case for the book Grokking Deep Learning. In their own words “If you passed high school math and can hack around in Python, I want to teach you Deep Learning.”
Another popular book is the appropriately named Deep Learning Book. This is a great resource for bottom up as it covers of all the required math for deep learning.
- Deep Learning by Yoshua Bengio, Ian Goodfellow and Aaron Courville (05/07/2015)
- Neural Networks and Deep Learning by Michael Nielsen (Dec 2014)
- Deep Learning by Microsoft Research (2013)
- Deep Learning Tutorial by LISA lab, University of Montreal (Jan 6 2015)
- neuraltalk by Andrej Karpathy : numpy-based RNN/LSTM implementation
- An introduction to genetic algorithms
- Artificial Intelligence: A Modern Approach
- Deep Learning in Neural Networks: An Overview
Videos and Lectures
Deep Learning Simplified is a great video series on Youtube. Watch the first video here:
This videos breaks down deep learning into everyday words and concepts.
- How To Create A Mind By Ray Kurzweil
- Deep Learning, Self-Taught Learning and Unsupervised Feature Learning By Andrew Ng
- Recent Developments in Deep Learning By Geoff Hinton
- The Unreasonable Effectiveness of Deep Learning by Yann LeCun
- Deep Learning of Representations by Yoshua bengio
- Principles of Hierarchical Temporal Memory by Jeff Hawkins
- Machine Learning Discussion Group – Deep Learning w/ Stanford AI Lab by Adam Coates
Top GitHub Projects
A curated list of awesome Deep Learning tutorials, projects and communities. A list of popular GitHub projects related to deep learning ranked by stars. Last Update: 2016.08.09.
|TensorFlow||29622||Computation using data flow graphs for scalable machine learning.|
|Caffe||11799||Caffe: a fast open framework for deep learning.|
|Neural Style||10148||Torch implementation of neural style algorithm.|
|Deep Dream||9042||Deep Dream.|
|Keras||7502||Deep Learning library for Python. Convnets, recurrent neural networks, and more. Runs on Theano and TensorFlow.|
|Roc AlphaGo||7170||An independent, student-led replication of DeepMind’s 2016 Nature publication, “Mastering the game of Go with deep neural networks and tree search” (Nature 529, 484-489, 28 Jan 2016).|
|TensorFlow Models||6671||Models built with TensorFlow|
|Neural Doodle||6275||Turn your two-bit doodles into fine artworks with deep neural networks, generate seamless textures from photos, transfer style from one image to another, perform example-based upscaling, but wait… there’s more! (An implementation of Semantic Style Transfer.)|
|CNTK||5957||Computational Network Toolkit (CNTK).|
|TensorFlow Examples||5872||TensorFlow tutorials and code examples for beginners.|
|Torch||5133||Torch7, Deep Learning Library.|
Deep Learning on Algorithmia:
- AWS AMI for Training Style Transfer Models
- Deep-Style.io Demo: Style Transfer Using Deep Learning
- Getting Started With Style Transfer
- Neural Style Transfer from the Command Line
- Lessons Learned from Deploying Deep Learning at Scale
- Why Deep Learning Matters and What’s Next for Artificial Intelligence