---
title: "Deep learning: definition and example – Carl-Uwe Würfel"
description: "Deep learning explained: what makes neural networks deep, how they learn, why the breakthrough came after 2010, and when simpler methods work better."
url: https://carl-wuerfel.de/en/glossary/deep-learning
author: Carl-Uwe Würfel
---

1. [Home](https://carl-wuerfel.de/en)
  2. [Knowledge](https://carl-wuerfel.de/en/knowledge)
  3. [Glossary](https://carl-wuerfel.de/en/glossary)
  4. Deep learning

# Deep learning

Deep learning is a branch of machine learning in which neural networks with many layers learn the relevant features on their own from raw data such as images, speech or text.

## On this page

  1. What “deep” means
  2. How a deep network is trained
  3. Why the breakthrough came after 2010
  4. The main architectures
  5. When deep learning fits and when it doesn’t
  6. What deep learning means for a business
  7. Where the term stops working
  8. Frequently asked questions
  9. Sources

The work shifts from training to deployment.

_Figure: A balance using the example of a maker of metal parts that has so far inspected surfaces by eye at the end of the line. On the left, the manual inspection. On the right, the new system: a camera photographs every part, and a pretrained convolutional network, adapted with a few thousand images labeled by inspectors, pulls out parts with scratches or dents; borderline cases still go to a person. Before launch, the system runs alongside manual inspection for several weeks so it is clear which defects it misses._

**Example**

A maker of metal parts has so far inspected surfaces by eye at the end of the line. Now a camera photographs every part, and a pretrained convolutional network, adapted with a few thousand images labeled by inspectors, pulls out parts with scratches or dents. Borderline cases still go to a person. Before launch, the system runs alongside manual inspection for several weeks so it is clear which defects it misses.

In classic [machine learning](https://carl-wuerfel.de/en/glossary/machine-learning), experts often decide up front what a model should pay attention to: for a credit check, say, income, loan term and payment history. Deep learning hands that step to the model. A deep [neural network](https://carl-wuerfel.de/en/glossary/neuronales-netz) receives the raw data, such as the pixels of a photo, and builds its own intermediate concepts layer by layer. In their 2015 review in Nature, Yann LeCun, Yoshua Bengio and Geoffrey Hinton describe deep learning as a method in which models composed of multiple processing layers learn representations of data with multiple levels of abstraction.

Almost everything in AI that has made headlines since 2012 rests on it: image recognition, speech recognition, machine translation and the [large language models](https://carl-wuerfel.de/en/glossary/llm) behind ChatGPT, Claude and Gemini. “Deep” refers to the number of stacked layers, not to deeper understanding.

For executives, the point is mostly to place the term correctly. Few mid-sized companies train their own deep networks; they use pretrained models through software or APIs. What matters then is a different set of questions: what the model is good for, how its output gets checked, what it costs to run and where the data ends up. And for many classic tasks on tabular data, deep learning is not the best choice at all.

## What “deep” means.

A neural network consists of layers of artificial neurons. The first layer takes the input, the last delivers the result, and in between sit so-called hidden layers. A network is called deep when several hidden layers are stacked one after another. There is no fixed cutoff: the standard textbook by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016) notes that there is no consensus on how much depth a model needs to qualify as deep.

Depth serves a purpose. Each layer works on what the previous one extracted and forms somewhat more abstract patterns from it. In image recognition, early layers respond to edges and color transitions, later ones to shapes like wheels or eyes, the last ones to whole objects. Researchers call this representation learning.

This is the practical difference from classic machine learning. There, a person picks the features a model looks at, and the result depends heavily on that choice. Deep learning finds the features itself. That saves manual work on images, speech and text, where good features are nearly impossible to describe by hand. The price is more data, more computing power and models whose inner workings are hard to interpret.

## How a deep network is trained.

Every connection between two neurons has a weight, a number that determines how strongly a signal is passed on. At the start, these weights are random. Training adjusts them in a loop until the network produces the desired results.

First, the network computes a prediction for an example through all its layers. A loss function measures how far that prediction is from the correct result. The error is then passed backward through the network to work out, for every weight, which direction it needs to move. This method is called backpropagation; David Rumelhart, Geoffrey Hinton and Ronald Williams popularized it in Nature in 1986. Finally, gradient descent shifts all weights a small step in the direction that reduces the error. This repeats over many examples, usually in several passes.

As with the rest of machine learning, the real test uses data the network never saw in training. Deep networks have so many weights that they can easily memorize training examples instead of finding general patterns. Remedies for this overfitting include more varied data, dedicated training techniques and a strict split between training, validation and test data.

## Why the breakthrough came after 2010.

The core ideas are old. Warren McCulloch and Walter Pitts described a formal model of a neuron in 1943, and backpropagation has been known since the 1980s. Deep networks still lagged other methods for a long time, for three reasons: too little data, too little computing power and training methods that broke down with many layers.

Around 2010, all three bottlenecks gave way. The internet supplied large labeled image collections such as ImageNet. Graphics processors (GPUs), built for video games, turned out to suit the many identical calculations of training, because they run thousands of them in parallel. And new techniques made deep networks easier to train, such as the ReLU activation function and a method called dropout.

The visible turning point came in 2012. Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton won the ImageNet image recognition challenge with a deep convolutional network trained on two graphics cards. Their top-5 error rate was 15.3 percent; the runner-up came in at 26.2 percent. After that, one research field after another switched to deep learning. In 2017 came the Transformer, the architecture today’s language models are built on.

## The main architectures.

Deep learning is a family of network architectures, each built for a type of data. For talks with vendors, knowing three is enough.

Convolutional neural networks (CNNs) are made for images. They slide small filters across an image and recognize patterns regardless of where they appear. They power visual inspection on production lines, document capture and medical imaging.

Recurrent networks process sequences step by step and carry an intermediate state along, for example with spoken language or sensor data. For years they were the standard for translation and speech recognition.

Transformers, introduced by Google researchers in 2017 in the paper “Attention Is All You Need,” rely on a mechanism called attention: every element of an input can relate directly to every other. That makes them strong on long texts and easy to train in parallel. Large language models, many image models and models for code are built on them today.

For buyers, the architecture matters less than whether it fits the data and whether the model was tested on your own examples.

## When deep learning fits and when it doesn’t.

Deep learning is strongest on unstructured data: photos, scans, audio, free text. Good features are hard to describe by hand there, and data volumes are often large. Tabular data, the kind stored in ERP and CRM systems, is a different story. In 2022, Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux compared methods on 45 medium-sized tabular datasets. Tree-based methods such as gradient boosting beat deep networks there, at lower computational cost.

There are also costs that planning often leaves out. Training a deep network from scratch takes many labeled examples. Training and operation both need computing power. And why a deep network reaches a particular result can only be explained to a limited degree, which matters for decisions about people or money.

That suggests a simple order. First, check whether a fixed rule or a classic method is enough. Then check whether a pretrained model covers the task, adapted with your own examples if needed. Building your own deep network from scratch only pays off for a clearly defined task, large amounts of proprietary data and an advantage that off-the-shelf models cannot offer.

## What deep learning means for a business.

In most companies, deep learning arrives as a finished model: as the language model behind an assistant, as text recognition in invoice processing, as image recognition in an inspection system. The vendor did the training. That is usually sensible: pretrained models bring general knowledge with them and can be adapted to a task with relatively few examples of your own. This is called transfer learning; especially broadly pretrained models are known as foundation models.

The work therefore shifts from training to deployment. Whoever uses such a model decides on data flows, meaning which data goes to which provider and which country, on how results are checked, on running costs per call and on how easily the model can be swapped later.

Good vendors raise these points without being asked. They can say which model they use and why, how it was tested on your examples, how errors surface in production and what happens when the provider changes or retires the model. A proposal that cites deep learning as a seal of quality says nothing about the solution itself.

## Where the term stops working

Deep learning is a learning method, not a product and not a quality tier. It is a branch of machine learning without a sharp boundary; neural networks are the tool, deep learning is learning with many layers. [Generative AI](https://carl-wuerfel.de/en/glossary/generative-ki) and large language models build on it but are only one part of it. An AI label on software does not mean deep learning; often fixed rules or classic statistics sit behind it.

## Frequently asked questions

What is the difference between deep learning and machine learning?

Deep learning is a branch of machine learning that works with multi-layered neural networks. Classic methods often need features a person defines in advance; deep learning learns them from raw data. In return, it needs more data and computing power.

Is ChatGPT deep learning?

Yes. The language models behind ChatGPT are very large deep neural networks built on the Transformer architecture. ChatGPT itself is the application that makes such a model usable, with an interface, rules and additional features.

Why does deep learning need GPUs?

Because training consists of a huge number of identical calculations, mostly matrix multiplications, that can run in parallel. GPUs handle thousands of these steps at once and are much faster at it than conventional processors. Large models also usually run on such chips in production.

How much data does deep learning need?

A network trained from scratch usually needs a very large number of labeled examples. Adapting a pretrained model often takes far fewer, because the model brings its general knowledge along. There is no fixed minimum; what counts is the result on held-out test data.

Is deep learning the same as generative AI?

No. Generative AI, which produces text or images, almost always runs on deep learning today. But deep learning is just as widely used for tasks that generate nothing, such as image recognition, speech recognition or fraud detection.

When should you avoid deep learning?

When the data is tabular and manageable in size, when there are few examples, or when every decision must be fully explainable. In those cases fixed rules or classic methods such as decision trees are usually cheaper and easier to audit.

## Further reading

  * [**AI glossary**](https://carl-wuerfel.de/en/glossary)
  * [**AI automation**](https://carl-wuerfel.de/en/ai-automation)
  * [**Machine learning**](https://carl-wuerfel.de/en/glossary/machine-learning)
  * [**Neural network**](https://carl-wuerfel.de/en/glossary/neuronales-netz)
  * [**LLM: Large Language Model**](https://carl-wuerfel.de/en/glossary/llm)
  * [**Artificial intelligence (AI)** Term](https://carl-wuerfel.de/en/glossary/kuenstliche-intelligenz)
  * [**Generative AI** Term](https://carl-wuerfel.de/en/glossary/generative-ki)
  * [**Foundation model** Term](https://carl-wuerfel.de/en/glossary/foundation-model)

## Sources

  1. [Yann LeCun, Yoshua Bengio, Geoffrey Hinton: Deep learning, Nature 521, 2015](https://doi.org/10.1038/nature14539)
  2. [Ian Goodfellow, Yoshua Bengio, Aaron Courville: Deep Learning, MIT Press, 2016](https://www.deeplearningbook.org/)
  3. [Warren S. McCulloch, Walter Pitts: A logical calculus of the ideas immanent in nervous activity, Bulletin of Mathematical Biophysics, 1943](https://doi.org/10.1007/BF02478259)
  4. [David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams: Learning representations by back-propagating errors, Nature 323, 1986](https://doi.org/10.1038/323533a0)
  5. [Alex Krizhevsky, Ilya Sutskever, Geoffrey Hinton: ImageNet Classification with Deep Convolutional Neural Networks, 2012](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks)
  6. [Ashish Vaswani et al.: Attention Is All You Need, 2017](https://arxiv.org/abs/1706.03762)
  7. [Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux: Why do tree-based models still outperform deep learning on tabular data?, 2022](https://arxiv.org/abs/2207.08815)