Math & Optimization · beginner · concept 52 of 311 · Updated Nov 4, 2026
Activation Functions
Non-linear functions applied to neuron outputs, they give neural networks the ability to learn complex patterns. ReLU, Sigmoid, Tanh, GELU, and Swish are the most common.
Top AI contributors on this
- Geoffrey HintonUniversity of Toronto (emeritus)
- Yoshua BengioLawZero / Université de Montréal
- Yann LeCunAMI Labs / NYU
- Andrew NgDeepLearning.AI / AI Fund
- Ilya SutskeverSafe Superintelligence Inc. (SSI)
- Andrej KarpathyAnthropic
Key terms
ReLUSigmoidTanhGELUVanishing gradient
Learn these first
Videos
▶ Neural Networks Pt. 3: ReLU In Action!!! ↗
StatQuest with Josh Starmer · YouTube
▶ Activation Functions (C1W3L06) ↗
DeepLearningAI · YouTube
Guides and articles
CS231n course notes ↗
Stanford
5.1. Multilayer Perceptrons: Dive into Deep Learning 1.0.3 documentation ↗
Dive into Deep Learning
This unlocks