Architectures · advanced · concept 66 of 176

🏗️Knowledge Distillation

Training a smaller 'student' model to mimic a larger 'teacher' model's behavior. The student learns from the teacher's soft outputs (probabilities), capturing knowledge that's lost with hard labels alone.