~8 min · advanced
큰 teacher model 의 *softmax 분포* 를 student model 이 모방하게 학습 — ground truth label 만 보다 더 풍부한 신호. student 가 작아도 teacher 의 *모르는 것까지* (어떤 클래스끼리 헷갈렸는지 — dark knowledge) 배움.
구현: teacher.predict 로 soft label 받고, student 의 loss 를 (CE with hard label) + (KL divergence with soft label) 로. temperature 도 한 hyperparameter — 클수록 soft 분포 더 평탄.
class <span class="dc">Distiller</span>(keras.Model):
def __init__(self, student, teacher, temperature=3.0, alpha=0.1):
super().__init__()
self.student = student
self.teacher = teacher
self.temperature = temperature
self.alpha = alpha
def train_step(self, data):
x, y = data
# Teacher predictions (soft labels)
teacher_pred = self.teacher(x, training=False)
# Student predictions
student_pred = self.student(x, training=True)
# Distillation loss (soft) + student loss (hard)
loss = (
self.alpha * hard_loss(y, student_pred) +
(1 - self.alpha) * soft_loss(teacher_pred, student_pred)
)
...아직 댓글이 없어요. 첫 댓글을 남겨보세요.