Linear Regression from Scratch — The Universal Loop

~14 min · training-loop, gradient-descent, regression

Level 0Level 0

0 XP0/78 lessons0/17 achievements

0/100 XP to next level100 XP to go0% complete

Build the universal four-step pattern

The best way to internalize GradientTape is to build linear regression with no Keras — just raw TF and manual gradient descent. The four-step pattern you write here is the same pattern model.fit runs internally.

Linear regression: find W and b minimizing the mean squared error of y_pred = W*x + b.

The four steps inside every training loop, ever:

Forward pass — compute predictions under the tape so TF records the ops.
Loss — compute the scalar loss, also under the tape.
Backward pass — tape.gradient(loss, [W, b]) applies chain rule automatically.
Update — subtract learning_rate × gradient from each parameter via assign_sub.

Code

Linear regression — Keras 없이·python

import tensorflow as tf
import numpy as np

# Synthetic data: y = 2x + 3 + noise
np.random.seed(42)
X = np.linspace(-1, 1, 100).astype(np.float32)
y_true = 2 * X + 3 + 0.1 * np.random.randn(100).astype(np.float32)

# Initialize parameters as Variables
W = tf.Variable(tf.random.normal([1]))
b = tf.Variable(tf.zeros([1]))
learning_rate = 0.1

for epoch in range(200):
    with tf.GradientTape() as tape:
        y_pred = W * X + b                                   # 1. forward
        loss = tf.reduce_mean(tf.square(y_true - y_pred))    # 2. loss

    dL_dW, dL_db = tape.gradient(loss, [W, b])              # 3. backward

    W.assign_sub(learning_rate * dL_dW)                      # 4. update
    b.assign_sub(learning_rate * dL_db)

    if epoch % 50 == 0:
        print(f"Epoch {epoch:3d}: loss={loss:.4f}, "
              f"W={W.numpy()[0]:.3f}, b={b.numpy()[0]:.3f}")

print(f"\nFinal: W={W.numpy()[0]:.3f} (true 2.0), "
      f"b={b.numpy()[0]:.3f} (true 3.0)")

Exercise

Modify the linear regression code to use tf.keras.optimizers.Adam(0.1) instead of manual assign_sub. Replace the update step with optimizer.apply_gradients(zip(grads, [W, b])) and confirm convergence is similar or faster.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousPersistent Tapes and Higher-Order Gradients Next →Gradient Clipping and stop_gradient

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.