Regression Metrics: MAE, RMSE, R²

Metrics encode a cost opinion

The same predictions can rank differently under different metrics. A metric is not a neutral scoreboard; it states which error sizes and directions deserve more weight. Draw the real error-cost curve before model comparison so the optimization target reflects the decision.

MAE stays in the original unit

Mean absolute error averages the absolute distance from truth. It is easy to explain as “three days wrong on average” and gives twice the penalty to twice the error. It is comparatively resistant to a few extreme failures and fits problems where cost grows roughly linearly.

RMSE emphasizes large misses

Root mean squared error squares errors before averaging, so large misses dominate. Use it when one severe failure costs much more than several small ones. Report MAE and RMSE together: a large gap tells you that a small tail of examples is controlling the result.

R² is a relative baseline comparison

R² measures improvement in squared error over predicting the training target mean. One is perfect, zero matches that mean baseline, and values can be negative. It is unitless but does not directly express business cost and should not be treated as universally comparable across unrelated datasets.

Symmetric vs asymmetric costs

Underpredicting inventory loses sales while overpredicting creates storage cost. If those costs differ, symmetric MAE or RMSE cannot select the operating point. Use quantile regression or a custom loss that implements the documented cost table.

The MAPE warning

Mean absolute percentage error divides by the true value. It is undefined at zero, explodes near zero, and behaves awkwardly for negatives. If a percentage is required, consider SMAPE or weighted MAPE and always pair it with absolute error in the original unit.

Do not hide groups in an average

Report errors by target range, time, and important user group with the sample count. Include median and high-percentile absolute error when tail risk matters. A good overall MAE can still conceal unacceptable failure for the cases the product most needs to serve.

Code

All three metrics in one block·python

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np

preds = model.predict(X_test)
mae = mean_absolute_error(y_test, preds)
rmse = np.sqrt(mean_squared_error(y_test, preds))
r2 = r2_score(y_test, preds)
print(f"MAE={mae:.3f}  RMSE={rmse:.3f}  R²={r2:.3f}")

Quantile regression for asymmetric costs·python

from sklearn.linear_model import QuantileRegressor

# Predict the 90th percentile so we under-stock less often
model = QuantileRegressor(quantile=0.9, alpha=0.1).fit(X_train, y_train)