Monitoring Pipelines That Run at Night

The first rule: alert on outcomes, not internals

Bad monitoring pages on-call when CPU spikes. Good monitoring pages on-call when data is wrong or late. The two are very different. CPU is an internal symptom; freshness and correctness are the SLAs that consumers actually care about.

The four signals every pipeline should emit

Freshness. When did this table last update? Compare against the SLA ("by 9 AM daily"). Alert if older than the threshold.
Volume. How many rows landed this run? Alert if outside the trailing-7-day band (e.g. ±3σ).
Schema. Did the columns change shape? Validation failures should page; soft drift should warn.
Distribution. Did the values shift? Mean / median / null-rate / cardinality compared to last week.

The escalation ladder

Not every signal is a page. Tune your alerting so:

Page on broken contracts: schema changes, data missing past SLA, validation hard-failures.
Slack on warning signals: row counts outside band, distribution drift.
Dashboard on everything: every run's metrics, plotted over time.

Code

A simple freshness/volume monitor — runs after every pipeline·python

import pandas as pd
from datetime import datetime, timedelta

def check_table_health(table_path: str, sla_hours: int = 24, volume_band: tuple = (0.7, 1.3)) -> list[str]:
    '''Return a list of human-readable problems. Empty list = healthy.'''
    df = pd.read_parquet(table_path)
    issues = []

    # Freshness — assumes a max(ingested_at) column
    last_update = df['ingested_at'].max()
    age = datetime.utcnow() - pd.Timestamp(last_update).to_pydatetime()
    if age > timedelta(hours=sla_hours):
        issues.append(f'stale: last update {last_update} ({age.total_seconds() / 3600:.1f}h ago)')

    # Volume — compare today to yesterday
    today = df.loc[df['ingested_at'].dt.date == datetime.utcnow().date()]
    yesterday = df.loc[df['ingested_at'].dt.date == (datetime.utcnow() - timedelta(days=1)).date()]
    if len(yesterday) > 0:
        ratio = len(today) / len(yesterday)
        if ratio < volume_band[0] or ratio > volume_band[1]:
            issues.append(f'volume out of band: today/yesterday = {ratio:.2f}')

    return issues

Exercise

For each of the four signals (freshness, volume, schema, distribution) on one of your tables, write down: (a) the SLA, (b) where you'd emit the metric, (c) the alert threshold, (d) whether it pages or Slacks. Most teams haven't done this exercise — it's where production-grade monitoring actually starts.