Incident Response

What "compromised" looks like

Most home-fleet compromises are caught by accident — an unexplained process at the top of top, a strange entry in authorized_keys, a 4 AM login from an IP you've never been to. The skill is recognizing the pattern early. The list below is what to scan for; the response section is what to do once you suspect something is wrong.

Red flags to scan for

Unknown keys in authorized_keys — keys you don't recognize, especially with comments referencing usernames or hosts you don't operate.
Logins at unusual times or from unusual IPs — 3 AM from a country you've never been to.
Unexpected processes — crypto miners, reverse shells, daemons you didn't install. top + ps aux + lsof -i.
Modified system files — /etc/ssh/, crontabs, /etc/passwd, /etc/sudoers.
New user accounts — accounts you don't recognize in /etc/passwd.
Outbound connections — established sessions to IPs that don't make sense for your usage.

Code

Quick triage commands·bash

# 1. authorized_keys audit on every fleet machine
ssh-keygen -l -f ~/.ssh/authorized_keys

# 2. Recent logins
last -50
lastb -20      # failed (Linux, sudo)
w

# 3. Unusual processes
ps -ef | grep -v "$(whoami)\|root" | head -30
ps aux --sort=-%cpu | head
lsof -i -P -n | grep ESTABLISHED

# 4. Crontab inventory
crontab -l
sudo ls -la /etc/cron.* 2>/dev/null
sudo find /var/spool/cron 2>/dev/null

# 5. New users
cut -d: -f1,3 /etc/passwd | grep -v 'nologin\|false'

# 6. Suspicious files modified recently
sudo find /etc -mtime -7 -type f
sudo find /usr/local/bin -mtime -30 -type f

# 7. Outbound connections summary
netstat -an | awk '/ESTABLISHED/ {print $5}' | sort | uniq -c | sort -rn

If compromise is confirmed·bash

# 1. ISOLATE — disconnect from the network
sudo ifconfig en0 down                  # macOS
sudo systemctl stop NetworkManager      # Linux (or `ifconfig eth0 down`)

# 2. Snapshot for forensics — disk image, log copies
# (out of scope for this lesson — but do it before any recovery)

# 3. Rotate ALL keys — not just on this machine
#    Assume agent forwarding may have spread the compromise

# 4. Audit authorized_keys on every server in the fleet
for host in $(cat ~/.fleet/all.txt); do
    echo "=== $host ==="
    ssh "$host" 'cat ~/.ssh/authorized_keys'
done

# 5. Review logs for the suspected timeframe
sudo grep sshd /var/log/auth.log | grep 'Accepted'

# 6. If you can't determine scope — wipe and rebuild
#    Better than "clean" with hidden persistence

Exercise

Today, while everything is fine, run the full triage block on every fleet machine. Bookmark the output. This is your baseline — what "normal" looks like for your fleet. The next time something feels off, diff against this baseline. Anomaly detection without a baseline is just guessing; with one, it's tractable.

What "compromised" looks like

Red flags to scan for

Code

External links

Exercise

Progress

Comments 0