tar + SSH Pipe

~10 min · tar, ssh, pipe, many-small-files

Level 0Pinger

0 XP0/101 lessons0/12 achievements

0/150 XP to next level150 XP to go0% complete

When rsync isn't there or is too slow

Two scenarios make tar | ssh faster than rsync:

Many thousands of tiny files. rsync's per-file metadata overhead becomes the bottleneck. tar streams everything as one continuous archive.
The remote side doesn't have rsync installed (rare these days, but sometimes).

The pattern is a Unix classic: tar creates the archive on stdout, the pipe feeds it to ssh which runs tar on the remote to extract. No intermediate files, no temporary disk space, just a stream.

Code

Push and pull through a tar pipe·bash

# Push: local → remote
tar czf - /path/to/source | ssh host 'tar xzf - -C /destination/'

# Pull: remote → local
ssh host 'tar czf - /path/to/source' | tar xzf - -C /local/dest/

# Add progress with pv (`brew install pv`)
tar cf - /source | pv | ssh host 'tar xf - -C /dest/'

# Skip compression on fast LAN, use a fast cipher to reduce CPU
tar cf - /source | ssh -c aes128-gcm@openssh.com host 'tar xf - -C /dest/'

External links

Exercise

Pick a directory with a lot of small files (a node_modules or a Python venv, copied to a scratch path). Time both approaches: time rsync -az source/ host:/tmp/x/ vs time (tar cf - source | ssh host 'tar xf - -C /tmp/y'). On a Gigabit LAN with 100k+ small files, tar usually wins by 2–5×.