C.W.K.
Stream
Lesson 05 of 10 · published

Pipes: The Unix Superpower

~12 min · pipe, composition, philosophy

Level 0Window Tourist
0 XP0/95 lessons0/14 achievements
0/100 XP to next level100 XP to go0% complete

The character that changed computing

The pipe character | says "connect the stdout of the command on the left to the stdin of the command on the right." Two small programs become a bigger program at the speed of typing. Doug McIlroy introduced it in 1973 and Bell Labs management reportedly applauded — they knew Unix had just become unstoppable.

The mental model

Each pipe stage is a process. They run in parallel, with the kernel buffering data between them. cmd1 | cmd2 | cmd3 launches three processes; each one reads as it can and produces as it goes. There is no intermediate file — everything is in-memory streams.

Reading pipes left-to-right

A pipe is a recipe in order: "do A, feed result to B, feed result to C." ps aux | grep python | grep -v grep | wc -l reads as: "list every process, keep only python ones, drop the grep itself, count the lines." Once you can write English in pipe form, you've internalized the model.

Pipe exit codes

By default a pipeline's exit code is the last command's. cat missing | wc -l exits 0 because wc succeeded. set -o pipefail changes the rule: any failed stage fails the whole pipeline. Always set this in scripts (we cover the full set -euo pipefail pattern in scripting).

When pipes are the wrong tool

Pipes stream byte by byte. Tools that need the whole input upfront — sort, jq with certain filters, anything random access — buffer it before producing output. That's fine for typical sizes but for multi-GB streams, prefer named pipes (FIFOs), or process substitution for parallel branches.

Code

Classic pipeline patterns·bash
# Top 10 most common shell commands
history | awk '{print $2}' | sort | uniq -c | sort -rn | head
# All running python processes (excluding grep)
ps aux | grep python | grep -v grep
# Word count for a project
find . -name '*.py' | xargs wc -l | tail -1
pipefail saves you·bash
set -o pipefail
cat missing.txt | wc -l   # exits 1 (cat failed) instead of 0
echo "$?"

External links

Exercise

Run history | awk '{print $2}' | sort | uniq -c | sort -rn | head. These are your most-used commands. Then ps aux | wc -l to count your processes. Then set -o pipefail; false | true; echo $? — see exit 1 instead of 0.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.