Pipes: the concurrency model you already use

Every pipeline stage runs at once, joined by a bounded buffer, and not every stage streams.

Transcript

Log lines flow from awk into sort while awk drops everything but the 500s; sort fills up and emits nothing until the last line arrives, then uniq -c counts 3 /pay and 1 /login.

# 500 errors per path, worst first
awk '$9 == 500 { print $7 }' access.log \
  | sort | uniq -c | sort -rn

On a six line combined format log (three 500s on /pay, one on /login, a 200 with a 500 byte body, a 404) it prints:

   3 /pay
   1 /login

Field 9 is the status, field 7 the path. Grepping for ' 500 ' would also catch that 200, and uniq -c on whole lines counts 1 each, because every timestamp is unique.

The mechanism#

The shell starts every stage at once and joins each stdout to the next stdin with a kernel pipe. sleep 2 | sleep 2 takes 2 seconds, not 4. Independent processes, no shared memory, bytes through a channel: a concurrency model in the strict sense, and Unix has had it since pipes arrived in 1973.

Backpressure is built in#

A pipe is a bounded buffer: 65,536 bytes on this Mac. When it is full, the writer blocks in write until the reader catches up. yes | sleep 2 runs 2 seconds and yes burns 0.00 s of CPU, asleep on a full pipe. Nothing piles up between stages.

Not every stage streams#

awk reads a line, decides, forgets it. sort cannot print its first line before it has read the last, so it holds everything. On a 444 MB log, /usr/bin/time -l measured a peak resident size of 1.6 MB for grep, 1.6 MB for awk '{print $7}' and 2.1 GB for sort. That is why the filter goes first: here sort only sees the 500s. uniq -c only merges neighbors, hence the sort before it.

Failure is quiet#

A pipeline's status is the last stage's. With set -o pipefail it is the rightmost non-zero status: (exit 3) | (exit 5) | true gives 5 in bash 3.2 and zsh 5.9. The cost: grep exits 1 when nothing matches, so a clean log fails the pipeline, and yes | head -1 fails with 141 (SIGPIPE). Under pipefail, decide which non-zero codes mean trouble. awk exits 0 on no match.

Rule of thumb#

Filter early, sort late, and know which stage holds everything. sort is the colleague who will not answer until you have completely finished talking, so let it hear as little as possible.