xargs -P: parallelism in one flag

xargs reads items and runs commands in parallel with -P, turning a serial loop into a worker pool.

Transcript

Ten jobs queue on stdin and are dealt to four workers the moment each one frees up, so the long job runs alongside nine short ones and the whole batch ends at three seconds.

time (printf '%s\n' 3 1 1 1 1 1 1 1 1 1 \
  | xargs -n 1 -P 4 sleep)

Ten jobs, twelve seconds of sleep in total. On macOS this finishes in about 3 seconds (3.03 to 3.05 s over three runs).

Why 3 and not 5#

Xargs does not run in rounds. With -n 1 it builds a command from one item and forks while fewer than -P children are running. When all slots are busy it waits, and the moment any child exits, the next item starts. Worker one takes sleep 3; the other three chew through the nine one second jobs meanwhile. Batches of four, (3,1,1,1), (1,1,1,1), (1,1), would take 5 seconds. Stdin is the queue, the slots are the pool.

Forget -n and nothing runs in parallel#

Xargs packs as many items as fit into one command line. Without -n 1 it runs sleep 3 1 1 1 1 1 1 1 1 1 once, macOS sleep adds its arguments, and you wait 12 seconds with -P 4 idle.

The boundary#

• Output order is completion order. Children share your stdout, so items 3 1 2 with -P 3 and sh -c 'sleep $0; echo $0' print 1, 2, 3. If order matters, write one file per item, or sort afterwards.
• Names with spaces: find . -print0 | xargs -0 splits on NUL, the one byte a path cannot contain.
• Failures: on macOS, if any command fails, xargs exits 1. A command that exits 255 makes xargs stop launching ("aborting"). Exit codes differ across platforms, so check your man page.
• -P 0 means as many processes as possible. -P caps concurrency, it does not create capacity: eight jobs fighting over one disk or one API rate limit still wait, just somewhere you cannot see.

Rule of thumb#

Set -P to the number of things that can truly run at once (cores for CPU work, the server's limit for calls) and cap the items per command: -n 1, -L 1 or -I, one line per command. A for loop with & and wait has no cap at all: that is not a worker pool, it is a fork bomb with good intentions.