GNU Parallel: The Good Parts

This article is basically a rant I wrote to Timo, before he told me to convert it into a blogpost. Here we go!

For years, I've been searching for a CLI tool that …

runs multiple commands concurrently
blocks until all the commands have finished
returns a unified exit code (If all commands pass, it passes; if any of the commands fails, the whole thing fails.)

Due to these requirements, I wasn't able to use &. (At least not without some more gymnastics to track the processes and calculate the exit code. It didn't seem worth it.)

Every now and again, I would stumble over GNU Parallel. Enticed by the promising name, I would look at the manual but not find a way to do what I want. There are 162 examples and most of them (I haven't actually looked at all of them in detail) do some wild shell gymnastics, like

# Simple network scanner
prips 130.229.16.0/20 | \
  parallel --timeout 2 -j0 \
    'ping -c 1 {} >/dev/null && echo {}' 2>/dev/null

I did not even cherry pick this so that I could be annoyed. This is literally the second example! I don't need a network scanner, I only want to run three scripts at once! Please!

At this point I often closed the page, since this tool is obviously too advanced and not what I was looking for. Sure, if I ever need to Download Apollo-11 images from NASA using jq or Grep n lines for m regular expressions, I might give it a go, but that's for another day.

Except that this week, I somehow cracked it. I don't know how I got the idea, but I noticed that some of the examples use ::: as a kind of argument separator. It turns out that we can pass the commands itself as arguments:

parallel ::: "pnpm lint" "pnpm check" "pnpm test"

I almost couldn't believe it, but this works! It runs the three commands, it prints their output separately, and it returns a unified exit code. Now that I knew what I was looking for, I still wasn't able to find an example of this invocation in between the 162 examples. Welp! But it works!

Most importantly: It really is faster. I save almost half the time:

time parallel ::: "pnpm lint" "pnpm check" "pnpm test"
  68.06s user 8.89s system 442% cpu 17.388 total

time ( pnpm lint; pnpm check; pnpm test; )
  27.05s user 5.65s system 115% cpu 28.221 total

If you ever need to parallelize something, I hope this knowledge will come in handy. Good luck!