GNU Parallel: The Good Parts
This article is basically a rant I wrote to Timo, before he told me to convert it into a blogpost. Here we go!
For years, I've been searching for a CLI tool that …
- runs multiple commands concurrently
- blocks until all the commands have finished
- returns a unified exit code (If all commands pass, it passes; if any of the commands fails, the whole thing fails.)
Due to these requirements, I wasn't able to use &
. (At least not without some more gymnastics to track the processes and calculate the exit code. It didn't seem worth it.)
Every now and again, I would stumble over GNU Parallel. Enticed by the promising name, I would look at the manual but not find a way to do what I want. There are 162 examples and most of them (I haven't actually looked at all of them in detail) do some wild shell gymnastics, like
# Simple network scanner
prips 130.229.16.0/20 | \
parallel --timeout 2 -j0 \
'ping -c 1 {} >/dev/null && echo {}' 2>/dev/null
I did not even cherry pick this so that I could be annoyed. This is literally the second example! I don't need a network scanner, I only want to run three scripts at once! Please!
At this point I often closed the page, since this tool is obviously too advanced and not what I was looking for. Sure, if I ever need to Download Apollo-11 images from NASA using jq or Grep n lines for m regular expressions, I might give it a go, but that's for another day.
Except that this week, I somehow cracked it. I don't know how I got the idea, but I noticed that some of the examples use :::
as a kind of argument separator. It turns out that we can pass the commands itself as arguments:
parallel ::: "pnpm lint" "pnpm check" "pnpm test"
I almost couldn't believe it, but this works! It runs the three commands, it prints their output separately, and it returns a unified exit code. Now that I knew what I was looking for, I still wasn't able to find an example of this invocation in between the 162 examples. Welp! But it works!
Most importantly: It really is faster. I save almost half the time:
time parallel ::: "pnpm lint" "pnpm check" "pnpm test"
68.06s user 8.89s system 442% cpu 17.388 total
time ( pnpm lint; pnpm check; pnpm test; )
27.05s user 5.65s system 115% cpu 28.221 total
If you ever need to parallelize something, I hope this knowledge will come in handy. Good luck!