Concurrent Shell Command Queues

Concurrent Shell Command Queues Christian Emmer Nov 6, 2026* · 10 min read How to run shell commands in parallel while limiting their concurrency. I had a use case recently where I wanted to speed up some Bash processing that I was doing, but still keep concurrency constrained. So I wondered: is there a straightforward way to create thread pools in Bash? The answer is: yes, but it takes some work to be semi-resilient and portable. You may ask: why do this in Bash at all? It seems silly to use a language with poor concurrency controls. You'd be right to ask this question, and my answer is: because I have a fondness for Bash , and I wanted the challenge. But you're still probably better off using another language, such as Python . Here is code that creates a command queue that accepts shell commands to execute. You do not need to know all the commands that you want executed up-front, you can submit commands to it over time. bash 1 # Start this shell's command queue. 2 # @param {number} $1 Maximum number of jobs to run at once 3 queue_start () { 4 # Validate that the parallelism is a valid limit for xargs 5 if [[ ! ${1:-} =~ ^[1-9][0-9]*$ ]]; then 6 echo "queue_start: size must be a positive integer, got ' ${1:-} '" >&2 7 return 1 8 fi 9 10 # Close this shell's copy (not the parent's) of an inherited or earlier 11 # command queue's file descriptor 12 if [[ -n ${_queue_fd:-} ]]; then 13 eval "exec ${_queue_fd} >&-" 14 unset _queue_fd _queue_pid 15 fi 16 17 # Find the first unused fd from 3 to 9 (zsh's redirect max) 18 local fd=3 19 while { : <& " ${fd} " ; } 2>/dev/null; do fd=$(( fd + 1 )); done 20 if [[ ${fd} -gt 9 ]]; then 21 echo "queue_start: no free fd from 3 to 9" >&2 22 return 1 23 fi 24 25 # Make a FIFO queue file in a private temporary directory 26 local dir 27 dir =$( mktemp -d " ${TMPDIR:-/tmp} /queue.XXXXXX" ) 28 mkfifo " ${dir} /queue" 29 30 # Find the shell to execute commands (the current shell) 31 local shell 32 shell=$(ps -o comm = -p "$$" ) 33 shell= ${shell#-} 34 35 # Start xargs in the background, reading NUL-separated commands from the 36 # queue file. Job messages are hidden by turning off job control and 37 # discarding the block's stderr. xargs gets the real stderr through a copy on 38 # the queue files's fd, until the file is opened. 39 eval "exec ${fd} >&2" 40 local opts=$- 41 set +m 42 { 43 # shellcheck disable=SC2016 44 xargs -0 -n 1 -P " $1 " " ${shell} " -c ' 45 # Output is captured rather than streamed so that command output stays separate 46 out=$(eval -- "$1" &1); rc=$? 47 [ -z "${out}" ] || printf "%s\n" "${out}" 48 # Coerce non-zero exit codes to 1, because a 255 code would stop xargs 49 [ "${rc}" -eq 0 ] || { printf "exit %d: %s\n" "${rc}" "$1" >&2; exit 1; 50 }' _ < " ${dir} /queue" 2>& " ${fd} " & 51 _queue_pid=$! 52 disown || : 53 } 2>/dev/null 54 if [[ ${opts} == *m* ]]; then set -m; fi 55 56 # Open the queue file for writing, which waits until xargs has opened it for reading 57 # With both ends open, the file can be safely deleted 58 eval "exec ${fd} >\"\${dir}/queue\"" 59 _queue_fd= ${fd} 60 rm -rf " ${dir} " 61 62 # Register an exit handler to terminate this PID's children, ensuring zsh behaves like bash 63 setopt localoptions posixtraps 2>/dev/null || : 64 trap _queue_stop EXIT 65 if [[ $- != *i* ]]; then 66 trap 'exit 129' HUP 67 trap 'exit 130' INT 68 trap 'exit 143' TERM 69 fi 70 } 71 72 # Queue a command on this shell's command queue. 73 # @param {string} $1 Command to run 74 queue_submit () { 75 if [[ -z ${_queue_fd:-} ]]; then 76 echo "queue_submit: no command queue started in this shell" >&2 77 return 1 78 fi 79 # Write the command to the queue file, blocking if it is full 80 # Use a printf process so that if xargs has exited, printf will get killed and not this shell 81 if [[ $- == *i* ]]; then 82 # Write the command in the foreground, because an interactive shell won't 83 # have any TERM/HUP/INT traps that could be delayed by a full queue 84 env printf '%s\0' " $1 " >& " ${_queue_fd} " 85 else 86 # Write the command in the background, so that a full queue doesn't delay any TERM/HUP/INT traps 87 env printf '%s\0' " $1 " >& " ${_queue_fd} " & 88 wait "$!" 89 fi || { 90 echo "queue_submit: couldn't queue the command; the command queue may have stopped" >&2 91 return 1 92 } 93 } 94 95 # Close the queue file so xargs finishes the jobs it has, then wait for it to exit 96 queue_wait () { 97 if [[ -z ${_queue_fd:-} ]]; then 98 echo "queue_wait: no command queue started in this shell" >&2 99 return 1 100 fi 101 eval "exec ${_queue_fd} >&-" 102 while kill -0 " ${_queue_pid} " 2>/dev/null; do sleep 0.1; done 103 unset _queue_fd _queue_pid 104 } 105 106 # Stop every process below the given PID. 107 # @param {number=} $1 PID whose descendants to stop, defaulting to this shell's PID 108 _queue_stop () { 109 # Default to this shell's PID, found from a child's parent PID because $$ is 110 # the parent's PID in a subshell 111 local parent= ${1:-$(exec sh -c 'echo " ${PPID} "')} 112 local pid 113 for pid in $( exec pgrep -P " ${parent} " ); do 114 # Pause the process so it can't start any new children 115 kill -STOP " ${pid} " 2>/dev/null || continue 116 # Terminate the process' descendants, then terminate the process 117 _queue_stop " ${pid} " 118 kill -TERM " ${pid} " 2>/dev/null || : 119 # Resume the process so it can respond to the termination 120 kill -CONT " ${pid} " 2>/dev/null || : 121 done 122 } 123 124 # Stop all background processes on exit, because they ignore Ctrl-C and don't receive TERM 125 if [[ $- != *i* ]]; then 126 trap _queue_stop EXIT 127 trap 'exit 129' HUP 128 trap 'exit 130' INT 129 trap 'exit 143' TERM 130 fi queue_start initializes the queue (managed by mkfifo ) and starts the executor (concurrency managed by xargs -P ), queue_submit submits commands to the queue, and queue_wait waits for the queue to drain before cleaning up. Calling queue_wait is optional, command processing will start as soon as you queue it with queue_submit . You can copy and paste this code into a script, or it's also safe to put in your dotfiles, as it only sets some of the exit traps for interactive shells. It works in Bash and Zsh, and it has as much defensive programming as I know how to add, but I'm confident it has failure modes that are difficult to predict or avoid. Example usage My original use case had to do with making a large number of web requests, so here's an example using Postman Echo : ● ● ● shell $ queue_start 3 $ for n in 1 2 3 4 5; do queue_submit "set -o pipefail; \ curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \ -d '{\"request\": ${n}}' \ https://postman-echo.com/post | \ jq -c .json" done $ queue_wait {"request":1} {"request":2} {"request":3} {"request":4} {"request":5} A more deterministic scenario demonstrating that concurrency is respected, only starting command three after command two has completed: ● ● ● shell $ queue_start 2 $ queue_submit 'sleep 1; echo first' $ queue_submit 'echo second' $ queue_submit 'echo third' $ queue_wait second third first Command queues are inherited by subshells: ● ● ● shell $ queue_start 2 $ ( queue_submit 'echo "submitted from a subshell"' ) $ queue_submit 'echo "submitted from the main shell"' $ queue_wait submitted from a subshell submitted from the main shell Different subshells can have their own command queue: ● ● ● shell $ ( queue_start 2 for n in 1 2 3 4; d