Managed concurrency in the bash shell

Thu, 23 May 2013 07:14:00 +0000

Concurrency is all the rage these days. I started to wonder about using it in my command shell (which is regrettably still bash) because so much of the infrastructure for job control is already in place. I’ve been using xargs -P for a while, but I wondered about doing it all in the shell.

“conc”

I came up with a super-simple function to do some experiments:

export CONC_MAX=2
    
conc() {
	local procs=(`jobs -p`)
	local proc_count=${#procs[*]}
    
	# Block until there is an open slot
	if ((proc_count >= CONC_MAX)); then
		wait
	fi
    
	# Start our task
	(eval "$@") &
}
export -f conc

To use it, just throw the word ‘conc’ in front of your commands. It will only use up to CONC_MAX processes. I matched that to the number of cores I have. It is especially useful in loops. For example, while this does some things:

Bash on galvanist

Managed concurrency in the bash shell

“conc”