Bash Metaprogramming

My current gig is doing Java enterprise integration work. You’d think that that would involve writing a lot of Java code. Not so much. It’s mostly dicking around with XML config files to wire together a bunch of different tools and frameworks. I’ve actually written more lines of Bash script than Java code. In fact, I’ve spent a fair amount of time working on scripts, and they’ve more than repaid the effort.

I’m dealing with data that gets passed through layers and layers of components, shuttling to various web services and back. There’s all this dynamic XML configuration and auto-generated Java code. When something gets garbled in translation, it’s great to have a four-line script to send a bare-bones HTTP POST to a web service and see what comes back.

Deploying a new version of our code is this tedious and unforgiving process of ftping the code down from the staging server, unpacking it to a deployment directory, shutting down the web server, backing up the old version, copying the new version into place, and restarting the web server. It’s actually more complicated than that, but I forget all the details - because I don’t have to remember them. It’s all scripted. I just run my script and hit return a bunch.

Computers have great memory and terrible judgement. My scripts remember all the nitpicky details of what I have to do, but they stop and ask me whenever a decision has to be made. We work well together.

I’ve never really thought of myself as a “Bash programmer”. Scripting is just an adjunct to “doing stuff on Linux machines.” But I’ve been doing enough of it lately that I decided to actually treat Bash as a programming language: Learn more about what you can do with it, and try to create re-usable code instead of single-use scripts. It turns out it has a ton of features that I’ve never used. Most surprisingly, it even allows some limited metaprogramming. For example:

$ name="Colin MacDonald"
$ city="Arlington"
$ state="VA"
$ echo $name
Colin MacDonald
$ vars="name city state"
$ for x in $vars ; do echo "$x=${!x}" ; done
name=Colin MacDonald
city=Arlington
state=VA

In the for loop, when x is “city”, ${!x} is $city. Ok, clever, but what do you use it for? Well, it lets me save configuration settings to a file with a generic function that goes something like:

function save_config () {
    for x in $vars ; do
        echo "$x=\"${!x}\"" >> $config_file
    done
}

And gets called like so:

save_config "name city state"

I’ve been able to extend this sort of meta-variable handling to parsing command-line parameters and prompting the user for input, so I now have a reusable library for managing script configuration in a fairly clean way. Now my script can just have:

load_config
process_options "[d]=debug" "[n]=name [c]=city [s]=state" "$@"
confirm_config "[name]=\"Name:\" [city]=\"City:\" [state]=\"State:\""
save_config "name city state"

When you run it, it sets the variables based on the command-line params, then prompts you to confirm the settings and add any missing ones.

$ ./my_script.sh  -n "John Smith"
Name: [John Smith] 
City: [] Arlington
city changed to 'Arlington'
State: [] VA
state changed to 'VA'

Those settings will be saved in the config file

$ cat .my_script.cfg 
name="John Smith"
city="Arlington"
state="VA"

And the next time you run your script, you get

$ ./my_script.sh
Name: [John Smith] 
City: [Arlington] 
State: [VA] 

In my world, these will be things like URLs and SOAP message definitions - things that will be different on each machine, and which I really only want to type in once. So, handy.

If you want to grab this library and a template of how to use it, they’re up on GitHub. If you want to learn more about metaprogramming tricks in Bash, pop the hood and dig around in there. The syntax is a bit strange and can be very finicky, so there’s some hard-won knowledge baked into the library.