Best Practices for managing BASH Scripts
--
Let’s face it, even with all the changes in how we run systems at scale, shell scripts are still a go-to tool in the DevOps & Site Reliability Engineers toolbelt. The simplicity of BASH has kept it relevant, but even though we rely on these shell scripts daily, they are often poorly managed.
Prevalent anti-practices such as being copied manually from server to server, edited live on a single server to change the behavior slightly for a one-off problem, resulting in that one instance of the script is different from all others. What’s worse is that a tiny change breaks the rest of the script, essentially leaving a ticking timebomb on the server.
Why do we let ourselves manage these scripts so poorly if we rely so heavily on them? This article proposes to use the same code management best practices we use for production applications.
Use Source Control
The first practice we should adopt is to manage the code for our shell scripts via Source Control. The most popular source control manager these days is Git. We can host our Git repository in multiple ways; most use a platform such as GitLab or GitHub.
These platforms bring many features that are useful for our needs.
Code Reviews
An essential feature is code reviews, Specifically, code reviews on Pull Requests (a.k.a Merge Requests). When we start using source control for our shell scripts, we don’t want just to put them in a single branch and let anyone push updates directly.
Outside of tracking who made the edits, that practice provides little value. Instead, we should take a stance of “no code is merged without a pull request.”
Pull requests offer the ability to review and approve changes going into scripts. They can also be the trigger for automated processes that validate the incoming code (more on this later).
Since shell scripts tend to be pretty small, this review and approval process can be very effective. Even more effective than the same process with much larger code bases.
Releases
Most source control platforms also offer the ability to create releases. A release is a tag or branch that preserves the code at a specific commit. Releases can also contain the project installation packaging at a particular commit.
A simple tar file might suffice for shell scripts as a packaging medium. If you have many scripts to install or configuration files that need specific placement and privileges, it may be better to create an Operating System specific package; using a packaging format such as RPM or Deb.
Creating a release and packaging for shell scripts makes it easier to deploy and track the installed version of a script. It can also help identify when someone modifies the script locally.
Use a formatter
One of the things I’ve enjoyed about writing applications in Go is the tool gofmt. This tool will format Go code using standard formatting. For Go developers, there is no argument about tabs vs. spaces, whether curly brackets should be “cuddled” or not.
Code either is formatted with gofmt, or it is not. If it is not, it is not ready to merge.
This philosophy is a practice that I recommend for any programming language, and luckily, there has been an effort to create such a tool for shell scripts called shfmt.
To install shfmt execute the go install command below.
$ go install mvdan.cc/sh/v3/cmd/shfmt@latest
Once installed, we can run shfmt against our shell scripts directly. Which can take our unformatted script like the below example and clean it up.
As we can see, shfmt made a few modifications around indentation and how to wrap sub-shell commands.
The hardest part of this process is getting a team to agree on tabs vs. spaces; and how many.
Use a linter
The great thing about a formatter is that it helps make sure regardless of who wrote the code. It all looks the same. But formatters mainly check if code follows best practices; formatters don’t detect bugs. That’s where the BASH linter ShellCheck comes into the picture.
Linters exist for almost every language; their goal is to ensure code quality. A linter will check for formatting issues, known anti-practices, security issues, and bugs. ShellCheck does this for shell scripts.
Installing ShellCheck is relatively simple, as there is an installer for most Operating Systems. Doing most of my coding on Mac, I installed it with HomeBrew.
$ brew install shellcheck
Once installed, we can run it against our simple script from earlier.
Having a linter for shell scripts is invaluable, as let’s face it. Across an entire Ops/SRE team, how many are shell scripting experts? How many know just enough to get the job done?
A linter is a great way to check for common mistakes. Mistakes that all levels can make.
Use Continuous Integration
Having shfmt and shellcheck installed locally are great for ensuring the scripts you develop are well-formatted and follow best practices. But how do you make sure a whole team uses these tools? How can you ensure all new code has run through these tools?
The answer is to make it part of your pull request process.
For most applications and packages these days, it is common to set up continuous integration (a.k.a build steps) for your repository. They will generally run linters, execute unit tests, and maybe some functional tests via BDD frameworks. This process ensures that any new code to the application is genuinely working and production-ready.
We can and should apply the same practice to shell scripts. At least for Linting and Formatting.
For those using GitHub, there is even a GitHub Action already built that runs both shfmt and shellcheck. All you need to do is enable it for your repository.
If you are not using GitHub Actions, check out the Docker Container pipelinecomponents/shellcheck, as it can easily integrate into other build systems.
Use Configuration Management
We no longer deploy applications across the farm by copying them from server to server.
So why not use the same tools we use to deploy our applications to our shell scripts. Whether Ansible, Chef, Puppet, or even Efs2, config management tools should perform software deployment, including shell scripts.
Summary
Every Operations focused team I’ve been on has invariably ended up with a collection of management scripts. Some in BASH, some in Python, some in Go, or whatever language made sense at the time.
I preferred creating a single Git repository and collecting all management scripts in one place. I would also make a single installable package such as an RPM.
Anytime an engineer added a new script, we tested it, packaged it, and deployed it across the farm.
While this did add some process to making modifications to scripts, adopting these practices was worth it overall. Not only could you rely on the existence of these scripts, you knew they worked the same every time.
