Just about every organization that does in-house software development will inevitably reach the point where they need a build server. Hopefully prior to that point you’ve already managed to either create or start the foundation for a well thought out build system. Regardless, at some point the development team will need a continuous integration (CI) system to automate, especially when multiple developers are touching the same code and components. Just remember you don’t need a build machine or CI server to have a quality build system.
While the particular CI system you choose is important, that is not the focus for today’s article. Instead, we’re going to lay out some best practices you should be following for the build server you use for hosting your CI system.
Minimize Environmental Dependencies
This is really a concern for your build system, but it’s worth reiterating here. Ideally, your build system should not require that anything special be installed on the machine where it runs be it your build server or any developer workstation. My minimizing environmental dependencies you can greatly simplify the build out process for your build servers and even developer workstations. This makes operations’ lives easier when setting up new or additional build servers.
One key technique towards achieving this separation is making sure that all dependencies required to build your applications from source code to deployable binary artifacts live in your source control system and are versioned right alongside the source code itself. This includes your build system as well. Then at build time your CI system simply pulls down the source code from source control along with all required dependencies and fires off your build system. No muss, no fuss. This also makes it easier to ensure that the process the CI system uses to build your software is the same process any developer would use.
Restrict Access
This can be a touchy subject, but I’ll come right out and say it: developers should not have access to build servers beyond read-only access for logs, artifacts, etc. Your operations staff and/or any Devopelers should be the only ones allowed to interactively login to any build servers or be able to change anything on any build servers. This implies that operations staff should have a fairly deep understanding of build systems, CI systems, etc. as well.
This may be too broad of a generalization, but most developers are good at getting things to work. That’s what makes them good developers. Unfortunately when it comes to system administration style tasks, they tend to be really bad at making documentation a priority. And as we’ll see in a bit, documentation around build servers is critical.
By restricting access to build servers, you can guarantee you have an independent, theoretically clean environment for building your applications from source code. This helps ensure that builds are repeatable and have not been tampered with. You can even take your process a step further and only release binaries to production that were produced by your CI system on your build server, which should be one of your ultimate goals. By doing so, you can guarantee all production source code was checked in to your source control system.
It’s worth noting that when we say “restrict access” we’re talking about access to the actual host that is running your CI system. Access to the CI system’s own configuration should be handled separately and open to developers in a self service fashion as much as possible.
Document Everything!
From the moment you begin setting up your CI server, you must begin religiously documenting it. Record every detail of what you installed, what version you used, how you configured it, any tweaks you made, any directories you created, etc. Anything you change beyond the base OS installation you have must be recorded in a well known document and disseminated amongst your development and operations teams.
Ideally your build system helped minimize all of the environmental dependencies and this document should be fairly lightweight. The goal of this documentation is to ensure that your organization can reliably reproduce any of its applications from source code to deployable binary artifacts until the end of time. Be sure to validate the documentation from time-to-time as well by spinning up new/replacement build servers and making sure they still build your source code as expected.
If you can automate your build server creation process, even better. Just make sure it’s a well defined process first!
Plan For Growth
As you define your process and get more and more projects/applications into your CI system, you’re going to begin to push the limits of your CI system’s underlying host. Treat it like a “tier 2” production service: monitor and track its CPU usage, memory usage, and especially its disk usage. Disk usage doesn’t just include disk space, it also includes things like disk queue lengths.
A busy CI system can easily chew through disk spindles on your build server. It’s doing a lot of repetitive random access tasks: pull down lots of little source files from source control to the local disk, compile those files, execute tests, then eventually wipe out that build’s working directory. A typical CI system in an organization with multiple developers committing source code changes multiple times a day will trigger dozens if not hundreds of CI builds. Work with your sysadmins to come up with an optimal disk solution which will typically involve multiple disk spindles setup in some sort of RAID configuration. Or shoot for SSDs if you can!
Considering how I/O intensive CI systems can be, we’ve had less than desirable results using virtual machines as build servers, but your mileage may vary. Either way, as your build server’s usage grows, think about expanding to using multiple build servers as part of a single build farm. Many CI systems support this concept of build farms with multiple “build agents”.
So in conclusion, some key build server best practices are:
- Minimize Environmental Dependencies
- Restrict Access
- Document Everything!
- Plan For Growth
With a solid build server running your CI system, you’ll have happy developers and fast, smooth release builds.
