Why Continuous Integration
A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build to detect integration errors as quickly as possible.
I have found most developers have a good intellectual understanding for the process of implementing Continuous Integration (CI) in their development practices. However, like Agile, unit testing, and unicorns they view CI as a mythical creature that exists only in the rainbow blogs of Silicon Valley startups.
Part of this gap exists because of a fundamental misunderstanding of the value of CI. CI is more than simply a time saving device meant to eliminate manual steps. CI’s value is actually a logical extension of capital ‘A’ Agile. Good CI allows developers to create software in a very fast build, deploy, test cycle and adjust direction based on observed outcomes. It is the “Empirical Process Control” my friend Larry Johnson regularly reminds me about. It is the ability to gather data early in the development process and change course quickly based on what we have learned.
Continuous delivery means minimizing lead time from idea to production and then feeding back to idea again
~Rolf Andrew Russell
Read on for a deep dive on how the Ticketmaster Resale team integrated Continuous Integration principles into their own development best practices…
Here are the criteria we used in selecting our tool set.
CI tools are typically expert systems not built with self-discovery in mind. Historically they have been write-only, xml based configurations that required an automation geek and bottle of pixie dust to make work. To fully integrate a CI tool into a team’s development process it cannot be an “expert system”. Any member of the team should be able to access its control panel and fairly quickly discern the tools purpose and operation.
Technology changes almost daily. A CI tool needs to keep up with those changes. We had a pretty good deployment tool developed in house when we started down the CI path but its developers, while extremely talented, had other duties that prevented them from regularly adding features.
No matter how good the tools documentation there are always edge cases that have not been considered ahead of time. Community is critical for self-support, bug reporting, and edge case documentation. Additionally, if there is not community support it is a sign the tool is not valuable will soon become obsolete resulting in the need to start over.
The CI system needs to be very clear on what it is doing. In other words, who did what and when. This is especially important in getting the infrastructure team’s buy-in for integrating the tool set into a production environment.
Even the most feature rich tools do not cover all use cases. A CI tools needs the ability to be extended through API’s, modules, or plug-in scripts.
CI tools are often built with specific audiences in mind. The tool needs to support the primary technology stack of those using it. Our team uses almost all Microsoft technologies. Some CI tools integrated with windows environments better than others. (Side Note: To all the Linux guys, I hear you snickering. Have you opened Visual Studio lately? I am not sure how you even develop anymore. My VS install just ate your Vi editor for breakfast. )
Network Security Integration
While not a necessity, network security integration is extremely helpful. Most of us have too many usernames and passwords as it is. Additionally, you do not want to be in the business of managing user credentials. Having direct network integrations saves time in the long run and allows for better accessibility.
High-level System Architecture
- Check-in Code and Tests – Code and tests are written in the development team’s language of choice. In our case that is primarily C# and MSTest.
- Auto-triggered Builds and Tests – Builds are triggered manually, upon on check-in, or with a cron job. At this point unit-tests, code coverage, and other code quality tools can also be run as part of the build processes.
- Packages, Reports, and Notifications Created – Upon successful build and test run a nuget package is created, code coverage and unit test results reported, and notifications sent to subscribed users.
- Deployments Completed – Release packages created either automatically or upon request and then pushed to the appropriate environment. Deployment can be either automatic or manual and is based off of users’ permission levels. Deployments happen over https through a client installed on the server allowing deployments to cross network domain boundaries.
For our build and test automation we chose TeamCity. Yes it is written in Java but we do not hold that against it. TeamCity fits all of the tool selection criteria from above.
TeamCity features strong support for Team Foundation Server version control system, Visual Studio project build configurations, and .Net unit testing. There was a bit of a learning curve on the initial setup but once a build configuration is in place any developer can easily kick off a build, modify build steps, or clone the entire configuration in order to create a brand new build.
Visual Studio Build Step Support
Version Control support
Build Package Publishing
One particular feature we have found incredibly valuable is the ability of TeamCity to automatically create nuget packages and host them on a local nuget server. The value of this becomes apparent when paired with the Octopus Deployment tool.
Other TeamCity Features
- APIs: RESTful API’s for customized automation and reporting
- Language Support: Java, .Net, Ruby, xCode, C++, Python, Node.js, and others. (oddly enough I have found some developers use languages other than c#)
- Version Control Support: Git, Mercurial, Subversion, TFS, etc… with flexible build triggering features.
- Code Quality Tracking: Duplicate code analysis, testing code coverage, static code analysis, etc… using a variety of 3rd party tools
- Unit test management
- Support for multiple build agents running on different servers
- Integrates directly with Octopus Deploy for automated release creation and integration environment deployments
- Email build notifications
- Desktop build notification widget
Octopus Deploy Features
We chose Octopus Deploy for our deployment automation tool. From their website:
“Octopus Deploy is a friendly deployment automation system for .NET developers. Octopus works with your build server to enable reliable, secure, automated releases of ASP.NET applications and Windows Services into test, staging and production environments, whether they are in the cloud or on-premises.”
My Mac friends often tell me how Apple’s product “just work”. Octopus is my product that “just works. Hyperbole maybe, but I love this tool. Every time I say I wish it could do X I go poke around and discover it has already been implemented. Outside of some load balancer integrations there was almost no custom code we needed to implement.
Octopus integrates with the TeamCity builds by through TeamCity’s nuget server. Octopus retrieves the nuget package from the build server, packages them up into numbered releases. These releases are then pushed from environment to environment. This means the bits you tested in your test environment are the same bits you deploy to production.
Deployments to specific machines in an environment are handled by deploy agents (called tentacles) installed on the target server . The Octopus server pushes the release package over https to the waiting agent. In the case of an IIS deployment the agent stores the package in a release numbered directory. Once ready it redirects IIS to this new directory. Rolling back is then simply a matter of pointing the IIS back at the original package.
I have been especially impressed with Octopus Deploy logging. There has rarely been a time when a deployment fails and I cannot immediately determine the cause by looking at the logging interface.
Other Octopus Features
- Work flow management
- Rolling deployments (1 to n servers at a time)
- Custom PowerShell scripts
- Security roles for Deployers, Environment managers, Project Managers, etc…
- Encrypted passwords for configuration transformations
Roadblocks and Hurdles
The biggest roadblock is process not technology. Implementing a CI stack like this requires internal marketing. This type of change involves a fair amount of up front investment in time and money. It can also be potentially threatening to those with entrenched interests. Unless there is general buy-in it will never be adopted.
We started by simply talking about the problem of deployment related bugs and what could be done to fix them. These conversations were non-threatening and simply presented as an opportunity. Different tools and options were offered and give team members a chance to think about the problem and become comfortable implementing some new practices.
Next we built a sample implementation using our most complex deployment pipeline. The ability to show a real life application of the process went a long way toward getting sign-off from the necessary parties. Once they saw how easy it was to create push-button deployments the tools sold themselves.
The next problem was time. Initially, our CI implementation was a side project squeezed between other tasks. In retrospect this was a bad idea. The project lost its initial momentum and nearly got forgotten. We ended up creating the CI install an actual project with a team lead working full-time to ensure it was completed. It turned out that is exactly what we needed. It took a number of weeks of heads down work, regular training sessions, and lots of inter team communication to make it happen. The bottom line: Don’t expect CI to happen on its own.
Finally we found repeated training necessary to help the different concepts sink in. Like any new technology or process there managing a CI system is a learning curve. Developers are busy with their regular jobs and implementing CI is a distraction to that.
Also, there are many moving pieces and it takes time to understand how all the pieces fit together. Rather the one long deep dive training session we found it better to do a series of training sessions over a number of weeks each focusing on a different component of the system. We have also followed up with more in-depth training for those who spend the most time managing deployments.
It has only been a couple of months and I hardly remember what it was like to do deployments manually. In the past deployments were repeatedly delayed because of they were so time consuming. We are now deploying constantly to our development and test environments. It is not unusual to deploy small feature changes several times a day.
We also approach production deployments with much less trepidation. The process is tried and proven. The automation steps tested repeatedly. We know that if there is a problem, rolling back is a simple as selecting the previous release package and redeploying it.
There continue to be challenges and huge areas for improvement. Some of things we plan on implementing next include:
- Integrating DB build script deployment and rollback generation
- Fully-automated integration testing for all of our major applications
- Deeper, auto-generated code metrics including testing code coverage
- More frequent production deployments