CI/CD Beyond Jenkins

So Why Not Jenkins?

The venerable Jenkins is the oldest CI/CD solution in this list, having its roots in the Hudson project of the early '00s. Despite its age, Jenkins is still incredibly well adopted for two main reasons: incredible flexibility and a large community with a large array of demonstrated deployments. If it hasn't been done with Jenkins, it probably can't be done.

However, deploying Jenkins on your own comes with a set of issues familiar to anyone who works with open source software: the cost of maintaining and extending Jenkins can exceed the cost of paying for a SaaS platform or 3rd-party support for a CI/CD service hosted on your private infrastructure.

Pros:

• Flexible: If you have enough time/talent, you can install Jenkins on practically any infrastructure and do a wide variety of tasks with it. Plus, no licensing fees!
• Community: A nearly two-decade history means Jenkins has a huge community of users creating resources and plugins to help your implementation. In most surveys, Jenkins is still the most popular CI/CD solution by market share.

Cons:

• DIY Installation: Your time is money. Unless someone on your team is a Jenkins expert, installing and configuring Jenkins could take weeks of developer time and may require an outside consultant if you're not DevOps savvy.
• DIY Support: You'll need to support your Jenkins installation or hire a 3rd party consultant to do so. Expect to dedicate a couple weeks of a full-time team member's time each year installing updates, troubleshooting issues, and extending your install.
• Infrastructure Costs: Remember to factor infrastructure costs in your final evaluation.

TravisCI and CircleCI: When CI/CD Got SaaS-y

Popular Software-as-a-service (SaaS) CI/CD arguably started with TravisCI, which became incredibly popular for offering free CI services to open source projects via TravisCI.org in the early 2010s. By the mid-2010s, a number of services began competing with TravisCI. CircleCI rose above many competitors by offering usage-based pricing, superior performance, and a host of other iterative improvements. While there are many salient differences between TravisCI and CircleCI, they have many similarities as part of this "second generation" of CI/CD providers.

Pros:

• Give Me That SaaS: Support, infrastructure management, maintenance, installation...they're all covered either by TravisCI's self-service SaaS platform or through an enterprise contract.
• Simple: Setup a manifest file following documentation or the legions of great online examples. Add a Github/GitLab webhook. Boom. You have CI/CD running on every commit.
• Good Documentation/Community: Both TravisCI and CircleCI have been around for years. They have extensive documentation and community resources.

Cons:

• Enterprise Deployment Options: While both TravisCI and CircleCI have options for deploying to your private cloud, your options may be limited depending on what their enterprise service teams are willing to do for you.
• No Self Deployment: Forget about deploying either service yourself; even with open source components, these are basically proprietary services.

Github Actions and GitLab CI/CD: The New Breed

Hosted version control providers like Github, GitLab, and Bitbucket are at the heart of modern development workflows. Virtually every developer checks code into one of these services each week, and if you're lucky enough to already have a CI/CD provider like Travis or CircleCI, you've probably already integrated them to run your tests on each pull/merge request created. So why not skip the middleman and have your VCS provider run your CI/CD tasks for you?

We've primarily used Github Actions and believe that it is a frontrunning innovator compared to GitLab CI/CD and Bitbucket Pipelines. However, many of its advantages (and drawbacks) are shared, so you can use our summary as a "rule of thumb" when considering all three services.

Pros:

• Direct Integration in Your VCS: Integrating a modern CI/CD service with your VCS provider isn't difficult, but it is nice having the two deeply connected.
• Community-sourced Actions [Unique to Github Actions]: This is one of our favorite features of Github Actions. Instead of having to write your own tasks, you can search an extensive community-sourced list of "actions" to get a jump-start on configuring the CI/CD process for your app. A wonderful way to get started quickly and get new ideas for the possibilities of CI/CD; this feature should only become more useful as the library of actions grows.
• Enterprise Deployment Options [Unique to GitLab]: GitLab has made support for private cloud/on-premise hosting a core part of its business model. Unfortunately Github and Bitbucket Pipelines don't offer this same option.
• Pricing: All three providers combine a base level of task runner time with most of their plans. If you have a team-level account with one of these VCS providers, chances are you can try out their CI/CD service today.

Cons:

• Enterprise Deployment/No Self Deployment: With the important exception of GitLab, neither Github or Bitbucket will let you deploy their CI/CD tooling in a private cloud or on premise. Bitbucket Data Center allows you to host their VCS service on your own infrastructure, but Pipelines isn't compatible with that service, so you'd have to use your own CI/CD provider like Jenkins or Bamboo.

Note that the ability to deploy GitLab in its entirety on the infrastructure of your choosing is what makes it a preferred solution for our customers running hybrid or private clouds. In this case you won't pay for task runner time, but you will pay the same per-seat fees.

Blow Down The House of Straw

Over a decade of innovation and competition has left an embarrassment of riches for developers looking for a CI/CD provider. So, if you're the big bad wolf looking for your team's best CI/CD option, how to choose?

The two most important factors in selecting a CI/CD provider are ease of setup and build performance.

To quantify ease of setup, ask yourself: how many builds does it take me to get a working configuration running for a simple project on the service? This is particularly important if your team manages fleets of varied applications, as is common for government agencies, higher education institutions, and software development agencies.

"Build performance" is a simpler metric: how long does it take for my tasks to run on this provider? However, it is strikingly difficult to compare CI/CD providers a priori on this scale. Build times are highly dependent on your particular application and the tasks you're running. For example, we've reaped huge performance benefits by moving to GitHub Actions for some complex projects, but have read test reports from other organizations that find it to be the slowest provider when testing a simple sample NodeJS project.

The best way to choose a provider is to try setting up a project. Disqualify the providers that don't meet hard requirements you may have; for example, if you're being forced to self-deploy on your company's private cloud, GitHub Actions is probably out. Then start evaluating the services that seem promising based on features you like, pricing, or another non-performance factor. A proof-of-concept deployment should be achievable on a self-service platform within a day; if it takes longer to set up a service, that may be an immediate disqualifier!

Remember that your organization is unique. Just because GitHub Actions is the newest, most exciting thing doesn't make it the perfect fit for a team operating in a highly regulated environment. Just because Jenkins is "free" doesn't mean you won't waste hundreds of hours configuring and maintaining it.

Use the chart below as a summary of our assessments and good luck hunting!