MacStadium Blog

All Blog Posts

Selecting a Scalable CI Pipeline for macOS

Scalable CI IMG

Software companies grow up fast, and that growth fuels the need for infrastructure that can support a product at scale. As a result, enterprise level CI generally comes on the heels of other milestones. This is natural – particularly in the notoriously tricky world of Mac-based CI. Not to mention the fact that local Mac mini "farms," which seem to be the default stepping off point for macOS and iOS CI, can grow to be quite large and still be very functional.

But at some point, most companies grow beyond their rudimentary build setup. They suffer major headaches and productivity loss. They make adjustments, but hit an internal, operational ceiling. Then, probably a single engineer is tasked with fixing it. At that point, said engineer pokes around online for a viable solution.

As a case in point, MacStadium hosted a panel discussion at AltConf 2019 (during Apple's annual WWDC) that focused on CI best practices. Top DevOps engineers from the likes of Pandora, Aspyr Media, and PSPDFKit were kind enough to share their insights on the matter.

Over the course of the discussion, one thing became abundantly clear: developing a viable CI system is nothing short of an organic process – often an afterthought upon which an entire company’s profitability can hinge. That is, at some point, CI emerges as being massively important to a company’s bottom line, but by the time that happens, a wide variety of factors will likely already be in play.

If that "coming of age" story sounds familiar, you're in the right place, and MacStadium is here to help.

In the time that we've spent working with all manor of software companies, a variety of factors combine to shape the best path forward for a given company in this position. Understanding what those factors are, and where your company falls on a continuum made up of these factors, will ease the pain of making this transition.  

Project Scale:

The scale of the project itself will almost certainly influence your CI system selections. In the most direct sense, the size of your codebase and frequency with which you kick off builds will determine your needs. But the beauty of moving up to an enterprise level system is that you have the freedom to make selections that do more than simply cover the bases. This is an opportunity to improve the work lives of colleagues. But in order to do so, it will help to think about team fit as much as you think about the technical integrations of the various components in your new CI pipeline on paper.

Real Cost of CI Delays:

Ultimately, you've been tasked with finding the greatest value that you can within the bounds of your current situation. It’s an all too common theme with young, fast-moving tech companies -- “We didn’t realize how much it was costing us, because we just didn’t have the resources to track it.” – Confucius

While a given company's internal variables are far too great for us to pin this down exactly, the following nearly always apply to the total cost of CI delays:

  • Idle developers
  • A break in focus for a developer who is mid-problem
  • Delay to delivery of the product
  • Average build time
  • Number of builds in a given time frame

Whatever figure you come to after considering the above, less the cost of the new CI system and the team that is required to manage it, represents your total potential savings.


Existing Infrastructure:

To retool or not to retool? That is the question. The answer may lie in your existing infrastructure. For example, Chef and Puppet are both Ruby-based languages, but with a different focus. Chef is designed with VMs as a primary use case, while Puppet deploys better to legacy infrastructure. They cover the same functionality, having a central server ensure that subordinate machines are up-to-date with the latest patches while not crashing the network, but the choice between these two will probably be driven by the existing hardware.


Existing Codebase:

Although CI is theoretically language agnostic, there are certainly patterns related to CI tool selection and a project's existing codebase. We see these first-hand as we aid engineers in settling on the best fit for their organization. To continue the example above, Ansible has had a massive adoption spike over Chef and Puppet in recent years by teams without Ruby experience. The YAML approach makes it easy to learn and adopt, even though most agree it is not quite as good as Chef or Puppet — the sacrifice in features is worth the reduced technical debt.


Team Culture:

Smaller teams may favor self-managed, open-source solutions to a point, and then need to retool when personnel limitations are reached. For example, over the course of our panel discussion with proven DevOps superstars, Jenkins was rightly described as being both "not scalable past a certain point" and "scalable to a massive size." These ostensibly conflicting perceptions of the same tool are really the product of differing team cultures, talent bases, use cases, and budgets. CI is meant to be a liberating force for development teams, so a good fit in terms of team culture both in the current moment, and hopefully well into the future, is essential.

Further, some teams value a pure open-source product, and won't consider solutions such as Buildkite or Bamboo, fearing vendor lock. For these teams, the investment in getting a Jenkins-type solution scalable for their use-case is time well spent. Others view the net gain in time-to-market that a paid service offers, being scalable out of the box, to be well worth the cost up front.

That said, for those teams that opt for an open-source solution, you certainly won't be in uncharted waters. Prime examples of this:

  • Microsoft maintains an extensive library of Chef scripts for updating macOS images
  • Codebase is a driver of automation tool selection
  • Dropbox can be used for hosting compiled builds
  • Automating builds is possible with Slack bot
  • Community opinions on built-in tool (such as Spotlight, which ought to be fully or partially disabled)


TLDR Summary:

Team culture is a major piece of the puzzle. So is pinning down the real cost of CI delays and malfunctions. Still though, there is no single "right" answer. And ultimately, MacStadium is ready to help with the journey, as teaming up with a datacenter with scalable infrastructure is the first logical step to a mature, enterprise-grade Apple CI system.

Return to Blog Home