Continuous Integration Services for iPhone Apps in 2018
The open source UI framework I've been working on in Swift (Theodolite) had used the BuddyBuild CI system. I loved how simple and easy to configure BuddyBuild was, and how fully-featured the integration was with GitHub.
So when BuddyBuild was acquired by Apple, and subsequently disabled all access to the platform for free and open source users, I was deeply disappointed, as I imagine others were as well. This thread popped up in one of the projects I contribute to, and it turned out there was quite a lot of interest in the topic.
As I search for another CI provider for Theodolite, I thought I'd write an up-to-date comparison of the current CI offerings for iOS applications. Here's what I'm looking for in a CI service:
- A hosted service. I don't want to set up a Mac Mini or tinker with a local machine.
- Full native support for Xcode builds.
- Support for running tests in addition to basic builds.
- Integration with GitHub repos.
- Reliable and fast.
Don't see your CI provider on the list and want me to include it? Add a link in the comments, and I'll take a look! Fix a bug mentioned here and want me to update the ratings? Shoot me an email, and I'll see what I can do!
Please note that I'm not being paid for any of these reviews. I opted into free trials where possible, or paid full price to test these services. I don't know any of the founders or employees of these companies.
In order to set an even playing field, I set up each of the below services with my Theodolite GitHub repo, and ran the following evaluations:
- Initial setup. How hard was it to set up with the repo?
- Build speed. I made a series of commits that broke the build, and waited for a notification from each provider that the build had broken. Take these measurements with a grain of salt, since queue times are not constant, and fluctuate a lot.
- Broken build behavior. I submitted commits with broken builds to the repo, and checked the error reporting in the CI. I did the same with a failed test.
- Pricing. I went over the pricing matrix of each service.
Travis CI is the grand old man of iOS CI. It's where most people started a few years ago, and it remains a primary workhorse for the iOS community.
I decided not to follow one of the many external how-to's for setting up Travis CI like this fantastic one at objc.io. Instead I just followed the documentation directly from the Travis CI website.
After logging in with GitHub, I was presented with this helpful screen:
So I went over to my Profile, and granted access to my GitHub account. Once there, I was quickly able to authorize access to the TheodoliteGroup organization I set up to own the Theodolite repo.
After that, enabling Travis CI in the repo required a bit of work. I had to add a .travis.yml file to the repo.
That's where things get a little complicated. Where other services nowadays automatically find your Xcode project and workspace files, Travis CI requires an actual config file to be present in the root of your project in order to build, and the documentation for creating that file is pretty minimal.
Luckily, the config file is pretty Spartan. Here's the first file I tried:
language: objective-c xcode_workspace: Theodolite.xcworkspace xcode_scheme: TheodoliteTests
Surprise! Xctool rejected the standard build command that Travis gave it:
ERROR: Unexpected action: build
A few Google searches later, I reached the conclusion that xctool (the recommended and default system for use in Travis CI) is fundamentally broken with their default configuration. Instead, I have to provide a script.
After several searches, and over a dozen test commits, I finally got something working. Here's the yml file I ended up with:
language: swift osx_image: xcode9.3 script: xcodebuild -scheme theodolite build test -destination "platform=iOS Simulator,name=iPhone 5s"
I struggled for several hours to get this little bit of config correct. I kept getting errors about having a valid SWIFT_VERSION... and finally found out that the default for Travis is to build with Xcode 8.3.3. Definitely a facepalm moment.
Anyway, long story short, I finally got it building with hooks from GitHub, but the path to get there was a fiddly, annoying voyage. I spent about three hours in total with the config to get something working over something that appears trivial. This mirrors issues I've had in the past with Travis.
Overall, I give Travis CI setup 2/5 stars: ★★☆☆☆. The ability to configure Travis CI is relatively simple, but the defaults and lack of hand-holding lead to some very frustrating setup.
Surprisingly, Travis CI was consistently the fastest service I tested, with an average reporting time of 2 minutes and 30 seconds after a breaking commit is made to an email notification in my inbox. I know that Travis has had lots of capacity issues in the past, and I personally have run into them, so take these numbers with a grain of salt.
Broken test #1: 2 min 40 s
Broken test #2: 2 min 38 s
Broken build #1: 2 min 22 s
Broken build #2: 2 min 20 s
Overall I give Travis CI build speed 5/5 stars: ★★★★★.
Broken Build Behavior
Travis's build failure dashboard provides information on the commit, the text log, and quick access to the build config. The lack of build artifacts, and color highlighting sets it slightly behind the front of the pack.
Overall I give Travis CI build break behavior 3/5 stars: ★★★☆☆.
Travis CI is free for open source projects, but if you need to use it for a private repo, here's what pricing looks like:
Travis CI's miserable setup process really holds it back. It really feels like it's showing its age, and Travis needs to start looking to their competitors for examples of how to do onboarding better. I give Travis CI 3/5 stars as a CI service: ★★★☆☆
After logging in with GitHub, and authorizing access to the TheodoliteGroup organization, I was able to add the Theodolite project to CircleCI. After creating the project, I was presented with this screen:
So similar to Travis, I have to commit a YAML file to my repo. Also, it appears the default config tries to use fastlane to build and test the app. I've never used fastlane before, so this should be interesting.
Here's the config.yml file I committed to my repo:
version: 2 jobs: build: macos: xcode: "9.2.0" steps: - checkout - run: name: Carthage Bootstrap command: carthage bootstrap - run: name: Build and run tests command: fastlane scan environment: SCAN_DEVICE: iPhone 5s SCAN_SCHEME: theodolite - store_test_results: path: test_output/report.xml - store_artifacts: path: /tmp/test-results destination: scan-test-results - store_artifacts: path: ~/Library/Logs/scan destination: scan-logs
This is all just based on the sample yml file that was presented in the intro docs and modifying it slightly for Theodolite. Notably I switched the iPhone simulator, changed from CocoaPods to Carthage, and updated the Xcode version (note that CircleCI doesn't have support for Xcode 9.3 yet EDIT: Circle CI has informed me I was looking at out of date docs, 9.3 is now supported!). The process of starting my account to a successful build and test took me about 20 minutes.
As a side-note, I had never used fastlane before, but it just worked. I happy to have my xcodebuild scripts all built for me instead of experimenting with them and always forgetting something.
Overall, I give Circle CI setup 4/5 stars: ★★★★☆. The setup and documentation for Circle was simple, easy to configure through the config file, and errors were much easier to diagnose. The presence of a config file ultimately slows down the process compared to the newer entrants.
Coming in third for build speed is Circle CI. With an average of 3 minutes 52 seconds, Circle CI was comfortably in the middle of the pack.
Broken test #1: 3 min 44 s
Broken test #2: 4 min 13 s
Broken build #1: 3 min 49 s
Broken build #2: 3 min 43 s
Overall I give Circle CI build speed 4/5 stars: ★★★★☆.
Broken Build Behavior
CircleCI's broken build dashboard is in the best in class category. It provides information about the commit, access to highlighted logs, timing information, direct access to configuration, and more.
Overall I give CircleCI build break behavior 5/5 stars: ★★★★★.
CircleCI provides free "Seed" level service to open source projects, but it does require that you email them. Here's their current pricing for paid plans:
CircleCI's simple configuration, reasonable performance, good pricing, and excellent bug report handling make this my second-favorite CI service I tested. I give CircleCI 4/5 stars as a CI service: ★★★★☆.
Logging in with my account and granting access to the TheodoliteGroup was easy. I then moved on to adding the app to Nevercode. I got to this screen, and was blocked. The GitHub via OAuth button did nothing in Safari.
Being an avid Safari user, this actually happens a fair amount to me in JS-heavy sites, so I opened Chrome and did the same thing. Unsurprisingly, it worked!
After navigating to the GitHub project I wanted to add, Nevercode then gave me the option of scanning the repo, which I did. This process is a bit longer than the other automatic detecting systems by folks like Bitrise and BuddyBuild, but so long as it works, I don't mind waiting a few extra minutes on setup. The majority of the time appeared to be spent in installing packages from Ruby dependencies used by Nevercode.
When that completed after 5-10 minutes, I was prompted to save the discovered configs, and run the build!
Similar to the setup step, a very large amount of time was spent on installation of Ruby packages:
I'm not sure if these installations are due to the fact that I have a "gh-pages" branch in my repo or not, but at this point, I was not prompted to select any specific builds.
The first build failed because Nevercode had picked my demo app in the repo to run in addition to the main Theodolite project, and my demo app has a compilation issue with Swift 4. I went over to the Settings menu and selected the Theodolite project instead of TheodoliteFeed, however for some reason Nevercode hadn't detected the workspace version of Theodolite, and only had the xcodeproj file. I selected it anyway, since it should build as well.
This configuration worked! I got a successful build after about a half hour since beginning the process.
Overall, I give Nevercode setup 3/5 stars: ★★★☆☆. My experience demonstrated several bugs (edit: Safari support, workspace detection), long build times due to unnecessary code installations as part of the build process, and a bit of confusion about how to fix issues that I found in the build.
However, getting started still didn't require a config file to be committed to my repo, and was minimally invasive. If Nevercode fixes the bugs in their platform, simplifies the startup process, and removes the unnecessary installation steps, there's nothing that will prevent them from moving near the front of the pack.
The performance issues I noted in the setup phase continued to plague Nevercode, and in some ways the comparison isn't really fair, since Nevercode is doing a hell of a lot more work than their competitors by installing huge Ruby packages to test my iOS codebase... Nevertheless, this is the data we have, and Nevercode came in dead last at an average of 6 minutes 55 seconds from failed commit to notification. That being said, that's still serviceable.
Broken test #1: 6 min 49 s
Broken test #2: 9 min 5 s
Broken build #1: 6 min 6 s
Broken build #2: 5 min 39 s
Overall, I give Nevercode build speed 3/5 stars: ★★★☆☆. From my inspection, it appears that Nevercode was kicking off the builds right about at the same time as its competitors, so it really seems like there's some low-hanging fruit here to speed up the builds for repos like mine. Also, if Nevercode doesn't try to install the world in your builds, you'll probably have similar build times to Travis and Circle.
Broken Build Behavior
Nevercode provides the standard build output, in addition to an artifacts tab where you can inspect the output of the build. This gives them a leg up on the basic log-only build failure systems.
Overall I give Nevercode broken build behavior 3/5 stars: ★★★☆☆.
Nevercode does not provide a free open source tier. Here's the pricing for the paid plans:
Nevercode's buggy UI (edit: in Safari), minimal feature set, mediocre performance, and high introductory pricing combine to make this difficult to recommend. I give it 2/5 stars as a CI service: ★★☆☆☆.
I pulled up the Bitrise website, and logged in using GitHub. Their welcome flow for adding a project looks like this:
This interface was amazingly simple, and gave me control of the OAuth process, something that other providers did for me without asking.
Bitrise identified my workspace files without issue and after clicking through the guided setup, it kicked off a build. I was surprised to see the first build succeed right off the bat! However, tests were failing because of the snapshot tests.
The snapshot tests for Theodolite are run on an iPhone 5S simulator. I started doing this back at Facebook to lessen the load of making changes that impact the UI. In order to update the simulator device that the tests are run with, I had to dig into the "workflow" that Bitrise had built for me.
I was pleasantly surprised to find that every step of the build process was represented here as a discrete step with deep configurability. I clicked into the Xcode Test workflow, and quickly found the simulator selection menu that I needed. After changing the value, I saved, and re-ran the build. Success! All in all I think it took me about 10 minutes to get to a working build with Bitrise, and no changes to my repo at all.
Overall, I give Bitrise setup 5/5 stars: ★★★★★. This was the smoothest, simplest configuration experience I've ever had with CI. It really reminded me of how simple BuddyBuild was, but in most ways it was even better, since their workflow system made the build process transparent, and easy to understand. Combine that with the configurability, and well thought-out UI, and we have a winning setup experience.
Bitrise came in second behind Travis CI with an average time of 2 minutes 55 seconds.
Broken test #1: 3 min 18 s
Broken test #2: 3 min 10 s
Broken build #1: 2 min 55 s
Broken build #2: 2 min 24 s
Overall, I give Bitrise build speed 4/5 stars: ★★★★☆.
Broken Build Behavior
Bitrise has an absolutely gorgeous UI for diagnosing build breaks. By default they show a trimmed, colorful log, provide access to the app build, artifacts, and quick access to workflows and build config all from one UI.
Overall I give Bitrise build break behavior 5/5 stars: ★★★★★.
Bitrise provides a free tier for small teams that's perfect for my open source projects. Their paid plans are priced based on how many concurrent builds you need.
Bitrise's focus on mobile, reasonable free tier, well thought-out UI, and killer setup makes it my preferred choice for CI provider. I give it 5/5 stars: ★★★★★.
I logged in with GitHub, and authorized the TheodoliteGroup organization. I was then welcomed with this pretty screen:
I added a new app, and was then asked to install their SDK into my app. Since this project isn't an app, just an open source framework, I went over to the Build area where I could configure the build process.
After selecting the repo, I was immediately presented with the Build configuration wizard. The wizard told me that I had to add a shared scheme (I already have it configured with shared schemes, so this appeared to be a bug), however I was able to click through to the Theodolite.xcodeproj file instead, which it was better able to understand. With this updated, I was able to save and kick off a build!
In the process of building, I quickly hit this error in the build output:
Looks like someone at Microsoft needs to change their API handling to authenticate their requests. I'm not sure if this is connected to the build failing ultimately, but it appears an infrastructural issue that other CI providers didn't run into tanked my first build:
The builds and tests all succeeded so far as I can tell though, so this appears to be the worst kind of error: infra flakiness. I restarted the build to see if this was a one-off issue, but sadly, the same error happened, and despite everything else working, App Center reported my build broken.
Overall, I give App Center setup 2/5 stars: ★★☆☆☆. I can't justify a higher rating because I never got a successful build on the platform, and the errors appeared to be infrastructure-related. However, the platform has a lot going for it, it had the snappiest UI of the services I tried in its setup menus, and appears to be much more than a basic CI service. I think this is one to keep an eye on due to its backer's size and experience (Microsoft).
Update 5/24/2018: On reporting the above errors to MS, here's the response I received: "Unfortunately, we don't support to build framework at this time. Sorry for the inconveniences." So it seems that what I'm trying to do here is outside their feature set. Considering that what I'm doing isn't that rare of a feature in other CI providers, I'm surprised about the lack of support.
Since App Center's build never succeeded (and therefore didn't issue a notification email to me), I instead used only the time it took to build the framework (not including the email, which it didn't send). That time averaged out to 3 minutes, 39 seconds, putting it between BitRise and Nevercode. That's still pretty fast.
Broken test #1: 4 min 20 s
Broken test #2: 4 min 25 s
Broken build #1: 3 min 4 s
Broken build #2: 2 min 48 s
Overall, I give App Center build speed 4/5 stars: ★★★★☆.
Broken Build Behavior
App Center's build error page is pretty bare-bones with just a single text log that you have to scroll through. Simple, yet workable. However, there doesn't appear to be additional artifacts available if you need them to diagnose your build or test issue. For example, when a snapshot test fails, you cannot see the snapshot test artifact that actually failed.
Overall, I give App Center broken build behavior 3/5 stars: ★★★☆☆.
In addition to a generous free tier that covers the needs of most open source projects, App Center lets you add the features you need, as you need them:
App Center is a more fully-featured application dashboard that handles things from analytics to user support. CI is just one of its many services, and it shows. The minimal build break flow, and the bugs I encountered made this product feel more like a beta than a finished product. However, it's fast, and its generous free tier means getting started and testing App Center costs you nothing, and if you like it, you can look at their other services which may help you simplify your deployment and monitoring systems.
Overall I'm rating App Center 3/5 stars as a CI service: ★★★☆☆.
Best in class
Easy setup, excellent error presentation, and a generous free tier make BitRise our preferred CI.
A more traditional configuration experience with great error handling and reasonable pricing makes CircleCI a close second.