Simulators, Emulators, and Devices -- Which is right for your Mobile CI Builds?

In the end, the answer to the question we posed in the title will be inextricably linked to the particulars of your operation. However, as MacStadium works with teams facing this very challenge, we have discovered a few insights that may help your team to navigate the ins and outs of this question.

Physical Devices

It is considered best practice to test applications on physical handsets before release. This is largely because of the many nuances inherent in not only the performance of the OS on the phone, but also within the internals of the device itself.

Advantages of Physical Devices for Mobile CI

The advantage of testing on physical devices is bulletproof confidence of an application’s performance on a given device.

Challenges of Physical Devices for Mobile CI

As enticing as this aforementioned certainty is, this is often an elusive target, as there is an enormous number of phones on the market, and maintaining a working set of devices that is representative of that variety is no small feat.

Emulators

Emulators are software that emulates the operating system that would be running on a given handset, and the code that emulates the internals of the device itself. Basically, it emulates the device and the OS.

Advantages of Emulators for Mobile CI

The real advantage of an emulator for mobile CI is that an entire bank of real devices can be emulated with a high degree of confidence and relatively little overhead, as compared to builds executed on physical devices.

Challenges of Emulators for Mobile CI

The challenge inherent in using emulators is that they may not offer an identical emulation of every device your team is targeting.

Additionally, although the overhead for running tests on emulators is significantly less than that it would be on physical devices, there is still significantly greater overhead to run emulator processes as compared to simulator processes, which translates into longer build times – often viewed as the arch-enemy of modern DevOps.  

In the end, we find that most teams who are focused exclusively on iOS CI choose to forego the extended build times that an emulator brings. These teams generally use a simulator instead.

Simulators

Simulators are software that simulates only the OS that would be running on a given device, with no attention paid to the underlying internals of a given device. The majority of teams that use MacStadium run their tests on Apple’s Xcode simulator.

Advantages of Simulators for Mobile CI

The real advantage of simulators for mobile CI is their lightweight nature, and by extension, their relatively fast execution rates.

Challenges of Simulators for Mobile CI

The lightweight nature of simulators comes at a cost, of course. Because simulators don’t concern themselves with replicating the underlying device at all, a good number of potential nuances can be whitewashed away, only to be caught at a later testing stage, or even in a beta release.

Android Emulators on a Mac?

Most MacStadium customers use Xcode’s simulator for their iOS builds, but there is a handful of customers doing some very interesting things with Android emulators running on Mac VMs. To read about one, in particular, check out this Medium post from the iFood team.

TL;DR

Where most of our customers are using a simulator, we find that it all depends on your team. If you’d like to find out more about what our customers are doing (not just with simulators, emulators, or physical devices), check out our customers page, and see how companies like Jamf and Dropbox are using MacStadium. Interested in learning more about what MacStadium can help your organization achieve? Reach out and talk to one of our sales engineers today to see how MacStadium can help.