AWS Device farm integration on iOS

iOS Jun 1, 2020

At Halodoc, we are constantly working hard to simplify healthcare services for our millions of customers in Indonesia. Since day-one, Innovation through Automation has always been our key driver and a fundamental development principle in our Software Development process. This has helped our teams to not only conserve their precious bandwidth and also become a lot more efficient in their delivery.

Our app is built for both the Android and iOS platforms and ensuring cross-platform compatibility while providing a high-quality app experience to our users is a pivotal goal for our mobile teams. We achieve this goal through rigorous app testing and make sure it runs successfully, across both these platforms, on a wide range of devices available in the market. We are currently leveraging AWS Device Farm which enables our developers and quality assurance (QA) engineers to test the performance of their websites and mobile apps on a significant number of real devices.

Why are Device Farm important for testing?

Any digital offering that can be viewed on a browser, needs to be compatible with all modern browsers, as well all the devices (in major circulation) for the mobile platforms they were built for. Technological products are driven by the intention to provide novelty, convenience, and speed to their customers. For any web-app or app to provide a truly universal experience, they must be seamlessly functional on whatever device/OS/browser combination their customer prefers. Having access to the latest hardware, operating systems, and platforms allows the testing process to be more streamlined, less expensive, and more innovative. In order to meet this objective, device availability becomes an imperative requirement. and organisations spend several hundreds (if not thousands) of dollars in procuring and maintaining these physical devices which are used for testing. This necessitates the need for a cloud-based device testing environment such as AWS Device Farm.

What is Device Farm?

AWS Device farm or device cloud is a cloud-based application testing environment that enables remote testing the performance of their websites and mobile apps on a significant number of real mobile devices and an extensive range of desktop browsers, without having to provision and manage any testing infrastructure. It offers access to both modern and legacy versions of devices from multiple manufacturers. It allows its customers (Developers & Testers) to test real-world customer scenarios, fine-tuning test environments with a broad set of device configurations, including language, location, app data, and app dependencies. It can identify errors across multiple devices and presents the events data for analysis that was gathered across hundreds of tests. Furthermore, the devices come with pre-installed OS and browsers for convenience and improved productivity.

When testing on a real device cloud, developer and QAs interacts with and controls mobile devices to see how the user interface appears and operates on every device. This allows for easy identification of functional issues and missing elements. App performance can also be gauged in the presence of hardware issues such as battery consumption, CPU usage, network strength and interruptions (incoming texts, notifications from other apps, etc.). Device farms offer testers the ability to manually evaluate app and website operability from the perspective of an end-user.

Benefits of using AWS Device Farm:

The following are the benefits of using an AWS Device Farm:  

a) it provides a huge variety of these devices on the cloud, (some of which may not be in major circulation), available on-demand 24x7, 365 days a year.

b) it eliminates the cost of procuring all these varied set of devices (for the purposes) and thus significantly reduces the Capital Expenditure for companies. Newer devices are made available at the same price structure.

c) Due to its pay-as-you-go on-demand pricing model aids in reducing the operating costs of these devices.

d) All tests execution are recorded(videos, screenshots and logs) and are available for replay. This will help immensely in further verification(eliminate false positives) and debugging.

Usage of AWS Device Farm:

The AWS Device Farm service enables us to run our tests concurrently on multiple desktop browsers or real mobile devices to speed up the execution of the test suite, and generates videos and logs to help us quickly identify issues with our app.

To run automated tests, many device clouds, including AWS Device farm offer parallel testing. This allows someone to execute test scripts on multiple devices, so as to save time. By scaling up on devices, they can run tests faster, get accurate results and debug quicker. This feature is built to benefit agile teams and enable them to release bug-free products within shorter deadlines.

There are two types of testing:

  1. Automated Testing
  2. Remote Access

We are using automated testing, which helps to test our app in parallel against a massive collection of physical devices in the AWS Cloud.

Pre Requisites :

  • User should have an AWS account and Identity and Access Management(IAM) user with permission to access Device Farm.
  • .ipa (iOS app archive) file of the app on which test needs to be run. Make sure that your .ipa file is built for an iOS device and not for a simulator.
  • iOS devices must be enabled for development and connected to a host running Xcode.

Step 1: Select UI Test target for the AWS Your iOS App

  1. Connect iOS Device in Xcode, and build the project.
  2. After build is successful, select the UI Test target from Debug Scheme to run on Device Farm.

3. Build Configuration setting for your app and your selected UI test target should be the same.

4. In build settings, select iPhone Developer certificate for selected UI test Target in the debug scheme.

Step 2: Package Your Test for AWS Device Farm

When packaging your test for upload to AWS Device Farm, make sure your iOS XCTest UI Runner test runner bundle is contained in a correctly formatted .ipa file.

Packing and Creating IPA

  1. Go into your build directory: ~/Library/Developer/Xcode/DerivedData/ProjectName/Build/Products/Debug-iphoneos/{YOUR_UI_TEST_TARGET_NAME-Runner}
  2. Create a new directory named "Payload"
  3. Move the {YOUR_UI_TEST_TARGET_NAME-Runner} folder into the "Payload" folder
  4. Right click on the Payload folder and click “Compress Payload”

5. Rename the Payload.zip file you just created to “Payload.ipa”

Step 3: Schedule a Run in AWS Device Farm

Create a project to run on Device Farm:

  1. Sign in to the AWS Device Farm console.
  2. Create a new Project and Enter a name of you project and click on create project button.

A project is a collection of all the Runs executed inside it. According to AWS, in this context, a run represents a specific build of your app, with a specific set of tests to be run on a specific set of devices. Here, it provides a variety of ways to track the results of our tests. Projects are differentiated by the operating system, application, or tests type. Since the are no limits for the number of projects we can create, it’s always better to differentiate on the operating system, the application, or both.

3. Create a new run.

4. Upload your .ipa file

5. Configure your test

Now we can configure your tests. There are a lot of options available for each Run. External frameworks like Appium or  Calabash are supported along with XCTest, XCUITest, and Instrumentation along with a Fuzz test.

To upload .ipa, choose XCTest UI as the test type and upload the .ipa file you created in step 2.

For execution environment, select Run your test in a custom environment. When you customize the test environment, you can specify the commands Device Farm should run to execute your tests. This ensures that tests on Device Farm run in a way that is similar to tests run on your local machine.

6. Select device that you would like to run your tests on Device Farm

Select the devices on which you’d like to test. If you like, you can create a new device pool for your test run to reuse in subsequent runs. In Device Farm, a device pool represents a collection of devices (where the tests are run) that typically share similar characteristics such as platform, manufacturer, or model. A curated pool of Top Devices is available out of the box or we can build our own. The list of available devices is quite large covering all iOS devices (from iOS 8.0 onward) plus a good collection of Android devices. There are no limits on how many devices you can include in a pool, but the tests will run simultaneously on a maximum of 5 devices at any point in time.

7. Select Specify device state

Specify device state page can be used for achieving any of the following:

  • To provide additional data for Device Farm to use during the run, Add extra data, choose Upload, and then browse to and choose the .zip file.
  • To install an additional app for Device Farm to use during the run, Install other apps, choose Upload, and then browse to and choose the .ipa file that contains the app. Repeat for any other apps you want to install. You can change the installation order by dragging and dropping the apps.
  • To preset the device latitude and longitude for the run, In Device location, type the coordinates.
  • To preset the device locale for the run, choose the locale in Device Locale.
  • To preset the network profile for the run, choose a profile in Network profile or choose create a new profile to create your own. This will helps us in throttling network bandwidth while testing.

The Execution time out will limit the maximum number of device minutes that your run can use by setting a timeout on each device. If execution exceeds your timeout, execution on that device will be forcibly stopped. You may also be able to view the Partial results of a run, if it's available.

Here we can able to edit the Application uploaded and the Runner .ipa uploaded and can able to edit the Devices selected.

8. Finally, review and start the test run.

Device Farm should start the run as soon as devices are available, typically within a few minutes. Until the run starts, Device Farm displays a calendar icon. After the run starts, results appear as tests are completed. During this time, Device Farm displays a progress icon.

Step 4: View XCTest UI Test Results

When the tests are completed we will be presented with a recap view of the test run. For each device, we can see which tests have succeeded and which have failed, plus detailed information about the device. Everything is recorded, we can access to performance graphs showing FPS, CPU, Memory and Threads plus all device logs and a video showing the device screen. This is especially useful during UI or Fuzz testing to better identify the problem.

We can now run our test. The test will run in parallel on every device in the pool. The total time of a test is calculated as the sum of the time spent by each device


Choose a device to examine its test suites. Here reviewing the test suite results for an Apple iPhone 11.  

Video displays a downloadable video recording of the test.

Suites displays information about the suites for the device and duration of the device taken. Inside the suites section contains Logs, Performance, Files and Screenshot of the device. Test results summarises the number of tests by outcome for each of the test suites run on the device.

The Logs section displays any information Device Farm logged for the device during the run.

The Performance section displays information about any performance data Device Farm generated for the device during the run.

Conclusion

AWS Device farm significantly improves the testing efforts for mobile/web applications and also helps in minimising the overall costs of testing. It greatly enhances our ability to accelerate our Go-To-Market plans for mobile/web applications and saves a huge amount of time in making a wide variety of devices available at our finger-tips 24X7 365 days a year. It is definitely a service worth  trying especially if you have your own test suite. It is easy to use and offers a very detailed views of the tests, providing all information need to reproduce the bug. Its strong point is the possibility to execute tests on a wide range of old and newer devices, allowing us to discover problems which could not be seen on our test devices.

Interesting in exploring with us

We are always looking out for top engineering talent across all roles for our tech team. If challenging problems that drive big impact enthral you, do reach out to us at careers.india@halodoc.com


About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 2500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, Gojek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalized for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.