Implementation of Percy Visual Testing Tool at Halodoc iOS Apps

iOS • Mar 22, 2024

Halodoc is continuously exploring innovative methods to enhance the user experience of mobile applications within the quality assurance process. We encountered the limitations of conventional testing approaches when it came to detecting visual discrepancies in our app. Despite employing manual inspections and functional test automation via our Apple iOS app's XCUITest framework, these methods often fell short in capturing the nuanced visual regressions that emerged after each deployment.

In order to ensure the visual consistency of our UI after implementing changes, we explored various AI-powered tools. After careful evaluation, we discovered that Percy Visual Testing stood out as the ideal solution for our needs. This blog delves into the necessity for Percy, addressing setup procedures, workflow, and how it enabled us to effectively address visual regression challenges.

Importance of Visual Testing

Visual testing, is a software testing technique used to verify the appearance and layout of a user interface (UI) in a software application. Unlike traditional functional testing that focuses on verifying the functionality of the application, visual testing specifically checks how the application looks to the end-user.

In visual testing, automated or manual tests are conducted to compare the expected visual elements of the UI, such as colors, fonts, images, layouts, and positioning, against the actual appearance of the application's UI. This comparison helps to identify any discrepancies, such as visual bugs, unintended changes, or layout issues, that may occur due to software updates, code changes, or application inconsistencies.

There is a need for visual regression testing post-deployment of a PRD (Product Requirements Document) on the stage or production environment. Although figma or other design tools provide an detailed UI designs for the specific feature outlined in the PRD, QA may not be aware of any unintended visual regressions it might introduce elsewhere on the application.

Percy Visual AI Testing Tool:

Percy is an all-in-one visual testing and review platform. It helps catch visual bugs and gain insights into UI changes on each commit. Our goal is to give confidence in the visual integrity of your UI every time deployment.

Benefits of using Percy for visual testing

Easy Integration with App Test Frameworks: Integrating Percy with testing frameworks such as XCUITest and Appium etc is straightforward. It seamlessly integrates with the existing app framework, requiring minimal adjustments to the framework's code and enabling simple invocation directly from within the test code with just a single line.
Capture baselines and Compare screenshot with other build test runs: Percy captures screenshot containing the all visible elements in the screen. These screenshot serve as a baseline for comparing new deployments, ensuring comprehensive visual comparison and accuracy.
Visual Diffs: After capturing baseline screenshots, Percy automatically captures new ones when changes occur in your application. It then compares them with the baseline images, highlighting any visual differences. If no changes are found, no user action is needed.
Dynamic Content Changes: If a failure stems from expected changes, SDETs can approve and merge the build without replacing the baseline. For defects related to dynamic content, SDETs can quickly approve them. If defects recur due to dynamic content, Percy's Ignore Regions feature allows SDETs to exclude specific areas for evaluation.
Review and Approve Changes: Percy provides a user interface for teams to review and approve visual changes. This helps in identifying legitimate design updates from unintended visual regressions.
Integrations with Slack, Git, and CI/CD Pipelines: Percy can be integrated with our existing GitLab and Jenkins and can be configured to be triggered whenever a new release is made to production.

Getting Started with Integrating Percy into your XCUITest

Install prerequisites

Sign up Percy Dashboard and Create new app project on percy dashboard post sign up
After you’ve created the project, you’ll be see a PERCY_TOKEN, please refer below screenshot. Set this token as an environment variable and Percy will use the PERCY_TOKEN to know which project to upload the screenshots to. You will need this PERCY_TOKEN in next steps.

Install Percy through command line.

Add 127.0.0.1 percy.cli to your /etc/hosts file on your macOS machine.

Use above command to open the /etc/hosts file, make sure 127.0.0.1 percy.cli to be added hosts file.

Configuration of Percy library into Test Automation.

At Halodoc, for iOS app test automation, we use a XCUITests with Cucumberish written on Swift language, so the below integration guide will be in that context, but Percy can also seamlessly integrate with a diverse range of language/tool bindings.

Configure of Percy SDK into the Projects Manifest dependencies file. Percy Integration to Test Projects can be done through two ways.

Open XCode > File > Add Package Dependencies. Search percy repositories URL https://github.com/percy/percy-xcui-swift.git and Press Add Packages

Integrate percy-xcui-swift dependency manually on SPM project dependencies file.

2. Configure Percy code in common module which can be access through all test targets. Please find below for reference

name (mandatory) - The screenshot name must be unique to each screenshot.
options - (optional) An object containing various snapshot options. Refer per-snapshot configuration options

Integration of Percy to app automation framework.

Cucumberish has already provided a way to organize your scenario execution by using tags in feature file. We can define each scenario with a useful tag. Later, we can decide which specific tag we want Cucumberish to execute. Tag starts with “@”. After “@” you can have any relevant text to define your tag like @VisualTests just above the scenarios you like to mark.

Providing screenName on every scenarios let say app home, make sure the screenName should be unique for every scenarios. And passing screenName to StepDefinition file to capture expected screenshot on every scenario

Abstracting the implementation from the feature specific step definition file

Inherit all StepDefinition methods on initialiser file

Run Percy Visual Testing.

Run your tests in local environment:

Open the terminal and set the Percy token:

Start your Percy Server by running the following command:

Run your XCUITests project either using Xcode or the below command.

Run your tests using BrowserStack App Automate:
When you run the script on App Automate, proceed to generate the application IPA and the test suite IPA files. For the App Automate session to work with App Percy, we need to provide App Percy parameters.For instance, in “appPercy”:

The env parameter can contain any of the following Percy environment variables.Execute tests using the following command post updating App Percy params:

Note : More information of run XCUITests app automation test cases through BrowserStack app automation can be found here

Exploring a Case Study on Percy's Visual Screen Validation Capabilities.

Upon successfully test cases run completes, check captured screenshot on Percy dashboard under previously created project. visit your project in Percy.

Approve the below initial screen as base screen if all visible information are expected

Upon running next build with visual changes to your application, Percy takes new screenshot. Compares above base screen with new screen based on provided name in scenario file. You can then see the comparisons between the two runs on the new build.

In the above screen comparison, we can see there are couple of differences on from base screen to feature build.

Text difference
Image difference

While textual differences can be detected through the existing framework setup, image differences cannot be achieved via the XCUITest framework. Similarly, visual changes of the application we are achieving by integrating Percy into the existing test framework.

Conclusion

In this blog, we've shared an overview of Integration of Percy at Halodoc, the journey in automation testing workflow has delivered substantial advantages across different aspects of our deployment process. Lets say, Percy facilitates automated monitoring of visual regressions on post-deployment, offering a comprehensive overview. SDETs seamlessly enhance automation with on-demand Percy tests, boosting efficiency. Migrating dynamic app screens to Percy has reduced test failure rates, improving reliability. Percy enables rapid testing of new in-app screens during development, even amid design changes. This has led to faster testing times and reduced development efforts compared to traditional methods and maintain the better user experience.

While Percy offers substantial benefits, it doesn't replace traditional tools like XCUITest, Appium etc, which are crucial for validating functionalities and serving as a foundation for tools like Percy. Thus, a synergistic approach combining Percy and traditional tools is vital for effective automation in quality engineering.

References

Join us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and In total have raised around USD $100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.