Revolutionizing Visual Testing using Automation and AI: Halodoc's Journey with Percy

In the rapidly evolving landscape of software development, ensuring flawless UI/UX remains a perpetual challenge for quality engineering.

At Halodoc, we encountered the limitations of conventional testing approaches in detecting the visual inconsistencies across our web applications. While manual inspections and functional test automation with our Java Selenium WebDriver based framework were helpful, they often fell short in comprehensively capturing the nuances of visual regressions post every deployment.

Why do we need Visual Regression Testing?

Visual regression occurs when a web application's visual appearance (layout, styling, colours, fonts) unintentionally deviates from established design standards or previous versions. Despite often mislabeled as mere cosmetic issues, visual regressions can significantly impact the credibility behind a web application when overlooked. In each of the below examples, the root cause lies in rendering or alignment issues.

Traditional web automation scripts focus more on functionality over visual discrepancies. They can be explicitly programmed to detect visual regressions, but this makes them unwieldy and non-scalable.

Even manual screenshot comparison tools struggle with detecting visual changes due to simplistic pixel-by-pixel comparison algorithms and an inability to adapt to dynamic content.

Percy: An AI-powered visual testing tool

Percy harnesses an AI powered Visual Engine employing computer vision and machine learning and offers Visual testing as a service by capturing DOM snapshots of the UI during testing and automatically highlights visual regressions found in the current build.

We can also make use of additional Percy features such as diff highlighting, anti-aliasing handling, page shift detection, and text stabilization to enhance the speed and accuracy of testing.

Percy Workflow at Halodoc

The first Percy build of the website under test will be used to create a reference base build for future testing. In this stage, Percy will capture DOM snapshots and reconstruct all the pages of the website, as annotated by the automation test scenarios.

After base Percy build creation, the subsequent deployments to application code can be tested using Percy in the following way:

• Referencing the baseline snapshots Percy captured during the first build, Percy automatically notifies of any detected regressions on subsequent builds.
• Below Percy dashboard displays a comprehensive view of the current build's page snapshot alongside the baseline image, the red vertical bars in the middle indicate UI areas with differences.

• Clicking on "Show Differences" highlights issues in red, simplifying detection. Zooming in, below we see the expected vs actual snapshot.

Once the SDET confirms these findings on the dashboard, Percy notifies the team using a slack alert.

Installing Percy

1. Define Use-Cases and Gauge Total Page Usage

Total Visual Snapshots Required per Build Test = Number of Browsers to Support × Number of Critical Web Pages

Here, critical pages refer to pages having high traffic or crucial user interactions, and should include the entire user journey.

3. Install prerequisites (for Java-Selenium-Cucumber framework)

• Download and Install LTS version for node.js, Git SCM and Maven.
• Develop an automation script for launching and navigating around your web application.
• Install Percy

3. Percy Registration & Configuration

• Proceed to register for Percy through Browserstack and choose a plan that matches your usage.
• Configuring your Percy dashboard as per project requirement.
• Select Visual Git for a SDET testing from the stage/production branch or select Git if for a SDE to dev test feature branch.
• Set Browser Selection to be handled by Percy, if paid BrowserStack Automate plan not acquired.

5. Environment setup

• From Project Settings > Token Section, copy the PERCY_TOKEN after setting the toggle to Full-access

• In your test launch environment, store the following environment variables. If having multiple websites, store website specific token in a vault, since we will replace the PERCY_TOKEN with this website specific token before test. Here, PERCY_BRANCH corresponds to the branch of the website that you want to test (stage, master, feature).

Automation using Percy

Below integration guide will be in the context of Selenium-Cucumber POM framework coded in Java, but Percy can also seamlessly integrate with a diverse range of other language/tool bindings.

• Add the below maven dependency in the pom.xml of the automation project

• Configuring Percy at the Base class level of your framework

This implementation ties Percy closely with the driver object and lets us invoke the method from any test method elsewhere in the project.

Taking a Percy Snapshot

percy.snapshot(name, options)

• name (required) - The snapshot name; must be unique to each snapshot
• options - (optional) An object containing various snapshot options

Integration into Automation Framework

• By default, enablePercy should be set to false, allowing tests to run without capturing screenshots when necessary, preventing accidental screenshot usage.

• Utilising a common helper function to support all Visual tool invocations across the project. This also decouples the Percy snapshot command from the test method.

• We can redirect to other tools, if supported, using defined user.properties or run time parameters. This is to ensure we're not locked into any specific tool and ensures the automation can be migrated to another tool with minimal changes.
• Abstracting the implementation from the feature specific step definition

• To simplify Cucumber scenarios, we utilize @Percy custom tags, allowing test invocation from any project location. If all Percy tests reside in the same feature file, direct invocation of the feature file is possible without the tag. For specific feature/page Percy testing, additional tags such as @Percy and @FeatureName can be incorporated.

• To enable local Percy testing, set the toggle to true in the properties file.

This guide assumes the user has a maven project with a test runner configured to run using cucumber tags. If not, refer to the cucumber documentation.

• If you're running from a remote setup, keep the property file value as false and pass the toggle value true as a run time parameter.

Above command will launch your test along with the percy server. It will end with Percy uploading the test screenshots.

Alternatively, we can setup Percy to be launched from Jenkins as well. This guide assumes that the user is familiar with how to setup a jenkins job to run automation tests.

• Post basic job setup, add the following in your Percy job groovy script

Optionally, the PERCY_TOKEN we obtained earlier can be stored in a vault as PERCY_WEBSITE_1,  if we have to test different websites, since each Percy project will have a different token.

• Add the following step in job pipeline script:

To make Percy notify the SDE and SDET team about testing progress, we can make use of Percy's official Slack Bot.

• Add Percy bot to slack from Slack Directory
• Make sure to have Owner privileges of the Percy project. From Percy Dashboard > Project Settings > Integrations, click on connect to slack and provider the necessary permissions from https://yourdomain.slack.com/apps/manage/permissions.
• Configure the channels and notification settings that Percy should notify the team

Conclusion

Percy cannot replace traditional functional test automation tools like Selenium, which remain essential for validating functionalities, backend integration and data integrity, and in fact Percy heavily relies on Selenium for navigation around the website. Therefore, a synergistic approach combining the strengths of both Percy and traditional functional testing tools is crucial for achieving optimal automation in quality engineering.

References

1. Github - Percy CLI
2. Browserstack - Percy Documentation
3. Selenium WebDriver Documentation
4. Cucumber Documentation
5. Jenkins - Building Maven Pipeline
6. Percy - State of Modern Visual Testing

Join us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and  if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and In total have raised around USD $100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.