Test automation reporting improvements through Allure reports

allure report • May 25, 2021

Reporting is an important element of a Test automation framework as it plays a crucial component in a well-designed test automation tool. The success of the designed test automation framework and its survival also depends on how effectively reporting mechanism is implemented.

Test automation report is the blueprint, that shows how the script was executed. It shows the scripts that were executed, steps in the test script which were executed, execution time, checkpoints that were passed, failed, skipped, and broken, if failed, skipped or broken what was the actual results and failure reason.

Test execution report should be detailed enough when there is a test execution pass as well as for other reasons. The report should be detailed enough to identify whether it’s a script failure or an application failure.

The test execution report also should be detailed enough even for other SDET to open a defect utilising the information provided in the report.

In this blog, we have outlined the reporting process and why we chose Allure report as a test automation report, some of the gaps in existing reporting tools and how allure report was successful in overcoming these challenges yet offering a plenitude of beneficial features and operators.

Tools considered

Multiple reporting tools tried and compared such as Testng, JUnit, Reportng and Maven Surefire reporting that facilitated test reporting ,but we choose Allure report from tool box.

Why did we choose Allure Report ?

Data-rich reports which will allow us to collect much more information.
Backed by a tree-like data structure, that represents a test execution process. Different tabs allow switching between the views of the original data structure thus giving a different perspective.
From QA/DEV perspective, shortens defect lifecycle. Test failures can be divided into bugs and broken tests, logs, steps, fixtures, attachments, timings.
From the Manager's perspective, provides a clear big picture of which features have been covered, where defects are clustered, how the timeline of execution looks like, and many other convenient things.
According to feature/stories, the test script will get categories. It will help when we have a lot of features and multiple SDET's are working on the same automation report. Devs can also categories feature which was assigned to them and accordingly act on it.
Each test case have information such as severity, description , duration ,test data and execution steps.
On the Suites tab a standard structural representation of the executed tests, grouped by suites and classes can be found.
Visualisation allows you to see different statistics collected from the test data: status breakdown or severity and duration diagrams.
Timeline tab visualise retrospective of tests execution, allure adaptors collect precise timings of tests, and here on this tab, they are arranged according to their sequential or parallel timing structure.
For the Behaviour-driven approach, this tab groups test results according to Epic, Feature, and Story tags.
The package's tab represents a tree-like layout of test results, grouped by different packages.
It supports TDD(test driven development) and BDD(behaviour driven development) approach as well as multiple languages such as Java, Python and Javascript.

Configuration steps for generating Allure report:

These are steps needed for generating Allure report

Add these annotations at test script level.

To implement into Maven automation repo, these changes are required into pom.xml

To implement into Gradle automation repo, these changes are required into build.gradle

Maintain Automation Reports: It is very important to maintain the report's history.

Current situation: For result reports, backtracking for automation result is not possible, if a Jenkins job build is gone, a report is also lost with that.
Solution: Slack is the main communication app in our organisation. Integration with a slack web-hook
To achieve this, need to add an incoming slack hook for the channel into the codebase.
Need to create a custom event listener.
Need to call slack API to post-test result object to the slack channel.

Summary

With a multitude of favorable features and operators, Allure report has ensured appropriate information from test scripts to necessary changes on test report data and finally sharing data-rich test report over a slack channel.

The functionality mentioned above helps us to find out issues in application and test script level. Allure report gives a clear picture to each level in tech and non-tech person in an organisation and hence we see great value in leveraging it for our Test automation report generation.

Interested in exploring with us?

We are always looking out for top engineering talent across all roles for our tech team. If challenging problems that drive big impact enthrall you, do reach out to us at careers.india@halodoc.com

About Halodoc

Halodoc is the number 1 all-around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 2500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allows patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, Gojek, and many more. We recently closed our Series B round and In total have raised $100million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalized for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.