HaloTestGenAI: AI-Powered Test Case Creation from PRD and UI Design

For every release item to be successful, it requires rigorous and smart testing to capture all the bugs that could potentially affect the users. Being in the Healthcare domain, it is very important that we, at Halodoc deliver high-quality products in shorter times. To achieve high efficiency, test cases have to be created covering all the potential areas where bugs could escape.

So far, Testcase Creation has always been a manual process where the SDET would write the entire test case list with his/her knowledge of the Release Item. This process can lead to human errors and can also be highly time-consuming.

The proposed Halodoc tool (HaloTestGenAI) uses AI to generate the test cases for Release Items. This would save up to 50% of SDET's Time in creating the test cases.

In this blog, we will discuss how it has been implemented in our Halodoc system to automate the test case creation process in Halodoc. We will look at several aspects, such as our AI input, AI output, time reduction, and script quality.

Test Case Creation: Manual Approach vs AI Approach

Traditionally when a new requirement comes, the SDET analyzes the requirement and starts writing the testcases manually, and based on review comments, adds more testcases and proceeds with testing. Typically when a SDET writes around 20 test cases covering the entire functionality of PRD it takes around 40 minutes approximately but using HaloTestGenAI, the same can be acheived in approximately 20 minutes saving upto 20 minutes of manual efforts in writing the testcases. This approach has several benefits to it. HaloTestGenAI improved the coverage by focusing on all the areas of the input data ensuring no edge case is missed. It also helps reduce SDETs' time up to 50%.

HaloTestGenAI Overview

HaloTestGenAI utilizes the OpenAI GPT-4o API to generate test cases with high accuracy and reliability. After analyzing various AI models available in the market, GPT-4o proved to be the most effective in delivering results that align closely with the specified requirements. HaloTestGenAI operates using three key inputs: the PRD screenshot, Figma screenshots, and a tailored prompt crafted according to the testing needs.

The above code snippet has the main functionality of HaloTestGenAI. Here get_response_from_openai_api method takes input of the prompt, PRD and Figma Images to generate the test cases.

The prompt is passed within the payload as content and the images are fetched from Halodoc's Cloud Storage. The URL mentioned calls the OpenAI and the model of the API is called in the payload.

The coverage and generation of specifically required test cases are largely dependant on fine tuning the prompt according to the need as the results are purely based on the prompt. (For Example. Giving the prompt as "Generate testcases" would generate random test cases. But giving the prompt as "Generate End to End Testcase analysing the UI Images in series" would result in more specific test cases).

Sample PRD Input

As stated above, HaloTestGenAI uses PRD and Figma images along with a prompt to generate the test cases.

Prompt

You are a specialist in software development engineer testing for web applications. Based on the given prompt and image data (here the Figma images are sequential starting from Figma1, then Figma2, analyze it and provide end-to-end testcases and Negative test case in Selenium feature file format.

Figma Image

PRD Image

Output of HaloTestGenAI

Based on the number of testcases requested, HaloTestGenAI will generate the requested number of testcases. Sample Output is given below,

Qualitative Analysis

HaloTestGenAI can be executed multiple times as needed by the SDET to achieve the desired test coverage. For analyzing the Efficiency of the Script, HaloTestGenAI was executed thrice on the sample PRD and it yielded a good amount of results. The qualitative analysis of the Script's Performance is given below,

Consolidative Report:

Based on the detailed analysis of the testcases generated by HaloTestGenAI and comparing it with the manual testcases written by the SDET, the following results have been obtained.

Considering 4 testcases missed, HaloTestGenAI could cover 18 testcases added by the Manual SDET which covers the entire functionality. This contributes to 81.81% coverage (arrived at by comparing HaloTestGenAI's test cases with test cases written by manual tester to cover entire functionality).

3 testcases which were newly given by HaloTestGenAI were very good and putting them up with the manual testcases given if it was part of 22 test cases - contributes to 13.63%.

With a high coverage rate of almost 82%, many critical bugs and vulnerabilities can be trapped with HaloTestGenAI in comparatively less time. The time analysis is provided in the next segment.

Time Analysis

The main benefit of having HaloTestGenAI generate the test cases is the reduction in time. With many PRDs coming and requirements getting complex, time management is very essential for building a better product. Based on the execution results on the sample PRD, the time analysis is stated below,

Formula to calculate time saved:

Time Saved = Total Time Taken by Complete Manual Efforts - Total Time Taken by AI-Driven Efforts

Traditional Method - Manual Time:

Total Testcases written manually: 22 * 2 = 44 mins approximately.

Time Taken for AI-Driven Testcase Creation: 

1. Taking Screenshot and uploading screenshot and adding the prompt - 5 minutes
2. Script Execution and Review of Testcases - 10 minutes
3. Adding missed testcases and also in-depth testcases - 4 * 1.5 mins (as analysis already done) - 6 minutes

Total Time: 5 + 10 + 6 = 21 minutes
Total Time Saved: 44 - 21 = 23 minutes (47.72%)

Why Human Intervention Remains Essential

With HaloTestGenAI, the SDETs can reduce up to 48% of the time and achieve up to 80% of the coverage from the image data of the PRD and fine-tuned prompt. But there are certain limitations to this approach. Since the AI relies fully on the image data and does not have backend product knowledge, it generates additional unnecessary test cases by analyzing the image data alone without product knowledge. This is not beneficial for the SDETs and also the product code cannot be fully exposed to the AI due to security reasons. This is the main reason why human intervention would still be needed. With the coverage of 80% and reduction of time up to 48%, when the SDET applies his/her knowledge of the product and fine-tunes the testcases, coverage can be increased up to 99%, reducing the TLC time and also being delivered with better quality.

Conclusion

Currently, AI is used across all types of industries. At Halodoc, AI is integrated into many of the systems already. From a QA perspective, when the team worked to understand how effectively AI can be utilised for making the system better, test case generation—an essential part of ensuring a quality product is delivered to users—was identified as a key area to leverage AI capabilities.

As a result, Halodoc's SDETs created HaloTestGenAI, a powerful AI-based Python script that generates test cases using the PRD, Figma images, and a tailored prompt.

Upon analysis, it was found that HaloTestGenAI delivered an exceptional coverage rate of up to 80%. However, it was also understood that for comprehensive testing, especially in complex systems like internal systems, human intervention is essential to bridge the contextual and architectural knowledge gaps.

Overall, using HaloTestGenAI helps SDETs reduce up to 50% of the time and effort required to create test cases.

References

• OpenAI GPT-4o

Join us

Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the number one all-around healthcare application in Indonesia. Our mission is to simplify and deliver quality healthcare across Indonesia, from Sabang to Merauke. Since 2016, Halodoc has been improving health literacy in Indonesia by providing user-friendly healthcare communication, education, and information (KIE). In parallel, our ecosystem has expanded to offer a range of services that facilitate convenient access to healthcare, starting with Homecare by Halodoc as a preventive care feature that allows users to conduct health tests privately and securely from the comfort of their homes; My Insurance, which allows users to access the benefits of cashless outpatient services in a more seamless way; Chat with Doctor, which allows users to consult with over 20,000 licensed physicians via chat, video or voice call; and Health Store features that allow users to purchase medicines, supplements and various health products from our network of over 4,900 trusted partner pharmacies. To deliver holistic health solutions in a fully digital way, Halodoc offers Digital Clinic services including Haloskin, a trusted dermatology care platform guided by experienced dermatologists.We are proud to be trusted by global and regional investors, including the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. With over USD 100 million raised to date, including our recent Series D, our team is committed to building the best personalized healthcare solutions — and we remain steadfast in our journey to simplify healthcare for all Indonesians.