Automated Sanity Suite Integration with Continuous Integration pipeline
In today's rapidly changing software development processes, the micro-service architecture has evolved into a powerful paradigm that provides scalability, flexibility, and faster development cycles. Integrating numerous micro-services seamlessly while maintaining overall system reliability is paramount. To meet this challenge, automated sanity suites and optimised build pipelines emerge as indispensable tools. They not only accelerate development but also ensure the delivery of high-quality software and services, meeting the escalating demands of today's business with ease.
At Halodoc, we prioritise the stability of our micro-services by including automated sanity checks into our continuous integration and deployment pipelines. These tests are crucial in ensuring the stability and dependability of our applications. This blog will walk you through the necessary steps and best practices for easily integrating automation sanity tests into the build pipeline.
Quick introduction to Manual Testing and Automation Testing:
Manual testing refers to manually creating and executing test cases without using automation tools. This approach aims to uncover software application issues, bugs, and defects, ensuring its quality and functionality. However, manual testing, characterised by its laborious nature and restricted scope, has been a bottleneck in development, leading to delays and the potential oversight of crucial functionalities. Additionally, the substantial resources and time invested in manual testing, alongside sluggish feedback loops, hinder the timely resolution of issues, undermining the overall effectiveness of the development lifecycle.
Automated testing simplifies the process by eliminating the need to manually test the suites, resulting in a seamless and uninterrupted workflow with only the initial effort required for test case generation. However, it requires a QA engineer to launch sanity test suites for report generation, which may become dependent on engineer availability as release frequency grows. Integrating sanity test suites into the build workflow improves their independence and reusability for developers, freeing up QA teams to focus on generating new test cases and improving overall test quality.
Benefits of Automated Sanity Testing
- Accelerate Testing Processes: Incorporating sanity testing into the CI/CD pipeline streamlines the process by swiftly evaluating each code change. This ensures early issue identification, provides prompt feedback to development teams, and ultimately enhances efficiency while reducing software delivery time.
- Identifying Critical Issues: Sanity check helps in detecting critical issues, such as configuration errors, missing dependencies, or database connectivity problems, before going to the production environment.
- Improved Test Coverage: Automation makes it possible to increase test coverage, ensuring that important functionalities and scenarios are properly tested.
- Performance Optimisation: Analysing the performance of an application during the sanity check can help identify bottlenecks, inefficient code, or database queries. Optimising performance early can prevent issues like slow response times or resource consumption problems under heavy loads.
- Error Handling and Exception Handling: Sanity check can uncover unhandled exceptions or improper error reporting that could lead to crashes or confusing error messages.
- Team Collaboration: Sanity check can facilitate better communication and collaboration among team members. It ensures that everyone is on the same page regarding project status, issues, and goals.
- Risk Mitigation: Identifying and addressing potential risks during the sanity check phase can help mitigate the likelihood of project delays, unexpected failures, or security breaches in the future.
- Confidence in Releases: Developers and stakeholders can release new versions or updates with greater confidence, knowing that the project has undergone a thorough review and validation process.
Teams can collaborate more effectively, and any issues can be addressed promptly, leading to a more collaborative and agile development process.
Steps to integrate automation suite with CI/CD:
Integrating an automation suite with CI/CD requires several essential steps to provide a seamless and automated testing process, especially when managing over 120+ microservices with frequent new releases on stage.
Step 1: Setting Up the Automation Suites.
In Halodoc, we manage more than 120+ microservices, each with its own test suite files in dedicated gitlab repositories. During this step, our focus is on creating straight-forward test scripts, configurations, and dependencies. Importantly, these suites are designed to run individually and can be run at any time using Jenkins jobs. This job is then integrated with the service CI/CD build pipeline as a downstream job along with the input parameters like (Filename=NamespaceServiceSuite.xml, Foldername=service-name).
Step 2: Organising Repositories and Naming Test Suites:
In this step, we match our repository structure to the Jenkins Shared Library (JSL) setup. Following our organization's best practices, each service has its own namespace and service name. Therefore, the repository structure and file names comply with the format: "<namespace><service-name>sanitysuite.xml".
Step 3: Configure Jenkins Shared Library (JSL) for Sanity Checks in CI/CD
In this phase, we improve the service build pipeline by adding a new stage that uses Groovy scripts. The Jenkins Shared Library Pipeline (JSLP) logic assures appropriate file format by verifying that the namespace and service name present in Jenkinsfile match the sanity suite file name. Once this validation is completed, the JSLP calls the backend test suite job to execute the sanity integration, depending on the automation input. The results are then conveniently conveyed using Slack notifications.
Step 4: Run Automated Sanity and Regression Tests
Following the adjustments, the automated sanity and regression tests stored in a designated Jenkins job location are activated. The subsequent detailed report is a crucial outcome. In case of test failures, prompt alerts are sent to both the respective service owner and the Software Development Engineer at Test (SDET). This proactive communication ensures swift issue resolution, maintaining the overall reliability and integrity of our services.
The above pipeline image illustrates the flow of code from development to deployment within Halodoc's CI/CD framework. It starts with developers committing their code and initiating a build job. After the deployment via ArgoCD and ensuring the pod is up, the code progresses to the sanity check step.
Conclusion
As a fast-phase approach to quality assurance in Halodoc, the combination of CICD and Sanity Test Suite goes beyond just converting manual testing to automated testing. Embracing automation suites, enforcing standardisation, and leveraging the power of Jenkins Shared Libraries (JSL) puts us at the vanguard of agility and precision in navigating our services' ever-changing world. It has significantly shortened developer wait times for sanity sign-offs by around 15%, according to observations, and improved total release timelines by about 20%. This positive impact extends beyond mere numbers. It reflects our unwavering dedication to providing high-quality, dependable services, which remains steadfast as we firmly march into the future.
As part of our future enhancements, failure in the sanity test will mark the CI/CD build as failure. Furthermore, any unsuccessful test cases within the sanity suite will automatically halt subsequent builds, not letting the code go to production unless successful. These measures demonstrate our commitment to upholding a robust and error-free development environment at Halodoc.
References
Join Us
Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resume at careers.india@halodoc.com.
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and In total have raised around USD$100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.