At Halodoc, solving a patient's problem has the highest precedence. We are constantly adding new features or improving our existing functionalities in 2-week sprint cycle. CI/CD is playing a vital role in our delivery and quality. We are using Jenkins + Fastlane as CI/CD tool. During the sprint cycle, we are releasing a large number of feature builds for testing. Reducing the Jenkins build time is the major problem to be solved.

Jenkins Stages

Before analysing the Jenkins build time, let us give a quick overview of Jenkins stages.

We are having the following stages in our build jobs

Feature build process:

1. Code coverage
2. Build generation
3. Test flight deployment

RC build process:

1. Code coverage check
2. Stage build generation
3. Stage test flight deployment
4. Production build generation
5. Production test flight deployment

What is the challenge?

We have multiple stages in our Jenkins jobs. Each stage has different dependencies and purposes which take more time than usual. The following picture summarizes the existing Jenkins jobs time.

Jobs	Completion Time
Feature Build	≈ 2 hours 45 minutes
RC Build	≈ 3 hours 50 minutes

Yes, the real challenge we were facing is build time. There are lots of feature builds generated for the quality assurance team to meet the requirements and quality of the product. This takes a long waiting period to get a build for verification.

How did we reduce the build time?

We started analysing the time taken for each stage and the commands which run on each stage.

Here is the list of key areas we identified for improvement in the report.

• Xcode Command line tool update
• Jenkins system's internal SSD usage & Pod Cache
• Parallel stage execution

Xcode Command line tool update

Xcode 13.3.1 command line tool got stuck in xcode -exportArchive command. It’s almost frozen in App.xcdistributionlogs state. It is supposed to be completed in 10 minutes but it takes 1 hour and 10 minutes to complete the process.

We have updated the Xcode command line tool with the latest 13.4.1 saving an hour of the build generation process approximately.

Jobs	Completion Time
Feature Build	≈ 1 hours 45 minutes
RC Build	≈ 2 hours 50 minutes

Jenkins system's internal SSD usage & Pod Cache

Another reason for Jenkins’ high build time is the external SSD checkout process. We are using an external SSD as a primary storage in the Jenkins server. The workspace checked out in the external SSD has 1000 GBPS read/write speed, whereas the internal SSD has 5000 GBPS read and 4000 GBPS write speed.

Data transfer speed influences both the build time and checkout process. So we marked internal SSD as primary storage.

As we are using cocoa-pods, enabling pod cache helps us to reduce the time in pod update sequence.

This saved 30 minutes in feature build jobs and 40 minutes in RC build jobs approximately.

Jobs	Completion Time
Feature Build	≈ 1 hours 15 minutes
RC Build	≈ 1 hours 40 minutes

Parallel stage execution

Even though the system has a capacity to handle multiple processes, all stages in the Jenkins job are executed serially. Jenkins also has an option to execute the stages parallel. Let's see the parallel stage configuration.

Parallel stage configuration

pipeline {
    agent iOS
    stages {
        stage('Release Candidate') {
            parallel {
                stage('Unit Test') {
                    agent {
                        label "iOS-2"
                    }
                    steps {
                          sh "run-tests.sh"
                    }
                    post {
                        always {
                            archive "**/unit-test-*.html"
                        }
                    }
                }
                stage('Stage Build') {
                    agent {
                        label "iOS-2"
                    }
                    steps {
                        sh "stage-release.sh"
                    }
                    post {
                        always {
                            // upload to test-flight
                        }
                    }
                }
                stage('Production Build') {
                    agent {
                        label "iOS-2"
                    }
                    steps {
                        sh "production-release.sh"
                    }
                    post {
                        always {
                            // upload to test-flight
                        }
                    }
                }
            }
        }
    }
}

With the above changes, we can achieve parallel execution. This helps to save around 45 minutes in RC jobs and 25 minutes in feature build jobs. This reduces the build completion time to around 50 minutes for each feature build and RC build.

Overall Optimisation

Initially feature build jobs took 2 hours 45 minutes and RC build jobs took 3 hours 50 minutes. The former has been reduced to 50 minutes and the latter has been reduced to 55 minutes.

Jobs	Completion Time
Feature Build	≈ 50 minutes
RC Build	≈ 55 minutes

Join us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek and many more. We recently closed our Series C round and In total have raised around USD$180 million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.