Liquibase Implementation

Introduction

Liquibase is an open-source database schema change management solution which enables you to manage revisions of your database changes easily . It works across various types of databases(MySQL, Postgres, MongoDB, etc.,)and supports various file formats for defining the DB structure.

At Halodoc we manage Schema updates via GitOps approach using Liquibase. Deploy and roll back changes for specific versions without needing to know what has already been deployed.

How Halodoc leveraged Liquibase features :

Liquibase  enable teams to release software faster by bringing Gitops approach to database changes.

a.Bringing database queries into version control system : Track every changes going into database via source code management(versioning).

b.Quality database changes by enabling code review process: At Halodoc, before executing the SQL queries, developers go through code review process by raising merge request. After approval/merge to master, CI server trigger the build and executes master changeset into database, when it’s much cheaper and easier to fix problems and also consistently deploy the same way in every environment, we have rollbacks always  prepared for every database changes.

Problem Statement:

Dev creates Jira tickets with query details and waits for the SRE team to approve and update the database. This process is very time consuming and hence slows down the development speed. Also there is no tracking of database changes.

Solution:

• Liquibase changelogs and changesets helps to maintain the version of changes in DBs to enable better tracking in place and keeps application and database version in sync.
• Easing developers to deploy changes faster after code review using automation job.

Installation process for Liquibase:

• To install the Liquibase Runner plugin in Jenkins, navigate to Manage Jenkins > Manage Plugins.
• Select the Available tab.
• After the plugin list has loaded, search for “Liquibase”. Select the checkbox next to “Liquibase Runner” and either choose “Install without restart” or “Download now and install after restart”.
• Next, set the location of Liquibase on your local Jenkins machine that is running the build.
• Go to Manage Jenkins > Global Tool Configuration > Liquibase Installations and create a new Global Liquibase installation:
• Finally, add a new Build Step to an existing Project in Jenkins. Or create a new one.

Repo's configuration at Halodoc:

Create the repo's for each services and provide  the access for all developer with developer grants .

Liquibase for Halodoc to perform DB changes:

• Always add the changesets to a changelog ( don’t change anything without Liquibase!) – changeset should be unique combining AUTHOR:ID(task) and filename (file with your changelog)
• Changeset in liquibase represents a single change to your database. Each changeset is executed in a transaction and will be either committed or rolled back depending on whether there is an error or not.
• Changelog files can be managed in two ways - either to create a single changelog file and add all of your changesets to that file, or create multiple files and split the changesets across them.
• Changelog file is a root file that contains a record of all your database changes (changesets). The changelog-file parameter helps you create a changelog file using the generate-changelog command or diff-changelog command, apply changes to your database, and keep track of your database information.
• Liquibase.properties file contains the information that Liquibase needs to connect to a particular database. It resides in the text-based Liquibase properties file, liquibase.properties, along with other properties that rarely change. The liquibase.properties file eliminates the need to enter properties through the command prompt as parameters, saving you time and potential typographical errors.
• How to create a table and how to perform DML operations in the Database using liquibase "formatted SQL statements" ,Here changeset value common for all DDL and DML statements example given below:

• If the change is for stage or prod environment then navigate to stage/prod folder → namespace → service as shown in the below screenshot.

SQL Formatted Changeset file

master_changelog.xml is required to be updated only at the beginning of every quarter with one version of sql file.Changeset incremented version should be updated in master_changelog.xmlGet your code review approved by Architects or Leads/DBAs & DE and merge it to master branch.After the above steps are successfully passed through the code review process, head to this Jenkins job.

Provide the following build parameters:

• Environment Type: stage/prod (Select one of the environment)
• Namespace: service Name (Select the namespace you target)
• Service: select the services under  (select the service under the above selected namespace where the DB changes needs to be performed)

LiquibaseAction: (Status / update / rollback)

When you deploy your changes, Liquibase creates two tables in your database: DATABASECHANGELOG and DATABASECHANGELOGLOCK.

DATABASECHANGELOG :Table tracks deployed changes so that you have a record. Liquibase compares the changesets in the changelog file with the DATABASECHANGELOG :Tracking table and deploys only new                 changesets.

Liquibase Workflow diagram :

• DBA after merging the MR ,developer will run the jenkins job.
• Once jenkins job  is started, it will connect to the vault cluster for database  credentials with help of liquibase properties files based on the repositories.
• Then Liquibase will execute the SQL queries in database.

Summary:

As we can see, by using Liquibase we can track database changes easily during  the release process.

Developers should follow the below workflow:

• Get your code reviewed by architects or Leads/DBAs & DE and merge it to master branch.
• Once the review is done execute the liquibase jenkins job to update the database.

Reference :

1.https://www.liquibase.org/

Join us

Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 4000+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek and many more. We recently closed our Series C round and In total have raised around USD$180 million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.