Migrating from Hazelcast to Redis @ Halodoc
Background
While deploying microservices, it is a common practice to have multiple instances of the same application running for high availability. But for some features, we would like to have only one instance of the application to operate in the critical section. For example, if there are multiple payment requests for the same order, then the system should only process one request while the others are queued. This can be enforced by using distributed locks. At Halodoc, we used Hazelcast for all the distributed locking needs.
A Few Use Cases
We use Hazelcast in an embedded mode where a Hazelcast instance is co-hosted with the application on the same JVM. Hazelcast nodes on different instances of the same microservice form a cluster. Hazelcast internally stores data in partitions and these partitions (and their replicas) are distributed across different instances.
This images will explain current existing Hazelcast implementation in our microservices cluster.
Like any distributed data framework, Hazelcast deployment (provisioning/de-provisioning/upgrade) involves the following steps:
- The shutdown signal sent to the Hazelcast instance.
- Hazelcast waits for the partition redistribution to complete.
- Hazelcast instance is shutdown.
- When the new version of the application starts, Hazelcast node tries to join the cluster.
- Data partitions are re-distributed to transfer some partitions to the newly added node.
This image explains the scenario where we add a new instance into the microservices cluster.
In this situation, we will have some problems when we move the microservice to Kubernetes cluster such as:
- We need to deploy/maintain the stateful application on Kubernetes.
- Rolling upgrade for Hazelcast is only supported in the enterprise edition.
- Increased time to do a rolling upgrade of our application, because Hazelcast must synchronise before the application starts running.
These issues necessitated the need to make our microservices stateless and moving Hazelcast out as a standalone cluster. But moving to a standalone Hazelcast cluster means our DevOps have to manage the cluster and also ensure the scaling, availability and monitoring aspects of the Hazelcast cluster. To overcome these problems, we decided to move the locking mechanism from Hazelcast to Redis. As we use AWS managed the Redis Elasticache service, the DevOps need not manage the cluster. With these changes, our microservices instances became completely stateless and could easily be deployed/scaled on Kubernetes. This image explains how we add a new instance to the existing microservice application.
Redisson Overview
Most of our microservices apps are created on Java. We implement the Redis lock using a library: Redisson. Redisson is a Redis Java client with an in-memory data grid feature. It provides a more convenient and comfortable way to work with Redis. Redisson objects offer a separation of concern, which allows us to maintain the focus on data modelling and application logic.
Implementation
For implementing the Redisson, you can use Maven or Gradle. In our case, we use Maven to implement the Redisson. You can add the Redisson in maven like this
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson</artifactId>
<version>3.12.1</version>
</dependency>
To implement in Java, it's quite simple: just instance the Config and get lock.
// Instance Config
Config config = Config.fromYAML(new File("config-file.yaml"));
RedissonClient redissonClient = Redisson.create(config);
// Create function for lock
public Boolean lock(final String key, final Integer maxLockDuration, final TimeUnit timeUnit) {
final RLock lock = redissonClient.getLock(key);
try {
final boolean lockAcquired = lock.tryLock(maxLockDuration, timeUnit);
return lockAcquired;
} catch (final InterruptedException e) {
log.error("An Error {} occurred while occurring lock on {} object.", e, key);
} finally {
lock.unlock()
}
return false;
}
First Step, you must instantiate Config class. It can read config from the file or you also can hard-code the config into the code. After that, you must create an instance of RedissonClient based on the Config instance. With RedissonClient, you can execute tryLock to get a lock from Redis. This lock has a duration time. The duration time should be added to tryLock.
Summary
- Migrating from Hazelcast to Redis made our microservice applications stateless and simplified our architecture.
- There is no need to implement or deploy a stateful application in Kubernetes.
- We only need to manage one Redis cluster for all applications.
Reference
- https://github.com/redisson/redisson
- https://engineering.datorama.com/moving-from-hazelcast-to-redis-b90a0769d1cb
- https://hazelcast.com/blog/rolling-upgrade-hazelcast-imdg-on-kubernetes/
We are always looking out to hire for all roles for our tech team. If challenging problems that drive big impact enthral you, do reach out to us at careers.india@halodoc.com
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke.
We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 1500+ pharmacies in 50 cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allows patients to book a doctor appointment inside our application.
We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates foundation, Singtel, UOB Ventures, Allianz, Gojek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission.
Our team work tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.