At Halodoc we've embraced the microservices based service architecture and all of the healthcare services provided by Halodoc like buying medicine, consulting with the doctor online, getting a lab checkup, or visiting a hospital are served by multiple fine-grained, interconnected services.
One of the primary advantages of microservices architecture is that failures can be isolated and graceful service degradation can be achieved as services are independent of each other. For example, during an outage at Halodoc, customers may not be able to order medicines online, but they can still consult a doctor or book a hospital appointment. To handle this kind of graceful service degradation, several failover logics need to be applied to handle temporary glitches and outages. Interdependent services fail together without failover logics and the system ends up with an alternative form of the Thundering Herd Problem.
In such a situation when one or more services are unavailable or exhibiting high latency, bombarding the downstream service continuously with the requests can result in a cascading failure across the enterprise. Service client retry logic only makes things worse for the service and can bring the services completely down. In such a catastrophic situation across multiple systems, a circuit breaker pattern can be put to place. The circuit breaker helps the system to survive gracefully when downstream services are either unavailable or are having high latency.
Why use Circuit Breaker ?
Despite making investments to build a robust infrastructure, many IT organizations continue to deal with database, hardware, and software downtime incidents at some point in time. Circuit breakers allow the system to handle a few of these failures gracefully. It helps in preventing cascading failures in a complex distributed system and enables resilience in systems where failure is inevitable by enabling to fail fast and rapid recovery.
There are several open-source libraries available for integrating the circuit breaker :
- Netflix Hystrix
- Sentinel by Alibaba
- Service Mesh like Istio, Linkerd, Cilium etc.
How a Circuit Breaker works ?
The circuit breaker wraps a function call inside the circuit breaker object, which continuously monitors for failures. Once failure reaches the defined threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the wrapped function getting called.
The circuit breaker, at any point in time, can be at any one of the following state :
CLOSED – It will be in the closed state and if failures exceed the threshold value configured at any point in time, the circuit will trip and will move to an open state.
OPEN – It will be in the open state when the failures are recorded and the threshold is reached i.e calls will start to fail fast without even executing the function calls.
HALF_OPEN – The circuit breaker does not make the wrapped function calls when it is OPEN. After a wait time duration has elapsed, it makes state transition from OPEN to HALF_OPEN and allows only a configurable number of calls. Further calls are rejected until all permitted calls have been completed. If the failure rate or slow call rate is greater than or equal to the configured threshold, the state changes back to OPEN. If the failure rate and slow call rate is below the threshold, the state changes back to CLOSED.
Why Resilience4j ?
The most popular framework which takes care of the resilience is Hystrix and Unfortunately, Hystrix is in sustenance mode. Resilience4j is a an alternative lightweight fault-tolerant library inspired from Netflix Hystrix and designed for Java 8 and functional programming. It is built on top of Vavr, a functional language extension to Java 8, which does not have any other external library dependencies.
Resilience4j provides higher-order functions (decorators) to enhance any functional interface, method reference or lambda expression. More than one decorator can be stacked on any functional interface, lambda expression or method reference. The advantage is that one has the choice to select the decorators they need and nothing else.
Resilience4j provides support for various build tools such as Maven and Gradle. In our case, we use Maven to implement.
In Java, a custom circuit breaker configuration needs to be defined as illustrated below :
A circuit breaker registry can be created with the custom global configuration as illustrated :
To decorate a protected function, which needs to be controlled by circuit breaker can be done as follows :
Resilience4j also supports a module for Micrometer for monitoring systems like InfluxDB or Prometheus.
Circuit breaker metrics can be registered as illustrated :
Metrics dashboard can be configured as per the data requirement. There are several properties that it supports out of the box
- State of the circuit breaker
- Failure call rates
- Slow call rates
- Not permitted calls
In this post, we discussed how introducing the circuit breaker to the system can help to ensure high service availability. We also briefly discussed how the circuit breaker pattern handles downtime and slowness of key services gracefully and helps those services recover by reducing load.
We are always looking out for top engineering talent across all roles for our tech team. If challenging problems that drive big impact enthral you, do reach out to us at firstname.lastname@example.org
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.