Distributed Locking In Go
Distributed locking is a critical concept in distributed systems. It involves the blocking of processes or threads attempting to access a shared resource that is currently in use by another thread or process. In this blog post, we will delve into the fundamental aspects of distributed locking, its significance, and the methods commonly employed to achieve it. Additionally, we will explore the implementation of distributed locking in the context of the Go programming language.
Background
In certain scenarios, it becomes crucial to safeguard shared resources from multiple processes trying to access them simultaneously. This requirement arises when tasks within a system demand a synchronized processing approach. Without such synchronization conflicts and data corruption may occur.
To address this challenge, synchronization mechanisms are employed. One common method is using locks or mutexes, which act as a way to guard shared resources. When a process needs access to a resource, it must first acquire the corresponding lock, ensuring exclusive access. Once the task is completed, the lock is released, allowing other waiting processes to access the resource.
Synchronous processing of critical tasks is particularly crucial in areas like concurrent programming, real-time systems, and database management, where data integrity and responsiveness are critical.
Example of Locking in Go
Here is a simplified example demonstrating how multiple processes accessing the same resource can happen at the same time:
In the above example, the shared resource account is accessed by multiple processes running on different goroutines (threads). In the absence of proper locking mechanisms, a second payment can occur before the first payment has completed its processing, resulting in the potential for erroneous data updates. To maintain data integrity and prevent such issues, employing appropriate locking techniques becomes essential.
To make the payment process work properly we need to add a locking mechanism so that each process can finish before another one is starting, hence the data integrity can be maintained. We can modify the previous example to have a lock like this:
We use the sync.Mutex type from Go's built-in sync package for the locking mechanism. With the locking mechanism in place, the correct result is achieved.
While sync.Mutex can be used for locking in Go, it is not suitable for distributed systems with multiple instances running concurrently. This limitation exists because sync.Mutex operates at the application level only. If requests are served by different instances, the locking mechanism will not work. In a distributed system, a distributed locking mechanism is required. In distributed locking, the lock needs to be placed outside the service itself so that multiple instances of a service share the same source of locking. Redis is a commonly used external storage system for implementing distributed locking.
Go Distributed Locking
When it comes to implementing distributed locking in Go, there are several ready-to-use libraries available, and one popular choice is redislock. Here's an example that demonstrates how to use this library:
Go Distributed Locking in Halodoc
In Halodoc, we implemented the distributed locking mechanism as a common library so that the implementation can be reused across services (following DRY principles). We created the locking mechanism to be generic and abstracted from the real implementation. By doing that, the user of the library does not need to be aware of the underlying implementation, they just need to use the provided functions to do the locking and unlocking. Another advantage of abstracting the real implementation is the common library can change the implementation and the user of the library does not need to change anything unless the function signature (contract) is changing. For example, if we decide to switch from using the redislock library to redsync, the client code remains unchanged as long as the exposed API remains the same.
In the common library, we expose an interface for distributed locking with the following signature:
We also provide a RedisLockService implementation that implements the BaseLockService interface. This implementation internally wraps the locking and unlocking logic from the redislock library and exposes it as an API to the client.
By utilizing the common library, clients can easily incorporate distributed locking into their code without worrying about the underlying implementation details. This approach allows for flexibility and future-proofing, as different distributed locking libraries can be seamlessly integrated by implementing the BaseLockService interface.
Common Distributed Locking Library Usage
In Halodoc we have many cases that need a locking mechanism, and since halodoc adopts a microservices approach in developing our systems we need to use distributed locking.
In one of the cases, we have a functionality that accepts payments through a payment gateway and distributes the payment to payable orders. The logic for distributing payments involves selecting unpaid order data and ordering them by the oldest order. We then allocate the payment to these orders.
In this scenario, there is a possibility that a user might make multiple payments simultaneously, especially if they use some kind of scheduler for money transfers. Without a locking mechanism, the second payment process could occur before the first process completes, resulting in the user paying for the same order twice. This inconsistency in data can be avoided by implementing a locking mechanism that ensures the second process waits for the first process to finish before proceeding, thereby maintaining data consistency.
Here's an example of how the distributed lock from the common library can be used on the client side to solve the case:
Summary
This blog post discusses the concept of distributed locking in Go, which ensures that only one process or thread can access a shared resource at a time in distributed systems. The post explains the need for distributed locking and provides an example of implementing it using the sync.Mutex type in Go. However, it highlights that sync.Mutex is not suitable for distributed systems with multiple instances running concurrently and introduces the use of external storage systems like Redis for distributed locking. The post mentions the redislock library as an example and demonstrates its usage. It further explains how the concept of distributed locking is implemented in the common library at Halodoc, providing an interface for distributed locking and an implementation using the redislock library. The post emphasizes the benefits of abstracting the internal implementation from the client, allowing for easy swapping of different distributed locking libraries without affecting the client code.
Join Us
Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek and many more. We recently closed our Series C round and In total have raised around USD$180 million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.