Implementation of MDC (Mapped Diagnostic Context) on Golang
What is MDC (Mapped Diagnostic Context)
When a multi-user web application is used by many users simultaneously, it is often difficult to see how each user interacts within your application without the additional context.
For example, in order to get a key attribute from a header, you need to include the header context in the method argument.
Log4j and Logback have a feature that handles these problems, called MDC (Mapped Diagnostic Context). An MDC is a map of String values with String keys that corresponded with current thread. The key attribute value can be added to the MDC header.
In short MDC is an instrument for distinguishing interleaved log output from different sources. Log output is typically interleaved when a server handles multiple clients concurrently.
The following is an example class for MDC. There are only static methods in the MDC class. Developers can store contextual information in a diagnostic context that can later be retrieved by logback components.
package org.slf4j;
public class MDC {
public static void put(String key, String val);
public static String get(String key);
public static void remove(String key);
public static void clear();
}Why is MDC important ?
Suppose you need to include context information to the log files that you're shipping. Without MDC you need to pass the context into every log method invocation, in order to display it in the logs.
In MDC, the context is passed into a map structure, and in logFormatter, you just call the context key to get the context value.
MDC.put("transaction-id", "abcd");
MDC.put("userId", "1234");For referring to the MDC context keys, we use the %X specifier that is used to print the current thread’s Mapped Diagnostic Context (MDC).
- Use
%Xto include the full contents of the Map. - Use
%X{key}to include the specified key.
As an example, we can refer to the transaction-id and sessionId keys created in the first section. Every log message will be appended with MDC information during application runtime.
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<layout>
<Pattern>%d{DATE} %p %X{transaction-id} %X{userId} %c - %m%n</Pattern>
</layout>
</appender>How does MDC works in Java ?
Java uses local storage in threads to store context values for every request that comes into the application, this local storage is known as ThreadLocal.
Prior to logging a message, the application will put context data into the MDC. In this case, MDC referred to a utility class that had MDCAdapter as an abstraction.
MDCAdapter on Java was implemented by :
- Slf4j (BasicMDCAdapter, NOPMDCAdapter)
- Logback (LogbackMDCAdapter)
MDCAdapter would save the information in ThreadLocal
What is ThreadLocal ?
We can write variables to this local storage and read them only within the same thread. Each thread will not see any modifications to ThreadLocal variable done by the other thread if two threads access the same code with a reference to the same ThreadLocal variable.
Let's take an example,
public class DemoRunnable implements Runnable {
private ThreadLocal<Integer> threadLocal = new ThreadLocal<Integer>();
@Override
public void run() {
threadLocal.set((Math.random() + 10));
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
}
System.out.println(threadLocal.get());
}
}public class Example {
public static void main(String[] args) {
DemoRunnable runnable = new DemoRunnable();
Thread thread1 = new Thread(runnable);
Thread thread2 = new Thread(runnable);
thread1.start();
thread2.start();
thread1.join();
thread2.join();
}
}This example creates a single DemoRunnable instance which is passed to two different threads. Both threads execute the run() method, and thus sets different values on the ThreadLocal instance. If the access to the set() call had been synchronized, and it had not been a ThreadLocal object, the second thread would have overridden the value set by the first thread.
However, since it is a ThreadLocal object, the two threads cannot see each other's values. Thus, they set and get different values.
How to display information context on Golang
In Java, ThreadLocal is a map-like structure that holds thread information, and MDC used ThreadLocal here.
But in Golang we don't know about the definition of ThreadLocal or any other local storage in this case inside a Goroutine.
To display information from the context we need to add it into a method arguments.
i.e :
This code below intends to display transaction-id and the amount for that transaction. We must pass context.Context to method arguments to get the context information of the request.
func getTransaction(amount float64, ctx *context.Context) {
logger.GetLogger().Info(fmt.Sprintf("Transaction-Id : %v - amount %v", ctx.Request.Header.Get("Transaction-Id"), amount))
//TODO BUSINESS LOGIC
}How does Halodoc implement MDC in Golang ?
To tackle these problems, Halodoc created a self-local storage based on map data structure.
Here MDC acts as a gateway between MDCAdapter and map structure to put or remove any key attribute inside map.
To handle value replacement problem inside map, we use gouroutine id as a differentiator among the same key.
i.e :
we want to store transaction-id inside a map for goroutine 1.
This it seems to be okay until goroutine 2 come in place and do it the same thing to store transaction-id inside map
Here value from goroutine 1 will be replaced by value from goroutine 2, it'll not become a problem if goroutine 1 had finished the process before goroutine 2 got executed.
But on a concurrent request system in application nowadays this problem can give misleading information.
To give unique key for every goroutine, we combine the key with the goroutine id
i.e :
instead we save the key as transaction-id, we can save as transaction-id-1, which 1 was goroutine id
This solution can give isolation level attributes between goroutine, so between goroutine-1 and goroutine-2 will not interfere with each other attribute value.
Conclusion
In Golang, there is no Threadlocal, so if we want to provide context information with each request, we mush pass the Context with every function method argument. However, this is cumbersome to implement in code.
In order to handle these problems, Halodoc uses a local map data structure and is able to store every request data based on the Goroutine id and the key we expect to store, so there is no need to pass context for each method parameter.
We've outlined here on how the MDC capability of Golang provides us seamless ability to inject additional context information into the logs without having to churn a lot of code for achieving the same. This has helped us to quick and effective debugging and also capture additional context information through logging.
Join Us
Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for data engineers/architects and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke.
We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 1500+ pharmacies in 50 cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allows patients to book a doctor appointment inside our application.
We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates foundation, Singtel, UOB Ventures, Allianz, Gojek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission.
Our team work tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.