Transporter - Centralised Logging System Improvement

Halodoc being at the forefront of modern healthcare technology, the significance of efficient log management cannot be overstated. In our pursuit of delivering exceptional healthcare experiences, we recognise that well-organised logs are integral to maintaining the integrity and performance of our systems. These records serve as a digital trail, guiding us through the intricacies of our applications and  helps for debugging purpose and also to meet the users' expectations.

What is Transporter?

Transporter, which is Halodoc's own implementation of the centralised logging system, consolidates all of the log data and pushes it to one central, accessible, and easy-to-use interface. Centralised logging is designed to make the life of engineers easier. Not only does it provide multiple features that allow you to easily collect log information, but it also helps you consolidate, analyse, and view that information quickly and clearly. No personal information are logged and no secret information is shared/stored.

Log Segregation

Logs refer to the recorded information, events, actions, errors and activities that occur within an application during its runtime. These logs are a way for developers and system administrators to monitor and troubleshoot the behaviour of an application. Logs are typically text-based records of events, and they can provide valuable insights into how an application is performing and interacting with its environment.

There are four types of logs that are maintained, namely:

1. Error Logs - Error logs provide crucial insights into app malfunctions and pinpointing the root causes behind glitches or crashes, aiding developers in swift issue resolution.
2. Info Logs - Info logs offer valuable context by capturing user interactions within the app, enabling a comprehensive understanding of user behaviour and aiding in optimising user experiences.
3. Debug Logs - Debug logs serve as a lifeline for developers, furnishing detailed step-by-step information during app execution, aiding in the identification and elimination of bugs and anomalies for seamless app performance.
4. Warn Logs - Warn logs highlight potential issues that may not necessarily lead to errors but require attention. These logs provide proactive insights, alerting developers to situations that could impact app functionality or user experience if left unaddressed.

How it works?

In its early stages, all network logs generated by the application, regardless of their specific types, were consolidated and stored within a single file. This file underwent a cyclical replacement process every 30 minutes, wherein the existing file was replaced with a new file bearing the same name. This systematic file rotation mechanism ensured efficient log management, enabling the continuous capture and store information for subsequent analysis and troubleshooting purposes.

Logging Enabled and Log Type Enabled are described under How we optimised the log reporting? section.

Transporter Instance: The entry point for any log to be printed.

Recorder: Records events in the storage system. Storage can be either file or DB. Each event type will have its own file and package, will have a file to store the recording info. It will be an abstraction over the underlying storage.

Filters: Applied while storing the logs and dispatching them. Logs can be filtered based on the configuration of the log type.

Enrichers: Applied before the recording phase to store some event info. It will also be applied during the dispatch phase to add info to the recorded logs. Each event type may have its own enrichers.

Dispatcher: Dispatcher will be HTTP layer for uploading logs to server.

Challenges

1. Imposed unwarranted strain on our logging backend service, because post debugging, it was still logging huge amount of data that were required during debugging.
2. Increased resource consumption impacted the system efficiency.
3. It leads to an more application's and backend's storage cost.

Recognising these challenges, it became evident that a more targeted and selective logging mechanism was essential for optimising performance, reducing resource overhead.

How we optimised the log reporting?

Even though the above method was helpful in many ways to improve user experience, there was a scope for optimising the log reporting to overcome the challenges mentioned above.

1. The changes was made such that, the enabling and disabling of storing the different types of logs was made on the fly without waiting for the app release. So, when there was a need for the specific type of the logs for the specific version of the app, this could be done with no waiting time.
2. Only the required logs will be stored in the files, so that the strain on our logging backend service is less. Hence contributing to app size inflation, adding  ~2 MB due to the accumulation logs which is no more required post debugging.
3. We have setup a root level configuration for the entire log level system that can be enabled/disabled based on the requirements.
4. If the root level configuration is enabled, this marks the generic gateway for logs to proceed towards file storage depending on the configuration of sub level configurations.
5. If the root level configuration is disabled, all the logs of the system are excluded from the logging file, effectively negating their storage.
6. This conditional mechanism, forms a pivotal determinant in orchestrating the routing of logs for storage or omission.

What have we achieved?

1. If our log files are growing then our app install size will also increase, so we are getting additional benefits on the install size if we stop logging the information post-debug session.
2. We've lowered storage costs by storing only essential logs for a specific version, rather than retaining all generated logs. This efficient approach helps reduce unnecessary data storage expenses while maintaining the necessary logs for each version.
3. Our approach has cut down API calls for log generation, subsequently lowering unnecessary API expenses. This strategy efficiently manages costs while maintaining essential log data for each version.
4. Disabling all log services post debugging session, we were able to save ~8GB of Backend storage.

Conclusion

The implementation of log storage optimisation not only leads to efficient log management and also contributes to the reduction of server-side load. By storing logs judiciously, the need for frequent server interactions diminishes, resulting in reduced server load. This dual benefit enhances both log handling and the overall performance of the application, leading to a more seamless user experience.

Furthermore, this optimised log management approach significantly improves the process of debugging and error rectification. With streamlined and relevant logs readily available, developers can swiftly pinpoint issues, trace their origins, and implement accurate corrective measures. This targeted approach accelerates the troubleshooting process, leading to faster resolution of errors and ultimately contributing to an overall, more robust and dependable application.

Join Us

Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resume at careers.india@halodoc.com.

About Halodoc

Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and in total have raised around USD$100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.