In-House Security: Protecting Microservices with Fine-Grained Access Control at Halodoc

Halodoc, Indonesia's top healthcare platform, understands the critical importance of trust in healthcare. The amount of data generated by patients, doctors, laboratories and by our applications is growing exponentially every day. One of the challenges with data of any size is managing who can perform operations on it, or in other words, authorising engineers to access internal patients' data. One of the key ways we achieve this is by adhering to the principle of least privilege or fine grained access to engineers.

Least privilege dictates that, ideally, an engineer should possess only the exact permissions with internal services for troubleshooting and debugging purposes and do their job duties at any given time. This minimises risk, whether protecting from a malicious actor, a compromised device, or even just an honest human mistake.

In this blog, we will explain how Halodoc internally uses a custom solution to provide least privilege to engineers in order to access production services securely.

Context

We use micro service architecture, and service can accept a request only if the request has a valid service authentication token. For developers to communicate with a service for any special use case (example: debugging, data correction, etc.) they have to use a respective service token.
Even though service tokens are rotated periodically, this approach poses a security risk as service tokens are long-lived and can be misused if exposed. Existing architecture had following limitations:

• It was challenging to audit engineers activities on production service API’s.
• We wanted to limit access to critical parts of the production services to only those who truly needed it.
• Engineers had full API access to production services, allowing them to perform CRUD operations as long as they had a service specific token.
• Along with periodic service token rotation, additional token rotation was required if the service token is revealed internally.

Considering existing limitations with the process at that time, we decided that we needed a centralised system to grant limited access to Halodoc services based on developer demand and at the same time audit the access and user activity.

To address security risk, we designed a solution that allows engineers to access production services without knowing individual service tokens and uses Google SSO authentication to access service. This also provides the administrator an audit trail of the engineer's activities on production services and advanced access control features to help provide limited access to individuals to service API endpoints/method.

The Solution

As previously said, administrators should be able to provide least privileged access to individual APIs and an audit trail of engineer activity with the use of our new method for accessing production services. Our new solution is designed with several goals in mind:

• User Friendly & Easy to use: Our developers use tools like curl or Postman to access services. Our new solution should integrate easily with existing procedures, requiring only minor adjustments to how services are accessed
• Centralised Access Control: We wanted a centralised approach for managing permissions (grant/revoke).
• Minimal Infrastructure Change: Our existing microservices architecture utilises service-to-service authentication handled at the service level (over 120 services). A core goal was to implement a solution that wouldn't require service-side code changes or any major changes to our existing infrastructure.
• Approval Process: Before access is granted, each access request must be properly justified by engineer and manually reviewed by an administrator. This improves security and compliance by guaranteeing that only those who actually require access to the services can use them.
• Better Audit log: Detailed logging of engineer activity on production services answers administrators the vital question: "Who is accessing which production service API endpoint using which http method, how many times and when?"
  
  

Architecture

To understand how our access control platform works, it's essential to grasp a few key terms.

1. Service Token: A token which is used for authentication in order to communicate with the service.
   
2. Access Token: A short-lived token generated by Google Single Sign-On (SSO) and used to fetch user information like username and email address.
   
3. Permission Store: We chose a relational database for a secure and centralised way to manage permissions. This permission storage acts as a central hub, storing information like the engineer's email, the service they can access, the specific API methods they're authorised to use, and also regular expressions  for API endpoints.
   
   Regex examples:
   - v1/api/.* : Allows access to endpoints like v1/api/email and v1/api/email/update.
   - v1/api/.*/update : To access v1/api/email/update and v1/api/address/update.

We designed a simple but effective solution which works as a proxy service between Halodoc services and developers,  which do not affect service-to-service communication at all, which effectively fulfils our designed goal “A solution which required minimal infrastructure changes and developer friendly”

Access Controller is the core system in this architecture and it has following components.

Interactive Platform:

This is user-friendly web interface for engineers and administrators to provision fine grained access to engineers to any service, this platform is integrated with Google Single Sign On (SSO) for authentication, enabling seamless login to the platform. Each login session generates a unique temporary access token which is used for authorisation in subsequent requests.

Proxy:

The Access Control Proxy acts as middleware between engineers and our internal Halodoc services. Instead of directly accessing services, engineer send requests to the Access Control Proxy (identified by wildcard DNS *.prod-developer.halodoc). Here's how the proxy performs operation on incoming request:

1. Access Token Authentication: The proxy intercepts the engineer's request and retrieves the access token included in the header Access-Token. Utilising the Google SDK, the proxy then verifies the token's validity and extracts user information like username and email address.
2. Access Validation: If the access token is valid, the proxy analyses the incoming request to determine if the engineer has the necessary permissions. This includes verifying access to the requested service, API endpoint, HTTP method and retention period.
3. Request Modification and Forwarding: Following successful authentication, the proxy creates a replica of the incoming request. The hostname is then replaced with the specific service endpoint the engineer is trying to access. The proxy retrieves the appropriate service token and adds it to the new request header under Service-Token: <service_token>, and finally forwards the modified request to the designated service.
4. Logging: Each request is logged for auditing purposes, including details such as the timestamp, email address, service name, API endpoint, and method.

User Experience

One of the core principles behind our access control platform was to minimise disruption to our engineers workflows and here's how we achieved this:

Familiar Tools with New Approach: Engineers can continue using their preferred tools like curl or Postman to interact with services. The transition process requires only minor adjustments. Instead of using the actual service domain ServiceA.prod.halodoc.com, engineers simply switch to the new proxy domain ServiceA.prod-developer.halodoc.com. Similarly, the service token is replaced with a temporary access token generated by Google SSO.

Example:

By keeping the core interaction pattern familiar, we ensure a smooth adoption, allowing them to quickly adapt to the new access control system without affecting productivity.

Security Enhancement:

Our engineers used to have direct VPN access to Halodoc's internal service subnet. We have blocked direct access to certain internal services since implementing our new solution. Now, engineers access is limited to use ‘access control proxy service’ which serves as a gateway for all internal services when communicating between employees and services. While service-to-service communication continues to function as before.

Conclusion

At Halodoc, we've enhanced the security with a new solution. By adhering to the principle of least privilege, our custom access control platform ensures that engineers access production services with precisely the permissions required for their tasks. Our approach accommodates their preferred tools like curl and Postman with minimal disruption and also seamlessly integrates with our existing infrastructure, avoiding service-side code changes.

Our commitment to balancing developer experience and centralised access control mechanisms have simplified administrators to manage permissions efficiently. Fine grained access control coupled with detailed auditing, gives the administrators an accurate account of user activities. These features effectively strengthen our Halodoc internal security by restricting direct access to internal services.

As we continue to innovate in improving security, Halodoc remains dedicated to ensuring that doctors and patients data remains safe and secure, empowering our team to deliver exceptional healthcare solutions without compromise.

Join us

Scalability, reliability, and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels, and if solving hard problems with challenging requirements is your forte, please reach out to us with your resume at careers.india@halodoc.com.

About Halodoc

Halodoc is the pioneer of a digital health ecosystem with a mission to simplify healthcare access by addressing users’ pain points through accessible, reliable, and quality services. The company’s commitment to promoting wellness is reflected in the comprehensive range of health solutions, covering preventive to curative approaches, all within a single application.Since 2016, Halodoc has been enhancing health literacy in Indonesia through user-friendly healthcare Communication, Education, and Information (KIE). Our expanding ecosystem offers a range of convenient healthcare services, including Home Lab for home care health test; My Insurance for seamless access to cashless outpatient service benefits; Chat with Doctor teleconsultation with 20,000+ licensed doctors and health workers; as well as Health Store for access to purchase medicines, supplements, and various health products from 4,900+ trusted partner pharmacies. Halodoc has been recognized as one of a telehealth platform making a positive impact on the healthcare sector and received the “supervised” status in the Regulatory Sandbox program by the Ministry of Health of the Republic of Indonesia. It reflects a strong commitment and partnership between Halodoc and the MoH, ensuring participatory supervision to safeguard Digital Health Innovation (IDK) organisers, consumers and healthcare workers as the partner of digital innovations. The platform has also received a string of prestigious awards, including being featured on CB Insights’ Digital Health 150 list in 2019–2020 and receiving the Indonesian government’s 2023 PPKM Award. Download Halodoc app via iOS and Android.