Scaling to New Heights: Auto-Scaling with Karpenter
At Halodoc we have been using AWS EKS (Managed Kubernetes) to deploy all of the applications. When the application experiences a traffic surge, we use cluster autoscaler with the AWS autoscaling group to scale or downscale the nodes. We accomplish that using a horizontal pod scaler (HPA). This autoscaling group used homogenous type of instances to provision, which mostly end up with fragmentation and over provisioned of the resources. For each product vertical, we have 7 node groups running with comparable instance types to provision in the AWS EKS cluster so that we can have better visibility on billings with the appropriate tags.
Due to our actions, the majority of our nodes were over provisioned, and AWS EC2 on-demand instances costs us more than 20% of our monthly bill. Even though our applications have been optimised to use the best resources, we noticed that our AWS nodes were over-provisioned. Therefore, we wanted to use different instance types and spot instances in production to lower the overall cost of our servers in the Kubernetes cluster.
Prerequisites for adopting autoscaler in production
Before adopting any new autoscaler in the production workload, we have to take the following things into consideration:
- Autoscaler should have a capability of launching the nodes as quickly as possible.
- Nodes provisioned by autoscaler should have proper tag in it to have better visibility on billing with respect to tag.
- Node provisioning should be according to the request made by application.
- There should not be any extra overhead while managing the tools.
- Upgrading of the node version should be easy, simple and quick.
- Zero downtime for any application while scaling up or down and on rescheduling pods between the nodes.
- Make sure all pods for a service don't get everything deployed on spot instances.
The AWS EKS node group doesn't have the capability of using spot and on-demand instances on the fly, and this also doesn't ensure that only some of the pods in the service go to spot instances.
By analysing different tools, we came across a tool called Karpenter as a autoscaler.
Karpenter overcomes these difficulties with ease. According to the resource requests made by the pods, Karpenter provides instances. Karpenter increases the cost-effectiveness and efficiency of running workloads by continuously monitoring, assessing, provisioning, and removing resources while simultaneously issuing orders to the underlying cloud provider.
Introduction
Karpenter, an open-source project for node provisioning in Kubernetes, is designed to function seamlessly across diverse Kubernetes clusters, whether they operate in various environments such as major cloud providers or on-premises setups. Serving as an inherent cluster autoscaler, Karpenter is constructed on top of AWS as an open-source tool. By intelligently selecting and deploying suitable nodes, Karpenter optimises the Kubernetes infrastructure, dynamically launching the necessary computing resources to effectively manage the cluster's applications. This tool significantly expedites and simplifies the process of provisioning compute resources for Kubernetes clusters.
Karpenter has 3 main components:
- Karpenter as a deployment
- Node Template
- Provisioner
Karpenter as a deployment
Before using Karpenter as an autoscaler, we must deploy a service that will continuously check other applications for workload consolidation or scalability in the Kubernetes cluster. During deployment, this will act as a bridge between your application and a provisioner.
Steps to deploy Karpenter
- Install the crd for the Karpenter deployment
2. Download official helm chart for karpenter in local:
3. Change the values.yaml file according to need such as cluster name, cluster endpoint, defaultInstanceProfile, resource request and limit. Login to EKS cluster and run the below command:
Node Template
A node template is simply a location where all of your configuration, such as ami ids, security groups, IAM roles, subnets, and so on, will be saved.
Example template
Once the template is created you can run following command to apply in Kubernetes cluster
Provisioner
Provisioner will help create a machine with the help of a node template. Provisioner will trigger the Node template and create a node. The provisioner will have all the labels, which will help manage any pod deployment.
Example template
Once the template is created you can run following command to apply in Kubernetes cluster
Configuring Provisioner
- Utilising spot instances:
Add annotation karpenter.sh/capacity-type in the karpenter provisioner to use spot or on-demand nodes. To use spot instances, we can create two provisioners, an on-demand provisioner and a spot provisioner, both of which refer to the same AWSNodeTemplate. Additionally, whenever a new node is created using this provisioner, add the capacity-spread tag with a specific value to each node by using it as a requirement. Once this is complete, the deployment file for your application should have the annotation below.
capacity-spread : This will ensure that all the pods will go to different capacity-spread nodes, which helps you to utilise the spot instance as well.
2. Utilising different instance types:
To use multiple instances for a node, use the karpenter provisioner's karpenter.k8s.aws/instance-size annotation. Once application deployment has occurred, nodes will be provisioned in accordance with the resource request specified in the deployment file. As a result, there will not be much resource over-provisioning. Additionally, Karpenter will keep an eye on the nodes and swap out expensive nodes for less expensive ones.
3. Upgrading of the nodes:
In the Karpenter provisioner, we can use ttlSecondsUntilExpired to upgrade each node. All nodes that are older than the specified value are rotated using this field. The result is that every node this provisioner creates will be replaced with a new node after 30 days if the value is 2592000 (60 * 60 * 24 * 30 Seconds). Also, the Pod Disruption Budget (PDB) may have an effect on this upgrade. The rotation of the node will not take place if the PDB for the application is 100%. Therefore, you must ensure that the minimum value that is available is lower than the value that is desired.
Conclusion
Karpenter responds quickly to the changes in application load, scheduling, and resource requirements, placing new workloads on multiple available compute resources which helped us on utilising multiple instances as per our requirement. Karpenter assisted with quicker resource provisioning, better resource allocation, and the cost-effective use of spot instances in the production environment.
After the implementation of Karpenter, the node provisioning time taken from earlier with the node group has reduced by upto 30 seconds. We were able to utilise multiple instance types along with spot instances in production which helped us save almost 20-25% of our EC2 instance cost. Karpenter might not provide immediate benefits to the user who has very small workloads because the same can be easily achieved using AWS Node groups.
Looking at all of the above benefits, I recommend a user who is running stateless application in Kubernetes with huge workload to evaluate Karpenter as an autoscaler to save on-demand cost and reduce creating multiple AWS autoscaling group for each business vertical.
References
- https://aws.amazon.com/blogs/aws/introducing-karpenter-an-open-source-high-performance-kubernetes-cluster-autoscaler/
- https://github.com/aws/karpenter/tree/main/charts
- https://karpenter.sh/
Join us
Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers at all levels and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek, Astra, Temasek, and many more. We recently closed our Series D round and In total have raised around USD$100+ million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.