Implementing RBAC for pods using IAM Roles for Service Accounts
Identity access management is the way in AWS to grant access to AWS services and collect and transmit data and credentials. Before the k8's era we had typical EC2 instances where our micro services were deployed and it required managing dozens of IAM policies designed to isolate access to sensitive data to only services that needed it.
When we migrated our services to managed kubernetes it was very difficult to manage permissions to AWS services. We had to follow the traditional way by having key based authentication for AWS services or we had to use kube2iam and KIAM solutions where both these solutions had their own challenges.
This blog describes what is RBAC and need of RBAC and best practices followed to implement role based access for pods using IAM roles for Service accounts
What is RBAC ?
Role-based access control (RBAC) is a method of regulating access to a computer or network resources based on the roles of individual users within the organisation.
RBAC authorization uses the rbac.authorization.k8s.io API group to drive authorization decisions, allowing you to dynamically configure policies through the Kubernetes API.
When creating a kubernetes cluster you can enable service account token volume projection to enhance security when we use a service account. This feature enables the kubelet to request and store the token on behalf of the pod, As the token approaches the end of its validity period the kubelet automatically rotates the token if it's older than 80 percent or more than 24 hrs. We, at Halodoc are using this feature to access AWS resources such as S3, Dynamo DB, SNS and other AWS resources through a fine grained permissions.
What is the need of RBAC ?
By having RBAC implemented in an organisation, we will have defined set of permission to users, with respect to EKS hosted cluster AWS provides a native solution called IAM roles for service account.
Advantages of having RBAC implemented is as follows
- Avoids the hardcoding any access keys and secret keys in our application properties.
- Secure way to access AWS services since the STS token generated expires every 24 hrs and new token gets generated.
- Least privilege: By using RBAC with service accounts there is no longer need of providing extended permissions on worker node IAM role. You can scope IAM permissions to service accounts only to particular pods who access the service accounts has those permissions. This feature also eliminates the need for third-party solutions such as kiam or kube2iam.
- Credential isolation: A container can only retrieve credentials for the IAM role that is associated with the service account to which it belongs. A container never has access to credentials that are intended for another container that belongs to another pod.
Disadvantages of using Access Keys over RBAC
- Access keys must be rotated in regular interval to maintain compliance
- Access Keys may get compromised easily
- Maintaining the access keys in secure way like vault involves much mechanism
- Access keys doesn’t have expiration until unless its rotated manually.
Background Information:
From Kubernetes version 1.12 projectedserviceaccounttoken feature was added which is an OIDC JSON web token which also includes service account identity information and supports configurable audience.
OIDC federation access allows it to assume IAM roles through STS secure token service. This enables authentication with an OIDC provider by receiving JSON web tokens which in turn can be used to assume an IAM role. Kubernetes on the other hand can issue so-called projected service account tokens which happen to be valid OIDC JWTs for pods. Our setup equips each pod with a cryptographically-signed token that can be verified by STS against the OIDC provider of your choice to establish the pod’s identity.
Workflow of IAM role for Service Accounts:
- When you launch an application on kubernetes with kubectl apply -f application-job.yaml, the yaml manifest is submitted to the API server with the Amazon EKS Pod Identity webhook configured.
- Kubernetes uses the service account set via serviceAccountName
- Since the service account has the annotation passed "eks.amazonaws.com/role-arn" in serviceaccount.yaml the webhook injects the necessary environment variables (AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN) and sets up aws-iam-token projected volumes.
- When application calls out to s3 to do any s3 operations the AWS SDK we use in the application code base performs STS assume role with web identity performs assume role that has s3 permissions attached. It receives temporary credentials that it uses to complete the S3 operation.
Preparation and setup of RBAC:
- Get the Kubernetes cluster version using
Kubectl version -short - Steps to configure OIDC provider if not created during cluster creation
a. Create an IAM OIDC identity provider.
b. Navigate to IAM console
c. Choose Identity Providers and then choose Create Provider
d. Select OpenID Connect for provider type
e. Under Audience type sts.amazonaws.com
f. Verify and create the provider - Creating OIDC provider using AWS CLI.
eksctl utils associate-iam-oidc-provider --name demo --region ap-southeast-1 --approve
The above command sets up OIDC provider ID for the cluster name demo in AWS Singapore region
4. Retrieve Open ID connect URL (OIDC) by below command.
aws eks describe-cluster --name demo --query cluster.identity.oidc.issuer --output text
5. Create IAM policy with required permissions for example i’m creating S3 list policy Create an IAM role and attach the policy created and add the trust policy as shown below
6. Create a Service account and associate the IAM role with the following annotation eks.amazonaws.com/role-arn
7. Now create a deployment file as shown in the below image and deploy using
kubectl apply -f app-deploy.yaml -n namespace
Supported SDK Version:
Verifying Role based access:
You can verify role based access from the below command where IAM role attached to your service account will be injected as environmental variable to a pod
Kubectl describe pod -- pod-name | grep aws
Example Output:
AWS_ROLE_ARN=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>AWS_WEB_IDENTITY_TOKEN_FILE/var/run/secrets/eks.amazonaws.com/serviceaccount/token
Service account token will be mounted in the below path
/var/run/secrets/eks.amazonaws.com/serviceaccount/
References:
Conclusion:
In this article, we have explained how to use RBAC for implementing IAM roles for service accounts in the EKS cluster to provide fine-grained permissions to pods and access AWS API securely.
Join us
Scalability, reliability and maintainability are the three pillars that govern what we build at Halodoc Tech. We are actively looking for engineers/architects and if solving hard problems with challenging requirements is your forte, please reach out to us with your resumé at careers.india@halodoc.com.
About Halodoc
Halodoc is the number 1 all around Healthcare application in Indonesia. Our mission is to simplify and bring quality healthcare across Indonesia, from Sabang to Merauke. We connect 20,000+ doctors with patients in need through our Tele-consultation service. We partner with 3500+ pharmacies in 100+ cities to bring medicine to your doorstep. We've also partnered with Indonesia's largest lab provider to provide lab home services, and to top it off we have recently launched a premium appointment service that partners with 500+ hospitals that allow patients to book a doctor appointment inside our application. We are extremely fortunate to be trusted by our investors, such as the Bill & Melinda Gates Foundation, Singtel, UOB Ventures, Allianz, GoJek and many more. We recently closed our Series B round and In total have raised USD$100million for our mission. Our team works tirelessly to make sure that we create the best healthcare solution personalised for all of our patient's needs, and are continuously on a path to simplify healthcare for Indonesia.