Connect to an EKS Private Endpoint with AWS ClientVPN

TABLE OF CONTENT

1. Introduction
2. AWS Elastic Kubernetes Service
3. EKS cluster Endpoint Access
4. Setting up the Kubernetes cluster
5. What is a VPN?
6. Difference between remote Access VPN and site-to-site VPN
7. Difference between a VPN server and a VPN client
8. AWS Client VPN
9. AWS Client VPN Endpoint
10. Summary of the Process
11. EKS Private Endpoint
12. Advantages of AWS Client VPN Service over running your own VPN server
13. Limitations of AWS Client VPN Service
14. RESOURCES

Introduction

When an EKS cluster is created, by default public access is enabled which means the Kubernetes API server is accessible from the internet. Although several actions could be taken to ensure the security of the cluster while it is public, restricting access to only private networks can be useful in situations where you want to add an extra layer of security.

In this article, we will go through how we can enable private access for our EKS cluster and access the API server through the use of an AWS-managed VPN service called AWS client VPN.

AWS Elastic Kubernetes Service

AWS offers a managed Kubernetes service called Amazon Elastic Kubernetes Service (EKS). EKS makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS. With EKS, you don’t need to worry about setting up and maintaining the underlying infrastructure, as AWS handles the provisioning and scaling of the control plane and worker nodes for you.

An EKS cluster consists of two VPCs: an AWS-managed VPC that hosts the Kubernetes control plane and a second customer-managed VPC that hosts the Kubernetes worker nodes where containers run.

The control plane consists of three master nodes each running in different availability zones to achieve high availability. These nodes are created in an AWS-managed account and so you won’t see them or be able to access them when your cluster is created.

The control plane provides an API Server endpoint that you use to communicate with your cluster.

All worker nodes need the ability to connect to the managed API server endpoint. This connection allows the worker node to register itself with the Kubernetes control plane and to receive requests to run application pods.

The worker nodes connect either to the public endpoint, or through the EKS-managed elastic network interfaces (ENIs) that are placed in the subnets that you provide when you create the cluster. The route that worker nodes take to connect is determined by whether you have enabled or disabled the private endpoint for your cluster. Even when the private endpoint is disabled, EKS still provisions ENIs to allow for actions that originate from the Kubernetes API server, such as kubectl exec and logs.

Source: De-mystifying cluster networking for Amazon EKS worker nodes | Containers

EKS Cluster Endpoint Access

What is the cluster API endpoint?

A cluster endpoint is a network interface that is used to access the Kubernetes API server of an EKS cluster. When you create a new cluster, Amazon EKS creates an endpoint for the managed Kubernetes API server that you use to communicate with your cluster using Kubernetes management tools such as kubectl.

EKS cluster endpoint access simply refers to the reachability of the endpoint and how it can be accessed by users or networks either publicly or privately.

Types of API Server Endpoint Access

1. Public Access: When public access is enabled, the API server endpoint is public to the internet. While it is accessible from the internet, access to the Kubernetes cluster endpoint is restricted by AWS Identity and Access Management (IAM) and Kubernetes role-based access control (RBAC) policies. It also allows for the specification of CIDR blocks that can access the endpoint.

When you choose to use Public access for your API server endpoint, EKS service creates a public endpoint for your API server, and therefore your EC2 worker nodes will communicate with your control plane using the Public endpoint. However, some of the pods running in your cluster (e.g. pods in kube-system namespace) will need to talk to the API server. They will do so by connecting to the IP addresses of the EKS Managed ENIs.

2. Private Access: This provides a private connection to the EKS cluster, the cluster and its resources are not accessible from the internet and all the communication between your nodes and the API Server stays within your VPC meaning that Kubernetes API requests to the API server that originates from within your cluster’s VPC use the private endpoint.

The cluster’s API server private endpoint is resolved by public DNS servers to private IP addresses gotten from the VPC CIDR range. I will show how this works using the dig command later in the article.

Any kubectl commands must come from within the VPC or a connected network.

Finally, a cluster that has been configured to only allow private access can only be accessed from the following:

• The VPC where the worker nodes reside
• Networks that have been peered with that VPC
• A network that has been connected to AWS through AWS Direct Connect (DX) or a virtual private network (VPN)

3. Public and Private Access: Kubernetes API requests within your cluster’s VPC (such as node-to-control plane communication) use the private endpoint.

While the cluster API server endpoint is accessible through the internet because it uses the public endpoint and you also have the option to restrict access to specific IP ranges.

Setting up the Kubernetes Cluster

First, we would need to set up the VPC and other AWS resources for the EKS cluster and we will be using a pre-built cloudFormation template.

Note: If you decide not to use the cloudformation template to set up the VPC for the cluster, when selecting CIDR ranges for your VPC only pick CIDRs in the Private IP range.

For more information on creating Public and Private subnets for an EKS Cluster, check out this link Creating a VPC for your Amazon EKS cluster — Amazon EKS

This template will help us create the required VPC resources for our cluster including the VPC, subnets, default security groups, route tables, routes, internet, and NAT gateways.

Most of the steps here will be done using a combination of the AWS CLI and the AWS MANAGEMENT CONSOLE.

Step 1: Navigate to the Aws management console >> CloudFormation >> Create Stack ,With new resources (standard)

Step 2: Create Stack, pick “template is ready” and then “Amazon S3 URL” as the template source. Copy and paste in the URL below.

https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/amazon-eks-vpc-private-subnets.yaml

Next, give the stack a name, I have given mine the name,EKS-VPC. As you can see the Worker network configuration has been taken care of and the VPC has a CIDR range of 192.168.0.0/16. Two public subnets and two private subnets will also be created with their respective CIDRs.

For Configure stack options, you can specify tags for the stack to make it easier to identify the resources that are created using this template.

Step 3: Create the EKS Cluster IAM Role. Navigate to Identity and Access Management(IAM) on the management console >> Create role

Select Aws Service as the Trusted Identity type

For the Use Case, Search for EKS and select EKS-cluster, then click Next

Add Permissions to the role, search for AmazonEKSClusterPolicy , then click Next

Finally, give the Role a name,EKS-Cluster-Role >> Create Role

Step 4: Create the EKS Cluster

Navigate to Elastic Kubernetes Service on the management console >> Add Cluster >> Create

Put in a name for your cluster, I’m using Demo-Cluster as mine. For the cluster service role select the IAM role that we just created >> Next.

In the specify networking step, under VPC, pick the VPC that was created in the previous step and select the four subnets in the VPC.

Leave the security group blank, this is because when an EKS cluster is created a default security group is created. We will make changes to the security group later.

For the cluster Endpoint access, we will pick private access. We’ll be able to access the cluster using a connected network and in this case it will be a Virtual Private Network(VPN)

You can leave the remaining steps as the default and move on to Review and create the cluster.

Step 5: Create a Node group. Now we have to create the nodes in the cluster. We will start by creating the Node group IAM role.

Navigate to IAM in the management console >> Create Role. Select Aws service as the Trusted entity type and pick EC2 as the use case >> Next.

Add permissions, AmazonEKSWorkerNodePolicy , AmazonEKS_CNI_Policy ,AmazonEC2ContainerRegistryReadOnly,Search for these permissions and select them >> Next

Give the role a name, in my case its EKS-Node-Role , Review and create the role.

Now navigate back to EKS on the management console, select the previously created cluster(Demo-cluster) >> Compute >> Add Node group

Give the node group a name Demo-Node-group and for the Node IAM role select the IAM role we just created.

In the compute and scaling configuration step we can leave everything as the default. Specify 2 as the desired, minimum and maximum node sizes if it's not specified that way already >> Next.

For the specify networking step, you’ll be required to pick the subnets where the nodes will reside. We want just the private subnets. This is because we want the resources in these nodes to not be accessible to the internet.

Now that we have created our cluster and configured our node groups. we need to set up a Virtual Private Network(VPN) so that we can access the private endpoint of our cluster from our computers.

What is a VPN?

VPN stands for Virtual Private Network and it is a technology that allows for the creation of a secure, encrypted connection over a public or shared network, such as the internet, making it possible for devices to communicate as if they were on the same private network. This can help keep sensitive data safe from unauthorized access and eavesdropping. In this case, users can access the resources in our private subnet remotely as if they were directly connected to the network. IPsec VPN (Internet Protocol Security) and SSL/TLS (Secure Socket Layer/Transport Layer Security) VPN are the two major VPN technologies.

Difference between Remote Access VPN and Site-to-Site VPN

A Remote Access VPN allows individual users to securely connect to a private network remotely, such as at home or while traveling. Remote access VPNs typically use software on a user’s device, such as a laptop or smartphone, to establish a secure connection with the private network.

A Site-to-Site VPN, on the other hand, connects entire networks to each other, allowing them to communicate as if they were on the same local network. Site-to-site VPNs typically use specialized hardware, such as a VPN gateway, to establish a secure connection between the networks. This type of VPN is commonly used by organizations that have multiple office locations or want to connect their cloud infrastructure to their on-premises data centers.

In summary, Remote Access VPNs are used for individual users to access a private network from remote locations, while Site-to-Site VPNs connect entire networks together.

Difference between a VPN Server and a VPN Client

A VPN server is a device or service that provides a secure connection to a remote user or network. The VPN server acts as the gateway to the private network, and it is responsible for authenticating incoming VPN connections, creating and managing the secure tunnel, and providing access to the private network resources.

A VPN client, on the other hand, is the device or software used by the user to establish a VPN connection to the VPN server. The client is responsible for initiating the VPN connection, authenticating with the VPN server, and maintaining the secure tunnel. VPN clients are typically software applications that run on a user’s device, such as a laptop, smartphone, or tablet. They are configured with the necessary information, such as the VPN server’s IP address and security settings, to establish a connection.

In summary, the VPN server is the device or service that provides the secure connection and manages the VPN network, while the VPN client is the device or software used to connect to the VPN server and access the private network.

In this case, the VPN connection is the connection between your VPC and your on-premise network/client device.

Aws offers two VPN services, AWS Client VPN (Remote access VPN) and AWS Site-to-Site VPN(site-to-site VPN).

In this article, we will focus on just the AWS Client VPN to establish a VPN connection between our devices(laptops) and the resources in our VPC.

AWS Client VPN

AWS Client VPN is a managed client-based OpenVPN Service that enables you to securely access your private AWS resources remotely using their Private IPs and hostnames.

Let’s say you have a computer like Windows/MAC/Linux, you can set up a client connection from there into the VPC and you can communicate with the resources in that VPC as if you were in the same network using AWS client VPN.

The AWS client VPN service runs the VPN server/protocol while your client computer will have a VPN client installed, but before a VPN connection is established between the server and the client, there needs to be a form of authentication.

There are a couple of options for authentication with the AWS Client VPN including;

1. Mutual Authentication using Certificates
2. Integrate a Directory service like AWS Directory Service

Mutual Authentication

Mutual authentication is a security feature that allows both the client and the server to authenticate each other’s identity before a connection is established.

When mutual authentication is used, the client must provide a valid certificate to the server during the initial connection. The server, in turn, must also provide a valid certificate to the client. The client then verifies the server’s certificate and establishes a secure connection only if the server’s certificate is valid and trusted.

This way, the VPN server can be sure that the client has the right credentials to connect to the network and the client can be sure that is connecting to the right VPN server.

We will use Mutual Authentication so let’s start by generating the necessary certificates.

Generate the Certificates

We will use EasyRSA to generate the server and client certificates and keys and upload them to AWS Certificate Manager.

You can follow these steps(Client authentication — AWS Client VPN (amazon.com)) in the AWS documentation to generate the certificates but I will still go through them.

Step 1: Download EasyRSA EasyRSA Release, and unzip it. I downloaded the windows version.

Step 2: Open a command prompt on your computer and navigate to the location where the EasyRSA-3.x folder was extracted.

cd EasyRSA-3.1.1/

Create a new PKI environment

./easyrsa init-pki

Build a new Certificate Authority(CA)

./easyrsa build-ca nopass

Create the server key and certificate

./easyrsa build-server-full server nopass

Create the client key and certificate and finally exit from the environment

./easyrsa build-client-full client1.domain.tld nopass

exit 

Now we have generated the keys and certificates, let's upload them to ACM.

Upload Certificates and Keys to ACM

We will upload these to ACM using the AWS CLI, so you have to make sure that your AWS Credentials(Access key, secret access key, region) are properly configured. You can confirm this by running the command below on your terminal.

aws configure

Create a new directory and copy the generated certificates and keys into that folder and cd into it. I created a folder named certs in my current working directory.

Then we copy the needed files into the folder to make it easier for us to upload them to ACM.

mkdir certs 
cp pki/ca.crt certs
cp pki/issued/server.crt certs
cp pki/private/server.key certs
cp pki/issued/client1.domain.tld.crt certs
cp pki/private/client1.domain.tld.key certs
cd certs

Upload the Server certificates and keys to ACM using the command below

aws acm import-certificate --certificate fileb://server.crt --private-key fileb://server.key --certificate-chain fileb://ca.crt

Upload the Client certificates and keys to ACM

aws acm import-certificate --certificate fileb://client1.domain.tld.crt --private-key fileb://client1.domain.tld.key --certificate-chain fileb://ca.crt

You can navigate to your AWS management console to confirm that they’ve been uploaded correctly.

AWS Client VPN Endpoint

We have already discussed the differences between a VPN server and a VPN client and we know that AWS CLIENT VPN runs the VPN server. But what is a VPN Endpoint?

A VPN endpoint is simply the VPN server that is located on the private network and is responsible for receiving and processing VPN connections.

The endpoint is an important part of the VPN connection, as it is the point of entry and exit for all traffic that travels over the VPN. It is responsible for encrypting and decrypting data, as well as authenticating users and devices that are trying to connect to the VPN.

Create the VPN endpoint

Navigate to the VPC section on the Management console >>Client VPN endpoints >> Create Client VPN endpoints

Give your VPN endpoint a name and also assign a CIDR block for the endpoint. When a client/user connects to the VPN it will be assigned an IP address from this CIDR range. I gave mine a CIDR range of 10.6.0.0/16 .

Select Mutual Authentication for Authentication Information, then select the server and client certificates we uploaded to ACM earlier.

Leave the Transport protocol as UDP and Enable Split Tunneling

Split tunneling: This is enabled when you don’t want to route internet traffic from your users over the VPN connection. Only the traffic that’s destined for your VPC gets routed over the VPN connection.

You can leave the rest of the configuration as the default and click Create client VPN endpoint.

You will see that it shows pending-associate, we have to associate the target networks which the VPN will be able to access.

Go to Target network associations >> Associate target network

We will associate the two private subnets in our EKS-VPC where we placed our node groups.

Select the name of the VPC >> select the first private subnet >>click Associate target network.

Repeat this process for the second private subnet.

Now we need to add an Authorization rule. Go to Authorization rules >> Add Authorization rule

Authorization Rule: This is where you can set the destination networks that you want to allow your client that is connected to the VPN to have access to. We will specify that the client VPN Endpoint is only authorized to serve requests whose destination lies within your VPC CIDR range(192.168.0.0/16) where we have deployed our EKS cluster.

Specify the VPC CIDR range, and select Allow access to all users >> Click Add authorization rule.

Next, download the Client Configuration File. Users use their VPN Client and their VPN configuration file to connect with the VPC and access internal applications in private subnets.

After it's been downloaded, you can open it with notepad or any other text editor and add the following lines to it between cipher AES-256-GCM and verb 3.

Be sure to specify the correct directory where the client cert and key were created.

--cert "{folder-path}\\EasyRSA-3.1.1\\pki\\issued\\client1.domain.tld.crt"
--key "{folder-path}\\EasyRSA-3.1.1\\pki\\private\\client1.domain.tld.key"

We’ve successfully set up the VPN endpoint/server now let's move on to setting up the VPN client on our device.

Download a VPN client

There are different options for VPN clients including AWS VPN client or Open VPN client.

I will use the AWS VPN client.

After it's downloaded, open it, then click on File >> Manage profiles >> Type in a display name.

Click on the folder icon and search for the client config file you just downloaded >> Add Profile >> Connect.

We are now connected to the VPN!

Connect to your EKS cluster

Make sure you have kubectl installed. (I am running this on an Ubuntu WSL on windows)

Specify your region code and the name of your cluster.

aws eks update-kubeconfig --region <region-code> --name <my-cluster>


#output
Added new context arn:aws:eks:<region-code>############::cluster/<my-cluster> to ~/.kube/config

kubectl config set-context <my-cluster>

#Output
Context "<my-cluster>" created

If you run Kubectl get nodes, you will see that it won't work, this is because we haven't configured the security group of the cluster to allow access to the VPN.

To make the changes to the security group, navigate to your EKS cluster on the management console, under Networking you will see the security group link, click on it. Next click on Edit Inbound rules, click on “Add Rule” and add a custom TCP rule that opens port 443 with a source of the VPC CIDR >> Save Rule

Now try kubectl get nodeagain and you will see that it works.

kubectl get node 
NAME                              STATUS   ROLES    AGE   VERSION
ip-192-168-179-187.ec2.internal   Ready    <none>   17m   v1.24.7-eks-fb459a0
ip-192-168-204-148.ec2.internal   Ready    <none>   17m   v1.24.7-eks-fb459a0

We have successfully connected to our EKS cluster through the private endpoint using a VPN!

Summary of the process

1. We create a VPC with two public and private subnets. we create an EKS cluster in the VPC and add the four subnets to the VPC.
2. We create a Node group where we will run our applications in just the two private subnets.
3. We create a VPN Endpoint which will be associated with the private subnets.
4. A VPN Elastic network interface(ENI) is created in the subnet and this is what enables the VPN connection to communicate with resources in the subnet.
5. We then have the client computer that will be running some VPN client software e.g openVPN, Aws Client VPN
6. The VPN client will establish a connection with the VPN endpoint over SSL/TLS(443) and that will be via the internet.
7. The VPN Endpoint will perform SNAT(Source Network Address Translation) from the CIDR Block Associated with the VPN Endpoint (10.6.0.0/16) to the CIDR block that is associated with the EKS VPC(192.168.0.0/16) and this is how the client would have access to the VPC.
8. Now the user will be able to access the API server private endpoint.

EKS Private Endpoint

Earlier it was mentioned that the cluster’s API server private endpoint is resolved by public DNS servers/public endpoint to private IP addresses gotten from the VPC CIDR range. Since the resolved addresses are always private IPs, clients without access to the private VPC may resolve the IP but are unable to connect to the cluster.

To show this, you run the dig command for the API Server endpoint.

To get the API server endpoint, Navigate to your EKS cluster on the console, in the overview section, you’ll see it there.

dig ###########################.gr7.us-east-1.eks.amazonaws.com


# output 
; <<>> DiG 9.16.1-Ubuntu <<>> ###########################.gr7.us-east-1.eks.amazonaws.co
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25785
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;###########################.gr7.us-east-1.eks.amazonaws.com. IN A

;; ANSWER SECTION:
###########################.gr7.us-east-1.eks.amazonaws.com. 0 IN A 192.168.149.177
###########################.gr7.us-east-1.eks.amazonaws.com. 0 IN A 192.168.89.243

;; Query time: 530 msec
;; SERVER: 172.29.0.1#53(172.29.0.1)
;; WHEN: Wed Jan 18 00:10:43 WAT 2023
;; MSG SIZE  rcvd: 178

You can see the endpoint got resolved to private addresses and this is because private access was enabled for the cluster.

Advantages of using AWS Client VPN service over running your own VPN Server

1. AWS client VPN is managed service and removes the overhead of installing your own VPN server.
2. It is highly scalable and scales based on the number of user connections to the VPN Endpoint.
3. It supports both AD authentication and certificate-based authentication
4. Granular Access Control, you can control access to specific networks using authorization rules and security groups
5. It allows split tunneling which basically allows only AWS-specific traffic through the VPN connection and all other internet-bound traffic will flow like normal and won’t use the VPN connection.

Limitations of AWS Client VPN service

1. Client CIDR and VPC CIDRs must not overlap.
2. VPN endpoint and associated VPC must be in the same AWS account
3. Associated Subnets must be in the same VPC.
4. We cannot associate multiple subnets from the same availability zone
5. Client VPN supports only IPV4 traffic

RESOURCES

1. Accessing VPC Resources Using AWS Client VPN | by Eric Tsang | Dev Genius
2. Network — EKS Best Practices Guides (aws.github.io)
3. De-mystifying cluster networking for Amazon EKS worker nodes | Containers
4. Amazon EKS cluster endpoint access control — Amazon EKS