Skip to main content

Configure Amazon VPC CNI

We'll start by configuring the Amazon VPC CNI. Our VPC has been reconfigured with the addition of a secondary CIDR with the range 100.64.0.0/16:

~$aws ec2 describe-vpcs --vpc-ids $VPC_ID | jq '.Vpcs[0].CidrBlockAssociationSet'
[
  {
    "AssociationId": "vpc-cidr-assoc-0ef3fae4a0abc4a42",
    "CidrBlock": "10.42.0.0/16",
    "CidrBlockState": {
      "State": "associated"
    }
  },
  {
    "AssociationId": "vpc-cidr-assoc-0a6577e1404081aef",
    "CidrBlock": "100.64.0.0/16",
    "CidrBlockState": {
      "State": "associated"
    }
  }
]
This means that we now have a separate CIDR range we can use in addition to the default CIDR range, which in the above output is 10.42.0.0/16. From this new CIDR range we have added 3 new subnets to the VPC which will be used for running our pods:
~$echo "The secondary subnet in AZ $SUBNET_AZ_1 is $SECONDARY_SUBNET_1"
~$echo "The secondary subnet in AZ $SUBNET_AZ_2 is $SECONDARY_SUBNET_2"
~$echo "The secondary subnet in AZ $SUBNET_AZ_3 is $SECONDARY_SUBNET_3"
To enable custom networking we have to set the AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG environment variable to true in the aws-node DaemonSet.
~$kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true
Then we'll create an ENIConfig custom resource for each subnet that pods will be deployed in:
~/environment/eks-workshop/modules/networking/custom-networking/provision/eniconfigs.yaml
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: $(SUBNET_AZ_1)
spec:
  securityGroups:
    - $(EKS_CLUSTER_SECURITY_GROUP_ID)
  subnet: $(SECONDARY_SUBNET_1)
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: $(SUBNET_AZ_2)
spec:
  securityGroups:
    - $(EKS_CLUSTER_SECURITY_GROUP_ID)
  subnet: $(SECONDARY_SUBNET_2)
---
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: $(SUBNET_AZ_3)
spec:
  securityGroups:
    - $(EKS_CLUSTER_SECURITY_GROUP_ID)
  subnet: $(SECONDARY_SUBNET_3)

Let's apply these to our cluster:
~$kubectl apply -k ~/environment/eks-workshop/modules/networking/custom-networking/provision
Confirm that the ENIConfig objects were created:
~$kubectl get ENIConfigs
Finally we'll update the aws-node DaemonSet to automatically apply the ENIConfig for an Availability Zone to any new Amazon EC2 nodes created in the EKS cluster.
~$kubectl set env daemonset aws-node -n kube-system ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone