Managing EKS Anywhere running on vSphere using Tanzu Mission Control

VMware Tanzu Mission Control™ is a centralized management platform for consistently operating and securing your Kubernetes infrastructure and modern applications across multiple teams and clouds.

In this post, I will demonstrate step by step instructions to deploy and EKS anywhere cluster on vSphere environment, attach it to Tanzu Mission Control and then perform few management function on EKS anywhere cluster using TMC console.

Below is the high level diagram that represent the components we will have eventually.

Pre-requirements

  • Administrative Machine (Linux or Mac)
  • Internet Connectivity from Administrative Machine
  • vSphere environment (in my case, I have vSphere 7.0u3). Check EKS anywhere doc for supported versions.
  • Tanzu Mission Control Access

Setting up EKS Anywhere

In this section, first we need to setup an eksctl cli based utility for deploying an EKS anywhere cluster on vSphere. You can have either Linux or Mac based administrative machine. I am using Ubuntu based administrative machine.

eksctl Installation and Configuration on an Administrative Machines

  • Run the command below to install eksctl
curl "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
    --silent --location \
    | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin/
  • Install the eksctl-anywhere plugin
export EKSA_RELEASE="0.8.2" OS="$(uname -s | tr A-Z a-z)" RELEASE_NUMBER=10
curl "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/${RELEASE_NUMBER}/artifacts/eks-a/v${EKSA_RELEASE}/${OS}/amd64/eksctl-anywhere-v${EKSA_RELEASE}-${OS}-amd64.tar.gz" \
    --silent --location \
    | tar xz ./eksctl-anywhere
sudo mv ./eksctl-anywhere /usr/local/bin/
  • Check the eksctl anywhere installation. Just hit the below command and it should work.
eksctl anywhere

EKS Anywhere Deployment

  • Generate an EKS anywhere deployment configuration file
  • Run the below command to generate an EKS anywhere cluster deployment yaml file
eksctl  anywhere generate clusterconfig demoekscluster -p vsphere > eksonvsphere.yam
  • Edit the Configuration file and update your vSphere environment specific values
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: demoekscluster
spec:
  clusterNetwork:
    cniConfig:
      cilium: {}
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneConfiguration:
    count: 2
    endpoint:
      host: "10.212.177.88"
    machineGroupRef:
      kind: VSphereMachineConfig
      name: demoekscluster-cp
  datacenterRef:
    kind: VSphereDatacenterConfig
    name: demoekscluster
  externalEtcdConfiguration:
    count: 3
    machineGroupRef:
      kind: VSphereMachineConfig
      name: demoekscluster-etcd
  kubernetesVersion: "1.22"
  managementCluster:
    name: demoekscluster
  workerNodeGroupConfigurations:
  - count: 2
    machineGroupRef:
      kind: VSphereMachineConfig
      name: demoekscluster
    name: md-0

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereDatacenterConfig
metadata:
  name: demoekscluster
spec:
  datacenter: "Datacenter"
  insecure: true
  network: "Workload"
  server: "vc-name or ip"
  thumbprint: ""

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig
metadata:
  name: demoekscluster-cp
spec:
  datastore: "LUN01"
  diskGiB: 25
  folder: "eksanywhere"
  memoryMiB: 8192
  numCPUs: 2
  osFamily: bottlerocket
  resourcePool: "ekspool"
  users:
  - name: ec2-user
    sshAuthorizedKeys:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa7jIRv8wIjq+362S8C2pPW1ia0cWrp8/R56oxy/LOtagZ2Hx9BtuEde8QNVPPK80R40gMRdYrlOx33m1DJ0WPeRpY8JzZ2xwEf8RKjijSyKd5XbviAKBLqtH2RUNw0aPpmqy2QSgjF1FdSDbDoSd2Q+LQ3BFdkPmNkDpJ6gaIW1Ei2zQeG0TeJ+VgW1hIHDUQNbKw0tSRkbaCffs/Ublgt6GeGV+CWClh1bZZUsmKUUpMKeUo7wXCQXnQibp22AoZ1I81/v9UntKZkOxv9FjBlyjyM4s2I5OVp64ypdkRtmex77iyl0I92668iSZ7msd8HhBhvSiGMWTy031zdLYVFlg0VeoPVfNABCfgPdHN+vPtdRDM6SJCALSL61IGT/FNk09fvkAS0ARSLxKLaHD2f8eLqFVbRjZ+QpfEJ1NUU2t7q4UEFMyiNvkYMx5JBqsE5++G/th8IKYKHb3jRR0RoSywut/OwLNq02d9dfZZ5U6WqAV13joVdA2GyxuFYIk= root@dt-vc-ubuntu

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig
metadata:
  name: demoekscluster
spec:
  datastore: "LUN01"
  diskGiB: 25
  folder: "eksanywhere"
  memoryMiB: 8192
  numCPUs: 2
  osFamily: bottlerocket
  resourcePool: "ekspool"
  users:
  - name: ec2-user
    sshAuthorizedKeys:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa7jIRv8wIjq+362S8C2pPW1ia0cWrp8/R56oxy/LOtagZ2Hx9BtuEde8QNVPPK80R40gMRdYrlOx33m1DJ0WPeRpY8JzZ2xwEf8RKjijSyKd5XbviAKBLqtH2RUNw0aPpmqy2QSgjF1FdSDbDoSd2Q+LQ3BFdkPmNkDpJ6gaIW1Ei2zQeG0TeJ+VgW1hIHDUQNbKw0tSRkbaCffs/Ublgt6GeGV+CWClh1bZZUsmKUUpMKeUo7wXCQXnQibp22AoZ1I81/v9UntKZkOxv9FjBlyjyM4s2I5OVp64ypdkRtmex77iyl0I92668iSZ7msd8HhBhvSiGMWTy031zdLYVFlg0VeoPVfNABCfgPdHN+vPtdRDM6SJCALSL61IGT/FNk09fvkAS0ARSLxKLaHD2f8eLqFVbRjZ+QpfEJ1NUU2t7q4UEFMyiNvkYMx5JBqsE5++G/th8IKYKHb3jRR0RoSywut/OwLNq02d9dfZZ5U6WqAV13joVdA2GyxuFYIk= root@dt-vc-ubuntu

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: VSphereMachineConfig
metadata:
  name: demoekscluster-etcd
spec:
  datastore: "LUN01"
  diskGiB: 25
  folder: "eksanywhere"
  memoryMiB: 8192
  numCPUs: 2
  osFamily: bottlerocket
  resourcePool: "ekspool"
  users:
  - name: ec2-user
    sshAuthorizedKeys:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDa7jIRv8wIjq+362S8C2pPW1ia0cWrp8/R56oxy/LOtagZ2Hx9BtuEde8QNVPPK80R40gMRdYrlOx33m1DJ0WPeRpY8JzZ2xwEf8RKjijSyKd5XbviAKBLqtH2RUNw0aPpmqy2QSgjF1FdSDbDoSd2Q+LQ3BFdkPmNkDpJ6gaIW1Ei2zQeG0TeJ+VgW1hIHDUQNbKw0tSRkbaCffs/Ublgt6GeGV+CWClh1bZZUsmKUUpMKeUo7wXCQXnQibp22AoZ1I81/v9UntKZkOxv9FjBlyjyM4s2I5OVp64ypdkRtmex77iyl0I92668iSZ7msd8HhBhvSiGMWTy031zdLYVFlg0VeoPVfNABCfgPdHN+vPtdRDM6SJCALSL61IGT/FNk09fvkAS0ARSLxKLaHD2f8eLqFVbRjZ+QpfEJ1NUU2t7q4UEFMyiNvkYMx5JBqsE5++G/th8IKYKHb3jRR0RoSywut/OwLNq02d9dfZZ5U6WqAV13joVdA2GyxuFYIk= root@dt-vc-ubuntu

---
  • Export the vSphere username and password environment variables. Replace your username and password here.
export EKSA_VSPHERE_USERNAME='billy'
export EKSA_VSPHERE_PASSWORD='t0p$ecret'
  • Run the deployment command. It will take approx 15 mins.
eksctl anywhere create cluster -f eksonvsphere.yaml 
Warning: VSphereDatacenterConfig configured in insecure mode
Performing setup and validations
Warning: VSphereDatacenterConfig configured in insecure mode
✅ Connected to server
✅ Authenticated to vSphere
✅ Datacenter validated
✅ Network validated
Creating template. This might take a while.
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Datastore validated
✅ Folder validated
✅ Resource pool validated
✅ Control plane and Workload templates validated
✅ Vsphere Provider setup is valid
✅ Validate certificate for registry mirror
✅ Create preflight validations pass
Creating new bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific post-setup
Creating new workload cluster
Installing networking on workload cluster
Installing storage class on workload cluster
Installing cluster-api providers on workload cluster
Installing EKS-A secrets on workload cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing AddonManager and GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!
  • Now we have successfully deployed and EKS Anywhere cluster on vSphere. It deploys the following vm’s.

Note that it used bottlerocket OS for deploying an EKS cluster. Below is the template that was used. You do not have to import that template in advance as it will be done during deployment.

Attaching EKS Anywhere to TMC via TMC GUI

  • Connect to an EKS anywhere cluster
export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
  • Run the below command and ensure that all pods are in running state
kubectl get po -A
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS       AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-56d9b67869-tzdkw       1/1     Running   0              149m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-69944f65c7-czqvz   1/1     Running   0              149m
capi-system                         capi-controller-manager-5b68598468-bfbdv                         1/1     Running   0              149m
capv-system                         capv-controller-manager-755bb9b895-wh4tf                         1/1     Running   0              149m
cert-manager                        cert-manager-76cd6884cc-4h7qh                                    1/1     Running   0              150m
cert-manager                        cert-manager-cainjector-7df6c99858-272c7                         1/1     Running   0              150m
cert-manager                        cert-manager-webhook-6bfb8f8cb5-4p6st                            1/1     Running   0              150m
eksa-system                         eksa-controller-manager-89b44749d-5khlv                          2/2     Running   0              148m
etcdadm-bootstrap-provider-system   etcdadm-bootstrap-provider-controller-manager-5d787b59-wsjgf     1/1     Running   0              149m
etcdadm-controller-system           etcdadm-controller-controller-manager-8694688769-46tx7           1/1     Running   0              149m
kube-system                         cilium-6qv7q                                                     1/1     Running   0              150m
kube-system                         cilium-76x5n                                                     1/1     Running   0              150m
kube-system                         cilium-gqj26                                                     1/1     Running   0              150m
kube-system                         cilium-operator-7698596ff4-mgknb                                 1/1     Running   0              150m
kube-system                         cilium-operator-7698596ff4-tf2p5                                 1/1     Running   0              150m
kube-system                         cilium-zf6nl                                                     1/1     Running   0              150m
kube-system                         coredns-55b469fd59-smmb9                                         1/1     Running   0              152m
kube-system                         coredns-55b469fd59-v2sst                                         1/1     Running   0              152m
kube-system                         kube-apiserver-10.212.177.229                                    1/1     Running   0              152m
kube-system                         kube-apiserver-10.212.177.232                                    1/1     Running   0              150m
kube-system                         kube-controller-manager-10.212.177.229                           1/1     Running   0              152m
kube-system                         kube-controller-manager-10.212.177.232                           1/1     Running   0              150m
kube-system                         kube-proxy-74f7t                                                 1/1     Running   0              151m
kube-system                         kube-proxy-rr4jh                                                 1/1     Running   0              151m
kube-system                         kube-proxy-t5fpd                                                 1/1     Running   0              150m
kube-system                         kube-proxy-vdqzs                                                 1/1     Running   0              152m
kube-system                         kube-scheduler-10.212.177.229                                    1/1     Running   0              152m
kube-system                         kube-scheduler-10.212.177.232                                    1/1     Running   0              150m
kube-system                         kube-vip-10.212.177.229                                          1/1     Running   0              152m
kube-system                         kube-vip-10.212.177.232                                          1/1     Running   0              150m
kube-system                         vsphere-cloud-controller-manager-626fc                           1/1     Running   0              151m
kube-system                         vsphere-cloud-controller-manager-jr59l                           1/1     Running   1 (151m ago)   152m
kube-system                         vsphere-cloud-controller-manager-mhjm9                           1/1     Running   0              151m
kube-system                         vsphere-cloud-controller-manager-pr6r5                           1/1     Running   0              150m
kube-system                         vsphere-csi-controller-5c84c8f794-fsd7x                          5/5     Running   0              152m
kube-system                         vsphere-csi-node-fs2qv                                           3/3     Running   0              151m
kube-system                         vsphere-csi-node-kqqft                                           3/3     Running   0              152m
kube-system                         vsphere-csi-node-m8422                                           3/3     Running   0              151m
kube-system                         vsphere-csi-node-w6jsx                                           3/3     Running   0              150m
  • Login to TMC console and click on Clusters option. Then click on Attach Cluster option
  • Fill the details like name, description etc. and Click Next
  • Grab the generated command and apply the yaml file to your EKS anywhere cluster. You will notice that it creates few resources.
namespace/vmware-system-tmc created
configmap/stack-config created
secret/tmc-access-secret created
customresourcedefinition.apiextensions.k8s.io/agents.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensionconfigs.intents.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensionintegrations.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensionresourceowners.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensions.clusters.tmc.cloud.vmware.com created
serviceaccount/extension-manager created
clusterrole.rbac.authorization.k8s.io/extension-manager-role created
clusterrolebinding.rbac.authorization.k8s.io/extension-manager-rolebinding created
service/extension-manager-service created
Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
deployment.apps/extension-manager created
serviceaccount/extension-updater-serviceaccount created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/vmware-system-tmc-agent-restricted created
clusterrole.rbac.authorization.k8s.io/extension-updater-clusterrole created
clusterrole.rbac.authorization.k8s.io/vmware-system-tmc-psp-agent-restricted created
clusterrolebinding.rbac.authorization.k8s.io/extension-updater-clusterrolebinding created
clusterrolebinding.rbac.authorization.k8s.io/vmware-system-tmc-psp-agent-restricted created
service/extension-updater created
deployment.apps/extension-updater created
serviceaccount/agent-updater created
clusterrole.rbac.authorization.k8s.io/agent-updater-role created
clusterrolebinding.rbac.authorization.k8s.io/agent-updater-rolebinding created
deployment.apps/agent-updater created
Warning: batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
Warning: spec.jobTemplate.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
cronjob.batch/agentupdater-workload created
  • Wait for Pod’s to come up under vmware-system-tmc namespace
kubectl get po -n vmware-system-tmc
NAME                                                              READY   STATUS      RESTARTS   AGE
agent-updater-5b6b6dc75b-z7kn2                                    1/1     Running     0          84s
agentupdater-workload-27503432--1-jjsgx                           0/1     Completed   0          35s
cluster-auth-pinniped-7b9b9ff6fd-6ckdk                            1/1     Running     0          42s
cluster-auth-pinniped-7b9b9ff6fd-fc2lv                            1/1     Running     0          42s
cluster-auth-pinniped-kube-cert-agent-656cd7cf49-gwgj5            1/1     Running     0          22s
cluster-health-extension-57666d7d7c-zhwlz                         1/1     Running     0          22s
cluster-secret-65677fd45c-szxzg                                   1/1     Running     0          36s
extension-manager-854686889c-v5jqz                                1/1     Running     0          84s
extension-updater-79489cd78d-945x8                                1/1     Running     0          84s
gatekeeper-operator-manager-bd78b5d79-rjr9m                       1/1     Running     0          23s
inspection-extension-5c7567d669-2pxf9                             1/1     Running     0          31s
intent-agent-78d44f97dc-d4d26                                     1/1     Running     0          20s
logs-collector-cluster-health-extension-20220417143220--1-rxjzf   1/1     Running     0          15s
logs-collector-cluster-secret-20220417143220--1-4zzpx             1/1     Running     0          15s
logs-collector-extension-manager-20220417143221--1-wswwr          1/1     Running     0          14s
logs-collector-gatekeeper-operator-20220417143220--1-5rdtr        1/1     Running     0          15s
logs-collector-inspection-20220417143220--1-lk82m                 1/1     Running     0          15s
logs-collector-intent-agent-20220417143220--1-bbx4d               1/1     Running     0          15s
logs-collector-package-deployment-20220417143220--1-7844l         1/1     Running     0          15s
logs-collector-policy-insight-extension-20220417143220--1-8fbvb   1/1     Running     0          15s
logs-collector-policy-sync-extension-20220417143220--1-swhkq      1/1     Running     0          15s
logs-collector-tmc-observer-20220417143220--1-5ccch               1/1     Running     0          15s
package-deployment-7dcc86d59d-7tw49                               1/1     Running     0          33s
policy-insight-extension-manager-5ddd66f98-tlg49                  1/1     Running     0          44s
policy-sync-extension-66c9dbf648-9n5tt                            1/1     Running     0          37s
sync-agent-6c8f4cf57d-v7hg4                                       1/1     Running     0          34s
tmc-observer-67cb4c7fc8-q7rdd                                     1/1     Running     0          30s
  • Go back to TMC console and Click Verify Connection button
  • Click on View your cluster button and you can see the cluster is now showing Healthy
  • Great, so we have completed Attaching an EKS Anywhere Cluster running on vSphere environment to TMC.

EKS Anywhere Cluster Management using TMC

Cluster Inspection

TMC can perform several management function on any conformant Kubernetes cluster. In this post, we will just run an inspection and package deployment and validate the result.

  • Login to TMC console
  • Click Inspections option from the left navigation menu
  • Click Run Inspection and select Conformance. TMC will start doing a conformance check against your EKS anywhere cluster. Wait for some time to get it completed.
  • Keep monitoring the status as it will take sometime to complete the test.
  • Eventually, you will see the following result

Package Deployment to an EKS Anywhere cluster using TMC

  • Login to TMC console
  • Click on Catalog option from left navigation pane. Select a Cluster
  • It will display the list of packages available for Install. Click on Cert Manager.
  • Click Install Package. Give a name and click Install Package button.
  • You will notice that the package start deploying resources on an EKS anywhere cluster. Wait for few sec to get it completed.

So, TMC greatly simplifies the Kubernetes cluster management operations like Inspection, package management, backup and restore, policies etc. Feel free to try out rest of the management options.

To know more about TMC, refer VMware official documentation here.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s