Deploying Tanzu Kubernetes Grid (TKG) Windows Workload Cluster

Starting with Tanzu Kubernetes Grid (TKG) 1.5.2, you can deploy Windows workload clusters. I tried this and on a vSphere 7.x environment and will share a step by step instructions with you in this post. In my lab environment, I have tried deploying two Windows workload clusters and one linux workload cluster managed using same management cluster. As you can see in the diagram below, Below are the cluster specifications:

  • TKG Mangement Cluster
    • Three Master nodes and Three worker node. Both are deployed with Ubuntu template. You can go with dev topology too.
  • TKG Windows Workload Cluster -1
    • 1 Master node that uses Ubuntu template
    • 2 Worker nodes – Both are using Windows 2019 image that we will be creating later in this post
  • TKG Windows Workload Cluster -2
    • 1 Master node that uses Ubuntu template
    • 1 Worker node – Using Windows 2019 image that we will be creating later in this post
  • TKG Linux Workload Cluster -3
    • 1 Master and 1 Worker node – Both are using Ubuntu image

Pre-requirements for Setting up Windows Workload Cluster

  • Windows Image
    • A Linux or Mac system to build Windows Image
    • Docker is installed and Running
    • Windows ISO file
    • VMware Tools for Windows ISO file
    • Windows Product Key (if not using trial)
    • Tanzu CLI
  • vSphere 6.7 U3 or a higher version
  • TKG Management Cluster is up and Running (not less than v1.5.2) with Ubuntu 2004 Kubernetes v1.22.5 OVA – You can follow the instructions here to deploy a management cluster.

Steps to Create a Windows Image

  • You should import Windows Server ISO and VMware tools ISO image on a vCenter Datacenter. In my case, I have created a folder named winiso and uploaded both ISO files there.
  • Once ISO files are uploaded, Note down the path. Ensure that you copy the full path shown under Path column above. e.g.
    [LUN01] winiso/en_windows_server_2019.iso
    [LUN01] winiso/VMware-tools-windows-12.0.0-19345655.iso
  • Create a yaml name builder.yaml with below content
apiVersion: v1
kind: Namespace
metadata:
 name: imagebuilder
---
apiVersion: v1
kind: Service
metadata:
 name: imagebuilder-wrs
 namespace: imagebuilder
spec:
 selector:
   app: image-builder-resource-kit
 type: NodePort
 ports:
 - port: 3000
   targetPort: 3000
   nodePort: 30008
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: image-builder-resource-kit
 namespace: imagebuilder
spec:
 selector:
   matchLabels:
     app: image-builder-resource-kit
 template:
   metadata:
     labels:
       app: image-builder-resource-kit
   spec:
     nodeSelector:
       kubernetes.io/os: linux
     containers:
     - name: windows-imagebuilder-resourcekit
       image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.22.5_vmware.1-tkg.1
       imagePullPolicy: Always
       ports:
         - containerPort: 3000
  • Apply the yaml file to the management cluster
$ kubectl apply -f builder.yaml
namespace/imagebuilder created
service/imagebuilder-wrs created
deployment.apps/image-builder-resource-kit created
  • Above step will deploys a container that allows your machine to retrieve the files it needs from an HTTP location during the image build process
  • Validate the pod status
$ kubectl get po -n imagebuilder
NAME                                          READY   STATUS    RESTARTS   AGE
image-builder-resource-kit-7f5cbc8cc9-lpmh5   1/1     Running   0          2d2h
  • Note down the Control plane IP Address of your management cluster.
  • Create a JSON file in an any empty folder named windows.json with the following configuration
{
 "unattend_timezone": "WINDOWS-TIMEZONE",
 "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
 "windows_updates_kbs": "",
 "kubernetes_semver": "v1.22.5",
 "cluster": "VSPHERE-CLUSTER-NAME",
 "template": "",
 "password": "VCENTER-PASSWORD",
 "folder": "",
 "runtime": "containerd",
 "username": "VCENTER-USERNAME",
 "datastore": "DATASTORE-NAME",
 "datacenter": "DATACENTER-NAME",
 "convert_to_template": "true",
 "vmtools_iso_path": "VMTOOLS-ISO-PATH",
 "insecure_connection": "true",
 "disable_hypervisor": "false",
 "network": "NETWORK",
 "linked_clone": "false",
 "os_iso_path": "OS-ISO-PATH",
 "resource_pool": "",
 "vcenter_server": "VCENTER-IP",
 "create_snapshot": "false",
 "netbios_host_name_compatibility": "false",
 "kubernetes_base_url": "http://CONTROLPLANE-IP:30008/files/kubernetes/",
 "containerd_url": "http://CONTROLPLANE-IP:30008/files/containerd/cri-containerd-v1.5.9+vmware.1.windows-amd64.tar",
 "containerd_sha256_windows": "0aa0df1c3d6c545b360e75fa4d20f604e1bb2ccd05b1651dc92941fdb4b49587",
 "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.5",
 "prepull": "false",
 "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
 "additional_download_files": "",
 "additional_executables": "true",
 "additional_executables_destination_path": "c:/k/antrea/",
 "additional_executables_list": "http://CONTROLPLANE-IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
 "load_additional_components": "true"
}

Where:

  1. WINDOWS-TIMEZONE is the Windows timezone for your environment. For example, GMT Standard Time.
  2. VSPHERE-CLUSTER-NAME is the name of your vSphere cluster. For example, VSAN-Cluster.
  3. VCENTER-PASSWORD is your vCenter password.
  4. VCENTER-USERNAME is your vCenter username.
  5. DATASTORE-NAME is the name of your datastore as it appears in vCenter. For example vsanDatastore.
  6. DATACENTER-NAME is the name of your data center as it appears in vCenter. For example VSAN-DC.
  7. VMTOOLS-ISO-PATH is the path to the VMware Tools ISO file structured like this: [DATASTORE-NAME] winiso/VMware-tools-windows-11.2.5-17337674.iso.
  8. NETWORK the name of a network or port group as it appears in vCenter Menu > Networking > Networks. For example, VM Network.
  9. OS-ISO-PATH is the path to the Windows Server 2019 ISO file structured like this: [DATASTORE-NAME] winiso/en-us_windows_server_2019_updated_aug_2021_x64_dvd_a6431a28.iso.
  10. VCENTER-IP is the IP address or FQDN of your vCenter server.
  11. CONTROLPLANE-IP is the management cluster IP address retrieved above.
  • Add the XML file that contains the Windows settings. Download this file to the same location where windows.json is created.
wget https://vdc-download.vmware.com/sampleExchange/v1/downloads/7719 -O autounattend.xml
  • Change the permission of autounattend.xml and windows.json to 777. Just to ensure that you dont see any problem with permission later.
  • Run the below command to start Windows image build
$ docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json --mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml -e PACKER_VAR_FILES="/windows.json" -e IB_OVFTOOL=1 -e IB_OVFTOOL_ARGS='--skipManifestCheck' -e PACKER_FLAGS='-force -on-error=ask' -t projects.registry.vmware.com/tkg/image-builder:v0.1.11_vmware.3 build-node-ova-vsphere-windows-2019 -e PACKER_LOG=1
  • Above commands take around an hour or more to complete the process. You can monitor the progress on the console and eventually the following output will be displayed.
hack/ensure-ansible.sh
hack/ensure-ansible-windows.sh
hack/ensure-packer.sh
hack/ensure-goss.sh
Right version of binary present
hack/ensure-ovftool.sh
packer build -var-file="/home/imagebuilder/packer/config/kubernetes.json"  -var-file="/home/imagebuilder/packer/config/windows/kubernetes.json"  -var-file="/home/imagebuilder/packer/config/containerd.json"  -var-file="/home/imagebuilder/packer/config/windows/containerd.json"  -var-file="/home/imagebuilder/packer/config/windows/docker.json"  -var-file="/home/imagebuilder/packer/config/windows/ansible-args-windows.json"  -var-file="/home/imagebuilder/packer/config/common.json"  -var-file="/home/imagebuilder/packer/config/windows/common.json"  -var-file="/home/imagebuilder/packer/config/windows/cloudbase-init.json"  -var-file="/home/imagebuilder/packer/config/goss-args.json"  -var-file="/home/imagebuilder/packer/config/additional_components.json"  -force -on-error=ask -var-file="packer/ova/packer-common.json" -var-file="/home/imagebuilder/packer/ova/windows-2019.json" -only=file -var-file="/windows.json"  packer/ova/packer-windows.json
2022/04/21 10:07:21 [INFO] Packer version: 1.7.2 [go1.16.3 linux amd64]
2022/04/21 10:07:21 [TRACE] discovering plugins in /home/imagebuilder/.local/bin
2022/04/21 10:07:21 [TRACE] discovering plugins in /home/imagebuilder/.packer.d/plugins
2022/04/21 10:07:21 [DEBUG] Discovered plugin: goss = /home/imagebuilder/.packer.d/plugins/packer-provisioner-goss
2022/04/21 10:07:21 using external provisioners [goss]
2022/04/21 10:07:21 [TRACE] discovering plugins in .
2022/04/21 10:07:21 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/04/21 10:07:21 [INFO] PACKER_CONFIG env var set; attempting to open config file: /home/imagebuilder/.packerconfig
2022/04/21 10:07:21 [WARN] Config file doesn't exist: /home/imagebuilder/.packerconfig
2022/04/21 10:07:21 [INFO] Setting cache directory: /home/imagebuilder/packer_cache
2022/04/21 10:07:22 [TRACE] Starting internal plugin packer-builder-file
-------------------
-------------------
------------------
    vsphere (shell-local): image-build-ova: create ova checksum windows-2019-kube-v1.22.5.ova.sha256
2022/04/21 11:14:44 [INFO] (telemetry) ending shell-local
Build 'vsphere' finished after 1 hour 7 minutes.

==> Wait completed after 1 hour 7 minutes
==> Wait completed after 1 hour 7 minutes
  • Once it is completed, you will also see the template on a vCenter.
  • Detail of the newly created Windows image

Deploying Windows Workload Cluster

  • Now, We can follow a normal process to create a TKG workload cluster. Ensure to add the following parameters in the cluster yaml file
IS_WINDOWS_WORKLOAD_CLUSTER: "true"
VSPHERE_WINDOWS_TEMPLATE: windows-2019-kube-v1.22.5
ENABLE_MHC: "false"
  • Also ensure to increase the cpu, RAM and disk size of worker nodes as Windows based worker node need it. You can get more detail about Workload Cluster Deployment here
  • Here is the sample file that I used for deployment of dev profile Windows workload cluster. Modify the bold text as per your env.
AVI_CA_DATA_B64: ""
AVI_CLOUD_NAME: ""
AVI_CONTROL_PLANE_HA_PROVIDER: ""
AVI_CONTROLLER: ""
AVI_DATA_NETWORK: ""
AVI_DATA_NETWORK_CIDR: ""
AVI_ENABLE: "false"
AVI_LABELS: ""
AVI_MANAGEMENT_CLUSTER_VIP_NETWORK_CIDR: ""
AVI_MANAGEMENT_CLUSTER_VIP_NETWORK_NAME: ""
AVI_PASSWORD: ""
AVI_SERVICE_ENGINE_GROUP: ""
AVI_USERNAME: ""
CLUSTER_CIDR: 100.96.0.0/11
CLUSTER_NAME: tkgwk1
CLUSTER_PLAN: dev
ENABLE_AUDIT_LOGGING: "false"
ENABLE_CEIP_PARTICIPATION: "false"
IDENTITY_MANAGEMENT_TYPE: none
INFRASTRUCTURE_PROVIDER: vsphere
LDAP_BIND_DN: ""
LDAP_BIND_PASSWORD: ""
LDAP_GROUP_SEARCH_BASE_DN: ""
LDAP_GROUP_SEARCH_FILTER: ""
LDAP_GROUP_SEARCH_GROUP_ATTRIBUTE: ""
LDAP_GROUP_SEARCH_NAME_ATTRIBUTE: cn
LDAP_GROUP_SEARCH_USER_ATTRIBUTE: DN
LDAP_HOST: ""
LDAP_ROOT_CA_DATA_B64: ""
LDAP_USER_SEARCH_BASE_DN: ""
LDAP_USER_SEARCH_FILTER: ""
LDAP_USER_SEARCH_NAME_ATTRIBUTE: ""
LDAP_USER_SEARCH_USERNAME: userPrincipalName
OIDC_IDENTITY_PROVIDER_CLIENT_ID: ""
OIDC_IDENTITY_PROVIDER_CLIENT_SECRET: ""
OIDC_IDENTITY_PROVIDER_GROUPS_CLAIM: ""
OIDC_IDENTITY_PROVIDER_ISSUER_URL: ""
OIDC_IDENTITY_PROVIDER_NAME: ""
OIDC_IDENTITY_PROVIDER_SCOPES: ""
OIDC_IDENTITY_PROVIDER_USERNAME_CLAIM: ""
OS_ARCH: amd64
OS_NAME: ubuntu
OS_VERSION: "20.04"
SERVICE_CIDR: 100.64.0.0/13
TKG_HTTP_PROXY_ENABLED: "false"
VSPHERE_CONTROL_PLANE_DISK_GIB: "80"
VSPHERE_CONTROL_PLANE_ENDPOINT: <control plane IP>
VSPHERE_CONTROL_PLANE_MEM_MIB: "4096"
VSPHERE_CONTROL_PLANE_NUM_CPUS: "2"
VSPHERE_DATACENTER: /Datacenter
VSPHERE_DATASTORE: /Datacenter/datastore/LUN01
VSPHERE_FOLDER: /Datacenter/vm/tkg-vsphere-tkg-mgmt
VSPHERE_NETWORK: /Datacenter/network/Workload
VSPHERE_PASSWORD: <your VC Password>
VSPHERE_RESOURCE_POOL: /Datacenter/host/Cluster/Resources/tkg-vsphere-tkg-Mgmt
VSPHERE_SERVER: <your-VCENTER IP or hostname>
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDoOAjJgmklW3Lc4dTB4doWgQUdfPRiPevDnFF1wm4LpUcu5+EiDr7IokPkS+tZJP1etM06mi6gvfsrzDJTQv0WgF0DK0lpzB5Wujul1HfXPXw910xYW40zn+GiATFposDUV/22Ceah3I5HwArGPOwr2BSWOce8D5ri/lzXYzURTL1343k3a2KOuleWWvULURKHCZcqSfwEVNqVxTTF6Jf5B/6xseXwg8PmwS/8Cr4h+FDAMeBhBEiRXhnS2BESiPrFKdSOLXN1qkjwvf7i6Q2yelOWryc5yLoD5grrCjSCklWtehkHt7bwvLnab+IIbMBhIXd5+jDGk3VY05ouWcRuUUVpFA0Kp2yq4xDQyw3aHhc1hx1xDSE6sSTXeTAeNUOelEF8ijqodnjGK/E1ipdYBSrAU/Uw617Q0+AjbM9vrd3+BxG5/kVJ2Tkwp9lFVm4BFgK1SoLGK3oPjneFGnLf7JNL2I9QuOpCmXowbXXYgcCk0HTpjhg1kXzczCqRM5LrpjOnozt5hor1+ebUaqNqzTxXnN+eQhJxmWbWHZSSSLwH0xvUMY7Vo1HDlzlUPBiqWmfE4MRE2fjg815nIYXhcRdknuPQDsPycgN1ly0dPHOToaXBBpsZjzfwOB+n4r+kkV58uqxRwdrD2fAJ6ZoiajHGd3T+cGQAVMq6w7txFQ== email@example.com
VSPHERE_TLS_THUMBPRINT: 63:A9:3B:38:94:9A:68:17:84:CA:E9:E9:6A:65:E7:0C:AE:5C:A9:2F
VSPHERE_USERNAME: administrator@vsphere.local
VSPHERE_WORKER_DISK_GIB: "80"
VSPHERE_WORKER_MEM_MIB: "8192"
VSPHERE_WORKER_NUM_CPUS: "8"
IS_WINDOWS_WORKLOAD_CLUSTER: "true"
VSPHERE_WINDOWS_TEMPLATE: windows-2019-kube-v1.22.5
ENABLE_MHC: "false"
  • Once the yaml is ready, Deploy Windows workload cluster.
$ tanzu cluster create -f wk1.yaml 

 Creating a windows workload cluster tkgwk1


 Creating a windows workload cluster tkgwk1

Validating configuration...
Warning: Pinniped configuration not found. Skipping pinniped configuration in workload cluster. Please refer to the documentation to check if you can configure pinniped on workload cluster manually
Creating workload cluster 'tkgwk1'...
Waiting for cluster to be initialized...
zero or multiple KCP objects found for the given cluster, 0 tkgwk1 default
cluster control plane is still being initialized: Cloning @ Machine/tkgwk1-control-plane-8f6n5
cluster control plane is still being initialized: PoweringOn @ Machine/tkgwk1-control-plane-8f6n5
cluster control plane is still being initialized: WaitingForKubeadmInit
Waiting for cluster nodes to be available...
Waiting for addons installation...
Waiting for packages to be up and running...

Workload cluster 'tkgwk1' created
  • If you want to deploy more Windows cluster, Follow the same approach and create them. You just need to edit the cluster yaml. In my case, I have created one more and just changed the name of the cluster and control plane IP address.

Deploy an ASP.NET Sample App on a Windows Workload Cluster

Let’s use the below yaml as sample file for deploying an asp .net application.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: aspnet
  name: aspnet
spec:
  replicas: 1
  selector:
    matchLabels:
      run: aspnet
  template:
    metadata:
      labels:
        run: aspnet
    spec:
      containers:
      - image: mcr.microsoft.com/dotnet/framework/samples:aspnetapp
        name: aspnet
      nodeSelector:
        kubernetes.io/os: windows
      tolerations:
      - key: "os"
        operator: "Equal"
        value: "windows"
        effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: aspnet
  name: aspnet
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: aspnet
  type: NodePort

  • Save the above yaml content and apply the yaml file.
kubectl apply -f winapp.yaml 
deployment.apps/aspnet created
service/aspnet created
root@dt-vc-ubuntu:~# kubectl get pod
NAME                      READY   STATUS              RESTARTS   AGE
aspnet-7b78876886-r5cqw   0/1     ContainerCreating   0          4s
  • It takes sometime to pull the image as it’s big. Once running, you can access the APP as it’s exposed over a nodeport. e.g. In your case, Node IP and port will be different.
  • We have deployed the sample app as shown in the below diagram and access it successfully.
  • Finally, I have created one more Linux based workload cluster and all are managed using single management cluster.
tanzu cluster list --include-management-cluster
  NAME         NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES       PLAN  
  tkgwk1       default     running  1/1           2/2      v1.22.5+vmware.1  <none>      dev   
  tkgwk2       default     running  1/1           1/1      v1.22.5+vmware.1  <none>      dev   
  tkgwk3       default     running  1/1           1/1      v1.22.5+vmware.1  <none>      dev   
  tkgnodemgmt  tkg-system  running  3/3           3/3      v1.22.5+vmware.1  management  prod  

Reference

Refer the VMware official doc here for more information about deploying Windows based workload clusters.

2 thoughts on “Deploying Tanzu Kubernetes Grid (TKG) Windows Workload Cluster

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s