Enabling Velero vSphere Operator Supervisor Service in a Supervisor Cluster for Backup and Restore

Couple of days back, I posted about enabling MinIO supervisor service, If you are interested to know about it, feel free to read here “https://mappslearning.com/2021/12/07/enabling-minio-supervisor-service-in-a-vsphere-with-tanzu-supervisor-cluster/

In this post, I will talk through the process of enabling Velero supervisor service and integrate it with MinIO for storing the backup data. I will only talk about enabling Velero and configuring it but not the backup and restore and that is for an upcoming post. Also, Data Mover will be covered in the later posts.

Velero Backup architecture in a vSphere with Tanzu Environment

So, Let’s get started.

Pre-requirements for Enabling Velero Supervisor Service

– MinIO is installed and configured, you can use other object storage but in this case, I am using    MinIO. Refer the steps listed in the above link for MinIO supervisor service configuration.

– Container image registry is available, I have used Harbor image registry that is deployed using tanzu packages on a TKG cluster

– Linux VM, this is where you will configure Velero client binary

Image Requirements

Ensure that the following images are pushed on a registry. You can check the version as this may change depending on when you are setting this up.

  • vsphereveleroplugin/velero-plugin-for-vsphere:v1.1.0
  • vsphereveleroplugin/backup-driver:v1.1.0
  • velero/velero:v1.6.0
  • velero/velero-plugin-for-aws:v1.1.0

Below is the list of commands that I used for pulling an image from the registry, tagging it and then pushing it to the local Harbor registry.

vsphereveleroplugin/velero-plugin-for-vsphere:v1.1.0

root@dinesh:~# docker pull vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0
1.1.0: Pulling from vsphereveleroplugin/velero-plugin-for-vsphere
da7391352a9b: Pull complete
14428a6d4bcd: Pull complete
2c2d948710f2: Pull complete
f2146ab07676: Pull complete
c4557402c340: Pull complete
388eaeed6014: Pull complete
7eecea8f518c: Pull complete
97a735219fcc: Pull complete
18aa56185e9c: Pull complete
9c341aded97e: Pull complete
7bb4d53eb7be: Pull complete
Digest: sha256:490fec9e61e852511bbc3cd39f4e553bb4c00a12aa824e75b75dbde482f7c35a
Status: Downloaded newer image for vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0
docker.io/vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0


root@dinesh:~# docker tag 4e7b3039617d 172.17.5.102/tanzutest01/velero-plugin-for-vsphere:1.1.0

root@dinesh:~# docker push 172.17.5.102/tanzutest01/velero-plugin-for-vsphere:1.1.0
The push refers to repository [172.17.5.102/tanzutest01/velero-plugin-for-vsphere]
9dd17956a5bb: Pushed
c270c4ce2cda: Pushed
539f7f3accdc: Pushed
ce4f411ba432: Pushed
ee519e5c2aa1: Pushed
c9ee3534e6b1: Pushed
5c18efdea10f: Pushed
e9a761582827: Pushed
f6253634dc78: Pushed
9069f84dbbe9: Pushed
bacd3af13903: Pushed
1.1.0: digest: sha256:490fec9e61e852511bbc3cd39f4e553bb4c00a12aa824e75b75dbde482f7c35a size: 2619

vsphereveleroplugin/backup-driver:v1.1.0

docker pull vsphereveleroplugin/backup-driver:1.1.0
1.1.0: Pulling from vsphereveleroplugin/backup-driver
da7391352a9b: Already exists
14428a6d4bcd: Already exists
2c2d948710f2: Already exists
3b239f0a857f: Pull complete
33753b27aee6: Pull complete
0995c912256b: Pull complete
97a735219fcc: Pull complete
Digest: sha256:dcce1a7b52ca0626d77323aa3173ed416a6b1b23b54fecfc2b5bcc82ceb1fcca
Status: Downloaded newer image for vsphereveleroplugin/backup-driver:1.1.0
docker.io/vsphereveleroplugin/backup-driver:1.1.0

root@dinesh:~# docker tag 484adcd3f3eb 172.17.5.102/tanzutest01/backup-driver:1.1.0

root@dinesh:~# docker push 172.17.5.102/tanzutest01/backup-driver:1.1.0
The push refers to repository [172.17.5.102/tanzutest01/backup-driver]
ce4f411ba432: Mounted from tanzutest01/velero-plugin-for-vsphere
95c0a87ff816: Pushed
7af1bb27d6e7: Pushed
cb7e71931f06: Pushed
f6253634dc78: Mounted from tanzutest01/velero-plugin-for-vsphere
9069f84dbbe9: Mounted from tanzutest01/velero-plugin-for-vsphere
bacd3af13903: Mounted from tanzutest01/velero-plugin-for-vsphere
1.1.0: digest: sha256:dcce1a7b52ca0626d77323aa3173ed416a6b1b23b54fecfc2b5bcc82ceb1fcca size: 1785

velero/velero:v1.6.0

docker pull velero/velero:v1.6.0

v1.6.0: Pulling from velero/velero

a70d879fa598: Pull complete

c4394a92d1f8: Pull complete

10e6159c56c0: Pull complete

f8bfd5750a30: Pull complete

ceae378c0e82: Pull complete

Digest: sha256:5c20fcc24e05f8a215189ec7364b341935963b0be6a170f6a2a9604d428baf2c

Status: Downloaded newer image for velero/velero:v1.6.0

docker.io/velero/velero:v1.6.0

root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# docker tag 236bc1f1c145 172.17.5.102/tanzutest01/velero:v1.6.0

root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# docker push 172.17.5.102/tanzutest01/velero:v1.6.0

The push refers to repository [172.17.5.102/tanzutest01/velero]

f1991a54fa25: Pushed

d70f824d4ee6: Pushed

346be19f13b0: Pushed

935f303ebf75: Pushed

0e64bafdc7ee: Pushed

v1.6.0: digest: sha256:0e19e318a199b77b6ee2f08286288ab35a10b15448e9a97ce152cb0d5d8e8507 size: 1366

velero/velero-plugin-for-aws:v1.1.0

docker pull velero/velero-plugin-for-aws:v1.1.0
v1.1.0: Pulling from velero/velero-plugin-for-aws
23884877105a: Pull complete
bc38caa0f5b9: Pull complete
2910811b6c42: Pull complete
36505266dcc6: Pull complete
935bc57d8d00: Pull complete
68e5c786df4a: Pull complete
Digest: sha256:e52d3545c3c52dbd061f0bf2ae8f7d6b21747d0a8bc64245fb58c8de54df9b33
Status: Downloaded newer image for velero/velero-plugin-for-aws:v1.1.0
docker.io/velero/velero-plugin-for-aws:v1.1.0

Tag and push this too.

Utilities/CLI Setup

Downloading and configuring Velero-vSphere CLI

root@dinesh:~# wget https://github.com/vmware-tanzu/velero-plugin-for-vsphere/releases/download/v1.1.0/velero-vsphere-1.1.0-linux-amd64.tar.gz
--2021-12-07 11:37:22--  https://github.com/vmware-tanzu/velero-plugin-for-vsphere/releases/download/v1.1.0/velero-vsphere-1.1.0-linux-amd64.tar.gz
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/229322479/a0264d00-444c-11eb-9abc-7f159f8348e5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211207%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211207T113722Z&X-Amz-Expires=300&X-Amz-Signature=8951f3c71145b4fee1db87350c220e2992d78882018bee133688b73d668f284a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=229322479&response-content-disposition=attachment%3B%20filename%3Dvelero-vsphere-1.1.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-12-07 11:37:22--  https://github-releases.githubusercontent.com/229322479/a0264d00-444c-11eb-9abc-7f159f8348e5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211207%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211207T113722Z&X-Amz-Expires=300&X-Amz-Signature=8951f3c71145b4fee1db87350c220e2992d78882018bee133688b73d668f284a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=229322479&response-content-disposition=attachment%3B%20filename%3Dvelero-vsphere-1.1.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22806963 (22M) [application/octet-stream]
Saving to: ‘velero-vsphere-1.1.0-linux-amd64.tar.gz’

velero-vsphere-1.1.0-linux-amd64.tar.gz          100%[=======================================================================================================>]  21.75M  89.5MB/s    in 0.2s

2021-12-07 11:37:22 (89.5 MB/s) - ‘velero-vsphere-1.1.0-linux-amd64.tar.gz’ saved [22806963/22806963]

Extract and copy the binary to the local bin

$ tar -xvzf velero-vsphere-1.1.0-linux-amd64.tar.gz
velero-vsphere-1.1.0-linux-amd64/
velero-vsphere-1.1.0-linux-amd64/velero-vsphere
root@dinesh:~# cd velero-vsphere-1.1.0-linux-amd64/
root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# ls
velero-vsphere
root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# cp velero-vsphere /usr/local/bin/
root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# velero-vsphere
Velero vSphere operator CLI is a tool to manage Velero and Velero plugin for vSphere on
Supervisor cluster of vSphere with Kubernetes.

Usage:
  velero-vsphere [command]

Available Commands:
  configure   Configure backup option(s)
  help        Help about any command
  install     Install Velero Instance
  uninstall   Uninstall Velero Instance

Flags:
      --enable-leader-election   Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.
  -h, --help                     help for velero-vsphere
      --kubeconfig string        Paths to a kubeconfig. Only required if out-of-cluster.
      --master --kubeconfig      (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
      --webhook-port int         Webhook server port (set to 0 to disable)

Use "velero-vsphere [command] --help" for more information about a command.

Download and Configure Velero CLI

# wget https://github.com/vmware-tanzu/velero/releases/download/v1.6.0/velero-v1.6.0-linux-amd64.tar.gz
--2021-12-07 11:42:50--  https://github.com/vmware-tanzu/velero/releases/download/v1.6.0/velero-v1.6.0-linux-amd64.tar.gz
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/99143276/8a7ba400-9c44-11eb-9920-2b003fb3b069?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211207%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211207T114251Z&X-Amz-Expires=300&X-Amz-Signature=1bf1389ea7e5bd236c7d849fec35d551cd870b13a384914c38446447eedc5ffe&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=99143276&response-content-disposition=attachment%3B%20filename%3Dvelero-v1.6.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream [following]
--2021-12-07 11:42:51--  https://github-releases.githubusercontent.com/99143276/8a7ba400-9c44-11eb-9920-2b003fb3b069?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211207%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211207T114251Z&X-Amz-Expires=300&X-Amz-Signature=1bf1389ea7e5bd236c7d849fec35d551cd870b13a384914c38446447eedc5ffe&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=99143276&response-content-disposition=attachment%3B%20filename%3Dvelero-v1.6.0-linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.111.154, 185.199.110.154, 185.199.109.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.111.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26962033 (26M) [application/octet-stream]
Saving to: ‘velero-v1.6.0-linux-amd64.tar.gz’

velero-v1.6.0-linux-amd64.tar.gz                 100%[=======================================================================================================>]  25.71M  49.3MB/s    in 0.5s

2021-12-07 11:42:51 (49.3 MB/s) - ‘velero-v1.6.0-linux-amd64.tar.gz’ saved [26962033/26962033]

Extract and copy the binary on the local bin

 $ tar -xzvf velero-v1.6.0-linux-amd64.tar.gz
velero-v1.6.0-linux-amd64/LICENSE
velero-v1.6.0-linux-amd64/examples/README.md
velero-v1.6.0-linux-amd64/examples/minio
velero-v1.6.0-linux-amd64/examples/minio/00-minio-deployment.yaml
velero-v1.6.0-linux-amd64/examples/nginx-app
velero-v1.6.0-linux-amd64/examples/nginx-app/README.md
velero-v1.6.0-linux-amd64/examples/nginx-app/base.yaml
velero-v1.6.0-linux-amd64/examples/nginx-app/with-pv.yaml
velero-v1.6.0-linux-amd64/velero
root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# cp velero-v1.6.0-linux-amd64/velero /usr/local/bin/
root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# velero
Velero is a tool for managing disaster recovery, specifically for Kubernetes
cluster resources. It provides a simple, configurable, and operationally robust
way to back up your application state and associated data.

Steps to enable Velero vSphere Operator Supervisor Service

– Login to vCenter UI using Administrative credentials

– Go to vSphere Cluster -> Configure and then look for Supervisor service

– You will notice that there are three services available as of today, we are interested in Velero vSphere Operator

As you can see in the above screenshot, it is disabled mode.

– Select it and click on the Enable option

– You will notice the above dialog box where you can select the version and fill the registry endpoint. Registry endpoint is an optional parameter, in case you don’t fill it, the image will be pulled from dockerhub. Ensure that your supervisor nodes have internet access for the same.

In many cases, I have found that the supervisor nodes do not have internet access, in that case, you can push the required images in a private registry. In this case, I have an embedded harbor registry enabled and will be using the same for images.

– Click Next and Accept the EULA

– Click on the Finish an wait for few mins to get pods up and running

kubectl get po -n velero-vsphere-domain-c175172 -o wide
NAME                                               READY   STATUS    RESTARTS   AGE     IP           NODE                               NOMINATED NODE   READINESS GATES
velero-vsphere-operator-5f65c58d89-588j8           1/1     Running   0          4m56s   10.244.0.4   423235dc88aa819e12d3f02ae605ed4b   <none>           <none>
velero-vsphere-operator-webhook-6cc75c7cb6-8xrfp   1/1     Running   0          4m56s   10.244.0.4   423235dc88aa819e12d3f02ae605ed4b   <none>           <none>
velero-vsphere-operator-webhook-6cc75c7cb6-gkm7v   1/1     Running   0          4m56s   10.244.0.3   42328685de52fe8710b7aadbb9378bee   <none>           <none>
velero-vsphere-operator-webhook-6cc75c7cb6-hjhl8   1/1     Running   0          4m56s   10.244.0.2   42326d181b972234351a13bf97c76d55   <none>           <none>

Note: You will not see the pods on vCenter UI as below.

This is because the pods are running on control plane nodes / Supervisor nodes and not on worker nodes. But if you want to see them, Click on the namespace and then click on the compute tab.

Now, we will be setting up the Velero server side components.

Set the supervisor namespace where you want to install server side components.

root@dinesh:~/velero-vsphere-1.1.0-linux-amd64# velero client config set namespace=tanzutest01

​​Create a file for storing MinIO credentials. Below is the reference file, you can update the content based on your environment. 

root@dinesh:~# cat velero-minio-credentials
[default]
aws_access_key_id = MZTF32ALZGLZ0ZNX
aws_secret_access_key = PT2BOZQKSJMS0QFFAGYIVRGBSLNP15FK

Now, I have created a sample script that can help you with installing velero components.

#!/bin/sh
 
NAMESPACE="tanzutest01"
BUCKET="demo-bucket"
REGION=minio
S3URL="https://172.17.5.104/"
PublicURL="https://172.17.5.104/"
VELERO_IMAGE=172.17.5.102/tanzutest01/velero:v1.6.0
VSPHERE_PLUGIN=172.17.5.102/tanzutest01/velero-plugin-for-vsphere:1.1.0
AWS_PLUGIN=172.17.5.102/tanzutest01/velero-plugin-for-aws:v1.1.0
 
./velero-vsphere install \
   --namespace $NAMESPACE \
   --image $VELERO_IMAGE \
   --use-private-registry \
   --provider aws \
   --plugins $AWS_PLUGIN,$VSPHERE_PLUGIN \
   --bucket $BUCKET \
   --secret-file ./velero-minio-credentials \
   --snapshot-location-config region=$REGION \
   --backup-location-config region=$REGION,s3ForcePathStyle="true",s3Url=$S3URL,publicUrl=$PublicURL

Just for the demonstration, I am not using the script and doing it manually.

Export the variables

root@dinesh:~# export NAMESPACE="tanzutest01"
root@dinesh:~# export BUCKET="demo-bucket"
root@dinesh:~# export REGION=minio
root@dinesh:~# export S3URL="https://172.17.5.104/"
root@dinesh:~# export PublicURL="https://172.17.5.104/"
root@dinesh:~# export VELERO_IMAGE=172.17.5.102/tanzutest01/velero:v1.5.1
root@dinesh:~# export VSPHERE_PLUGIN=172.17.5.102/tanzutest01/velero-plugin-for-vsphere:1.1.0
root@dinesh:~# export AWS_PLUGIN=172.17.5.102/tanzutest01/velero-plugin-for-aws:v1.1.0

Execute the command to install Velero 

root@dinesh:~# velero-vsphere install --namespace $NAMESPACE --image $VELERO_IMAGE --use-private-registry --provider aws --plugins $AWS_PLUGIN,$VSPHERE_PLUGIN --bucket $BUCKET --secret-file ./velero-minio-credentials --snapshot-location-config region=$REGION --backup-location-config region=$REGION,s3ForcePathStyle="true",s3Url=$S3URL,publicUrl=$PublicURL
I1207 12:05:33.793415 1662976 request.go:621] Throttling request took 1.029056394s, request: GET:https://172.17.5.97:6443/apis/apiextensions.k8s.io/v1beta1?timeout=32s

Send the request to the operator about installing Velero in namespace tanzutest01

If you notice in the output below, a backup driver and velero pods are installed.

root@dinesh:~# k get po -n tanzutest01
NAME                                         READY   STATUS    RESTARTS   AGE
backup-driver-f9f5465f4-58m8l                1/1     Running   0          2m21s
demo-minio-tenant-console-67df6485d9-gw49z   1/1     Running   0          5h34m
demo-minio-tenant-zone-0-0                   1/1     Running   0          5h36m
demo-minio-tenant-zone-0-1                   1/1     Running   0          5h36m
demo-minio-tenant-zone-0-2                   1/1     Running   0          5h36m
demo-minio-tenant-zone-0-3                   1/1     Running   0          5h36m
nginx                                        1/1     Running   0          6d21h
velero-f8d46bb46-8mhqr                       1/1     Running   0          16m

Check the status of install

root@dinesh:~# kubectl -n tanzutest01 get veleroservice default -o json | jq '.status'
{
  "enabled": true,
  "installphase": "Completed",
  "version": "v1.5.1"
}


I will talk about backup and restore process and Data Mover part in the upcoming blogs.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s