Understanding Etcd Encryption in a TKGm and TKGs Clusters

Etcd is a key/value store database used in a Kubernetes cluster to store the data. It’s one of the key components that stores all the objects created in a Kubernetes cluster including secrets.

By default, etcd encryption is not enabled and as a result, you can see the stored secrets in a non encrypted format and it really increases the attack surface. 

In this blog post, we will learn about etcd encryption in a TKGm and TKGs cluster. We will also understand which type of cluster comes with the default encryption and what type of encryption is being used.

In the below screenshot, when you create a secret, base64 encoded data is set by api-server and stored in a etcd database in a plain text format.

Once encryption is enabled, kube-api server uses the encryption method and store the encrypted data in an etcd cluster.

I found a very good article about the same on a Tanzu Developer Center, you can find the link below.

https://tanzu.vmware.com/developer/guides/platform-security-secret-management/

Let’s start understanding etcd encryption in a TKGm cluster first:

In order to check the etcd encryption at rest, we will need a utility called “etcdctl”. You can either have this utility on a TKGm master node or on a node from where you can reach out to the master nodes.

By default, etcdctl utility is not available on TKGm master nodes.

You can install etcdctl on a ubuntu machine by running the following command. You can do this on a TKGm master node or on a separate machine.

 $ apt-get install etcd-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  etcd-client
0 upgraded, 1 newly installed, 0 to remove and 93 not upgraded.
Need to get 4,563 kB of archives.
After this operation, 17.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/universe amd64 etcd-client amd64 3.                                                  2.26+dfsg-6 [4,563 kB]
Fetched 4,563 kB in 2s (2,737 kB/s)
Selecting previously unselected package etcd-client.
(Reading database ... 170642 files and directories currently installed.)
Preparing to unpack .../etcd-client_3.2.26+dfsg-6_amd64.deb ...
Unpacking etcd-client (3.2.26+dfsg-6) ...
Setting up etcd-client (3.2.26+dfsg-6) ...
Processing triggers for man-db (2.9.1-1) …

– Validate the version installed

root@dinesh:~# etcdctl version
etcdctl version: 3.4.13
API version: 3.4

If you have installed this on a master node, then run the below command to validate the etcd health first:

$ ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key --endpoints=https://172.168.20.101:2379 endpoint health
https://172.168.20.101:2379 is healthy: successfully committed proposal: took = 2.671401ms

Note: If you are running this command outside of the master node, you need to copy those certificates and mention the locations accordingly.

Now let’s test the secret encryption

– Create a secret 

– Run the below command to create a secret

$ kubectl create secret generic demo –from-literal=pass=12345

secret/demo created

– Run the below command to check the secret in a etcd database

$ ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt --key=/etc/kubernetes/pki/apiserver-etcd-client.key --endpoints=https://172.168.20.101:2379 get /registry/secrets/default/demo
/registry/secrets/default/demo
k8s

v1Secret▒
▒
demodefault"*$7e174636-3495-4a79-a8d8-62dd5acf4d342▒▒▒▒z▒_
kubectl-createUpdatev▒▒▒▒FieldsV1:-
+{"f:data":{".":{},"f:pass":{}},"f:type":{}}
pass12345Opaque"

If you notice the output above, the secret is shown in a plain text format. This means that the data is not encrypted in an etcd database by default and you need to encrypt this.

You can see the Kubernetes documentation to encrypt etcd data at rest in a TKGm cluster.

https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

Let’s understand etcd encryption in a TKGs cluster now:

For the demonstration purpose, I will be using vSphere with the Tanzu supervisor cluster. In case of the vSphere with Tanzu clusters, etcd encryption is enabled by default and we will be validating it now.

Note: You won’t be able to create secrets and few other objects on a supervisor cluster from outside, for that you need to login to supervisor nodes.

If you need the steps to get root password and login on a supervisor node, you can follow my other blog post here. https://mappslearning.com/2021/12/01/ssh-login-into-vsphere-with-tanzu-supervisor-cluster-nodes/

– Login to the supervisor node

– Check the etcd health, If you are on supervisor node, then you don’t need to specify certificate parameters. Run the below command:-

root@42326d181b972234351a13bf97c76d55 [ /etc/kubernetes/manifests ]# etcdctl endpoint health --write-out table
+------------------------+--------+------------+-------+
|        ENDPOINT        | HEALTH |    TOOK    | ERROR |
+------------------------+--------+------------+-------+
| https://127.0.0.1:2379 |   true | 9.614124ms |       |
+------------------------+--------+------------+-------+

– create a secret in a default namespace

 $ kubectl create secret generic demo --from-literal=pass=12345
secret/demo created

– Check the encryption status in an etcd database, run the below command

root@42326d181b972234351a13bf97c76d55 [ /etc/kubernetes/manifests ]# etcdctl get /registry/secrets/default/demo
/registry/secrets/default/demo
k8s:enc:aescbc:v1:key1:Jv.▒▒>▒66▒▒p"%!▒m/5▒[▒▒e▒%▒!▒oP▒▒r▒▒▒▒▒▒▒B▒j▒▒▒▒f▒K▒▒▒▒▒▒▒▒y▒▒▒▒&-▒▒L[▒▒▒▒t▒▒▒<j▒▒h4;▒|"▒▒գh▒~X1Ɂ▒▒D▒▒]Z▒3▒▒▒▒▒▒eM▒д▒▒ϧ▒F▒▒+▒▒˞▒▒۲&l▒Vb▒w▒IY▒9u▒2}% 0▒▒▒▒▒▒▒d▒j▒3 H▒`▒▒LY▒▒▒8▒eb▒▒V▒[A}▒▒▒4▒H▒▒K▒
root@42326d181b972234351a13bf97c76d55 [ /etc/kubernetes/manifests ]#

Since it is encrypted, so you can not read that now, but you if notice closely, it is using aescbc encryption method.

Now, let’s also validate where the encryption file and encryption type is.

 – Grep the encryption configuration in a kube-api server yaml file

root@42326d181b972234351a13bf97c76d55 [ /etc/kubernetes/manifests ]#  cat kube-apiserver.yaml | grep encryption
    - --experimental-encryption-provider-config=/etc/vmware/wcp/encryption-config.yaml

– Open the file

root@42326d181b972234351a13bf97c76d55 [ /etc/kubernetes/manifests ]# cat /etc/vmware/wcp/encryption-config.yaml
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
    - secrets
    providers:
    # 1.1.34 Ensure that the encryption provider is set to aescbc
    - aescbc:
        keys:
        - name: key1
          secret: <Your encryption secret will be shown here>

So above file also contains the encryption method as “aescbc” and the secret key.

So in summary:

By default, etcd database in TKGm clusters is not encrypted by default and we need to do it but it’s encrypted on TKGs clusters. In the upcoming post, I will also write about encrypting etcd in TKGm clusters.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s