Running Sonobuoy test using tanzu conformance option in a Tanzu Community Edition (TCE) created Standalone cluster

Sonobuoy is an opensource tool that helps to check the k8s cluster conformance. You can run Sonobuoy utility on your laptop and perform the test or other option is through tanzu cli that is part of Tanzu Community Edition (TCE).
In this tutorial, I will walk you through the later option and will run the end to end test using Tanzu CLI against Tanzu Standalone cluster running on a laptop.

Pre-requirement

$ tanzu                
Tanzu CLI

Usage:
  tanzu [command]

Available command groups:

  Admin
    builder                 Build Tanzu components 
	
  Build
    accelerator             Manage accelerators in a Kubernetes cluster 
    apps                    Applications on Kubernetes 
	
  Run
    cluster                 Kubernetes cluster operations 
    conformance             Run Sonobuoy conformance tests against clusters 
    diagnostics             Cluster diagnostics 
    imagepullsecret         Manage image pull secret operations. Image pull secrets enable the package and package repository consumers to authenticate to private registries. 
    kubernetes-release      Kubernetes release operations 
    management-cluster      Kubernetes management cluster operations 
    package                 Tanzu package management 
    standalone-cluster      Create clusters without a dedicated management cluster 
	
  System
    completion              Output shell completion code 
    config                  Configuration for the CLI 
    init                    Initialize the CLI 
    login                   Login to the platform 
    plugin                  Manage CLI plugins 
    update                  Update the CLI 
    version                 Version information 
	

Flags:
  -h, --help   help for tanzu

Use "tanzu [command] --help" for more information about a command. 

Logged in to tkgmgmtcluster

Running E2E Test

The Kubernetes end-to-end testing plugin (the e2e plugin) is used to run tests which are maintained by the upstream Kubernetes community. We can run the conformance test just by running tanzu conformance run command but before we run, let’s validate what resources will be created on a cluster. Run below command:

  $ tanzu conformance gen   
---
apiVersion: v1
kind: Namespace
metadata:
  name: sonobuoy
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    component: sonobuoy
  name: sonobuoy-serviceaccount
  namespace: sonobuoy
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    component: sonobuoy
    namespace: sonobuoy
  name: sonobuoy-serviceaccount-sonobuoy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sonobuoy-serviceaccount-sonobuoy
subjects:
- kind: ServiceAccount
  name: sonobuoy-serviceaccount
  namespace: sonobuoy
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    component: sonobuoy
    namespace: sonobuoy
  name: sonobuoy-serviceaccount-sonobuoy
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '/metrics'
  - '/logs'
  - '/logs/*'
  verbs:
  - 'get'
---
apiVersion: v1
data:
  config.json: |
    {"Description":"DEFAULT","UUID":"","Version":"v0.53.2","ResultsDir":"/tmp/sonobuoy","Resources":["apiservices","certificatesigningrequests","clusterrolebindings","clusterroles","componentstatuses","configmaps","controllerrevisions","cronjobs","customresourcedefinitions","daemonsets","deployments","endpoints","ingresses","jobs","leases","limitranges","mutatingwebhookconfigurations","namespaces","networkpolicies","nodes","persistentvolumeclaims","persistentvolumes","poddisruptionbudgets","pods","podlogs","podsecuritypolicies","podtemplates","priorityclasses","replicasets","replicationcontrollers","resourcequotas","rolebindings","roles","servergroups","serverversion","serviceaccounts","services","statefulsets","storageclasses","validatingwebhookconfigurations","volumeattachments"],"Filters":{"Namespaces":".*","LabelSelector":""},"Limits":{"PodLogs":{"Namespaces":"","SonobuoyNamespace":true,"FieldSelectors":[],"LabelSelector":"","Previous":false,"SinceSeconds":null,"SinceTime":null,"Timestamps":false,"TailLines":null,"LimitBytes":null,"LimitSize":"","LimitTime":""}},"QPS":30,"Burst":50,"Server":{"bindaddress":"0.0.0.0","bindport":8080,"advertiseaddress":"","timeoutseconds":21600},"Plugins":null,"PluginSearchPath":["./plugins.d","/etc/sonobuoy/plugins.d","~/sonobuoy/plugins.d"],"Namespace":"sonobuoy","WorkerImage":"sonobuoy/sonobuoy:v0.53.2","ImagePullPolicy":"IfNotPresent","ImagePullSecrets":"","ProgressUpdatesPort":"8099"}
kind: ConfigMap
metadata:
  labels:
    component: sonobuoy
  name: sonobuoy-config-cm
  namespace: sonobuoy
---
apiVersion: v1
data:
  plugin-0.yaml: |
    podSpec:
      containers: []
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Never
      serviceAccountName: sonobuoy-serviceaccount
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
        operator: Exists
      - key: CriticalAddonsOnly
        operator: Exists
      - key: kubernetes.io/e2e-evict-taint-key
        operator: Exists
    sonobuoy-config:
      driver: Job
      plugin-name: e2e
      result-format: junit
    spec:
      command:
      - /run_e2e.sh
      env:
      - name: E2E_EXTRA_ARGS
        value: --progress-report-url=http://localhost:8099/progress
      - name: E2E_FOCUS
        value: \[Conformance\]
      - name: E2E_PARALLEL
        value: "false"
      - name: E2E_SKIP
        value: \[Disruptive\]|NoExecuteTaintManager
      - name: E2E_USE_GO_RUNNER
        value: "true"
      - name: SONOBUOY_K8S_VERSION
        value: v1.21.2
      image: k8s.gcr.io/conformance:v1.21.2
      imagePullPolicy: IfNotPresent
      name: e2e
      resources: {}
      volumeMounts:
      - mountPath: /tmp/results
        name: results
  plugin-1.yaml: |
    sonobuoy-config:
      driver: DaemonSet
      plugin-name: systemd-logs
      result-format: raw
    spec:
      command:
      - /bin/sh
      - -c
      - /get_systemd_logs.sh; while true; do echo "Plugin is complete. Sleeping indefinitely
        to avoid container exit and automatic restarts from Kubernetes"; sleep 3600; done
      env:
      - name: CHROOT_DIR
        value: /node
      - name: NODE_NAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
      - name: RESULTS_DIR
        value: /tmp/results
      - name: SONOBUOY_K8S_VERSION
        value: v1.21.2
      image: sonobuoy/systemd-logs:v0.3
      imagePullPolicy: IfNotPresent
      name: systemd-logs
      resources: {}
      securityContext:
        privileged: true
      volumeMounts:
      - mountPath: /tmp/results
        name: results
      - mountPath: /node
        name: root
kind: ConfigMap
metadata:
  labels:
    component: sonobuoy
  name: sonobuoy-plugins-cm
  namespace: sonobuoy
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    component: sonobuoy
    run: sonobuoy-master
    sonobuoy-component: aggregator
    tier: analysis
  name: sonobuoy
  namespace: sonobuoy
spec:
  containers:
  - env:
    - name: SONOBUOY_ADVERTISE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    image: sonobuoy/sonobuoy:v0.53.2
    imagePullPolicy: IfNotPresent
    name: kube-sonobuoy
    volumeMounts:
    - mountPath: /etc/sonobuoy
      name: sonobuoy-config-volume
    - mountPath: /plugins.d
      name: sonobuoy-plugins-volume
    - mountPath: /tmp/sonobuoy
      name: output-volume
  restartPolicy: Never
  serviceAccountName: sonobuoy-serviceaccount
  tolerations:
  - key: "kubernetes.io/e2e-evict-taint-key"
    operator: "Exists"
  volumes:
  - configMap:
      name: sonobuoy-config-cm
    name: sonobuoy-config-volume
  - configMap:
      name: sonobuoy-plugins-cm
    name: sonobuoy-plugins-volume
  - emptyDir: {}
    name: output-volume
---
apiVersion: v1
kind: Service
metadata:
  labels:
    component: sonobuoy
    sonobuoy-component: aggregator
  name: sonobuoy-aggregator
  namespace: sonobuoy
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    sonobuoy-component: aggregator
  type: ClusterIP

So, As you can see the above specification, that is what going to run on a cluster and start testing the conformance parameters.

Now, Lets run the test.

$ tanzu conformance run
INFO[0000] created object                                name=sonobuoy namespace= resource=namespaces
INFO[0000] created object                                name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] created object                                name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] created object                                name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] created object                                name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object                                name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object                                name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] created object                                name=sonobuoy-aggregator namespace=sonobuoy resource=services

Check the status of test:

$ tanzu conformance status
The Sonobuoy aggregator is in the 'Pending' state. This is normal as the pod is created and begins to run, but if this state persists, use kubectl to debug further.

And you see above because pods are still being created, Check the status.

$ k get po -n sonobuoy  
NAME                                                      READY   STATUS              RESTARTS   AGE
sonobuoy                                                  1/1     Running             0          20s
sonobuoy-e2e-job-0619425a176943e5                         0/2     ContainerCreating   0          12s
sonobuoy-systemd-logs-daemon-set-8cdf7bf30e3a4e40-djgmk   0/2     ContainerCreating   0          12s
sonobuoy-systemd-logs-daemon-set-8cdf7bf30e3a4e40-l57fc   0/2     ContainerCreating   0          12s

Wait for few mins to run the pods and then run the command again and you will the status:

$ tanzu conformance status
         PLUGIN     STATUS   RESULT   COUNT             PROGRESS
            e2e    running                1   2/337 (0 failures)
   systemd-logs   complete                2    

Sonobuoy is still running. Runs can take 60 minutes or more depending on cluster and plugin configuration.

As you can see, first systemd-logs plugin will be executed and then e2e plugin. It says about 60 mins but in my case, it took around 3 hours. So wait.

Eventually, when all tests are run, see the status again.

$ tanzu conformance status                                
         PLUGIN     STATUS   RESULT   COUNT               PROGRESS
            e2e   complete   failed       1   337/337 (3 failures)
   systemd-logs   complete   passed       2                       

In my case, there are three test failed. Before you dig, Lets explore few more useful options.

Fetch the Test Result

Run the below command to retrive the test result.

$ tanzu conformance retrieve .                                                                           
202110301357_sonobuoy_827a65bd-8fd3-4046-8b2e-31dbc1a91f53.tar.gz

Validate the result

$ tanzu conformance results 202110301357_sonobuoy_827a65bd-8fd3-4046-8b2e-31dbc1a91f53.tar.gz                                                                             
Plugin: e2e
Status: failed
Total: 5771
Passed: 334
Failed: 3
Skipped: 5434

Failed tests:
[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]
[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]
[sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance]

Plugin: systemd-logs
Status: passed
Total: 2
Passed: 2
Failed: 0
Skipped: 0

Check the logs

Run below command to check the status

$ tanzu conformance logs

View Detailed Report with failure message

Add --mode detailed option to see the result in detail

$ tanzu conformance results 202110301357_sonobuoy_827a65bd-8fd3-4046-8b2e-31dbc1a91f53.tar.gz --mode detailed

Finally, If you don’t want to run full end to end test or change some paramters like image etc, see the available options.

$ tanzu conformance run --help
Starts a Sonobuoy run by launching the Sonobuoy aggregator and plugin pods.

Usage:
  tanzu conformance run [flags]

Flags:
      --aggregator-node-selector nodeSelectors   Node selectors to add to the aggregator. Values can be given multiple times and are in the form key:value (default map[])
      --config Sonobuoy config                   Path to a sonobuoy configuration JSON file.
      --context string                           Context in the kubeconfig to use.
      --dns-namespace string                     The namespace to check for DNS pods during preflight checks. (default "kube-system")
      --dns-pod-labels strings                   The label selectors to use for locating DNS pods during preflight checks. Can be specified multiple times or as a comma-separated list. (default [k8s-app=kube-dns,k8s-app=coredns])
      --e2e-focus envModifier                    Specify the E2E_FOCUS value for the e2e plugin, specifying which tests to run. Shorthand for --plugin-env=e2e.E2E_FOCUS=<string> (default \[Conformance\])
      --e2e-repo-config yaml-filepath            Specify a yaml file acting as KUBE_TEST_REPO_LIST, overriding registries for test images.
      --e2e-skip envModifier                     Specify the E2E_SKIP value for the e2e plugin, specifying which tests to skip. Shorthand for --plugin-env=e2e.E2E_SKIP=<string> (default \[Disruptive\]|NoExecuteTaintManager)
  -f, --file -                                   If set, loads the file as if it were the output from sonobuoy gen. Set to - to read from stdin.
  -h, --help                                     help for run
      --image-pull-policy string                 Set the ImagePullPolicy for the Sonobuoy image and all plugins. Valid options are Always, Never, IfNotPresent. (default "IfNotPresent")
      --kube-conformance-image image             Container image override for the e2e plugin. Shorthand for --plugin-image=e2e:<string> (default map[])
      --kubeconfig Kubeconfig                    Path to explicit kubeconfig file.
      --kubernetes-version string                Use default E2E image, but override the version. Default is 'auto', which will be set to your cluster's version if detected, erroring otherwise. 'ignore' will try version resolution but ignore errors. 'latest' will find the latest dev image/version upstream.
  -m, --mode Mode                                What mode to run the e2e plugin in. Valid modes are [non-disruptive-conformance certified-conformance quick]. (default non-disruptive-conformance)
  -n, --namespace string                         The namespace to run Sonobuoy in. Only one Sonobuoy run can exist per namespace simultaneously. (default "sonobuoy")
  -p, --plugin pluginList                        Which plugins to run. Can either point to a URL, local file/directory, or be one of the known plugins (e2e or systemd-logs). Can be specified multiple times to run multiple plugins.
      --plugin-env pluginenvvar                  Set env vars on plugins. Values can be given multiple times and are in the form plugin.env=value (default map[])
      --plugin-image plugin:image                Override a plugins image from what is in its definition (e.g. myPlugin:testimage) (default map[])
      --rbac RBACMode                            Whether to enable RBAC on Sonobuoy. Valid modes are Enable, Disable, and Detect (query the server to see whether to enable RBAC). (default Detect)
      --show-default-podspec                     If true, include the default pod spec used for plugins in the output.
      --skip-preflight                           If true, skip all checks before starting the sonobuoy run.
      --sonobuoy-image string                    Container image override for the sonobuoy worker and aggregator. (default "sonobuoy/sonobuoy:v0.53.2")
      --ssh-key yamlFile                         Path to the private key enabling SSH to cluster nodes. May be required by some tests from the e2e plugin.
      --ssh-user envModifier                     SSH user for ssh-key. Required if running e2e plugin with certain tests that require SSH access to nodes.
      --systemd-logs-image image                 Container image override for the systemd-logs plugin. Shorthand for --plugin-image=systemd-logs:<string> (default map[])
      --timeout int                              How long (in seconds) Sonobuoy aggregator will wait for plugins to complete before exiting. 0 indicates no timeout. (default 21600)
      --wait int[=1440]                          How long (in minutes) for the CLI to wait for sonobuoy run to be completed or fail, where 0 indicates do not wait. If specified, the default wait time is 1 day.
      --wait-output string                       Specify the type of output Sonobuoy should produce when --wait is used. Valid modes are silent, spinner, or progress (default "Silent")

Global Flags:
  -v, --verbose int32   Number for the log level verbosity(0-9)
 

Cleanup

Run below command to cleanup Sonobuoy run

$ tanzu conformance delete    
INFO[0000] deleted                                       kind=namespace namespace=sonobuoy
INFO[0000] deleted                                       kind=clusterrolebindings
INFO[0000] deleted                                       kind=clusterroles

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s