Kubeflow Test Infrastructure Playbook

This is a playbook for build cops to help deal with problems with the CI infrastructure.

GCP Quota errors

1. There is a stack driver dashboard named CI Health that can be used to check various quota usages like CPUs

2. List regional quotas to see which quotas are running hot

   gcloud compute regions describe --project=kubeflow-ci ${REGION}
   

3. Check if we are leaking Kubeflow deployments and this is causing us to run out of quota.

   gcloud --project=kubeflow-ci --format="table(name,createTime:sort=1,location,status)" container clusters list
   gcloud --project=kubeflow-ci deployment-manager deployments list --format="table(name,insertTime:sort=1)" 
   

   • Deployments created by the E2E tests should be GC'd after O(2) hours
   • So if there are resources older than O(2) hours it indicates that there is a problem with garbage collection

4. To access to k8s resources make sure to get credentials and set the default namespace to kubeflow-test-infra:

gcloud --project kubelow-ci container clusters get-credentials kubeflow-testing --zone $ZONE
kubectl config set-context $(kubectl config current-context) --namespace=kubeflow-test-infra

1. Check if the cron job to GC resources is running in the test cluster

   kubectl get cronjobs
   NAME                 SCHEDULE       SUSPEND   ACTIVE    LAST SCHEDULE   AGE	
   cleanup-ci           0 */2 * * *    False     0         <none>          14m
   

   • The cron job is defined in cleanup-ci-cron.jsonnet

   • If the cron job is not configured then start it.

2. Look for recent runs of the cron job and figure out whether the are running successfully

    kubectl get jobs -l app=cleanup-ci-kubeflow-ci-deployment --sort-by=".metadata.creationTimestamp"
   

   • Jobs triggered by cron will match the regex cleanup-ci-??????????

   • Check that the job ran successfully

   • The pods associated with the job can be fetched via labels

     kubectl logs -l job-name=${JOBNAME}
     

   • In stackdriver you can use the following query (change the value of job-name)

     resource.type="k8s_container"
     resource.labels.project_id="kubeflow-ci"
     resource.labels.cluster_name="kubeflow-testing"
     labels."k8s-pod/job-name"="cleanup-ci-kubeflow-ci-deployment-1586916000"
     

3. Do a oneoff run of the cleanup job

   cd test-infra
   kubectl create -f cleanup-ci-kubeflow-ci-deployment.yaml
   

   • You can adjust the command line arguments in order to do more aggressive garbage collection then usual

GCP SSL Certificate Quota Errors

This is a helpful command to list SSL certificates and their associated domain

gcloud beta --project=kubeflow-ci-deployment compute ssl-certificates list  --format="table(name,managed.domains,creationTimestamp:sort=1)"

GCP Stockouts

1. Find a region with available quota

   gcloud --project=kubeflow-ci-deployment compute regions describe ${REGION}
   

   • We currently have quota in us-east1 and us-central1

2. Change the kfctl tests to use the other zone

   • Currently this is hardcoded in python in kfctl_go_test_utils.py

NFS Volume Is Out Of Disk Space.

1. Use stackdriver to check the disk usage

   • TODO(jlewi): We should add this to the Kubeflow CI Health Dashboard

2. There are two ways to free up disk space

   1. Delete old directories on the NFS share

   2. Delete and recreate the NFS share

   • Both options are outlined below

Deleting old directories on the NFS share

1. Start a shell in the debug worker

   kubectl exec -it debug-worker-0 /bin/bash
   

2. Delete old directories

   cd /mnt/test-data-volume
   find -maxdepth 1 -type d ! -path . -mtime +7 -exec rm -rf {} ";"
   

Deleting and recreating the NFS share

1. Delete the PV and pvc

   kubectl -n kubeflow-test-infra delete pvc nfs-external
   kubectl -n kubeflow-test-infra delete pv gcfs
   kubectl -n kubeflow-test-infra delete pods --all=true
   

   • We delete the pods since the pods will be mounting the volume which will prevent deletion of the PV and PVC

2. Wait for them to be deleted

   • Most likely we will need to override delete protection because there will be some pods still mounting it

   • Dump the yaml

     kubectl -n kubeflow-test-infra get pvc nfs-external -o yaml > /tmp/nfs-external.yaml
     

   • Delete the finalizer kubernetes.io/pvc-protection in nfs-external.yaml

     ...
     finalizers:
     labels:
     ...
     

     • Make sure you have the field finalizers and its an empty list

   • Update the object

     kubectl -n kubeflow-test-infra apply -f /tmp/nfs-external.yaml
     

     • Alternatively you can use kubectl edit to remove finalizers.

   • Similarly, make sure you remove finzlizers from pv (i.e., gcfs)

3. If pv/pvc deleteion still stalls, delete all pods in kubeflow-test-infra manually

   kubectl -n kubeflow-test-infra delete pods --all
   

4. Delete the nfs deployment

   gcloud --project=kubeflow-ci deployment-manager deployments delete kubeflow-ci-nfs
   

5. Recreate the NFS share

   cd test-infra/gcp_configs
   gcloud --project=kubeflow-ci deployment-manager deployments create kubeflow-ci-nfs --config=gcfs.yaml
   

6. Get the IP address of the new NFS share

   gcloud beta --project=kubeflow-ci filestore instances list
   

7. Set the IP address in the PV

   • Edit test-infra/base/pvc.yaml
   • Change the server address of the persistent volume to the new PV

8. Recreate the PV and PVC

   cd test-infra/
   kustomize build base | kubectl apply -f -
   

9. Make sure the debug-worker-0 pod is able to successfully mount the PV

• You may need to restart the debug worker if it isn't running

  ks apply kubeflow-ci -c debug-worker
  

  • If you already deleted the pod debug-worker-0 make sure it is restarted and is healthy. Otherwise, if it stalls in terminated state, force delete it as follows:

  kubectl delete pods debug-worker-0 --grace-period=0 --force
  

  • Connect to debug-worker-0 to make sure it is able to mount the PV

  kubectl exec -it debug-worker-0 /bin/bash
  ls /secret
  

Garbage Collecting Old K8s Resources

./go/cmd/cleanup/main.go provides a simple go program that can delete all kinds of a given resource e.g. (Tekton PipelineRuns) in a given namespace

• The program allows setting a minimum age for resources to be eligible for deletion
• This program appears to work when kubectl delete all fails; probably because it handles pagination better