Bring Your Own Kubernetes (BYOK8s) Upgrade

​

Prerequisites

• You must have access to the Administrative Server.
• You must verify the Service Account used for the upgrade has the correct permissions.
• Air-gapped servers must have the Docker images added to their Docker image repository.

​

Upgrading

1. Log in to the service account on the Administrative Server running your Anaconda Enterprise software.
2. Create a backup of your cluster configurations.
   This backup is the only reliable way to undo this upgrade later. Some Workbench components run forward database migrations when they start under the new chart version (confirmed for the git-storage service), and helm rollback does not reverse those migrations. Once a migration has run, rolling the release back to the prior version leaves that component unable to start against the now-newer database schema.Confirm the backup completed successfully before continuing. If you need to undo this upgrade after it has run, restore from this backup rather than relying on helm rollback alone.
3. Unpack the Helm Charts provided by Anaconda onto the Administrative Server:
   # Replace <HELM_CHARTS> with the Helm Charts you received from Anaconda
   tar xvzf <HELM_CHARTS>
   
4. Save your current configurations with the extract_config.sh script delivered with your Helm Chart by running the following command:
   # Replace <NAMESPACE> with the namespace Anaconda Enterprise is installed in
   NAMESPACE=<NAMESPACE> ./extract_config.sh
   
   The extract_config.sh script creates a file called helm_values.yaml and saves it in the directory where the script was run.
5. Verify the information captured in helm_values.yaml file is correct and contains all of your current cluster configuration settings.
6. Begin the upgrade by running the following command:
   helm upgrade --values ./helm_values.yaml anaconda-enterprise ./Anaconda-Enterprise/
   
   If the upgrade is successful, your output will look like this:
   Release "anaconda-enterprise" has been upgraded. Happy Helming!
   NAME: anaconda-enterprise
   LAST DEPLOYED: Wed Dec  7 21:52:34 2022
   NAMESPACE: aaron
   STATUS: deployed
   REVISION: 10
   TEST SUITE: None
   
   The duration of the upgrade depends on both the cluster node count and the speed of the new docker images loading into each node. You can view the status of the upgrade process by running the following command:
   kubectl get pods
   
   Once all pods display a Running status and each pod is running all their containers, you may return to using the cluster as normal.

​

Troubleshooting a failed rollback

Error: conflict occurred while applying object default/kubernetes-dashboard-csrf /v1, Kind=Secret: Apply failed with 1 conflict: conflict with "dashboard" using v1: .data.csrf

1. Find the revision number you want to roll back to:
   helm history anaconda-enterprise --name <NAMESPACE>
   
2. Delete the conflicting secret, then retry the rollback:
   kubectl delete secret kubernetes-dashboard-csrf --name <NAMESPACE>
   helm rollback anaconda-enterprise <REVISION> --name <NAMESPACE>
   
   Kubernetes and Helm recreate the secret cleanly once there is no longer a live value to conflict with.
3. Restart the Workbench Dashboard pod:
   kubectl delete pod --name <NAMESPACE> -l app.kubernetes.io/name=anaconda-enterprise-ops-dashboard
   
   The Dashboard process reads and caches its CSRF token only once, at startup, so it will not notice that the secret changed until it restarts. If this label selector does not match any pods in your environment, find the pod name first and delete it directly:
   kubectl get pods --name <NAMESPACE> | grep dashboard
   kubectl delete pod --name <NAMESPACE> <POD_NAME>
   
4. Confirm the rollback succeeded:
   kubectl get secret kubernetes-dashboard-csrf --name <NAMESPACE> -o jsonpath='{.data.csrf}'
   kubectl get pods --name <NAMESPACE> | grep dashboard
   
   The csrf field should return a non-empty base64 value, and the Dashboard pod should show 1/1 Running.