> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Bring Your Own Kubernetes (BYOK8s) Upgrade

export const Comments = ({children}) => {
  return <div class="my-4 px-5 py-4 overflow-hidden rounded-2xl flex gap-3 border border-zinc-500/20 bg-zinc-50/50 dark:border-zinc-500/30 dark:bg-zinc-500/10" data-callout-type="comments">
      <div class="w-4">
        <svg width="14" height="14" viewBox="0 0 640 640" fill="currentColor" xmlns="http://www.w3.org/2000/svg" class="w-5 h-5" aria-label="Comments">
            <path d="M320 112C434.9 112 528 205.1 528 320C528 434.9 434.9 528 320 528C205.1 528 112 434.9 112 320C112 205.1 205.1 112 320 112zM320 576C461.4 576 576 461.4 576 320C576 178.6 461.4 64 320 64C178.6 64 64 178.6 64 320C64 461.4 178.6 576 320 576zM280 400C266.7 400 256 410.7 256 424C256 437.3 266.7 448 280 448L360 448C373.3 448 384 437.3 384 424C384 410.7 373.3 400 360 400L352 400L352 312C352 298.7 341.3 288 328 288L280 288C266.7 288 256 298.7 256 312C256 325.3 266.7 336 280 336L304 336L304 400L280 400zM320 256C337.7 256 352 241.7 352 224C352 206.3 337.7 192 320 192C302.3 192 288 206.3 288 224C288 241.7 302.3 256 320 256z" />
        </svg>
      </div>
      <div class="text-sm prose min-w-0 w-full">
        {children}
      </div>
    </div>;
};

[Contact the Anaconda implementation team](https://anaconda.zendesk.com/hc/en-us/requests/new) before you begin for assistance upgrading your version of Anaconda Enterprise 5. It is intended for you to follow along with these instructions as an Anaconda Implementation team member guides you through the upgrade process. They will provide you with a Helm Chart archive that contains all the necessary components needed for upgrading.

<Warning>
  * Project sessions are terminated during the upgrade process! Because of this, it is important to stop all sessions prior to upgrading. If you do not, sessions that are terminated as part of the upgrade process must be restarted manually post upgrade.

  * Sessions can be stopped programmatically using `ae5-tools` by running the following command in a terminal that has access your Anaconda Enterprise cluster over the network:

    ```
    ae5 session list --columns=id --no-header | xargs -n1 ae5 session stop --yes
    ```
</Warning>

## Prerequisites

* You must have access to the [Administrative Server](../zero-g).
* You must verify the Service Account used for the upgrade has the [correct permissions](../guides/rbac).
* Air-gapped servers must have the Docker images added to their Docker image repository.

## Upgrading

Upgrade your BYOK8s cluster by performing the following steps:

1. Log in to the service account on the Administrative Server running your Anaconda Enterprise software.

2. [Create a backup of your cluster configurations](./backup-restore#run-the-backup-script).

   <Note>
     This backup is the only reliable way to undo this upgrade later. Some Workbench components run forward database migrations when they start under the new chart version (confirmed for the git-storage service), and `helm rollback` does not reverse those migrations. Once a migration has run, rolling the release back to the prior version leaves that component unable to start against the now-newer database schema.

     Confirm the backup completed successfully before continuing. If you need to undo this upgrade after it has run, restore from this backup rather than relying on `helm rollback` alone.
   </Note>

3. Unpack the Helm Charts provided by Anaconda onto the Administrative Server:

   ```
   # Replace <HELM_CHARTS> with the Helm Charts you received from Anaconda
   tar xvzf <HELM_CHARTS>
   ```

4. Save your current configurations with the `extract_config.sh` script delivered with your Helm Chart by running the following command:

   ```
   # Replace <NAMESPACE> with the namespace Anaconda Enterprise is installed in
   NAMESPACE=<NAMESPACE> ./extract_config.sh
   ```

   The `extract_config.sh` script creates a file called `helm_values.yaml` and saves it in the directory where the script was run.

5. Verify the information captured in `helm_values.yaml` file is correct and contains all of your current cluster configuration settings.

6. Begin the upgrade by running the following command:

   ```
   helm upgrade --values ./helm_values.yaml anaconda-enterprise ./Anaconda-Enterprise/
   ```

   If the upgrade is successful, your output will look like this:

   ```
   Release "anaconda-enterprise" has been upgraded. Happy Helming!
   NAME: anaconda-enterprise
   LAST DEPLOYED: Wed Dec  7 21:52:34 2022
   NAMESPACE: aaron
   STATUS: deployed
   REVISION: 10
   TEST SUITE: None
   ```

   The duration of the upgrade depends on both the cluster node count and the speed of the new docker images loading into each node. You can view the status of the upgrade process by running the following command:

   ```
   kubectl get pods
   ```

   Once all pods display a **Running** status and each pod is running all their containers, you may return to using the cluster as normal.

## Troubleshooting a failed rollback

Rolling back a Workbench upgrade with `helm rollback` can fail with a conflict on the `kubernetes-dashboard-csrf` secret:

```txt wrap theme={null}
Error: conflict occurred while applying object default/kubernetes-dashboard-csrf /v1, Kind=Secret: Apply failed with 1 conflict: conflict with "dashboard" using v1: .data.csrf
```

<Accordion title="Why this happens">
  The `kubernetes-dashboard-csrf` secret is written by two different owners: the *Helm chart* and the *Kubernetes Dashboard application* itself, which generates a real CSRF token at runtime under its own field manager.

  * `helm upgrade` re-renders the chart on every run, so it stays in sync with whatever the Dashboard has since written.
  * `helm rollback` does not re-render anything; it reapplies the exact secret content captured at the time the target revision was installed or upgraded to.

  By the time you need to roll back, that captured content has almost always diverged from what the Dashboard has written since, and Kubernetes' server-side apply rejects the conflicting write. Deleting the secret before retrying removes the stale content so the rollback can apply cleanly, and restarting the Dashboard pod ensures it picks up the resulting token instead of continuing to use the one it cached at startup.
</Accordion>

This affects Workbench chart versions prior to `ae-helm-chart-1.1-364`. Use the following steps to complete the rollback:

1. Find the revision number you want to roll back to:

   ```sh theme={null}
   helm history anaconda-enterprise --name <NAMESPACE>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

2. Delete the conflicting secret, then retry the rollback:

   ```sh theme={null}
   kubectl delete secret kubernetes-dashboard-csrf --name <NAMESPACE>
   helm rollback anaconda-enterprise <REVISION> --name <NAMESPACE>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.<br />
     Replace \<REVISION> with the revision number from the previous step.
   </Comments>

   Kubernetes and Helm recreate the secret cleanly once there is no longer a live value to conflict with.

3. Restart the Workbench Dashboard pod:

   ```sh theme={null}
   kubectl delete pod --name <NAMESPACE> -l app.kubernetes.io/name=anaconda-enterprise-ops-dashboard
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   The Dashboard process reads and caches its CSRF token only once, at startup, so it will not notice that the secret changed until it restarts.

   If this label selector does not match any pods in your environment, find the pod name first and delete it directly:

   ```sh theme={null}
   kubectl get pods --name <NAMESPACE> | grep dashboard
   kubectl delete pod --name <NAMESPACE> <POD_NAME>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.<br />
     Replace \<POD\_NAME> with the name of the Dashboard pod from the first command.
   </Comments>

4. Confirm the rollback succeeded:

   ```sh theme={null}
   kubectl get secret kubernetes-dashboard-csrf --name <NAMESPACE> -o jsonpath='{.data.csrf}'
   kubectl get pods --name <NAMESPACE> | grep dashboard
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   The `csrf` field should return a non-empty base64 value, and the Dashboard pod should show `1/1 Running`.