> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Bring Your Own Kubernetes (BYOK8s) Upgrade

export const Comments = ({children}) => {
  return <div class="my-4 px-5 py-4 overflow-hidden rounded-2xl flex gap-3 border border-zinc-500/20 bg-zinc-50/50 dark:border-zinc-500/30 dark:bg-zinc-500/10" data-callout-type="comments">
      <div class="w-4">
        <svg width="14" height="14" viewBox="0 0 640 640" fill="currentColor" xmlns="http://www.w3.org/2000/svg" class="w-5 h-5" aria-label="Comments">
            <path d="M320 112C434.9 112 528 205.1 528 320C528 434.9 434.9 528 320 528C205.1 528 112 434.9 112 320C112 205.1 205.1 112 320 112zM320 576C461.4 576 576 461.4 576 320C576 178.6 461.4 64 320 64C178.6 64 64 178.6 64 320C64 461.4 178.6 576 320 576zM280 400C266.7 400 256 410.7 256 424C256 437.3 266.7 448 280 448L360 448C373.3 448 384 437.3 384 424C384 410.7 373.3 400 360 400L352 400L352 312C352 298.7 341.3 288 328 288L280 288C266.7 288 256 298.7 256 312C256 325.3 266.7 336 280 336L304 336L304 400L280 400zM320 256C337.7 256 352 241.7 352 224C352 206.3 337.7 192 320 192C302.3 192 288 206.3 288 224C288 241.7 302.3 256 320 256z" />
        </svg>
      </div>
      <div class="text-sm prose min-w-0 w-full">
        {children}
      </div>
    </div>;
};

[Contact the Anaconda implementation team](https://anaconda.zendesk.com/hc/en-us/requests/new) before you begin for assistance upgrading your version of Data Science & AI Workbench. Follow along with these instructions as an Anaconda implementation team member guides you through the upgrade process. They will provide you with a Helm chart archive that contains all the components needed for upgrading.

<Warning>
  * Current versions of Workbench no longer support Postgres versions 9.6 and older. To align with these changes, you must update your Postgres to a supported version. [Verify your Postgres version is compatible before you begin your upgrade](/docs/data-science/latest/admin/advanced/postgres-upgrade).
  * It is important to [stop all sessions](/docs/data-science/latest/admin/session-mgmt#stopping-sessions) prior to upgrading. If you do not, sessions that are terminated as part of the upgrade process must be restarted manually post upgrade.
  * Sessions can also be stopped programmatically using `ae5-tools` by running the following command in a terminal that has access to your Workbench cluster over the network:

    ```sh theme={null}
    ae5 session list --columns=id --no-header | xargs -n1 ae5 session stop --yes
    ```
</Warning>

## Prerequisites

* You must have access to the [Administrative Server](/docs/data-science/latest/environment-prep/byok8s-prep#administration-server).
* You must verify the Service Account used for the upgrade has the [correct permissions](/docs/data-science/latest/environment-prep/byok8s-prep#namespace-service-account-rbac).
* Air-gapped servers must have the Docker images added to their Docker image repository.

## Upgrading

Upgrade your BYOK8s cluster by performing the following steps:

1. Log in to the service account on the Administrative Server running your Workbench software.

2. [Create a backup of your cluster configurations](/docs/data-science/latest/install/migrate/backup-restore#run-the-backup-script).

   <Note>
     This backup is the only reliable way to undo this upgrade later. Some Workbench components run forward database migrations when they start under the new chart version (confirmed for the git-storage service), and `helm rollback` does not reverse those migrations. Once a migration has run, rolling the release back to the prior version leaves that component unable to start against the now-newer database schema.

     Confirm the backup completed successfully before continuing. If you need to undo this upgrade after it has run, restore from this backup rather than relying on `helm rollback` alone.
   </Note>

3. Unpack the Helm charts provided by Anaconda onto the Administrative Server:

   ```sh theme={null}
   tar xvzf <HELM_CHARTS>
   ```

   <Comments>
     Replace \<HELM\_CHARTS> with the Helm charts you received from Anaconda.
   </Comments>

4. Save your current configurations with the `extract_config.sh` script delivered with your Helm chart:

   ```sh theme={null}
   NAMESPACE=<NAMESPACE> ./extract_config.sh
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   The `extract_config.sh` script creates a file called `helm_values.yaml` and saves it in the directory where the script was run.

5. Verify the information captured in the `helm_values.yaml` file is correct and contains all of your current cluster configuration settings.

6. Begin the upgrade:

   ```sh theme={null}
   helm upgrade --values ./helm_values.yaml anaconda-enterprise ./Anaconda-Enterprise/
   ```

   If the upgrade is successful, your output will look like this:

   ```
   Release "anaconda-enterprise" has been upgraded. Happy Helming!
   NAME: anaconda-enterprise
   LAST DEPLOYED: Wed Dec  7 21:52:34 2022
   NAMESPACE: <YOUR_NAMESPACE>
   STATUS: deployed
   REVISION: 10
   TEST SUITE: None
   ```

   The duration of the upgrade depends on both the cluster node count and the speed of the new Docker images loading into each node.

## Verifying your upgrade

1. Verify all pods are running:

   ```sh theme={null}
   sudo kubectl get pods
   ```
2. Open a web browser and navigate to your Authentication Center.

   ```txt theme={null}
   https://<FQDN>/auth/admin
   ```

   <Comments>
     Replace \<FQDN> with the fully qualified domain name of your Workbench server.
   </Comments>
3. Select **Users** from the left-hand navigation, then click **View all users** and verify your users’ data is present.
4. Open a new tab in your web browser and navigate to your Workbench URL. Log in using the same credentials you used for your previous installation.
5. Review the **Projects** list to verify that all project data has been restored.
6. Verify your deployments have returned to a **started** state.

<Warning>
  If you are upgrading from Workbench version 5.6.2 or older *and* have RStudio installed, *you must* [reinstall RStudio](/docs/data-science/latest/admin/advanced/rstudio-install). This is due to the upgrade of the Workbench base image from UBI8 to UBI9 in version 5.7. UBI9 requires a different binary for RStudio.
</Warning>

## Troubleshooting a failed rollback

Rolling back a Workbench upgrade with `helm rollback` can fail with a conflict on the `kubernetes-dashboard-csrf` secret:

```txt wrap theme={null}
Error: conflict occurred while applying object default/kubernetes-dashboard-csrf /v1, Kind=Secret: Apply failed with 1 conflict: conflict with "dashboard" using v1: .data.csrf
```

<Accordion title="Why this happens">
  The `kubernetes-dashboard-csrf` secret is written by two different owners: the *Helm chart* and the *Kubernetes Dashboard application* itself, which generates a real CSRF token at runtime under its own field manager.

  * `helm upgrade` re-renders the chart on every run, so it stays in sync with whatever the Dashboard has since written.
  * `helm rollback` does not re-render anything; it reapplies the exact secret content captured at the time the target revision was installed or upgraded to.

  By the time you need to roll back, that captured content has almost always diverged from what the Dashboard has written since, and Kubernetes' server-side apply rejects the conflicting write. Deleting the secret before retrying removes the stale content so the rollback can apply cleanly, and restarting the Dashboard pod ensures it picks up the resulting token instead of continuing to use the one it cached at startup.
</Accordion>

This affects Workbench chart versions prior to `ae-helm-chart-1.1-364`. Use the following steps to complete the rollback:

1. Find the revision number you want to roll back to:

   ```sh theme={null}
   helm history anaconda-enterprise --name <NAMESPACE>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

2. Delete the conflicting secret, then retry the rollback:

   ```sh theme={null}
   kubectl delete secret kubernetes-dashboard-csrf --name <NAMESPACE>
   helm rollback anaconda-enterprise <REVISION> --name <NAMESPACE>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.<br />
     Replace \<REVISION> with the revision number from the previous step.
   </Comments>

   Kubernetes and Helm recreate the secret cleanly once there is no longer a live value to conflict with.

3. Restart the Workbench Dashboard pod:

   ```sh theme={null}
   kubectl delete pod --name <NAMESPACE> -l app.kubernetes.io/name=anaconda-enterprise-ops-dashboard
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   The Dashboard process reads and caches its CSRF token only once, at startup, so it will not notice that the secret changed until it restarts.

   If this label selector does not match any pods in your environment, find the pod name first and delete it directly:

   ```sh theme={null}
   kubectl get pods --name <NAMESPACE> | grep dashboard
   kubectl delete pod --name <NAMESPACE> <POD_NAME>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.<br />
     Replace \<POD\_NAME> with the name of the Dashboard pod from the first command.
   </Comments>

4. Confirm the rollback succeeded:

   ```sh theme={null}
   kubectl get secret kubernetes-dashboard-csrf --name <NAMESPACE> -o jsonpath='{.data.csrf}'
   kubectl get pods --name <NAMESPACE> | grep dashboard
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   The `csrf` field should return a non-empty base64 value, and the Dashboard pod should show `1/1 Running`.