> ## Documentation Index
> Fetch the complete documentation index at: https://anaconda.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

export const Comments = ({children}) => {
  return <div class="my-4 px-5 py-4 overflow-hidden rounded-2xl flex gap-3 border border-zinc-500/20 bg-zinc-50/50 dark:border-zinc-500/30 dark:bg-zinc-500/10" data-callout-type="comments">
      <div class="w-4">
        <svg width="14" height="14" viewBox="0 0 640 640" fill="currentColor" xmlns="http://www.w3.org/2000/svg" class="w-5 h-5" aria-label="Comments">
            <path d="M320 112C434.9 112 528 205.1 528 320C528 434.9 434.9 528 320 528C205.1 528 112 434.9 112 320C112 205.1 205.1 112 320 112zM320 576C461.4 576 576 461.4 576 320C576 178.6 461.4 64 320 64C178.6 64 64 178.6 64 320C64 461.4 178.6 576 320 576zM280 400C266.7 400 256 410.7 256 424C256 437.3 266.7 448 280 448L360 448C373.3 448 384 437.3 384 424C384 410.7 373.3 400 360 400L352 400L352 312C352 298.7 341.3 288 328 288L280 288C266.7 288 256 298.7 256 312C256 325.3 266.7 336 280 336L304 336L304 400L280 400zM320 256C337.7 256 352 241.7 352 224C352 206.3 337.7 192 320 192C302.3 192 288 206.3 288 224C288 241.7 302.3 256 320 256z" />
        </svg>
      </div>
      <div class="text-sm prose min-w-0 w-full">
        {children}
      </div>
    </div>;
};

Anaconda Enterprise provides detailed logs and monitoring information related to the Kubernetes services and containers it uses. You can use the Operations Center and Kubernetes CLI to access this information, to help diagnose and debug errors that you or other users may encounter while using the platform.

***

## The Anaconda Enterprise cluster

As an Operations Center Admin, you can use the [Operations Center](./admin/consoles#the-operations-center) to configure and monitor the platform.

**To access the Operations Center:**

1. Log in to Anaconda Enterprise, select the **Menu** icon <Icon icon="user" iconType="regular" /> in the top right corner, and click the **Administrative Console** link displayed at the bottom of the slide out window.

2. Click **Manage Resources**.

3. Login to the Operations Center using the Administrator credentials [configured after installation](./install/config).

**To view resource utilization:**

1. Select **Servers** in the menu on the left.

2. Click on the Private IP address of the Anaconda Enterprise master node, and select **SSH login as root**.

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/u4MsRLDCM8Pedgc_/images/ssh_master_node.png?fit=max&auto=format&n=u4MsRLDCM8Pedgc_&q=85&s=8047348d82826e2b101435fbf6c62432" alt="" width="2032" height="960" data-path="images/ssh_master_node.png" />
   </Frame>

3. To display the current resource utilization of each *node* in the cluster, run this command:

   ```
   kubectl top nodes --heapster-namespace=monitoring

   ```

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/q5KdI2mr6ZKMYRZp/images/node_utilization.png?fit=max&auto=format&n=q5KdI2mr6ZKMYRZp&q=85&s=c5d51760a9081fe15781107167ab8487" alt="" width="1924" height="1014" data-path="images/node_utilization.png" />
   </Frame>

   <Note>
     This is actual resource utilization, not limits or requests.
   </Note>

4. To view utilization *and requests* for a particular node, run the `kubectl describe node` command against the IP address for the node (listed under `NAME`). For example:

   ```
   kubectl describe node 172.31.25.175

   ```

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/q5KdI2mr6ZKMYRZp/images/node_requests.png?fit=max&auto=format&n=q5KdI2mr6ZKMYRZp&q=85&s=dc5b200bdd57abed2256fe057e995046" alt="" width="2640" height="1494" data-path="images/node_requests.png" />
   </Frame>

5. To view the resource utilization per *pod*, run this command:

   ```
   kubectl top pods --heapster-namespace=monitoring

   ```

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/GAawxvWy-HUVSCqf/images/pod_utilization.png?fit=max&auto=format&n=GAawxvWy-HUVSCqf&q=85&s=47007926120c9a332678eb9d0386fdec" alt="" width="2018" height="946" data-path="images/pod_utilization.png" />
   </Frame>

6. To view the current status of all pods in the cluster, run `kubectl get pods`.

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/opbTXGcYjx4zM8zO/images/get_pods.png?fit=max&auto=format&n=opbTXGcYjx4zM8zO&q=85&s=6e3c770713da21027cfece2eadb56dc4" alt="" width="2510" height="1236" data-path="images/get_pods.png" />
   </Frame>

   The following table summarizes common pod states:

   | Status                | Description                                                              |
   | :-------------------- | :----------------------------------------------------------------------- |
   | Running               | The pod has been bound to a node, and at least one container is running. |
   | Pending               | The pod is waiting for one or more container images to be created.       |
   | Terminating           | The pod is in the process of being terminated.                           |
   | Error                 | An error has occurred with the pod.                                      |
   | Init:CrashLoopBackoff | The pod failed to start, and will make another attempt in a few minutes. |

7. To view information for a particular pod, run the `kubectl describe pod` command against the pod (listed under `NAME`). For example:

   ```
   kubectl describe pod anaconda-session-89747d7fdb154b89b182d5eaa25b2e59-7f497db55wl9g
   ```

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/atV1FF4bOzhiwIx6/images/describe_pod.png?fit=max&auto=format&n=atV1FF4bOzhiwIx6&q=85&s=f454ec630bf772401aea2191cc24fb64" alt="" width="2634" height="1588" data-path="images/describe_pod.png" />
   </Frame>

You can also use the Operations Center **Logs** to gain insights into pod behavior and troubleshoot issues. See logging for more information.

***

## User errors

If a user experiences issues within a Notebook session, have them send you the name of the pod associated with their project session. They can obtain this information by running the `hostname` command from within a Jupyter Notebook or terminal window.

<Frame>
  <img src="https://mintcdn.com/anaconda-29683c67/q5KdI2mr6ZKMYRZp/images/notebook_hostname.png?fit=max&auto=format&n=q5KdI2mr6ZKMYRZp&q=85&s=969baba74d252658ea0640c49285348b" alt="" width="2132" height="826" data-path="images/notebook_hostname.png" />
</Frame>

<Frame>
  <img src="https://mintcdn.com/anaconda-29683c67/u4MsRLDCM8Pedgc_/images/terminal_hostname.png?fit=max&auto=format&n=u4MsRLDCM8Pedgc_&q=85&s=2fde8aa2cb719fc7fa214a17acc709f0" alt="" width="1886" height="626" data-path="images/terminal_hostname.png" />
</Frame>

You can then use the commands described above or the Operation Center's **Monitoring** and **Logs** features to investigate the issue. See Monitoring sessions and deployments for more information.

<Frame>
  <img src="https://mintcdn.com/anaconda-29683c67/eSsEBS6xLmJK8V-1/images/monitoring-pods.png?fit=max&auto=format&n=eSsEBS6xLmJK8V-1&q=85&s=369fdeed7682a324430dcb7ee738f1ca" alt="" width="2216" height="1236" data-path="images/monitoring-pods.png" />
</Frame>

<Tip>
  As an Administrator, you can also use the [Authentication Center](./admin/consoles#the-authentication-center) to impersonate a user to try to reproduce the problem they are experiencing.
</Tip>

**To access the Authentication Center:**

1. Login to Anaconda Enterprise, click the **Menu** icon <Icon icon="user" iconType="regular" /> in the top right corner, then click the **Administrative Console** link in the bottom of the slideout menu.

2. Click **Manage Users**.

3. In the Manage menu on the left, click **Users**.

4. On the **Lookup** tab, click **View all users** to list every user in the system, or search the user database for all users that match the criteria you enter, based on their first name, last name, or email address.

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/QCWY8EsGZWJYinOU/images/impersonate_users.png?fit=max&auto=format&n=QCWY8EsGZWJYinOU&q=85&s=ee2ca511ac1816a5a5e435273c1616b9" alt="" width="1948" height="1102" data-path="images/impersonate_users.png" />
   </Frame>

5. Click **Impersonate** in the row of **Actions** for the user to display a table of all **Applications** this user has interacted with on the platform, including editor sessions and deployments.

   <Frame>
     <img src="https://mintcdn.com/anaconda-29683c67/kBuj3mPdZoBk8YoG/images/user_applications.png?fit=max&auto=format&n=kBuj3mPdZoBk8YoG&q=85&s=3ee7806b4e164866930bd58359ab89fc" alt="" width="2368" height="1422" data-path="images/user_applications.png" />
   </Frame>

6. Click the **Anaconda Platform** lik to interact with Anaconda Enterprise as the user.

See Managing users for more information on managing users.

***

## Editor sessions

To help you troubleshoot issues with editor sessions, it might be helpful to understand what is happening "behind the scenes".

* When a user starts a session, Anaconda Enterprise launches the appropriate editor for them to work with their project files. In the background, the editor environment and other services are running in Docker containers.
* To improve startup time for projects, the editor container includes conda environments for each of the [project template environments](./data-science-workflows/templates/main) provided by the platform. These environments are stored in `/opt/continuum/anaconda/envs`, along with any custom environments created *during the editor session*.
* The project repository is cloned into `/opt/continuum/project`. (Only changes to files in this directory can be saved to the repository.)
* The `anaconda-project prepare` command runs, scans the project's `anaconda-project.yml` file for new packages and environments, and installs them into the running session.

  During this phase, you can monitor the progress by watching the output of `/opt/continuum/preparing`.

  When this process completes, the `/opt/continuum/prepare.log` is created.

<Warning>
  Any changes made to the container image will be lost when the session stops, so any packages installed from the command line are available during the current session only. To persist package installs across sessions, they must be added to the project's `anaconda-project.yml` file.
</Warning>

## Helm rollback fails with a kubernetes-dashboard-csrf conflict

Running `helm rollback anaconda-enterprise <REVISION>` can fail with an error similar to:

```txt theme={null}
Error: conflict occurred while applying object default/kubernetes-dashboard-csrf /v1, Kind=Secret: Apply failed with 1 conflict: conflict with "dashboard" using v1: .data.csrf
```

This occurs because the `kubernetes-dashboard-csrf` secret is managed by both the Helm chart and the Kubernetes Dashboard application, which writes its own CSRF token to the same secret at runtime. `helm rollback` reapplies the secret content captured at the time of the target revision, which is typically out of date by the time a rollback is needed, causing Kubernetes to reject the conflicting write. This affects Workbench chart versions prior to `ae-helm-chart-1.1-364`, on both K3s and BYOK8s installations.

To resolve it:

1. Delete the conflicting secret and retry the rollback:

   ```sh theme={null}
   kubectl delete secret kubernetes-dashboard-csrf --name <NAMESPACE>
   helm rollback anaconda-enterprise <REVISION> --name <NAMESPACE>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.<br />
     Replace \<REVISION> with the target revision number.
   </Comments>

2. Restart the Workbench Dashboard pod so it picks up the recreated secret (it caches its token at startup and will not notice the change otherwise):

   ```sh theme={null}
   kubectl delete pod --name <NAMESPACE> -l app.kubernetes.io/name=anaconda-enterprise-ops-dashboard
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   If no pods match this selector in your environment, locate the pod by name and delete it directly:

   ```sh theme={null}
   kubectl get pods --name <NAMESPACE> | grep dashboard
   kubectl delete pod --name <NAMESPACE> <POD_NAME>
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.<br />
     Replace \<POD\_NAME> with the name of the Dashboard pod from the first command.
   </Comments>

3. Confirm the secret and pod are healthy:

   ```sh theme={null}
   kubectl get secret kubernetes-dashboard-csrf --name <NAMESPACE> -o jsonpath='{.data.csrf}'
   kubectl get pods --name <NAMESPACE> | grep dashboard
   ```

   <Comments>
     Replace \<NAMESPACE> with the namespace Workbench is installed in.
   </Comments>

   The `csrf` field should return a non-empty base64 value, and the Dashboard pod should show `1/1 Running`.

See the [Bring Your Own Kubernetes (BYOK8s) Upgrade](/docs/data-science/5.6.1/install/migrate/byok8s-upgrade#troubleshooting-a-failed-rollback) guide for the full walkthrough, including how to find the revision to roll back to.
