Preparing a Gravity environment for Workbench

Determining the resource requirements for a Kubernetes cluster depends on a number of different factors, including what type of applications you are going to be running, the number of users that are active at once, and the workloads you will be managing within the cluster. Data Science & AI Workbench’s performance is tightly coupled with the health of your Kubernetes stack, so it is important to allocate enough resources to manage your users workloads. Generally speaking, your system should contain at least 1 CPU, 1GB of RAM, and 5GB of disk space for each project session or deployment.

To install Workbench successfully, your systems must meet or exceed the requirements listed below. Anaconda has created a pre-installation checklist to help prepare you for installation. The checklist verifies that your cluster has the necessary resources are reserved and is ready to install Workbench. Anaconda’s Implementation team will review the checklist with you prior to your installation.

You can initially install Workbench on up to five nodes. Once initial installation is complete, you can add or remove nodes as needed. Anaconda recommends having one master and one worker node per cluster. For more information, see Adding and removing nodes.

For historical information and details regarding Anaconda’s policies related to Gravity, see our Gravity update policy.

​

Hardware requirements

Anaconda’s hardware recommendations ensure a reliable and performant Kubernetes cluster.

The following are minimum specifications for the master and worker nodes, as well as the entire cluster.

mkfs.xfs -n ftype=1 <PATH/TO/YOUR/DEVICE>

​

Multi-node synchronization

When installing Workbench on a system with multiple nodes, verify that the clock of each node is in sync with the others prior to installation. Anaconda recommends using the Network Time Protocol (NTP) to synchronize computer system clocks automatically over a network. For step-by-step instructions, see How to Synchronize Time with Chrony NTP in Linux.

​

Disk IOPS requirements

Master and worker nodes require a minimum of 3000 concurrent Input/Output operations Per Second (IOPS).

​

Cloud performance requirements

Requirements for running Workbench in the cloud relate to compute power and disk performance. Make sure your chosen cloud platform meets these minimum specifications:

​

Operating system requirements

Workbench currently supports the following Linux versions:

• RHEL/CentOS 7.x, 8.x
• Ubuntu 16.04
• SUSE 12 SP2, 12 SP3, 12 SP5 (Requires you set DefaultTasksMax=infinity in /etc/systemd/system.conf)

​

Security requirements

• If your Linux system utilizes an antivirus scanner, make sure the scanner excludes the /var/lib/gravity volume from its security scans.
• Installation requires that you have sudo access.
• Nodes running CentOS or RHEL must make sure that Security Enhanced Linux (SELinux) set to either disabled or permissive mode in the /etc/selinux/config file.

getenforce

​

Kernel module requirements

Kubernetes relies on certain functionalities provided by the Linux kernel. The Workbench installer verifies that the following kernel modules (required for Kubernetes to function properly) are present, and notifies you if any are not loaded.

# Replace <MODULE_NAME> with a module name
lsmod | grep <MODULE_NAME>

# Replace <MODULE_NAME> with a module name
sudo modprobe <MODULE_NAME>

# Replace <MODULE_NAME> with a module name
sudo echo -e '<MODULE_NAME>' > /etc/modules-load.d/<MODULE_NAME>.conf

​

System control settings

Workbench requires the following Linux sysctl settings to function properly:

# Replace <SYSCTL_SETTING> with a system control setting
sudo sysctl -w <SYSCTL_SETTING>=1

# Replace  <SYSCTL_SETTING> with a system control setting
sudo echo -e "<SYSCTL_SETTING> = 1" > /etc/sysctl.d/10-<SYSCTL_SETTING>.conf

​

GPU requirements

Workbench requires that you install a supported version of the NVIDIA Compute Unified Device Architecture (CUDA) driver on the host OS of any GPU worked node.

Currently, Workbench supports the following CUDA driver versions:

• CUDA 10.2
• CUDA 11.2
• CUDA 11.4
• CUDA 11.6

You can obtain the driver you need a few different ways.

• Use the package manager or the Nvidia runfile to download the file directly.
• For SLES, CentOS, and RHEL, you can get a supported driver using rpm (local) or rpm (network).
• For Ubunutu, you can get a driver using deb (local) or deb (network).

GPU deployments should use one of the following models:

• Tesla V100 (recommended)
• Tesla P100 (adequate)

Theoretically, Workbench will work with any GPU card compatible with the CUDA drivers, as long as they are properly installed. Other cards supported by CUDA 11.6:

• A-Series: NVIDIA A100, NVIDIA A40, NVIDIA A30, NVIDIA A10
• RTX-Series: RTX 8000, RTX 6000, NVIDIA RTX A6000, NVIDIA RTX A5000, NVIDIA RTX A4000, NVIDIA T1000, NVIDIA T600, NVIDIA T400
• HGX-Series: HGX A100, HGX-2
• T-Series: Tesla T4
• P-Series: Tesla P40, Tesla P6, Tesla P4
• K-Series: Tesla K80, Tesla K520, Tesla K40c, Tesla K40m, Tesla K40s, Tesla K40st, Tesla K40t, Tesla K20Xm, Tesla K20m, Tesla K20s, Tesla K20c, Tesla K10, Tesla K8
• M-Class: M60, M40 24GB, M40, M6, M4

Support for GPUs in Kubernetes is still a work in progress, and each cloud vendor provides different recommendations. For more information about GPUs, see Understanding GPUs.

​

Network requirements

Workbench requires the following network ports to be externally accessible:

These ports need to be externally accessible during installation only, and can be closed after completing the install process:

The following ports are used for cluster operation, and must be open internally, between cluster nodes:

Make sure that the firewall is permanently set to keep the required ports open, and will save these settings across reboots. Then restart the firewall to load your changed settings.

You’ll also need to update your firewall settings to ensure that the 10.244.0.0/16 pod subnet and 10.100.0.0/16 service subnet are accessible to every node in the cluster, and grant all nodes the ability to communicate via their primary interface.

For example, if you’re using iptables:

# Replace <NODE_IP> with the internal IP address(es) used by all nodes in the cluster to connect to the master node
iptables -A INPUT -s 10.244.0.0/16 -j ACCEPT
iptables -A INPUT -s 10.100.0.0/16 -j ACCEPT
iptables -A INPUT -s <NODE_IP> -j ACCEPT

If you plan to use online package mirroring, allowlist the following domains in your network’s firewall settings:

• repo.anaconda.com
• anaconda.org
• conda.anaconda.org
• binstar-cio-packages-prod.s3.amazonaws.com

To use Workbench in conjucntion with Anaconda Navigator in online mode, allowlist the following sites in your network’s firewall settings as well:

• https://repo.anaconda.com (or for older versions of Navigator and conda)
• https://conda.anaconda.org if any users will use conda-forge and other channels on Anaconda.org
• google-public-dns-a.google.com (8.8.8.8:53) to check internet connectivity with Google Public DNS

​

TLS/SSL certificate requirements

Workbench uses certificates to provide transport layer security for the cluster. Self-signed certificates are generated during the initial installation. Once installation is complete, you can configure the platform to use your organizational TLS/SSL certificates.

You can purchase certificates commercially, or generate them using your organization’s internal public key infrastructure (PKI) system. When using an internal PKI-signed setup, the CA certificate is inserted into the Kubernetes secret.

In either case, the configuration will include the following:

• A certificate for the root certificate authority (CA)
• An intermediate certificate chain
• A server certificate
• A certificate private key

For more information about TLS/SSL certificates, see Updating TLS/SSL certificates.

​

DNS requirements

Workbench assigns unique URL addresses to deployments by combining a dynamically generated universally unique identifier (UUID) with your organization’s domain name, like this: https://uuid001.anaconda.yourdomain.com.

This requires the use of wildcard DNS entries that apply to a set of domain names such as *.anaconda.yourdomain.com.

For example, if you are using the domain name anaconda.yourdomain.com with a master node IP address of 12.34.56.78, the DNS entries would be as follows:

anaconda.yourdomain.com IN A 12.34.56.78
*.anaconda.yourdomain.com IN A 12.34.56.78

The master node’s hostname and the wildcard domains must be resolvable with DNS from the master nodes, worker nodes, and the end user machines. To ensure the master node can resolve its own hostname, distribute any /etc/hosts entries to the gravity environment.

sudo systemctl status dnsmasq

sudo systemctl stop dnsmasq
sudo systemctl disable dnsmasq

​

Browser requirements

Workbench supports the following web browsers:

• Chrome 39+
• Firefox 49+
• Safari 10+

The minimum browser screen size for using the platform is 800 pixels wide and 600 pixels high.

​

Verifying system requirements

The installer performs pre-installation checks, and only allows installation to continue on nodes that are configured correctly, and include the required kernel modules. If you want to perform the system check yourself prior to installation, you can run the following commands from the installer directory, ~/anaconda-enterprise-<VERSION>, on your intended master and worker nodes:

To perform system checks on the master node, run the following command as sudo or root user:

sudo ./gravity check --profile ae-master

To perform system checks on a worker node, run the following command as sudo or root user:

sudo ./gravity check --profile ae-worker

If all of the system checks pass and all requirements are met, the output from the above commands will be empty. If the system checks fail and some requirements are not met, the output will indicate which system checks failed.

​

Pre-installation checklist

Anaconda has created this pre-installation checklist to help you verify that you have properly prepared your environment prior to installation. You can run the system verification checks to automatically verify many of the requirements for you.