基于NVIDIA GPU虚拟化与VMware PVRDMA技术的Kubernetes集群部署参考指南（适用于ML和HPC工作负载）

创建于2019年8月15日

简介

本文档是面向RoCE加速的机器学习（ML）和HPC应用在Kubernetes（k8s）集群上的参考部署指南（RDG），结合NVIDIA vGPU和VMware PVRDMA技术、NVIDIA ConnectX®-4/5 VPI PCI Express网卡以及运行NVIDIA Onyx软件的NVIDIA Spectrum交换机。

本文档描述了构建包含1个Kubernetes主节点和4个工作节点的K8s集群的过程。

本文档假定已安装以下软件和驱动程序：

• VMware ESXi 6.7 Update 2, build 13006603
• VMware vCenter 6.7 Update 2, build 13007421
• NVIDIA Virtual GPU Software v9.0
• Distributed Switch 6.6.0
• ConnectX® Ethernet Driver for VMware® ESXi Server 4.17.13.1-1vmw.670.2.48.13006603
• SN2700 Open Ethernet Switch with NVIDIA Onyx
• CentOS 7.6 as Virtual Machine OS
• Kubernetes v1.15
• Kubeflow MPI Operator v1alpha2

参考资料

• Using GPUs with Virtual Machines on vSphere – Part 3: Installing the NVIDIA GRID Technology
• Virtual GPU Software Supported 产品
• NVIDIA Collective Communications Library (NCCL)
• What is RDMA over Converged Ethernet (RoCE)?
• Recommended Network Configuration Examples for RoCE Deployment
• GitHub - uber/horovod: Distributed training framework for TensorFlow, Keras, PyTorch, and MXNet
• Get Docker Engine - Community for CentOS
• Kubernetes (K8s)
• Kubeflow
• Kubeflow MPI Operator
• vSphere Command-Line Interface Concepts and Examples

组件概述

NVIDIA Virtual GPU (vGPU™)

NVIDIA vGPU软件产品支持图形和计算工作负载的GPU虚拟化。在本参考设计中，使用了NVIDIA Virtual Compute Server许可证。Virtual Compute Server使数据中心管理员能够在虚拟化环境中运行AI工作负载，从而提高安全性、利用率和可管理性。IT管理员可以使用VMware vSphere等虚拟机管理程序虚拟化工具（包括vCenter和vMotion）来管理其所有数据中心应用程序，包括在NVIDIA GPU上运行的计算密集型AI应用程序。NVIDIA Virtual Compute Server提供GPU共享（多个虚拟机可由单个GPU驱动）和GPU聚合（一个或多个GPU可驱动一个虚拟机）等功能。

虚拟机的远程直接内存访问

vSphere 6.5（及更高版本）引入了对具有ParaVirtualized RDMA（PVRDMA）网卡的虚拟机之间远程直接内存访问（RDMA）通信的支持。

RDMA

RDMA允许从一台计算机的内存直接访问另一台计算机的内存，而无需涉及操作系统或CPU。内存传输卸载到支持RDMA的主机通道适配器（HCA）。PVRDMA网卡在虚拟环境中提供远程直接内存访问。

在vSphere中使用RDMA

在vSphere中，虚拟机可以使用PVRDMA网卡与其他使用PVRDMA设备的虚拟机通信。这些虚拟机必须连接到同一vSphere Distributed Switch。PVRDMA设备会自动选择虚拟机之间的通信方式。

• 对于在同一ESXi主机上运行且无论是否具有物理RDMA设备的虚拟机，数据传输是两个虚拟机之间的memcpy。在这种情况下，不使用物理RDMA硬件。
• 对于位于不同ESXi主机上且具有物理RDMA连接的虚拟机，物理RDMA设备必须作为分布式交换机的上行链路。在这种情况下，通过PVRDMA的虚拟机间通信使用底层物理RDMA设备。
• 对于在不同ESXi主机上运行的两个虚拟机，如果至少有一个主机没有物理RDMA设备，则通信会回退到基于TCP的通道，性能会降低。

PVRDMA架构

加速虚拟机数据

Kubernetes

Kubernetes (K8s) 是一个开源容器编排系统，用于容器化应用程序的部署自动化、扩展和管理。

Kubeflow MPI Operator

Kubeflow是一个基于Google内部机器学习流水线的云原生机器学习平台。Kubeflow MPI Operator使得运行allreduce风格的分布式训练变得简单。

请参考官方文档 kubeflow.org。

Docker

Docker是一种执行操作系统级虚拟化的计算机程序。它用于运行称为容器的软件包，这些容器彼此隔离，并捆绑自己的应用程序、工具、库和配置文件。Docker容器由指定其精确内容的镜像创建，并且可以相互通信。

所有容器由单个操作系统内核运行，因此比虚拟机更轻量。

TensorFlow

TensorFlow是由Google Brain团队开发的开源软件库，用于进行机器学习和深度神经网络研究。

该库通过使用数据流图进行数值计算，其中图中的节点表示数学运算，图边表示在节点之间通信的多维数据数组（张量）。

TensorFlow支持Cuda和cuDNN（需注册）。

Horovod

Horovod是用于TensorFlow、Keras、PyTorch和MXNet的分布式训练框架。其目标是

Horovod is to make distributed Deep Learning fast and easy to use.

vSphere Distributed Switch

A vSphere Distributed Switch provides centralized management and monitoring of the networking configuration of all hosts that are associated with the switch. You must set up a distributed switch on a vCenter Server system, and its settings will be propagated to all hosts that are associated with the switch.

NVIDIA's Machine Learning

NVIDIA Ethernet solutions accelerate many of the world's leading artificial intelligence and machine learning platforms and wide range of applications, ranging from security, finance, and image and voice recognition, to self-driving cars and smart cities. NVIDIA solutions enable companies and organizations such as Baidu, NVIDIA, JD.com, Facebook, PayPal and more to leverage machine learning platforms to enhance their competitive advantage.

In this post we will show how to build most efficient Machine Learning cluster enhanced by RoCE over 100 Gbps Ethernet network.

Solution Overview

Setup

Before you start, make sure you are familiar with VMware vSphere and vCenter deploy and manage procedures.

This guide does not contain step-by-step instructions for performing all of the required standard vSphere and vCenter installation and configuration tasks because they often depend on customer requirements.

Make sure you are aware of the Kubernetes and Kubeflow MPI Operator, and the Uber Horovod distributed training framework (see GitHub - uber/horovod: Distributed training framework for TensorFlow, Keras, PyTorch, and MXNet for more info).

The below hardware specifications are used in this solution.

Solution Logical Design

Workload ESXi Logical Design

Bill of Materials

Note: This document does not cover the Management Cluster ESXi server configuration. Make sure to configure at least one ESXi server for vCenter and Kubernetes Master VMs. This document does not cover storage components. Make sure to configure your storage components to match your requirements.

The below table specifies the hardware components used in this solution:

Physical Network Wiring

Configuration

Network and Security Configuration

Before starting the configuration process make sure that the below requirements are met:

• Full network connectivity between all Kubernetes Virtual Machines by VMs Management network
• Full network connectivity between all Worker Kubernetes Virtual Machines by High-Speed Ethernet network
• Certain ports are open on your machines:
  • For Kubernetes cluster see here
  • CNI ports. As example Calico requires TCP port 179 for inbound and outbound traffic
  • The port 7070, which is the default port number used by NVIDIA GRID License Server
  • The port 28250 (incoming/outgoing), which is the default port number used by PVRDMA

vSphere Network Architecture - Management Cluster

vSphere Network Architecture - Workload Cluster

The below table provides details of Workload Cluster ESXi server names and their network configuration:

ESXi Server	Server Name	High-Speed Ethernet Network 192.168.0.0/24	Management Network 172.16.31.0/24
ESXi-m01	sc2esx21	none	eno0: From DHCP (reserved)
ESXi-w01	sc2esx24	vmk3: 192.168.0.24	eno0: From DHCP (reserved)
ESXi-w02	sc2esx25	vmk3: 192.168.0.25	eno0: From DHCP (reserved)
ESXi-w03	sc2esx26	vmk3: 192.168.0.26	eno0: From DHCP (reserved)
ESXi-w04	sc2esx27	vmk3: 192.168.0.27	eno0: From DHCP (reserved)

The below table provides details of VM names and their network configuration:

VM Name	...	...	...

VM	Server Name	High-Speed Ethernet Network (192.168.0.0/24)	Management Network (172.16.7.0/24)
VM-01	sckubw01	192.168.0.51	eno0: From DHCP (reserved)
VM-02	sckubw02	192.168.0.52	eno0: From DHCP (reserved)
VM-03	sckubw03	192.168.0.53	eno0: From DHCP (reserved)
VM-04	sckubw04	192.168.0.54	eno0: From DHCP (reserved)
VM-vCenter	sc2vc03	none	eno0: From DHCP (reserved)
VM-K8sMaster	sckubm01	none	eno0: From DHCP (reserved)

Host Configuration

注意： 所有工作节点必须具有相同的配置和相同的PCIe卡插槽位置。

在配置主机之前，请确保满足以下硬件和软件要求：

1. 服务器平台搭载基于NVIDIA ConnectX®-4/5 HCA设备的网卡。
2. 来自NVIDIA Scale-Out SN2000以太网交换机系列的NVIDIA交换机。
3. 已安装并配置VMware vSphere 6.7 u2集群。
4. 已安装并配置VMware vCenter 6.7 u2。
5. 来自支持的NVIDIA GPU列表的GPU卡。
6. 已授权的NVIDIA GRID（包括vSphere的VIB和Guest OS驱动程序）。
7. 在所有主机和虚拟机上安装并激活NTP。
8. 安装权限：安装过程需要在目标机器上具有管理员权限。

Virtual Machine Configuration

Master节点配置

VMware建议Kubernetes Master节点至少2个节点，并分配以下资源以支持最多50个节点：

k8s Nodes	CPU	Memory	Disk
Up to 50	4	16 GB	50 GB SSD

注意： 在本方案中，我们仅安装一个Kubernetes Master节点。

对于VMware vSphere上的Kubernetes部署，Master节点大小建议适用于vSphere虚拟机管理程序上托管的虚拟机。您应根据预期的工作负载、增长和系统要求来确定这些虚拟机的大小。

Worker节点要求

工作节点的大小严重依赖于您的工作负载。这些节点必须满足使用kubeadm安装Kubernetes的基本要求以及您的应用程序要求（更多信息请参见此处）。

CPU	Memory	Disk
4	24 GB	50 GB SSD

对于基于vSphere的部署，请使用上述工作节点大小指南作为起点。

由于大多数基于vSphere的部署最终可能会在单个虚拟机管理程序上托管多个工作节点，因此工作节点的大小必须考虑虚拟机管理程序/硬件故障的整体影响、可能受影响的节点数量以及这些节点上的工作负载数量。

建议在扩展和扩展工作节点之间找到平衡。

Virtual Machine OS

在本方案中，我们为所有Kubernetes节点使用CentOS 7.6。

Deployment

Workload Cluster Host and VM Configuration

PVRDMA Configuration

要在vSphere 6.5/6.7中使用PVRDMA，您的环境必须满足以下配置要求。

要为ESXi主机配置PVRDMA适配器，请参阅如何在VMware vSphere 6.5/6.7中配置PVRDMA。

vGPU Deployment and Configuration

NVIDIA vGPU作为许可产品在支持的Tesla GPU上提供。有关推荐的服务器平台和支持的GPU列表，请参阅支持虚拟机管理程序的发行说明，网址为NVIDIA Virtual GPU Software 文档。

注意： 试用许可证可用于PoC环境。可从https://www.nvidia.com/en-us/data-center/virtual-gpu-technology/下载（点击“FREE TRIAL”）。

要使用NVIDIA vGPU软件驱动程序进行vSphere部署，请遵循NVIDIA官方VIRTUAL GPU SOFTWARE User Guide中的Installing and Configuring the NVIDIA Virtual GPU Manager for VMware vSphere部分。您还可以参考VMware博客上的Using GPUs with Virtual Machines on vSphere – Part 3: Installing the NVIDIA GRID Technology。

Installing NVIDIA Container Toolkit

要安装

the NVIDIA Container Toolkit on CentOS 7.6 OS run the following command line:

VM Console

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | sudo tee /etc/yum.repos.d/nvidia-docker.repo
yum install -y nvidia-container-toolkit
systemctl restart docker

Application Deployment and Configuration

Kubernetes Cluster Deployment

You can create a single control-plane cluster using one of the below options:

• Creating a single control-plane cluster with kubeadm
• VMware Essential PKS
• VMware Enterprise PKS

Cluster Deployment Verification

To verify that the Kubernetes cluster is running, executing the following command on the Master Node:

kubectl get nodes -o wide

Install Kubeflow MPI Operator

To install Kubeflow MPI Operator, run the following command on a Kubernetes Master VM:

Kubernetes Master VM Console

curl https://raw.githubusercontent.com/Mellanox/mpi-operator/single_deploy/deploy/mpi-operator.yaml -O
kubectl apply -f mpi-operator.yaml

Running the Application

Before you start, make sure you are familiar with the following:

• Kubernetes application deployment
• Getting started with Kubeflow and MPI Operator

To create a MPI job:

1. Define an MPIJob config file. Below is an example of how to run a distributed TensorFlow training job with Horovod framework and RoCE acceleration:

   Kubernetes Master VM Console

   apiVersion: kubeflow.org/v1alpha2
   kind: MPIJob
   metadata:
     name: tensorflow-benchmarks
   spec:
     slotsPerWorker: 1
     cleanPodPolicy: Running
     mpiReplicaSpecs:
       Launcher:
         replicas: 1
         template:
            spec:
              containers:
              - image: mpioperator/tensorflow-benchmarks:latest
                name: tensorflow-benchmarks
                command:
                - mpirun
                - --allow-run-as-root
                - -np
                - "4"
                - -bind-to
                - none
                - -map-by
                - slot
                - -x
                - NCCL_DEBUG=INFO
                - -x
                - NCCL_IB_DISABLE=0
                - -x
                - NCCL_IB_GDR_LEVEL=0
                - -x
                - HOROVOD_MPI_THREADS_DISABLE=1
                - -x
                - LD_LIBRARY_PATH
                - -x
                - PATH
                - -mca
                - pml
                - ob1
                - -mca
                - btl
                - ^openib
                - python
                - scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
                - --model=resnet50
                - --batch_size=32
                - --variable_update=horovod
                - --use_fp16
                - --xla=True
                - --num_batches=10000
       Worker:
         replicas: 4
         template:
           spec:
             containers:
             - image: mpioperator/tensorflow-benchmarks:latest
               name: tensorflow-benchmarks
               securityContext:
                 privileged: true
               volumeMounts:
               - mountPath: /dev/infiniband
                 name: infiniband
               resources:
                 limits:
                   nvidia.com/gpu: 1
            volumes:
            - name: infiniband
              hostPath:
                path: /dev/infiniband
   

   Warning: See Tensorflow benchmark example config file for launching a multi-node TensorFlow benchmark training job. Please see the tensorflow-benchmarks-roce.yaml * (TBD) * file adapted for our environment and includes all 4 nodes.

   To change job so that it would run on TCP, modify the following parameters: NCCL_IB_DISABLE=0 to NCCL_IB_DISABLE=1 and HOROVOD_MPI_THREADS_DISABLE=1 to HOROVOD_MPI_THREADS_DISABLE=0

2. Deploy the MPIJob resource to start the training session with the following command:

   Kubernetes Master VM Console

   kubectl create -f tensorflow-benchmarks-roce.yaml
   

3. Once the MPIJob resource is created, you can monitor the job status from the "Status" section. You can also see the created pods matching the specified number of GPUs. Below is a sample output of a successful job:

   Kubernetes Master VM Console

   kubectl get -o yaml mpijob tensorflow-benchmarks-roce
   

   The training session should run for 100 steps and takes a few minutes on a GPU cluster. You can inspect the logs to see the training progress. When the job starts, access the logs from the launcher pod using the following command:

   Kubernetes Master VM Console

   PODNAME=$(kubectl get pods -l mpi_job_name=tensorflow-benchmarks,mpi_role_type=launcher -o name)
   kubectl logs -f ${PODNAME}
   

Performance Testing

The following are the results of our performance tests:

Done!

Authors

Boris KovalevBoris Kovalev has worked for the past several years as a 解决方案 Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the NVIDIA Documents website.