超擎数智超擎数智

RDG：通过RoCE网络使用RAPIDS加速器加速Apache Spark 3.0

创建于2020年6月14日。本参考部署指南（RDG）演示了在Mellanox NVIDIA RoCE/UCX端到端25 Gb/s以太网解决方案上，使用RAPIDS Accelerator for Apache Spark部署GPU和网络加速的Apache Spark 3.0多节点集群的步骤。

文档目录

创建于 Jun 14, 2020

Introduction

本参考部署指南（RDG）演示了在Mellanox NVIDIA RoCE/UCX端到端25 Gb/s以太网解决方案上，使用RAPIDS Accelerator for Apache Spark部署GPU和网络加速的Apache Spark 3.0多节点集群的步骤。

本RDG提供了预构建的Spark 3.0集群（三个物理节点，运行Ubuntu 18.04）的安装指南，并包含逐步操作说明，以使用Mellanox推荐设置（在主机和交换机侧）为RoCE流量准备网络。

HDFS集群包括2个数据节点和1个名称节点服务器。

Abbreviation and Acronym List

Term Definition
AOC Active Optical Cable
DAC Direct Attach Copper cable
DHCP Dynamic Host Configuration Protocol
DNS Domain Name System
GDR GPUDirect
GPU Graphics Processing Unit
OFED OpenFabrics Enterprise Distribution
MPI Message Passing Interface
NGC NVIDIA GPU Cloud
RDG Reference Deployment Guide
RDMA Remote Direct Memory Access
RoCE RDMA over Converged Ethernet
GPU Graphics Processing Unit
UCX Unified Communication X

References

Key Components and Technologies

  • NVIDIA® T4 GPU

    The NVIDIA® T4 GPU accelerates diverse cloud workloads, including high-performance computing, deep-learning training and inference, machine learning, data analytics, and graphics. Based on the NVIDIA Turing™ architecture and packaged in an energy-efficient 70-watt, small PCIe form factor, T4 is optimized for mainstream computing environments and features multi-precision Turing Tensor Cores and new RT Cores. Combined with accelerated containerized software stacks from NGC, T4 delivers revolutionary performance at scale.

  • NVIDIA Cumulus Linux

    Cumulus Linux is the only open network OS that allows you to affordably build and efficiently operate your network like the world’s largest data center operators, unlocking web-scale networking for businesses of all sizes.

  • NVIDIA Mellanox Ethernet Network Interface Cards enable the highest performance and efficiency for data center hyper-scale, public and private clouds, storage, machine learning and deep learning, artificial intelligence, big data and Telco platforms and applications.

  • NVIDIA Mellanox Spectrum Open 以太网交换机

    The Mellanox Spectrum® switch family provides the most efficient network solution for the ever-increasing performance demands of data center applications. The Spectrum product family includes a broad portfolio of Top-of-Rack (TOR) and aggregation switches that range from 16 to 128 physical ports, with Ethernet data rates of 1GbE, 10GbE, 25GbE, 40GbE, 50GbE, 100GbE and 200GbE per port. Spectrum Ethernet switches are ideal to build cost-effective and scalable data center network fabrics that can scale from a few nodes to tens-of-thousands of nodes.

  • NVIDIA Mellanox LinkX Ethernet Cables and Transceivers

    Mellanox LinkX® cables and transceivers make 100Gb/s deployments as easy and as universal as 10Gb/s links. Because Mellanox offers one of industry’s broadest portfolio of 10, 25, 40, 50,100 and 200Gb/s Direct Attach Copper cables (DACs), Copper Splitter cables, Active Optical Cables (AOCs) and Transceivers, every data center reach from 0.5m to 10km is supported. To maximize system performance, Mellanox tests every product in an end-to-end environment ensuring a Bit Error Rate of less than 1e-15. A BER of 1e-15 is 1000x better than many competitors.

  • Spark GPU Scheduling Overview

    Spark 3.0 now supports GPU scheduling as long as you are using a cluster manager that supports it. You can have Spark request GPUs and assign them to task. The exact configs you use will vary depending on your cluster manager.

    840px-Spark_3.0 (1).jpg

  • About RAPIDS

    The RAPIDS suite of open source software libraries and APIs provide the ability to execute end-to-end data science and analytics pipelines entirely on GPUs. Licensed under Apache 2.0, RAPIDS is incubated by NVIDIA® based on extensive hardware and data science experience. RAPIDS utilizes NVIDIA CUDA® primitives for low-level compute optimization, and exposes GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.

    RAPIDS also focuses on common data preparation tasks for analytics and data science. This includes a familiar dataframe API that integrates with a variety of machine learning algorithms for end-to-end pipeline accelerations without paying typical serialization costs. RAPIDS also includes support for multi-node, multi-GPU deployments, enabling vastly accelerated processing and training on much larger dataset sizes.

    1_PysdpYKKYbIFUPLJJBkQsQ.png

  • RAPIDS Accelerator For Apache Spark

    As data scientists shift from using traditional analytics to leveraging AI applications that better model complex market demands, traditional CPU-based processing can no longer keep up without compromising either speed or cost. The growing adoption of AI in analytics has created the need for a new framework to process data quickly and ensure cost efficiency with GPUs.

    The RAPIDS Accelerator for Apache Spark combines the power of the cuDF library with Apache Spark to enable GPU-accelerated processing.

noreferrer">RAPIDS cuDF 库和 Spark 分布式计算框架的规模。RAPIDS 加速器库还内置了基于 UCX 的加速 shuffle，可配置利用 GPU 到 GPU 通信和 RDMA 能力。

switches guide. For more information, please refer to the Mellanox Onyx User Manual at https://docs.mellanox.com/category/onyx.

First we need to update the switch OS to the latest ONYX software version. For more details go to HowTo Upgrade MLNX-OS Software on Mellanox switch systems.

For lossless configuration and for Mellanox Onyx version 3.8.2004 and above run:

Switch console

# switch (config) #roce lossless

(PFC+ECN) to run RoCE on lossless fabric, run the "roce" command

To see the RoCE configuration run:

Switch console

# show roce

To monitor the RoCE counters run:

Switch console

# show interface ethernet counters roce

Master and Worker Servers Installation and Configuration

Add a SPARK User and Grant Root Privileges on the Master and Worker Servers

To add a user, run the following command:

Server Console

# adduser sparkuser

To grant the user root privileges, run:

Server Console

# visudo

The command above leads you to the /etc/sudoers.tmp file, where you can view the following code:

Server Console

# User privilege specification
root    ALL=(ALL:ALL) ALL

After the root user line, you will add your new user with the same format for you to grant admin privileges.

Server Console

sparkuser ALL=(ALL:ALL)ALL

Once you've added the permission, save and exit the file by using the following process. (In Ubuntu 18.04, nano is the default editor, so we need to use the ctrl+x, y keystrokes to save and exit the file.)

Hold ctrl and press x. At the prompt, press y and then hit enter to save and exit the file.

Now this user will be able to run commands (like update, using the sudo):

Server Console

# su - sparkuser
Updating Ubuntu Software Packages on the Master and Worker Servers

To update/upgrade Ubuntu software packages, run the following commands:

Server Console

> sudo apt-get update            # Fetches the list of available update
> sudo apt-get upgrade -y        # Strictly upgrades the current packages
Installing General Dependencies on the Master and Worker Servers

To install general dependencies, run the commands below or copy-paste each line.

Server Console

> sudo apt-get install git wget scala maven openssh-server openssh-client
Running LLDP service on the Master and Worker Servers

To run LLDP service on server, run the commands below or copy-paste each line.

Server Console

> sudo service lldpd start
> sudo systemctl enable lldpd
Installing Java 8 (Recommended Oracle Java) on the Master and Worker Servers

To install Java 8 software packages, run the following commands:

Server Console

> sudo apt-get install python-software-properties
> sudo add-apt-repository ppa:webupd8team/java
> sudo apt-get update
> sudo apt-get install oracle-java8-installer
Adding Entries to Host files on the Master and Worker Servers

To edit the host files:

Server Console

> sudo vim /etc/hosts

Now add entries of namenoder (master) and worker servers.

Example:

Server Console

127.0.0.1 localhost
127.0.1.1 sl-spark-n01.vwd.clx sl-spark-n01

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.11.71 sl-spark-n01l
192.168.11.72 sl-spark-n02l
192.168.11.73 sl-spark-n03l
Configuring SSH

We will configure passwordless SSH access from Master to all Slave nodes.

  1. Install OpenSSH Server-Client on both Master and Slaves nodes

    Server Console

    > sudo apt-get install openssh-server openssh-client

  2. On Master node - Generate Key Pairs

    Server Console

    > sudo ssh-keygen -t rsa -P ""

  3. Copy the content of .ssh/id_rsa.pub file on master to .ssh/authorized_keys file on all nodes (Master and Slave nodes)

    Server Console

    > cp /home/sparkuser/.ssh/id_rsa.pub /home/sparkuser/.ssh/authorized_keys
> scp /home/sparkuser/.ssh/authorized_keys sparkuser@192.168.1.72:/home/sparkuser/.ssh/authorized_keys
> scp /home/sparkuser/.ssh/authorized_keys sparkuser@192.168.1.73:/home/sparkuser/.ssh/authorized_keys

    Set the permissions for your user with

the chmod command:

Server Console

> chmod 0600 ~/.ssh/authorized_keys

  • The new user is now able to SSH without needing to enter a password every time. Verify everything is set up correctly by using the sparkuser user to SSH to localhost and all Slave nodes from Master node:

    Server Console

    > ssh sl-spark-n01l
> ssh sl-spark-n02l
> ssh sl-spark-n03l

Install the NVIDIA Drivers

The last NVIDIA drivers must be installed. To install them, you can use the Ubuntu built (when installing the additional drivers) after updating the driver packages.

  1. Go to NVIDIA’s website (https://www.nvidia.co.uk/Download/index.aspx?lang=uk). Select your Product, OS and CUDA version.

    image2020-7-12_13-57-28.png

  2. Download the driver's latest version.

    image2020-7-12_13-49-33.png

  3. Once you accept the download, please follow the steps listed below.

    Server Console

    sudo dpkg -i nvidia-driver-local-repo-ubuntu1804-440.95.01_1.0-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda-drivers

  4. Once installed using additional drivers, restart your computer.

    Server Console

    sudo reboot

Verify the Installation

Make sure the NVIDIA driver can work properly with the installed GPU card.

Server Console

lsmod | grep nvidia

image2020-7-13_16-36-46.png

Run the nvidia-debugdump utility to collect internal GPU information.

Server Console

nvidia-debugdump -l

image2020-7-13_16-40-37.png

Run the nvidia-smi utility to check the NVIDIA System Management Interface.

Server Console

nvidia-smi

image2020-7-13_16-41-14.png

Installing MLNX_OFED for Ubuntu on the Master and Workers

This section describes how to install and test MLNX_OFED for Linux package on a single host machine with Mellanox ConnectX®-5 adapter card installed.

For more information click on Mellanox OFED for Linux User Manual.

  1. Verify that the system has a Mellanox network adapter (HCA/NIC) installed:

    Server Console

    lspci -v | grep Mellanox

    The following example shows a system with an installed Mellanox HCA:

    image2020-7-9_11-7-24.png

  2. The Mellanox Technologies Ltd. public repository provides all packages required for InfiniBand, Ethernet and RDMA.

    Subscribe your system to the Mellanox Technologies repository by downloading and installing Mellanox Technologies GPG-KEY.

    Server Console

    sudo wget -qO - https://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -

  3. Download the repository configuration file of the desired product.

    • Go to the main repository https://linux.mellanox.com/public/repo
    • Choose the repository that suits your needs (mlnx_ofed, mlnx_en or mlnx_rdma_minimal).
    • Choose your Operating System Under the "latest" folder
    • Download the repository configuration file "mellanox_mlnx_ofed.repo" or "mellanox_mlnx_ofed.list"

    The commands below provide examples of how to configure a 'mlnx_ofed' repository for Ubuntu18.04.

    Server Console

    cd /etc/apt/sources.list.d/
sudo wget https://linux.mellanox.com/public/repo/mlnx_ofed/latest/ubuntu18.04/mellanox_mlnx_ofed.list

  4. Remove the distribution InfiniBand packages

    Server Console

    sudo apt-get remove libipathverbs1 librdmacm1 libibverbs1 libmthca1 libopenmpi-dev openmpi-bin openmpi-common openmpi-doc libmlx4-1 rdmacm-utils ibverbs-utils infiniband-diags ibutils perftest

    Since the distro InfiniBand packages conflict with the packages included in the MLNX_OFED/MLNX_EN driver, make sure that all distro InfiniBand RPMs are uninstalled.

  5. Installing Mellanox OFED

    After setting up a repository, install the following metadata package:

    Server Console

    apt-get install mlnx-ofed-all

  6. Firmware update.

    All the installation options above do not automatically update the firmware on your system.

    To update the firmware to the version included in the configured repository, you can either:

    1. Install the "mlnx-fw-updater" package:

      Server Console

      apt-get install mlnx-fw-updater

    2. Update the firmware to the latest version available on Mellanox Technologies’ website. (www.mellanox.com -> Support/Education -> Firmware 下载).

      Instruction on how to update the firmware are available in the MLNX_ODED User Manual section “Updating Firmware After Installation”.

Reboot after the installation finished successfully:

Server Console

reboot

Check that the port modes are Ethernet: ConnectX®-5 ports can be individually configured to work as Infiniband or Ethernet ports

Server Console

ibv_devinfo

This is the desired configuration:

11.PNG

If you see the following - You need to change the interfaces port type to Ethernet - next optional step

image2020-8-9_13-54-38.png

Change the mode to Ethernet. (optional step) To change the interfaces port type to Ethernet mode, use the mlxconfig command after the driver is loaded.

a. Start mst and see ports names:

Server Console

mst start
mst status

b. Change the port mode to Ethernet:

Server Console

mlxconfig -d /dev/mst/mt4121_pciconf0 s LINK_TYPE_P1=2
    Port 1 set to ETH mode
reboot

c. Query Ethernet devices and print information on them that is available for use from userspace:

Server Console

ibv_devinfo

d. Run the ibdev2netdev utility to see all the associations between the Ethernet devices and the InfiniBand devices/ports:

Server Console

ibdev2netdev

e. Configure the network interface:

Server Console

ifconfig ens3f0 192.168.11.71 netmask 255.255.255.0

f. Insert the lines below to the /etc/netplan/01-netcfg.yam file after the following lines:

Server Console

vim /etc/netplan/01-netcfg.yaml

The new lines:

Sample

ens3f0:
    dhcp4: no
    address:
        - 192.168.11.71/24

Example:

image2020-7-12_16-42-43.png

g. Check the network configuration is set correctly:

Server Console

ifconfig -a

Lossless Fabric with L3 (DSCP) Configuration

This post provides a configuration example for Mellanox devices installed with MLNX_OFED running RoCE over a lossless network, in DSCP-based QoS mode.

For this configuration you need to know what is your network interface (for example ens3f0) and its parent Mellanox device (for example mlx5_0).

To get the information, run the ibdev2netdev command:

Shell

# ibdev2netdev
#s# add output here

Configuration:

Shell

# mlnx_qos -i ens3f0 --trust dscp
# echo 106 > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
# cma_roce_tos -d mlx5_0 -t 106
# sysctl -w net.ipv4.tcp_ecn=1
# mlnx_qos -i ens3f0 --pfc 0,0,0,1,0,0,0,0

Installing Mellanox OFED GPUDirect RDMA

The below listed software are required for this component. The software needs to be installed and running:

  1. NVIDIA compatible driver.
  2. MLNX_OFED (latest).

Please check with NVIDIA support for the above NVIDIA driver and other relevant information.

  1. Download last nv_peer_memory:

    Server Console

    # cd
# git clone https://github.com/Mellanox/nv_peer_memory

  2. Build source packages (src.rpm for RPM based OS and tarball for DEB based OS), use the build_module.sh script:

    Node cli

    # ./build_module.sh

Building source rpm for nvidia_peer_memory...
Building debian tarball for nvidia-peer-memory...

Built: /tmp/nvidia_peer_memory-1.0-9.src.rpm
Built: /tmp/nvidia-peer-memory_1.0.orig.tar.gz

  3. Install the built packages:

    Node cli

    # cd /tmp
# tar xzf /tmp/nvidia-peer-memory_1.0.orig.tar.gz
# cd nvidia-peer-memory-1.0
# dpkg-buildpackage -us -uc
# dpkg -i <path to generated deb files>

    (e.g. dpkg -i nv-peer-memory_1.0-9_all.deb nv-peer-memory-dkms_1.0-9_all.deb)

After a successful installation:

  1. nv_peer_mem.ko kernel module is installed.
  2. Service file /etc/init.d/nv_peer_mem to control the kernel module by start/stop/status will be added.
  3. /etc/infiniband/nv_peer_mem.conf configuration file to control whether kernel module will be loaded on boot (default value is YES).

Important Note:

Use nvidia-smi topo -m to make sure that this is the case:

20.png

We recommend that both the NIC and the GPU are physically placed on the same PCI switch (to achieve better performance):

安装Open Fabrics Enterprise Distribution (OFED) 性能测试工具

Perftools是一组基于 uverbs 编写的测试工具，旨在用作性能微基准测试。这些测试可用于硬件或软件调优以及功能测试。该集合包含一系列带宽和延迟基准测试，例如：

  • Send - ib_send_bw 和 ib_send_lat
  • RDMA Read - ib_read_bw 和 ib_read_lat
  • RDMA Write - ib_write_bw 和 ib_write_lat
  • RDMA Atomic - ib_atomic_bw 和 ib_atomic_lat
  • Native Ethernet（使用MOFED2时）- raw_ethernet_bw, raw_ethernet_lat

要安装 perftools，请运行以下命令：

服务器控制台

cd
git clone https://github.com/linux-rdma/perftest
cd perftest/
make clean
./autogen.sh
./configure --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc CUDA_H_PATH=/usr/local/cuda/include/cuda.h
make

我们将运行一个RDMA Write - ib_write_bw 带宽压力基准测试（基于RoCE）。

服务器 ib_write_bw -a -d mlx5_0 &
客户端 ib_write_bw -a -F $server_IP -d mlx5_0 --report_gbits

image2020-7-14_13-10-52.png

image2020-7-14_13-11-32.png

要运行GPUDirect基准测试，请使用 --use_cuda= 参数运行基准测试。

服务器 ib_write_bw -a -d mlx5_0 --use_cuda=0 &
客户端 ib_write_bw -a -F $server_IP -d mlx5_0 --use_cuda=0 --report_gbits

image2020-7-14_13-7-41.png

image2020-7-14_13-6-42.png

安装Unified Communication X (UCX)

Unified Communication X (UCX) 为消息传递（MPI）、PGAS/OpenSHMEM 库以及RPC/数据密集型应用提供了优化的通信层。

UCX利用高速网络进行节点间通信，并利用共享内存机制实现高效的节点内通信。

要安装UCX，请访问 https://github.com/openucx/ucx/releases/tag/v1.8.1-rc4。此外，下载并安装一个deb包。

服务器控制台

cd
wget https://github.com/openucx/ucx/releases/download/v1.8.1/ucx-v1.8.1-ubuntu18.04-mofed5.0-cuda10.2.deb
sudo dpkg -i ucx-v1.8.1-ubuntu18.04-mofed5.0-cuda10.2.deb

验证UCX

要验证UCX，可以运行 ucx_info 命令：

服务器控制台

ucx_info -d | less

image2020-7-13_13-2-15.png

image2020-7-13_14-32-57.png

image2020-7-13_14-33-59.png

image2020-7-13_14-34-39.png

此外，您可以通过运行以下命令来运行UCX基准测试：

基于TCP

服务器 CX_NET_DEVICES=mlx5_0:1 UCX_TLS=cuda,cuda_copy,tcp ucx_perftest -m cuda -s 1000000 -t tag_lat
客户端 CX_NET_DEVICES=mlx5_0:1 UCX_TLS=cuda,cuda_copy,tcp ucx_perftest -m cuda -s 1000000 -t tag_lat sl-spark-n01l

示例结果：

image2020-7-14_13-35-40.png

......

image2020-7-14_13-37-24.png

基于RC（RDMA可靠连接）

服务器 UCX_NET_DEVICES=mlx5_0:1 UCX_RNDV_SCHEME=get_zcopy UCX_TLS=cuda,cuda_copy,rc ucx_perftest -m cuda -s 1000000 -t tag_lat
客户端 UCX_NET_DEVICES=mlx5_0:1 UCX_RNDV_SCHEME=get_zcopy UCX_TLS=cuda,cuda_copy,rc ucx_perftest -m cuda -s 1000000 -t tag_lat sl-spark-n01l

示例结果：

image2020-7-14_13-35-40.png

image2020-7-14_13-40-59.png

.....

image2020-7-14_13-39-33.png

安装Nvidia Toolkit 10.2 (CUDA)

前提条件

  1. 确认你拥有支持CUDA的GPU。 检查你的GPU是否列在Your GPU Compute Capability部分。

  2. 确认系统已安装正确的内核头文件和内核开发包。

安装过程

通过运行以下命令安装CUDA：

服务器控制台
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda
安装后操作

必需操作 - 环境配置

安装后必须执行一些操作，才能使用CUDA工具包和驱动程序。

  • “PATH”变量需要包含/usr/local/cuda-10.2/bin。

  • 此外，使用.run文件安装方法时，“LD_LIBRARY_PATH”变量需要包含/usr/local/cuda-10.2/lib64。

  • 更新你的bash文件。

    服务器控制台

    sudo vim ~/.bashrc

  • 这将在文本编辑器中打开你的bash文件，滚动到底部并添加以下行：

    服务器控制台

    export CUDA_HOME=/usr/local/cuda-10.2
export PATH=/usr/local/cuda-10.2/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

  • 保存并关闭文本文件后，返回原始终端并输入以下命令重新加载.bashrc文件。

    服务器控制台

    source ~/.bashrc

  • 检查路径是否正确修改。

    服务器控制台

    echo $CUDA_HOME
echo $PATH
echo $LD_LIBRARY_PATH
推荐操作 - 完整性检查

建议执行以下操作以验证安装的完整性。

  • 安装可写示例（http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#install-samples） 为了修改、编译和运行示例，必须以“写入”权限安装示例。提供了一个方便的安装脚本：

    服务器控制台

    cuda-install-samples-10.2.sh ~

    此脚本将cuda-samples-10-2包安装到你的主目录中。

  • 在继续之前，验证安装过程是否成功（http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#verify-installation）。 验证CUDA工具包能否正确找到并与支持CUDA的硬件通信非常重要。为此，你需要编译并运行一些包含的示例程序。

  • 验证NVIDIA CUDA Toolkit能否正确查询GPU设备。 你应该从~/NVIDIA_CUDA-10.2_Samples/1_Utilities/deviceQuery目录编译源代码。生成的二进制文件位于~/NVIDIA_CUDA-10.2_Samples/bin下。

    服务器控制台

    cd ~/NVIDIA_CUDA-10.2_Samples/1_Utilities/deviceQuery/
make
cd ~/NVIDIA_CUDA-10.2_Samples
./bin/x86_64/linux/release/deviceQuery

  • 运行bandwidthTest程序以确保系统和支持CUDA的设备能够正确通信。

    服务器控制台

    cd ~/NVIDIA_CUDA-9.1_Samples/1_Utilities/bandwidthTest/
make
cd ~/NVIDIA_CUDA-9.1_Samples/
./bin/x86_64/linux/release/bandwidthTest

注意：bandwidthTest的设备描述和结果可能因系统而异。 重要的是倒数第二行确认所有必要测试均已通过。 如果测试未通过，请确保你的系统上有支持CUDA的NVIDIA GPU，并确保其正确安装。 如果在链接步骤中遇到困难（例如找不到库），请查阅CUDA示例目录中doc文件夹中的Linux发行说明。

应用程序部署和配置

下载并安装Hadoop

下载Hadoop

  1. 安装Hadoop集群通常涉及在集群中的所有机器上解压软件。 从链接下载hadoop-3.2.1.tar.gz到主节点机器上进行安装。

    服务器控制台

    cd /opt/
wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz

    下载完成后，解压文件以启动Hadoop安装：

    服务器控制台

    tar -zxvf hadoop-3.2.1.tar.gz

    Hadoop二进制文件现在位于/opt/hadoop-3.2.1目录中。

  2. 为hadoop-3.2.1文件夹创建符号链接，并为你的用户设置hadoop-3.2.1和hadoop文件夹的权限：

服务器控制台
sudo ln -s hadoop-3.2.1/ hadoop
sudo chown -R user hadoop
sudo chown -R user hadoop-3.2.1/
配置Hadoop环境变量 (bashrc)

...

Configuration Changes in .bashrc file

Edit the .bashrc shell configuration file:

sudo vim .bashrc

Define the Hadoop environment variables by adding the following content to the end of the file:

#Hadoop Related Options
export HADOOP_HOME=/opt/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

Once you save and close the text file, you can return to your original terminal and type this command to reload your .bashrc file.

source ~/.bashrc

Check that the paths have been properly modified.

echo $HADOOP_HOME
echo $PATH
echo $YARN_HOME

Configuration Changes in hadoop-env.sh file

The hadoop-env.sh file serves as a master file to configure YARN, HDFS and Hadoop-related project settings.

When setting up a multi-node Hadoop cluster, you need to define which Java implementation is to be utilized. Edit the hadoop-env.sh file:

sudo vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Un-comment the $JAVA_HOME variable (i.e., remove the # sign) and add the full path to the OpenJDK installation on your system.

We have installed the OpenJDK version 8.

Add the following line:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

The path needs to match with the location of the Java installation on your system.

image2020-7-14_20-18-49.png

Configuration Changes in core-site.xml file

The core-site.xml file defines HDFS and Hadoop core properties.

We need to specify the URL for your NameNode, and the temporary directory Hadoop uses for the map and reduce process.

Open the core-site.xml file in a text editor:

sudo vim $HADOOP_HOME/etc/hadoop/core-site.xml

Add the following configuration to override the default values for the temporary directory and add your HDFS URL to replace the default local file system setting:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="https://networking-docs.nvidia.com/sol/configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://sl-spark-n01l:9123</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/data/hadoop_tmp/</value>
</property>

</configuration>

Note: Do not forget to create directory in the location you specified for your temporary data.

In our environment we will create a /hadoop_tmp/ directory on a /dev/nvme0n1p1/data mounted NVMe local drive.

sudo mkdir /data/hadoop_tmp/

Configuration Changes in the hdfs-site.xml File

The properties in the hdfs-site.xml file govern the location for storing node metadata, fsimage file, and edit log file. Configure the file by defining the NameNode and DataNode information.

In addition, the default dfs.replication value of 3 needs to be changed to 1 to match the single node setup.

Use the following command to edit the hdfs-site.xml file:

sudo vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Add the following configuration to the file:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="https://networking-docs.nvidia.com/sol/configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.rpc-bind-host</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.https-bind-host</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.servicerpc-bind-host</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
  <value>false</value>
</property>

<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>

<property>
  <name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name>
  <value>1000</value>
</property>

<property>
  <name>dfs.datanode.dns.interface</name>
  <value>ens3f0</value>
</property>

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

<property>
  <name>dfs.client.use.datanode.hostname</name>
  <value>true</value>
</property>

</configuration>

mapred-site.xml 文件配置更改

使用以下命令编辑 mapred-site.xml 文件：

服务器控制台

sudo vim $HADOOP_HOME/etc/hadoop/mapred-site.xml

添加以下配置：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="https://networking-docs.nvidia.com/sol/configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>40960</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>81920</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx3072m</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx6144m</value>
</property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/hadoop/</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/hadoop/</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/hadoop/</value>
</property>

</configuration>

yarn-site.xml 文件配置更改

yarn-site.xml 文件用于定义与 YARN 相关的设置。它包含 Node Manager、Resource Manager、Container、Application Master 的配置，以及配置 YARN 以支持 GPU 隔离：https://hadoop.apache.org/docs/r3.1.3/hadoop-yarn/hadoop-yarn-site/UsingGpus.html

使用以下命令编辑 yarn-site.xml 文件：

服务器控制台

sudo vim $HADOOP_HOME/etc/hadoop/yarn-site.xml

添加以下配置：

<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-vcores</name>
  <value>32</value>
</property>

<property>
  <name>yarn.nodemanager.log.retain-seconds</name>
  <value>100800</value>
</property>

<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>32</value>
</property>

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>94000</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>93000</value>
</property>
<property>
  <name>yarn.nodemanager.pmem-check-enabled</name>
  <value>false</value>
</property>
<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>
<property>
  <name>yarn.nodemanager.disk-health-checker.enable</name>
  <value>false</value>
</property>
<property>
  <name>yarn.log-aggregation-enable</name>
  <value>false</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
  <value>100</value>
</property>
<property>
 <name>yarn.nodemanager.resource-plugins</name>
 <value>yarn.io/gpu</value>
</property>

<!--<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>400</value>
</property>
-->

<!--
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/dev/shm/spark/</value>
</property>
-->
</configuration>

将 hadoop 目录从 NameNode 复制到 DataNodes

如果不使用共享目录，则需要将 /opt/hadoop-3.2.1 目录从 NameNode 复制到 DataNodes，创建符号链接 hadoop 指向 /opt/hadoop-3.2.1 目录，并将权限设置为 sparkuser。

将 NameNode 上的 /opt/hadoop-3.2.1 文件内容复制到所有 DataNodes 上的 /opt/hadoop-3.2.1 文件：

服务器控制台

scp -r /opt/hadoop-3.2.1 sparkuser@192.168.11.72:/opt/hadoop-3.2.1
scp -r /opt/hadoop-3.2.1 sparkuser@192.168.11.73:/opt/hadoop-3.2.1

为 hadoop-3.2.1 目录创建符号链接，并将 hadoop-3.2.1 和 hadoop 目录的权限设置为 sparkuser（在所有 DataNodes 上执行）：

服务器控制台

ssh sl-spark-n02l
cd /opt
sudo ln -s hadoop-3.2.1/ hadoop
sudo chown -R user hadoop
sudo chown -R user hadoop-3.2.1/
exit
ssh sl-spark-n03l
cd /opt
sudo ln -s hadoop-3.2.1/ hadoop
sudo chown -R user hadoop
sudo chown -R user hadoop-3.2.1/
exit

启动 Hadoop 集群

创建 YARN cgroup 目录

在 NameNode 和所有 DataNodes 上运行以下命令创建 YARN cgroup 目录：

服务器控制台

sudo mkdir -p /sys/fs/cgroup/devices/hadoop-yarn

格式化HDFS NameNode

首次使用前需格式化NameNode。以sparkuser用户身份运行以下命令格式化NameNode。

服务器控制台

hdfs namenode -format

将GPU设置为独占模式

所有GPU需配置为EXCLUSIVE_PROCESS模式，以便在YARN上使用RAPIDS插件运行Spark，前提是Spark应用运行在带GPU的节点上。否则无法确保同一时间只有一个执行器访问GPU。注意，即使启用了GPU调度支持，此设置仍然必要。

在所有YARN节点上，确保GPU处于EXCLUSIVE_PROCESS模式：

  1. 运行nvidia-smi查看GPU数量及索引。
  2. 将每个GPU索引设置为EXCLUSIVE_PROCESS模式： nvidia-smi -c EXCLUSIVE_PROCESS -i $<GPU索引>

示例：

服务器控制台

sudo nvidia-smi -c EXCLUSIVE_PROCESS -i 0
sudo nvidia-smi -c EXCLUSIVE_PROCESS -i 1

说明：要将GPU从EXCLUSIVE_PROCESS模式移除，对每个GPU运行以下命令： nvidia-smi -c 0 -i $<GPU索引>

启动Hadoop集群

格式化NameNode后，使用start-dfs.sh脚本启动HDFS。在NameNode上导航到/opt/hadoop/sbin目录，执行以下命令启动NameNode和DataNode：

服务器控制台

cd /opt/hadoop
./start-dfs.sh

系统需要一些时间启动必要节点。

在NameNode和DataNode上运行以下命令，检查所有HDFS守护进程是否作为Java进程运行：

服务器控制台

jps

NameNode上：

image2020-7-15_16-46-39.png

DataNode上：

image2020-7-15_10-27-27.png

NameNode、DataNode和Secondary NameNode启动后，在NameNode上执行YARN启动脚本start-yarn.sh启动YARN资源管理器和节点管理器：

服务器控制台

./start-yarn.sh

与之前命令类似，输出会提示进程正在启动。

说明：可以使用start-all.sh脚本同时启动HDFS和YARN。

在NameNode和DataNode上输入以下命令，检查所有HDFS和YARN守护进程是否作为Java进程运行：

服务器控制台

jps

如果一切正常，运行中的Java进程列表将包含所有HDFS和YARN守护进程。

NameNode上：

image2020-7-15_16-44-9.png

DataNode上：

image2020-7-15_10-32-3.png

说明：要停止YARN和HDFS并重新启动，运行以下命令： 服务器控制台

cd /opt/hadoop
./stop-all.sh
rm -rf /opt/hadoop/logs/*
./start-all.sh
jps

您已成功安装Hadoop集群。

安装Apache Spark

下载Apache Spark（主节点）

访问下载 | Apache Spark，选择Spark版本和包类型，下载spark-3.0.0-bin-hadoop3.2.tgz文件，并将其复制到/opt文件夹。

image2020-6-23_16-34-52.png

可选：也可以通过以下命令下载Spark：

cd /opt
wget http://mirror.cogentco.com/pub/apache/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz

设置Apache Spark

使用tar命令解压压缩文件：

服务器控制台

tar -xvf spark-3.0.0-bin-hadoop3.2.tgz

为spark-3.0.0-bin-hadoop3.2文件夹创建符号链接，并为用户设置spark-3.0.0-bin-hadoop3.2和spark文件夹的权限：

服务器控制台

sudo ln -s spark-3.0.0-bin-hadoop3.2/ spark
sudo chown -R user spark
sudo chown -R user spark-3.0.0-bin-hadoop3.2/

配置Spark环境变量（bashrc）

编辑

the .bashrc shell configuration file:

Server Console

sudo vim .bashrc

Define the Spark environment variables by adding the following content to the end of the file:

Server Console

#Spark Related Options
echo "export SPARK_HOME=/opt/spark"
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin"
echo "export PYSPARK_PYTHON=/usr/bin/python3"

Once you save and close the text file, you can return to your original terminal and type this command to reload your .bashrc file.

Server Console

$ source ~/.bashrc

Check that the path have been properly modified.

Server Console

echo $SPARK_HOME
echo $PATH
echo $PYSPARK_PYTHON

Run a sample Spark job in local mode to ensure Spark is working

Server Console

SPARK_HOME/bin/run-example SparkPi 10

Install the Spark Rapids Plugin jars

Create /opt/sparkRapidsPlugin and set the permissions for your user to the directory:

Server Console

sudo mkdir -p /opt/sparkRapidsPlugin
sudo chown -R sparkuser sparkRapidsPlugin

Download the 2 jars from the link above and put them into the /opt/sparkRapidsPlugin directory. Make sure to pull the cudf jar version for the cuda version you are running 10.2.

Note Server Console In case, you want to use stable cuDF and RAPIDS plugins versions, you can download them from: https://github.com/NVIDIA/spark-rapids/blob/branch-0.2/docs/version/stable-release.md Today 22/7/2020 are stable versions: cudf-0.14-cuda10-2 and rapids-4-spark_2.12-0.1.0 In our environment we will use last for today versions.

To download cuDF plugin run following commands:

Server Console

cd /opt/sparkRapidsPlugin
wget https://oss.sonatype.org/service/local/repositories/snapshots/content/ai/rapids/cudf/0.15-SNAPSHOT/cudf-0.15-20200710.084843-7-cuda10-2.jar

To download RAPIDS plugin for Apache Spark, run the following commands:

Server Console

sudo git clone https://github.com/NVIDIA/spark-rapids.git
cd spark-rapids/
sudo git checkout 8f038ae
sudo mvn package -DskipTests
sudo cp dist/target/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar
sudo cp /opt/spark-rapids/integration_tests/target/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar /opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar

Install the GPU Discovery Script

  • Download the getGpusResource.sh script and install on all the nodes.

  • Put it into a local folder. You may put it in the same directory as the plugin jars (/opt/sparkRapidsPlugin in our example)

  • Add GPU addresses to the getGpusResource.sh script. See the Example (we will provide two GPUs that we have):

    Server Console

    vim /opt/sparkRapidsPlugin/getGpusResource.sh

    Sample output:

    Sample

    #!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# This script is a basic example script to get resource information about NVIDIA GPUs.
# It assumes the drivers are properly installed and the nvidia-smi command is available.
# It is not guaranteed to work on all setups so please test and customize as needed
# for your environment. It can be passed into SPARK via the config
# spark.{driver/executor}.resource.gpu.discoveryScript to allow the driver or executor to discover
# the GPUs it was allocated. It assumes you are running within an isolated container where the
# GPUs are allocated exclusively to that driver or executor.
# It outputs a JSON formatted string that is expected by the
# spark.{driver/executor}.resource.gpu.discoveryScript config.
#
# Example output: {"name": "gpu", "addresses":["0","1","2","3","4","5","6","7"]}

echo {\"name\": \"gpu\", \"addresses\":[\"0\", \"1\"]}

Running Apache Spark shell on YARN with RAPIDS/UCX

YARN requires Spark, the Spark Rapids Plugin jars and discovery script to be installed on a launcher(Master) node. YARN handles shipping them to the nodes as needed.

To running Apache Spark shell with RAPIDS and UCX plugins we will create new start-shell.sh file in the /opt/sparkRapidsPlugin directory.

Server Console

vim /opt/sparkRapidsPlugin start-shell.sh

In our use case we are using the following configs when running Spark on YARN.

Server Console

export YARN_CONF_DIR=/opt/hadoop/

/opt/spark/bin/spark-shell \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.task.resource.gpu.amount=0.25 \
--conf spark.executor.resource.gpu.amount=1 \
--conf spark.rapids.sql.concurrentGpuTasks=1 \
--conf spark.rapids.memory.pinnedPool.size=2G \
--conf spark.locality.wait=0s \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.shuffle.partitions=150 \
--conf spark.sql.files.maxPartitionBytes=212MB \
--conf spark.rapids.sql.batchSizeBytes=212M \
--conf spark.scheduler.minRegisteredResourcesRatio=1.0 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.device.count=15 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.host.count=25 \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.shuffle.manager=com.nvidia.spark.RapidsShuffleManager \
--conf spark.rapids.shuffle.transport.enabled=true \
--conf spark.executorEnv.UCX_RNDV_SCHEME=get_zcopy \
--conf spark.executorEnv.UCX_LOG_LEVEL=debug \
--conf spark.driver.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--files ./getGpusResources.sh \

--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar

Run the ***start-shell.sh*** script.

##### **Server Console**

```bash
bash ./start-shell.sh
Example Join Operation:

Once you have started your spark shell you can run the following commands to do a basic join and look at the UI to see that it runs on the GPU.

Server Console
val df = sc.makeRDD(1 to 10000000, 6).toDF
val df2 = sc.makeRDD(1 to 10000000, 6).toDF
df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").count

Go to the Spark UI and click on the application you ran and on the "SQL" tab. If you click the operation "count at ...", you should see the graph of Spark Execs and some of those should have the label Gpu... For instance, in the screenshot below you will see GpuRowToColumn, GpuFilter, GpuColumnarExchange, those all indicate things that would run on the GPU.

Stop Apache Spark shell

To stop Apache Spark shell click on "CTRL-C".

Example TPCH Application

To run TPCH application, please download the generated dataset and run following commands:

Server Console
sudo git clone https://github.com/databricks/tpch-dbgen.git
cd tpch-dbgen
sudo make
sudo dbgen -s 30
sudo hdfs dfs -put *.tbl

Create 3 new files spark-submit-tcp.sh, spark-submit-rc.sh, spark-submit-rc-ucx.sh in the /opt/sparkRapidsPlugin directory.

Server Console
sudo vim /opt/sparkRapidsPlugin/spark-submit-tcp.sh
sudo vim /opt/sparkRapidsPlugin/spark-submit-rc.sh
sudo vim /opt/sparkRapidsPlugin/spark-submit-rc-ucx.sh

For TCP without GPU/RAPIDS/UCX running we are using the spark-submit-tcp.sh file with the following configs when running Spark on YARN.

Server Console
export YARN_CONF_DIR=/opt/hadoop/
/opt/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.memory=4G \
--conf spark.executor.cores=15 \
--conf spark.task.cpus=1 \
--conf spark.locality.wait=0s \
--conf spark.sql.files.maxPartitionBytes=512m \
--conf spark.sql.shuffle.partitions=100 \
--conf spark.driver.extraClassPath=//usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar \
com.nvidia.spark.rapids.tests.tpch.CSV 22

For RDMA/RC with GPU/RAPIDS and without UCX running we are using the spark-submit-rc.sh file with the following configs when running Spark on YARN.

Server Console
export YARN_CONF_DIR=/opt/hadoop/
/opt/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.task.resource.gpu.amount=0.25 \
--conf spark.executor.resource.gpu.amount=1 \
--conf spark.rapids.sql.concurrentGpuTasks=1 \
--conf spark.rapids.memory.pinnedPool.size=2G \
--conf spark.locality.wait=0s \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.shuffle.partitions=150 \
--conf spark.sql.files.maxPartitionBytes=212MB \
--conf spark.rapids.sql.batchSizeBytes=212M \
--conf spark.scheduler.minRegisteredResourcesRatio=1.0 \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.shuffle.manager=com.nvidia.spark.RapidsShuffleManager \
--conf spark.driver.extraClassPath=//usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--files ./getGpusResources.sh \
--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar \
com.nvidia.spark.rapids.tests.tpch.CSV 22

For RDMA/RC with GPU/RAPIDS and UCX running we are using the spark-submit-rc-ucx.sh file with the following configs when running Spark on YARN.

Server Console
export YARN_CONF_DIR=/opt/hadoop/
/opt/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.task.resource.gpu.amount=0.25 \
--conf spark.executor.resource.gpu.amount=1 \
--conf spark.rapids.sql.concurrentGpuTasks=1 \
--conf spark.rapids.memory.pinnedPool.size=2G \
--conf spark.locality.wait=0s \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.shuffle.partitions=150 \
--conf spark.sql.files.maxPartitionBytes=212MB \
--conf spark.rapids.sql.batchSizeBytes=212M \
--conf spark.scheduler.minRegisteredResourcesRatio=1.0 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.device.count=15 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.host.count=25 \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.shuffle.manager=com.nvidia.spark.RapidsShuffleManager \
--conf spark.executorEnv.UCX_TLS=cuda_copy,cuda_ipc,tcp,rc \
--conf spark.executorEnv.UCX_RNDV_SCHEME=get_zcopy \
--conf spark.executorEnv.UCX_LOG_LEVEL=debug \
--conf spark.rapids.shuffle.transport.enabled=true \
--conf spark.driver.extraClassPath=//usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--files ./getGpusResources.sh \
--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar \
com.nvidia.spark.rapids.tests.tpch.CSV 22

Done!

Appendix

General Recommendations

  1. Fewer large input files are better than many of small files. You may not have control over this but it is worth knowing.
  2. Larger input sizes spark.sql.files.maxPartitionBytes=512m are generally better as long as things fit into the GPU.
  3. The GPU does better with larger data chunks as long as they fit into memory. When using the default spark.sql.shuffle.partitions=200 it may be beneficial to make this smaller. Base this on the amount of data the task is reading. Start with 512MB / task.
  4. Out of GPU Memory. GPU out of memory can show up in multiple ways. You can see an error that it is out of memory or it can also manifest as it just crashes. Generally this means your partition size is too big, go back to the Configuration section for the partition size and/or the number of partitions. Possibly reduce the number of concurrent gpu tasks to 1. The Spark UI may give you a hint at the size of the data. Look at either the input data or the shuffle data size for the stage that failed.

RAPIDS Accelerator for Apache Spark Tuning Guide

Tuning a Spark

调整作业的配置设置通常可以提升作业性能，对于使用RAPIDS Accelerator插件的Apache Spark作业也是如此。

本文档提供了如何调整Spark作业配置设置以在使用RAPIDS Accelerator插件时获得更好性能的指南。

监控

由于插件无需更改API即可运行，查看GPU上运行情况的最简单方法是查看Spark Web UI中的“SQL”选项卡。SQL选项卡仅在执行查询后才会显示。转到UI中的SQL选项卡，点击感兴趣的查询，它会显示一个包含详细信息的DAG图。您还可以向下滚动并展开“Details”部分以查看文本表示。

如果您想通过代码查看Spark计划，可以使用explain()函数调用。例如：query.explain()将打印Spark的物理计划，您可以查看哪些节点被替换为GPU调用。

调试

目前，最佳调试方式与Spark常规调试方式相同。查看UI和日志文件以了解失败原因。如果遇到GPU段错误，请查找hs_err_pid.log文件。为确保hs_err_pid.log文件进入YARN应用程序日志，您可以添加配置：--conf spark.executor.extraJavaOptions="-XX:ErrorFile=<LOG_DIR>/hs_err_pid_%p.log"。

如果您想了解某个操作未在GPU上运行的原因，可以启用配置：--conf spark.rapids.sql.explain=NOT_ON_GPU。然后，Driver日志中会输出一条日志消息，说明Spark操作无法在GPU上运行的原因。

作者
BK.jpg Boris KovalevBoris Kovalev过去几年担任解决方案架构师，专注于NVIDIA Networking/Mellanox技术，负责复杂的机器学习、大数据和基于VMware的高级云研究与设计。此前，他在多家公司（最近在VMware）担任高级顾问和解决方案架构师超过20年。他撰写了多份参考设计，涵盖VMware、机器学习、Kubernetes和容器解决方案，可在NVIDIA文档网站上获取。
image2020-11-17_13-55-51.png Peter RudenkoPeter Rudenko是NVIDIA高性能计算（HPC）团队的软件工程师，专注于加速数据密集型应用，开发UCX通信库及其他大数据解决方案。