RDG：通过RoCE网络使用RAPIDS加速器加速Apache Spark 3.0

本指南不涉及服务器的存储方面。您应根据用例（数据集大小）配置合适的存储组件。

如果您使用 NVIDIA Mellanox Onyx 网络操作系统

如果您不熟悉 Mellanox 交换机软件，请按照 HowTo Get Started with Mellanox 操作。

switches guide. For more information, please refer to the Mellanox Onyx User Manual at https://docs.mellanox.com/category/onyx.

First we need to update the switch OS to the latest ONYX software version. For more details go to HowTo Upgrade MLNX-OS Software on Mellanox switch systems.

For lossless configuration and for Mellanox Onyx version 3.8.2004 and above run:

Switch console

# switch (config) #roce lossless

(PFC+ECN) to run RoCE on lossless fabric, run the "roce" command

To see the RoCE configuration run:

Switch console

# show roce

To monitor the RoCE counters run:

Switch console

# show interface ethernet counters roce

Master and Worker Servers Installation and Configuration

Add a SPARK User and Grant Root Privileges on the Master and Worker Servers

To add a user, run the following command:

Server Console

# adduser sparkuser

To grant the user root privileges, run:

Server Console

# visudo

The command above leads you to the /etc/sudoers.tmp file, where you can view the following code:

Server Console

# User privilege specification
root    ALL=(ALL:ALL) ALL

After the root user line, you will add your new user with the same format for you to grant admin privileges.

Server Console

sparkuser ALL=(ALL:ALL)ALL

Once you've added the permission, save and exit the file by using the following process. (In Ubuntu 18.04, nano is the default editor, so we need to use the ctrl+x, y keystrokes to save and exit the file.)

Hold ctrl and press x. At the prompt, press y and then hit enter to save and exit the file.

Now this user will be able to run commands (like update, using the sudo):

Server Console

# su - sparkuser

Updating Ubuntu Software Packages on the Master and Worker Servers

To update/upgrade Ubuntu software packages, run the following commands:

Server Console

> sudo apt-get update            # Fetches the list of available update
> sudo apt-get upgrade -y        # Strictly upgrades the current packages

Installing General Dependencies on the Master and Worker Servers

To install general dependencies, run the commands below or copy-paste each line.

Server Console

> sudo apt-get install git wget scala maven openssh-server openssh-client

Running LLDP service on the Master and Worker Servers

To run LLDP service on server, run the commands below or copy-paste each line.

Server Console

> sudo service lldpd start
> sudo systemctl enable lldpd

Installing Java 8 (Recommended Oracle Java) on the Master and Worker Servers

To install Java 8 software packages, run the following commands:

Server Console

> sudo apt-get install python-software-properties
> sudo add-apt-repository ppa:webupd8team/java
> sudo apt-get update
> sudo apt-get install oracle-java8-installer

Adding Entries to Host files on the Master and Worker Servers

To edit the host files:

Server Console

> sudo vim /etc/hosts

Now add entries of namenoder (master) and worker servers.

Example:

Server Console

127.0.0.1 localhost
127.0.1.1 sl-spark-n01.vwd.clx sl-spark-n01

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.11.71 sl-spark-n01l
192.168.11.72 sl-spark-n02l
192.168.11.73 sl-spark-n03l

Configuring SSH

We will configure passwordless SSH access from Master to all Slave nodes.

1. Install OpenSSH Server-Client on both Master and Slaves nodes

   Server Console

   > sudo apt-get install openssh-server openssh-client
   

2. On Master node - Generate Key Pairs

   Server Console

   > sudo ssh-keygen -t rsa -P ""
   

3. Copy the content of .ssh/id_rsa.pub file on master to .ssh/authorized_keys file on all nodes (Master and Slave nodes)

   Server Console

   > cp /home/sparkuser/.ssh/id_rsa.pub /home/sparkuser/.ssh/authorized_keys
   > scp /home/sparkuser/.ssh/authorized_keys sparkuser@192.168.1.72:/home/sparkuser/.ssh/authorized_keys
   > scp /home/sparkuser/.ssh/authorized_keys sparkuser@192.168.1.73:/home/sparkuser/.ssh/authorized_keys
   

   Set the permissions for your user with

the chmod command:

Server Console

> chmod 0600 ~/.ssh/authorized_keys

• The new user is now able to SSH without needing to enter a password every time. Verify everything is set up correctly by using the sparkuser user to SSH to localhost and all Slave nodes from Master node:

  Server Console

  > ssh sl-spark-n01l
  > ssh sl-spark-n02l
  > ssh sl-spark-n03l
  

Install the NVIDIA Drivers

The last NVIDIA drivers must be installed. To install them, you can use the Ubuntu built (when installing the additional drivers) after updating the driver packages.

1. Go to NVIDIA’s website (https://www.nvidia.co.uk/Download/index.aspx?lang=uk). Select your Product, OS and CUDA version.

   

2. Download the driver's latest version.

   

3. Once you accept the download, please follow the steps listed below.

   Server Console

   sudo dpkg -i nvidia-driver-local-repo-ubuntu1804-440.95.01_1.0-1_amd64.deb
   sudo apt-get update
   sudo apt-get install cuda-drivers
   

4. Once installed using additional drivers, restart your computer.

   Server Console

   sudo reboot
   

Verify the Installation

Make sure the NVIDIA driver can work properly with the installed GPU card.

Server Console

lsmod | grep nvidia

Run the nvidia-debugdump utility to collect internal GPU information.

Server Console

nvidia-debugdump -l

Run the nvidia-smi utility to check the NVIDIA System Management Interface.

Server Console

nvidia-smi

Installing MLNX_OFED for Ubuntu on the Master and Workers

This section describes how to install and test MLNX_OFED for Linux package on a single host machine with Mellanox ConnectX®-5 adapter card installed.

For more information click on Mellanox OFED for Linux User Manual.

1. Verify that the system has a Mellanox network adapter (HCA/NIC) installed:

   Server Console

   lspci -v | grep Mellanox
   

   The following example shows a system with an installed Mellanox HCA:

   

2. The Mellanox Technologies Ltd. public repository provides all packages required for InfiniBand, Ethernet and RDMA.

   Subscribe your system to the Mellanox Technologies repository by downloading and installing Mellanox Technologies GPG-KEY.

   Server Console

   sudo wget -qO - https://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -
   

3. Download the repository configuration file of the desired product.

   • Go to the main repository https://linux.mellanox.com/public/repo
   • Choose the repository that suits your needs (mlnx_ofed, mlnx_en or mlnx_rdma_minimal).
   • Choose your Operating System Under the "latest" folder
   • Download the repository configuration file "mellanox_mlnx_ofed.repo" or "mellanox_mlnx_ofed.list"

   The commands below provide examples of how to configure a 'mlnx_ofed' repository for Ubuntu18.04.

   Server Console

   cd /etc/apt/sources.list.d/
   sudo wget https://linux.mellanox.com/public/repo/mlnx_ofed/latest/ubuntu18.04/mellanox_mlnx_ofed.list
   

4. Remove the distribution InfiniBand packages

   Server Console

   sudo apt-get remove libipathverbs1 librdmacm1 libibverbs1 libmthca1 libopenmpi-dev openmpi-bin openmpi-common openmpi-doc libmlx4-1 rdmacm-utils ibverbs-utils infiniband-diags ibutils perftest
   

   Since the distro InfiniBand packages conflict with the packages included in the MLNX_OFED/MLNX_EN driver, make sure that all distro InfiniBand RPMs are uninstalled.

5. Installing Mellanox OFED

   After setting up a repository, install the following metadata package:

   Server Console

   apt-get install mlnx-ofed-all
   

6. Firmware update.

   All the installation options above do not automatically update the firmware on your system.

   To update the firmware to the version included in the configured repository, you can either:

   1. Install the "mlnx-fw-updater" package:

      Server Console

      apt-get install mlnx-fw-updater
      

   2. Update the firmware to the latest version available on Mellanox Technologies’ website. (www.mellanox.com -> Support/Education -> Firmware 下载).

      Instruction on how to update the firmware are available in the MLNX_ODED User Manual section “Updating Firmware After Installation”.

Reboot after the installation finished successfully:

Server Console

reboot

Check that the port modes are Ethernet: ConnectX®-5 ports can be individually configured to work as Infiniband or Ethernet ports

Server Console

ibv_devinfo

This is the desired configuration:

If you see the following - You need to change the interfaces port type to Ethernet - next optional step

Change the mode to Ethernet. (optional step) To change the interfaces port type to Ethernet mode, use the mlxconfig command after the driver is loaded.

a. Start mst and see ports names:

Server Console

mst start
mst status

b. Change the port mode to Ethernet:

Server Console

mlxconfig -d /dev/mst/mt4121_pciconf0 s LINK_TYPE_P1=2
    Port 1 set to ETH mode
reboot

c. Query Ethernet devices and print information on them that is available for use from userspace:

Server Console

ibv_devinfo

d. Run the ibdev2netdev utility to see all the associations between the Ethernet devices and the InfiniBand devices/ports:

Server Console

ibdev2netdev

e. Configure the network interface:

Server Console

ifconfig ens3f0 192.168.11.71 netmask 255.255.255.0

f. Insert the lines below to the /etc/netplan/01-netcfg.yam file after the following lines:

Server Console

vim /etc/netplan/01-netcfg.yaml

The new lines:

Sample

ens3f0:
    dhcp4: no
    address:
        - 192.168.11.71/24

Example:

g. Check the network configuration is set correctly:

Server Console

ifconfig -a

Lossless Fabric with L3 (DSCP) Configuration

This post provides a configuration example for Mellanox devices installed with MLNX_OFED running RoCE over a lossless network, in DSCP-based QoS mode.

For this configuration you need to know what is your network interface (for example ens3f0) and its parent Mellanox device (for example mlx5_0).

To get the information, run the ibdev2netdev command:

Shell

# ibdev2netdev
#s# add output here

Configuration:

Shell

# mlnx_qos -i ens3f0 --trust dscp
# echo 106 > /sys/class/infiniband/mlx5_0/tc/1/traffic_class
# cma_roce_tos -d mlx5_0 -t 106
# sysctl -w net.ipv4.tcp_ecn=1
# mlnx_qos -i ens3f0 --pfc 0,0,0,1,0,0,0,0

Installing Mellanox OFED GPUDirect RDMA

The below listed software are required for this component. The software needs to be installed and running:

1. NVIDIA compatible driver.
2. MLNX_OFED (latest).

Please check with NVIDIA support for the above NVIDIA driver and other relevant information.

1. Download last nv_peer_memory:

   Server Console

   # cd
   # git clone https://github.com/Mellanox/nv_peer_memory
   

2. Build source packages (src.rpm for RPM based OS and tarball for DEB based OS), use the build_module.sh script:

   Node cli

   # ./build_module.sh
   
   Building source rpm for nvidia_peer_memory...
   Building debian tarball for nvidia-peer-memory...
   
   Built: /tmp/nvidia_peer_memory-1.0-9.src.rpm
   Built: /tmp/nvidia-peer-memory_1.0.orig.tar.gz
   

3. Install the built packages:

   Node cli

   # cd /tmp
   # tar xzf /tmp/nvidia-peer-memory_1.0.orig.tar.gz
   # cd nvidia-peer-memory-1.0
   # dpkg-buildpackage -us -uc
   # dpkg -i <path to generated deb files>
   

   (e.g. dpkg -i nv-peer-memory_1.0-9_all.deb nv-peer-memory-dkms_1.0-9_all.deb)

After a successful installation:

1. nv_peer_mem.ko kernel module is installed.
2. Service file /etc/init.d/nv_peer_mem to control the kernel module by start/stop/status will be added.
3. /etc/infiniband/nv_peer_mem.conf configuration file to control whether kernel module will be loaded on boot (default value is YES).

Important Note:

Use nvidia-smi topo -m to make sure that this is the case:

We recommend that both the NIC and the GPU are physically placed on the same PCI switch (to achieve better performance):

安装Open Fabrics Enterprise Distribution (OFED) 性能测试工具

Perftools是一组基于 uverbs 编写的测试工具，旨在用作性能微基准测试。这些测试可用于硬件或软件调优以及功能测试。该集合包含一系列带宽和延迟基准测试，例如：

• Send - ib_send_bw 和 ib_send_lat
• RDMA Read - ib_read_bw 和 ib_read_lat
• RDMA Write - ib_write_bw 和 ib_write_lat
• RDMA Atomic - ib_atomic_bw 和 ib_atomic_lat
• Native Ethernet（使用MOFED2时）- raw_ethernet_bw, raw_ethernet_lat

要安装 perftools，请运行以下命令：

服务器控制台

cd
git clone https://github.com/linux-rdma/perftest
cd perftest/
make clean
./autogen.sh
./configure --prefix=/usr --libdir=/usr/lib64 --sysconfdir=/etc CUDA_H_PATH=/usr/local/cuda/include/cuda.h
make

我们将运行一个RDMA Write - ib_write_bw 带宽压力基准测试（基于RoCE）。

要运行GPUDirect基准测试，请使用 --use_cuda= 参数运行基准测试。

安装Unified Communication X (UCX)

Unified Communication X (UCX) 为消息传递（MPI）、PGAS/OpenSHMEM 库以及RPC/数据密集型应用提供了优化的通信层。

UCX利用高速网络进行节点间通信，并利用共享内存机制实现高效的节点内通信。

要安装UCX，请访问 https://github.com/openucx/ucx/releases/tag/v1.8.1-rc4。此外，下载并安装一个deb包。

服务器控制台

cd
wget https://github.com/openucx/ucx/releases/download/v1.8.1/ucx-v1.8.1-ubuntu18.04-mofed5.0-cuda10.2.deb
sudo dpkg -i ucx-v1.8.1-ubuntu18.04-mofed5.0-cuda10.2.deb

验证UCX

要验证UCX，可以运行 ucx_info 命令：

服务器控制台

ucx_info -d | less

此外，您可以通过运行以下命令来运行UCX基准测试：

基于TCP

示例结果：

......

基于RC（RDMA可靠连接）

示例结果：

.....

安装Nvidia Toolkit 10.2 (CUDA)

前提条件

1. 确认你拥有支持CUDA的GPU。 检查你的GPU是否列在Your GPU Compute Capability部分。

2. 确认系统已安装正确的内核头文件和内核开发包。

安装过程

通过运行以下命令安装CUDA：

服务器控制台

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

安装后操作

必需操作 - 环境配置

安装后必须执行一些操作，才能使用CUDA工具包和驱动程序。

• “PATH”变量需要包含/usr/local/cuda-10.2/bin。

• 此外，使用.run文件安装方法时，“LD_LIBRARY_PATH”变量需要包含/usr/local/cuda-10.2/lib64。

• 更新你的bash文件。

  服务器控制台

  sudo vim ~/.bashrc
  

• 这将在文本编辑器中打开你的bash文件，滚动到底部并添加以下行：

  服务器控制台

  export CUDA_HOME=/usr/local/cuda-10.2
  export PATH=/usr/local/cuda-10.2/bin${PATH:+:${PATH}}
  export LD_LIBRARY_PATH=/usr/local/cuda-10.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
  

• 保存并关闭文本文件后，返回原始终端并输入以下命令重新加载.bashrc文件。

  服务器控制台

  source ~/.bashrc
  

• 检查路径是否正确修改。

  服务器控制台

  echo $CUDA_HOME
  echo $PATH
  echo $LD_LIBRARY_PATH
  

推荐操作 - 完整性检查

建议执行以下操作以验证安装的完整性。

• 安装可写示例（http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#install-samples） 为了修改、编译和运行示例，必须以“写入”权限安装示例。提供了一个方便的安装脚本：

  服务器控制台

  cuda-install-samples-10.2.sh ~
  

  此脚本将cuda-samples-10-2包安装到你的主目录中。

• 在继续之前，验证安装过程是否成功（http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#verify-installation）。 验证CUDA工具包能否正确找到并与支持CUDA的硬件通信非常重要。为此，你需要编译并运行一些包含的示例程序。

• 验证NVIDIA CUDA Toolkit能否正确查询GPU设备。 你应该从~/NVIDIA_CUDA-10.2_Samples/1_Utilities/deviceQuery目录编译源代码。生成的二进制文件位于~/NVIDIA_CUDA-10.2_Samples/bin下。

  服务器控制台

  cd ~/NVIDIA_CUDA-10.2_Samples/1_Utilities/deviceQuery/
  make
  cd ~/NVIDIA_CUDA-10.2_Samples
  ./bin/x86_64/linux/release/deviceQuery
  

• 运行bandwidthTest程序以确保系统和支持CUDA的设备能够正确通信。

  服务器控制台

  cd ~/NVIDIA_CUDA-9.1_Samples/1_Utilities/bandwidthTest/
  make
  cd ~/NVIDIA_CUDA-9.1_Samples/
  ./bin/x86_64/linux/release/bandwidthTest
  

注意：bandwidthTest的设备描述和结果可能因系统而异。 重要的是倒数第二行确认所有必要测试均已通过。 如果测试未通过，请确保你的系统上有支持CUDA的NVIDIA GPU，并确保其正确安装。 如果在链接步骤中遇到困难（例如找不到库），请查阅CUDA示例目录中doc文件夹中的Linux发行说明。

应用程序部署和配置

下载并安装Hadoop

下载Hadoop

1. 安装Hadoop集群通常涉及在集群中的所有机器上解压软件。 从链接下载hadoop-3.2.1.tar.gz到主节点机器上进行安装。

   服务器控制台

   cd /opt/
   wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
   

   下载完成后，解压文件以启动Hadoop安装：

   服务器控制台

   tar -zxvf hadoop-3.2.1.tar.gz
   

   Hadoop二进制文件现在位于/opt/hadoop-3.2.1目录中。

2. 为hadoop-3.2.1文件夹创建符号链接，并为你的用户设置hadoop-3.2.1和hadoop文件夹的权限：

服务器控制台

sudo ln -s hadoop-3.2.1/ hadoop
sudo chown -R user hadoop
sudo chown -R user hadoop-3.2.1/

配置Hadoop环境变量 (bashrc)

...

Configuration Changes in .bashrc file

Edit the .bashrc shell configuration file:

sudo vim .bashrc

Define the Hadoop environment variables by adding the following content to the end of the file:

#Hadoop Related Options
export HADOOP_HOME=/opt/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

Once you save and close the text file, you can return to your original terminal and type this command to reload your .bashrc file.

source ~/.bashrc

Check that the paths have been properly modified.

echo $HADOOP_HOME
echo $PATH
echo $YARN_HOME

Configuration Changes in hadoop-env.sh file

The hadoop-env.sh file serves as a master file to configure YARN, HDFS and Hadoop-related project settings.

When setting up a multi-node Hadoop cluster, you need to define which Java implementation is to be utilized. Edit the hadoop-env.sh file:

sudo vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Un-comment the $JAVA_HOME variable (i.e., remove the # sign) and add the full path to the OpenJDK installation on your system.

We have installed the OpenJDK version 8.

Add the following line:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

The path needs to match with the location of the Java installation on your system.

Configuration Changes in core-site.xml file

The core-site.xml file defines HDFS and Hadoop core properties.

We need to specify the URL for your NameNode, and the temporary directory Hadoop uses for the map and reduce process.

Open the core-site.xml file in a text editor:

sudo vim $HADOOP_HOME/etc/hadoop/core-site.xml

Add the following configuration to override the default values for the temporary directory and add your HDFS URL to replace the default local file system setting:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="https://networking-docs.nvidia.com/sol/configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://sl-spark-n01l:9123</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/data/hadoop_tmp/</value>
</property>

</configuration>

Note: Do not forget to create directory in the location you specified for your temporary data.

In our environment we will create a /hadoop_tmp/ directory on a /dev/nvme0n1p1/data mounted NVMe local drive.

sudo mkdir /data/hadoop_tmp/

Configuration Changes in the hdfs-site.xml File

The properties in the hdfs-site.xml file govern the location for storing node metadata, fsimage file, and edit log file. Configure the file by defining the NameNode and DataNode information.

In addition, the default dfs.replication value of 3 needs to be changed to 1 to match the single node setup.

Use the following command to edit the hdfs-site.xml file:

sudo vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Add the following configuration to the file:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="https://networking-docs.nvidia.com/sol/configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.rpc-bind-host</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.https-bind-host</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.servicerpc-bind-host</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>dfs.namenode.datanode.registration.ip-hostname-check</name>
  <value>false</value>
</property>

<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>

<property>
  <name>dfs.datanode.directoryscan.throttle.limit.ms.per.sec</name>
  <value>1000</value>
</property>

<property>
  <name>dfs.datanode.dns.interface</name>
  <value>ens3f0</value>
</property>

<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>

<property>
  <name>dfs.client.use.datanode.hostname</name>
  <value>true</value>
</property>

</configuration>

mapred-site.xml 文件配置更改

使用以下命令编辑 mapred-site.xml 文件：

服务器控制台

sudo vim $HADOOP_HOME/etc/hadoop/mapred-site.xml

添加以下配置：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="https://networking-docs.nvidia.com/sol/configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
  <name>mapreduce.map.memory.mb</name>
  <value>40960</value>
</property>
<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>81920</value>
</property>
<property>
  <name>mapreduce.map.java.opts</name>
  <value>-Xmx3072m</value>
</property>
<property>
  <name>mapreduce.reduce.java.opts</name>
  <value>-Xmx6144m</value>
</property>
<property>
  <name>yarn.app.mapreduce.am.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/hadoop/</value>
</property>
<property>
  <name>mapreduce.map.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/hadoop/</value>
</property>
<property>
  <name>mapreduce.reduce.env</name>
  <value>HADOOP_MAPRED_HOME=/opt/hadoop/</value>
</property>

</configuration>

yarn-site.xml 文件配置更改

yarn-site.xml 文件用于定义与 YARN 相关的设置。它包含 Node Manager、Resource Manager、Container、Application Master 的配置，以及配置 YARN 以支持 GPU 隔离：https://hadoop.apache.org/docs/r3.1.3/hadoop-yarn/hadoop-yarn-site/UsingGpus.html

使用以下命令编辑 yarn-site.xml 文件：

服务器控制台

sudo vim $HADOOP_HOME/etc/hadoop/yarn-site.xml

添加以下配置：

<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>sl-spark-n01l</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-vcores</name>
  <value>32</value>
</property>

<property>
  <name>yarn.nodemanager.log.retain-seconds</name>
  <value>100800</value>
</property>

<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>32</value>
</property>

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>94000</value>
</property>

<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>93000</value>
</property>
<property>
  <name>yarn.nodemanager.pmem-check-enabled</name>
  <value>false</value>
</property>
<property>
  <name>yarn.nodemanager.vmem-check-enabled</name>
  <value>false</value>
</property>
<property>
  <name>yarn.nodemanager.disk-health-checker.enable</name>
  <value>false</value>
</property>
<property>
  <name>yarn.log-aggregation-enable</name>
  <value>false</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage</name>
  <value>100</value>
</property>
<property>
 <name>yarn.nodemanager.resource-plugins</name>
 <value>yarn.io/gpu</value>
</property>

<!--<property>
<name>yarn.nodemanager.delete.debug-delay-sec</name>
<value>400</value>
</property>
-->

<!--
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/dev/shm/spark/</value>
</property>
-->
</configuration>

将 hadoop 目录从 NameNode 复制到 DataNodes

如果不使用共享目录，则需要将 /opt/hadoop-3.2.1 目录从 NameNode 复制到 DataNodes，创建符号链接 hadoop 指向 /opt/hadoop-3.2.1 目录，并将权限设置为 sparkuser。

将 NameNode 上的 /opt/hadoop-3.2.1 文件内容复制到所有 DataNodes 上的 /opt/hadoop-3.2.1 文件：

服务器控制台

scp -r /opt/hadoop-3.2.1 sparkuser@192.168.11.72:/opt/hadoop-3.2.1
scp -r /opt/hadoop-3.2.1 sparkuser@192.168.11.73:/opt/hadoop-3.2.1

为 hadoop-3.2.1 目录创建符号链接，并将 hadoop-3.2.1 和 hadoop 目录的权限设置为 sparkuser（在所有 DataNodes 上执行）：

服务器控制台

ssh sl-spark-n02l
cd /opt
sudo ln -s hadoop-3.2.1/ hadoop
sudo chown -R user hadoop
sudo chown -R user hadoop-3.2.1/
exit
ssh sl-spark-n03l
cd /opt
sudo ln -s hadoop-3.2.1/ hadoop
sudo chown -R user hadoop
sudo chown -R user hadoop-3.2.1/
exit

启动 Hadoop 集群

创建 YARN cgroup 目录

在 NameNode 和所有 DataNodes 上运行以下命令创建 YARN cgroup 目录：

服务器控制台

sudo mkdir -p /sys/fs/cgroup/devices/hadoop-yarn

格式化HDFS NameNode

首次使用前需格式化NameNode。以sparkuser用户身份运行以下命令格式化NameNode。

服务器控制台

hdfs namenode -format

将GPU设置为独占模式

所有GPU需配置为EXCLUSIVE_PROCESS模式，以便在YARN上使用RAPIDS插件运行Spark，前提是Spark应用运行在带GPU的节点上。否则无法确保同一时间只有一个执行器访问GPU。注意，即使启用了GPU调度支持，此设置仍然必要。

在所有YARN节点上，确保GPU处于EXCLUSIVE_PROCESS模式：

1. 运行nvidia-smi查看GPU数量及索引。
2. 将每个GPU索引设置为EXCLUSIVE_PROCESS模式： nvidia-smi -c EXCLUSIVE_PROCESS -i $<GPU索引>

示例：

服务器控制台

sudo nvidia-smi -c EXCLUSIVE_PROCESS -i 0
sudo nvidia-smi -c EXCLUSIVE_PROCESS -i 1

说明：要将GPU从EXCLUSIVE_PROCESS模式移除，对每个GPU运行以下命令： nvidia-smi -c 0 -i $<GPU索引>

启动Hadoop集群

格式化NameNode后，使用start-dfs.sh脚本启动HDFS。在NameNode上导航到/opt/hadoop/sbin目录，执行以下命令启动NameNode和DataNode：

服务器控制台

cd /opt/hadoop
./start-dfs.sh

系统需要一些时间启动必要节点。

在NameNode和DataNode上运行以下命令，检查所有HDFS守护进程是否作为Java进程运行：

服务器控制台

jps

NameNode上：

DataNode上：

NameNode、DataNode和Secondary NameNode启动后，在NameNode上执行YARN启动脚本start-yarn.sh启动YARN资源管理器和节点管理器：

服务器控制台

./start-yarn.sh

与之前命令类似，输出会提示进程正在启动。

说明：可以使用start-all.sh脚本同时启动HDFS和YARN。

在NameNode和DataNode上输入以下命令，检查所有HDFS和YARN守护进程是否作为Java进程运行：

服务器控制台

jps

如果一切正常，运行中的Java进程列表将包含所有HDFS和YARN守护进程。

NameNode上：

DataNode上：

说明：要停止YARN和HDFS并重新启动，运行以下命令： 服务器控制台

cd /opt/hadoop
./stop-all.sh
rm -rf /opt/hadoop/logs/*
./start-all.sh
jps

您已成功安装Hadoop集群。

安装Apache Spark

下载Apache Spark（主节点）

访问下载 | Apache Spark，选择Spark版本和包类型，下载spark-3.0.0-bin-hadoop3.2.tgz文件，并将其复制到/opt文件夹。

可选：也可以通过以下命令下载Spark：

cd /opt
wget http://mirror.cogentco.com/pub/apache/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz

设置Apache Spark

使用tar命令解压压缩文件：

服务器控制台

tar -xvf spark-3.0.0-bin-hadoop3.2.tgz

为spark-3.0.0-bin-hadoop3.2文件夹创建符号链接，并为用户设置spark-3.0.0-bin-hadoop3.2和spark文件夹的权限：

服务器控制台

sudo ln -s spark-3.0.0-bin-hadoop3.2/ spark
sudo chown -R user spark
sudo chown -R user spark-3.0.0-bin-hadoop3.2/

配置Spark环境变量（bashrc）

编辑

the .bashrc shell configuration file:

Server Console

sudo vim .bashrc

Define the Spark environment variables by adding the following content to the end of the file:

Server Console

#Spark Related Options
echo "export SPARK_HOME=/opt/spark"
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin"
echo "export PYSPARK_PYTHON=/usr/bin/python3"

Once you save and close the text file, you can return to your original terminal and type this command to reload your .bashrc file.

Server Console

$ source ~/.bashrc

Check that the path have been properly modified.

Server Console

echo $SPARK_HOME
echo $PATH
echo $PYSPARK_PYTHON

Run a sample Spark job in local mode to ensure Spark is working

Server Console

SPARK_HOME/bin/run-example SparkPi 10

Install the Spark Rapids Plugin jars

Create /opt/sparkRapidsPlugin and set the permissions for your user to the directory:

Server Console

sudo mkdir -p /opt/sparkRapidsPlugin
sudo chown -R sparkuser sparkRapidsPlugin

Download the 2 jars from the link above and put them into the /opt/sparkRapidsPlugin directory. Make sure to pull the cudf jar version for the cuda version you are running 10.2.

Note Server Console In case, you want to use stable cuDF and RAPIDS plugins versions, you can download them from: https://github.com/NVIDIA/spark-rapids/blob/branch-0.2/docs/version/stable-release.md Today 22/7/2020 are stable versions: cudf-0.14-cuda10-2 and rapids-4-spark_2.12-0.1.0 In our environment we will use last for today versions.

To download cuDF plugin run following commands:

Server Console

cd /opt/sparkRapidsPlugin
wget https://oss.sonatype.org/service/local/repositories/snapshots/content/ai/rapids/cudf/0.15-SNAPSHOT/cudf-0.15-20200710.084843-7-cuda10-2.jar

To download RAPIDS plugin for Apache Spark, run the following commands:

Server Console

sudo git clone https://github.com/NVIDIA/spark-rapids.git
cd spark-rapids/
sudo git checkout 8f038ae
sudo mvn package -DskipTests
sudo cp dist/target/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar
sudo cp /opt/spark-rapids/integration_tests/target/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar /opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar

Install the GPU Discovery Script

• Download the getGpusResource.sh script and install on all the nodes.

• Put it into a local folder. You may put it in the same directory as the plugin jars (/opt/sparkRapidsPlugin in our example)

• Add GPU addresses to the getGpusResource.sh script. See the Example (we will provide two GPUs that we have):

  Server Console

  vim /opt/sparkRapidsPlugin/getGpusResource.sh
  

  Sample output:

  Sample

  #!/usr/bin/env bash
  
  #
  # Licensed to the Apache Software Foundation (ASF) under one or more
  # contributor license agreements. See the NOTICE file distributed with
  # this work for additional information regarding copyright ownership.
  # The ASF licenses this file to You under the Apache License, Version 2.0
  # (the "License"); you may not use this file except in compliance with
  # the License. You may obtain a copy of the License at
  #
  # http://www.apache.org/licenses/LICENSE-2.0
  #
  # Unless required by applicable law or agreed to in writing, software
  # distributed under the License is distributed on an "AS IS" BASIS,
  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  # See the License for the specific language governing permissions and
  # limitations under the License.
  #
  
  # This script is a basic example script to get resource information about NVIDIA GPUs.
  # It assumes the drivers are properly installed and the nvidia-smi command is available.
  # It is not guaranteed to work on all setups so please test and customize as needed
  # for your environment. It can be passed into SPARK via the config
  # spark.{driver/executor}.resource.gpu.discoveryScript to allow the driver or executor to discover
  # the GPUs it was allocated. It assumes you are running within an isolated container where the
  # GPUs are allocated exclusively to that driver or executor.
  # It outputs a JSON formatted string that is expected by the
  # spark.{driver/executor}.resource.gpu.discoveryScript config.
  #
  # Example output: {"name": "gpu", "addresses":["0","1","2","3","4","5","6","7"]}
  
  echo {\"name\": \"gpu\", \"addresses\":[\"0\", \"1\"]}
  

Running Apache Spark shell on YARN with RAPIDS/UCX

YARN requires Spark, the Spark Rapids Plugin jars and discovery script to be installed on a launcher(Master) node. YARN handles shipping them to the nodes as needed.

To running Apache Spark shell with RAPIDS and UCX plugins we will create new start-shell.sh file in the /opt/sparkRapidsPlugin directory.

Server Console

vim /opt/sparkRapidsPlugin start-shell.sh

In our use case we are using the following configs when running Spark on YARN.

Server Console

export YARN_CONF_DIR=/opt/hadoop/

/opt/spark/bin/spark-shell \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.task.resource.gpu.amount=0.25 \
--conf spark.executor.resource.gpu.amount=1 \
--conf spark.rapids.sql.concurrentGpuTasks=1 \
--conf spark.rapids.memory.pinnedPool.size=2G \
--conf spark.locality.wait=0s \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.shuffle.partitions=150 \
--conf spark.sql.files.maxPartitionBytes=212MB \
--conf spark.rapids.sql.batchSizeBytes=212M \
--conf spark.scheduler.minRegisteredResourcesRatio=1.0 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.device.count=15 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.host.count=25 \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.shuffle.manager=com.nvidia.spark.RapidsShuffleManager \
--conf spark.rapids.shuffle.transport.enabled=true \
--conf spark.executorEnv.UCX_RNDV_SCHEME=get_zcopy \
--conf spark.executorEnv.UCX_LOG_LEVEL=debug \
--conf spark.driver.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--files ./getGpusResources.sh \

--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar

Run the ***start-shell.sh*** script.

##### **Server Console**

```bash
bash ./start-shell.sh

Example Join Operation:

Once you have started your spark shell you can run the following commands to do a basic join and look at the UI to see that it runs on the GPU.

Server Console

val df = sc.makeRDD(1 to 10000000, 6).toDF
val df2 = sc.makeRDD(1 to 10000000, 6).toDF
df.select( $"value" as "a").join(df2.select($"value" as "b"), $"a" === $"b").count

Go to the Spark UI and click on the application you ran and on the "SQL" tab. If you click the operation "count at ...", you should see the graph of Spark Execs and some of those should have the label Gpu... For instance, in the screenshot below you will see GpuRowToColumn, GpuFilter, GpuColumnarExchange, those all indicate things that would run on the GPU.

Stop Apache Spark shell

To stop Apache Spark shell click on "CTRL-C".

Example TPCH Application

To run TPCH application, please download the generated dataset and run following commands:

Server Console

sudo git clone https://github.com/databricks/tpch-dbgen.git
cd tpch-dbgen
sudo make
sudo dbgen -s 30
sudo hdfs dfs -put *.tbl

Create 3 new files spark-submit-tcp.sh, spark-submit-rc.sh, spark-submit-rc-ucx.sh in the /opt/sparkRapidsPlugin directory.

Server Console

sudo vim /opt/sparkRapidsPlugin/spark-submit-tcp.sh
sudo vim /opt/sparkRapidsPlugin/spark-submit-rc.sh
sudo vim /opt/sparkRapidsPlugin/spark-submit-rc-ucx.sh

For TCP without GPU/RAPIDS/UCX running we are using the spark-submit-tcp.sh file with the following configs when running Spark on YARN.

Server Console

export YARN_CONF_DIR=/opt/hadoop/
/opt/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.memory=4G \
--conf spark.executor.cores=15 \
--conf spark.task.cpus=1 \
--conf spark.locality.wait=0s \
--conf spark.sql.files.maxPartitionBytes=512m \
--conf spark.sql.shuffle.partitions=100 \
--conf spark.driver.extraClassPath=//usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar \
com.nvidia.spark.rapids.tests.tpch.CSV 22

For RDMA/RC with GPU/RAPIDS and without UCX running we are using the spark-submit-rc.sh file with the following configs when running Spark on YARN.

Server Console

export YARN_CONF_DIR=/opt/hadoop/
/opt/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.task.resource.gpu.amount=0.25 \
--conf spark.executor.resource.gpu.amount=1 \
--conf spark.rapids.sql.concurrentGpuTasks=1 \
--conf spark.rapids.memory.pinnedPool.size=2G \
--conf spark.locality.wait=0s \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.shuffle.partitions=150 \
--conf spark.sql.files.maxPartitionBytes=212MB \
--conf spark.rapids.sql.batchSizeBytes=212M \
--conf spark.scheduler.minRegisteredResourcesRatio=1.0 \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.shuffle.manager=com.nvidia.spark.RapidsShuffleManager \
--conf spark.driver.extraClassPath=//usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--files ./getGpusResources.sh \
--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar \
com.nvidia.spark.rapids.tests.tpch.CSV 22

For RDMA/RC with GPU/RAPIDS and UCX running we are using the spark-submit-rc-ucx.sh file with the following configs when running Spark on YARN.

Server Console

export YARN_CONF_DIR=/opt/hadoop/
/opt/spark/bin/spark-submit \
--master yarn \
--num-executors 6 \
--executor-memory 40G \
--driver-memory 2G \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.task.resource.gpu.amount=0.25 \
--conf spark.executor.resource.gpu.amount=1 \
--conf spark.rapids.sql.concurrentGpuTasks=1 \
--conf spark.rapids.memory.pinnedPool.size=2G \
--conf spark.locality.wait=0s \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.sql.shuffle.partitions=150 \
--conf spark.sql.files.maxPartitionBytes=212MB \
--conf spark.rapids.sql.batchSizeBytes=212M \
--conf spark.scheduler.minRegisteredResourcesRatio=1.0 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.device.count=15 \
--conf spark.rapids.shuffle.ucx.bounceBuffers.host.count=25 \
--conf spark.plugins=com.nvidia.spark.SQLPlugin \
--conf spark.resources.discoveryPlugin=com.nvidia.spark.ExclusiveModeGpuDiscoveryPlugin \
--conf spark.shuffle.manager=com.nvidia.spark.RapidsShuffleManager \
--conf spark.executorEnv.UCX_TLS=cuda_copy,cuda_ipc,tcp,rc \
--conf spark.executorEnv.UCX_RNDV_SCHEME=get_zcopy \
--conf spark.executorEnv.UCX_LOG_LEVEL=debug \
--conf spark.rapids.shuffle.transport.enabled=true \
--conf spark.driver.extraClassPath=//usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.extraClassPath=/usr/lib/:rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar:rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar:cudf-0.15-20200710.084843-7-cuda10-2.jar \
--conf spark.executor.resource.gpu.discoveryScript=./getGpusResources.sh \
--files ./getGpusResources.sh \
--jars /opt/sparkRapidsPlugin/rapids-4-spark_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/rapids-4-spark-integration-tests_2.12-0.2.0-SNAPSHOT.jar,/opt/sparkRapidsPlugin/cudf-0.15-20200710.084843-7-cuda10-2.jar \
com.nvidia.spark.rapids.tests.tpch.CSV 22

Done!

Appendix

General Recommendations

1. Fewer large input files are better than many of small files. You may not have control over this but it is worth knowing.
2. Larger input sizes spark.sql.files.maxPartitionBytes=512m are generally better as long as things fit into the GPU.
3. The GPU does better with larger data chunks as long as they fit into memory. When using the default spark.sql.shuffle.partitions=200 it may be beneficial to make this smaller. Base this on the amount of data the task is reading. Start with 512MB / task.
4. Out of GPU Memory. GPU out of memory can show up in multiple ways. You can see an error that it is out of memory or it can also manifest as it just crashes. Generally this means your partition size is too big, go back to the Configuration section for the partition size and/or the number of partitions. Possibly reduce the number of concurrent gpu tasks to 1. The Spark UI may give you a hint at the size of the data. Look at either the input data or the shuffle data size for the stage that failed.

RAPIDS Accelerator for Apache Spark Tuning Guide

Tuning a Spark

调整作业的配置设置通常可以提升作业性能，对于使用RAPIDS Accelerator插件的Apache Spark作业也是如此。

本文档提供了如何调整Spark作业配置设置以在使用RAPIDS Accelerator插件时获得更好性能的指南。

监控

由于插件无需更改API即可运行，查看GPU上运行情况的最简单方法是查看Spark Web UI中的“SQL”选项卡。SQL选项卡仅在执行查询后才会显示。转到UI中的SQL选项卡，点击感兴趣的查询，它会显示一个包含详细信息的DAG图。您还可以向下滚动并展开“Details”部分以查看文本表示。

如果您想通过代码查看Spark计划，可以使用explain()函数调用。例如：query.explain()将打印Spark的物理计划，您可以查看哪些节点被替换为GPU调用。

调试

目前，最佳调试方式与Spark常规调试方式相同。查看UI和日志文件以了解失败原因。如果遇到GPU段错误，请查找hs_err_pid.log文件。为确保hs_err_pid.log文件进入YARN应用程序日志，您可以添加配置：--conf spark.executor.extraJavaOptions="-XX:ErrorFile=<LOG_DIR>/hs_err_pid_%p.log"。

如果您想了解某个操作未在GPU上运行的原因，可以启用配置：--conf spark.rapids.sql.explain=NOT_ON_GPU。然后，Driver日志中会输出一条日志消息，说明Spark操作无法在GPU上运行的原因。

作者

	Boris Kovalev Boris Kovalev过去几年担任解决方案架构师，专注于NVIDIA Networking/Mellanox技术，负责复杂的机器学习、大数据和基于VMware的高级云研究与设计。此前，他在多家公司（最近在VMware）担任高级顾问和解决方案架构师超过20年。他撰写了多份参考设计，涵盖VMware、机器学习、Kubernetes和容器解决方案，可在NVIDIA文档网站上获取。
	Peter Rudenko Peter Rudenko是NVIDIA高性能计算（HPC）团队的软件工程师，专注于加速数据密集型应用，开发UCX通信库及其他大数据解决方案。