QSG: NVIDIA BlueField DPU with DPDK Acceleration.

Created on Jun 30, 2019 Introduction BlueField's Ethernet DPU solutions combine Arm® processing power with advanced network offloads to accelerate a multitude

文档目录

Created on Jun 30, 2019

Introduction

BlueField's Ethernet DPU solutions combine Arm® processing power with advanced network offloads to accelerate a multitude of security, networking and storage applications.

Based on the BlueField system-on-chip (SoC) programmable network engine, BlueField DPU integrate Mellanox ConnectX network controller with Arm multicore processors and cryptology options to deliver an innovative and high-performance programmable platform.

Through this QSG (Quick Start Guide) article you will be able to quickly set up a simple test lab which will allow you to experiment and explore BlueField DPU capabilities. The setup includes a bare metal server hosting BlueField 25GbE DPU connected back-to-back to another bare metal server with ConnectX-5 NIC.

Equipment

Qty.	Item	Comments
2	x86 servers running CentOS 7.5	-
1	ConnectX-5 NIC (FW: 16.24.0328 and up)	-
1	BlueField 25GbE DPU with Arm processor (FW: 18.24.1000 and up)	-
1	25GB/s SFP28 networking cable	-
1	QSA28 Adapter (100GB-to-25GB Adapter)	-
1	UART-to-USB cable	For DPU USB console access
1	Mini USB Type B cable	For DPU USB Console Access
1	bfb image to be loaded on the BlueField DPU Arm.	Image contains:

CentOS 7.5 over Arm architecture
MLNX_OFED_LINUX-4.4-2.5.11.0 (compiled with DPDK and BlueField flags)
DPDK libraries version 18.11
Pre-configured network settings for easy DPU management |

Physical Configuration Diagram

The following diagram shows the physical configuration between the BlueField DPU and the ConnectX-5 NIC.

Physical Configuration Instructions

Insert ConnectX-5 NIC into Server A
Insert BlueField DPU into Server B
Connect the DPU UART cable from DPU to its hosting Server B USB port #1
Connect Mini USB cable from DPU to its hosting Server B USB port #2
Insert QSA28 Adapter into Server B 100GB port
Connect one of the DPU ports to the ConnectX-5 using 25GB cable
Connect both servers to a management LAN

DPU Server Preparations

Install useful packages:

yum install -y screen minicom net-tools tcpdump usbutils net-utils sshpass iptables-services fio epel-release iperf wget git unzip perl pciutils lsof tcl libxml2-python tk unbound gtk2 atk cairo gcc-gfortran tcsh nvme-cli

Install MLNX_OFED with the mentioned flags below.
```
tar -xzvf MLNX_OFED_LINUX-4.4-2.5.11.0-rhel7.5-x86_64.tgz
cd MLNX_OFED_LINUX-4.4-2.5.11.0-rhel7.5-x86_64
./mlnxofedinstall --with-nvmf --bluefield --without-fw-update
```
Note:
- Use automatic FM update during MLNX_OFED installation ONLY if the current BlueField FW version is 18.24.1000 (GA) or above. In case its older, manually burn the FW after MLNX_OFED installation.
- In case you plan to run DPDK on the host itself, install MLNX_OFED with "--dpdk --upstream-libs" flags
- The "bluefield" flag is not required in some of the MLNX_OFED versions and OS releases

Enable SR-IOV on FW (Optional):

mlxconfig -y -d /dev/mst/mt41682_pciconf0 s SRIOV_EN=1
mlxconfig -y -d /dev/mst/mt41682_pciconf0 s NUM_OF_VFS=8

Uncomment the following line in /etc/modprobe.d/rshim.conf to set the default RShim access method to the DPU:
```
options rshim backend_driver=rshim_usb
```
Note:
- When this line is commented, the first available driver will be used.
- When the mini USB cable is not used, the driver should be rshim_pcie
Create a file named ifcfg-tmfifo_net0 under /etc/sysconfig/network-scripts/ directory, with the following content:
```
NAME="tmfifo_net0"
HWADDR="00:1a:ca:ff:ff:02"
ONBOOT=yes
NETBOOT=yes
IPV6INIT=yes
BOOTPROTO=static
TYPE=Ethernet
IPADDR=192.168.100.1
NETMASK=255.255.255.0
```
ifcfg-tmfifo_net0 is the network interface used for IP communication between the x86 host and the DPU with rshim_net driver.

Enable ip forwarding and set a NAT rule to allow the DPU OS to communicate with the outer world via the host management interface:

sysctl -w net.ipv4.ip_forward=1
systemctl enable iptables
systemctl start iptables
iptables -F
iptables -t nat -A POSTROUTING -o <host_mgmt_interfac> -j MASQUERADE
service iptables save

Disable Firewall and Network Manager services

systemctl disable firewalld.service
systemctl disable NetworkManager

Reboot the server. Once server is up again, run some configuration sanity tests:
- To verify BlueField FW version:
```
mlxfwmanager
```

mlxfwmanager --query

To verify RShim modules are loaded:
```
lsmod | grep -i rshim
```
To identify the backend active RShim driver:
```
cat /dev/rshim0/misc
```
To verify the NAT rule for DPU communication with the outer world:
```
iptables -L -n -t nat
```
Contact NVIDIA Support to download the BlueField Quick Start bfb image
Place the bfb image on the host and load it on the BlueField DPU using the following command:
```
cat /home/CentOS7.5ALT-aarch64-QuickStart.bfb > /dev/rshim0/boot
```

Note: Use one of the console access methods described in the "DPU Management Methods" section below to follow the image installation process. When bfb image installation process is completed, connect to the DPU with the console or SSH, as described below, and test connectivity from the SmartNIC OS to the outer world.

DPU Management Methods

To manage the DPU from the hosting server, use one of the following methods.

Access Credentials: root\centos

Console over USB-Serial / Serial Port properties: Rate 115200, HW Flow Control OFF.
```
minicom -D /dev/ttyUSB0
```
Console over USB (rshim_usb) / Serial Port properties: Rate 115200, HW Flow Control OFF
```
minicom -D /dev/rshim0/console
```
SSH (rshim_net)
```
ssh root@192.168.100.2
```

Note: DPU OS is configured to obtain internet access via its hosting server.

Basic DPU Management Operations

How-to soft reset the DPU via host server:

Issue the following command on the server:
```
echo "SW_RESET 1" > /dev/rshim0/misc
```
Follow the boot process using console access:
```
minicom -D /dev/ttyUSB0
```

Note: This is not a power cycle of the DPU.

How-to load BlueField DPU OS image from hosting server:

Place bfb image on the server
Issue the following command on the server:
```
cat <image>.bfb > /dev/rshim0/boot
```
Follow the installation and boot process using the console access:
```
minicom -D /dev/ttyUSB0
```

DPU Modes

BlueField DPU can operate in two modes:

SEPERATED_HOST (default mode)

In this mode, both the x86 hosting server and the DPU Arm OS act as separated entities; Both can use and communicate with each other or the network via the ConnectX-5 module of the DPU.
EMBEDDED

In this mode, the x86 hosting server communicates with the outer world only through the DPU Arm.

Note: In order to identify the DPU current mode, issue the following commands on the x86 hosting server:
x86_host#mst start
x86_host#mlxconfig -d /dev/mst/mt41682_pciconf0 q | grep -i model

Switching to EMBEDDED Mode from SEPERATED_HOST Mode

On the x86 host:

Enable EMBEDDED mode:

x86_host #mst start
x86_host #mlxconfig -d /dev/mst/mt41682_pciconf0 s INTERNAL_CPU_MODEL=1

Reboot.

Verify:

x86_host#mst start
x86_host#mlxconfig -d /dev/mst/mt41682_pciconf0 q | grep -i model

On the Arm:

Issue the following commands to enable ECPF parameters (applied per port):

Note: The ECPF parameters will be available for setting only after completing the previous steps.

Snic#mst start
Snic#mst status -v
Snic#mlxconfig -d /dev/mst/mt41682_pciconf0 s ECPF_ESWITCH_MANAGER=1 ECPF_PAGE_SUPPLIER=1
Snic#mlxconfig -d /dev/mst/mt41682_pciconf0.1 s ECPF_ESWITCH_MANAGER=1 ECPF_PAGE_SUPPLIER=1

Reboot the x86 host.

Verify configuration:

Once the procedure is completed, issue the following command on both the x86 host and the Arm:

/opt/mellanox/iproute2/sbin/rdma link

On x86 host: the output should list two net devices.

1/1: mlx5_0/1: state ACTIVE physical_state LINK_UP netdev ens1f0
2/1: mlx5_1/1: state DOWN physical_state DISABLED netdev ens1f1

On Arm: the output should list four representors:

1/1: mlx5_0/1: state ACTIVE physical_state LINK_UP netdev rep0-ffff
2/1: mlx5_1/1: state ACTIVE physical_state LINK_UP netdev rep0-0
3/1: mlx5_2/1: state DOWN physical_state DISABLED netdev rep1-ffff
4/1: mlx5_3/1: state DOWN physical_state DISABLED netdev rep1-0

Issue

在 Arm 上执行以下命令，查看 PCI 设备与其 representor 之间的对应关系：

mst start
mst status -v

PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
BlueField(rev:0) NA 03:00.1 mlx5_3,mlx5_2 net-eth2,net-rep1-ffff -1
BlueField(rev:0) NA 03:00.0 mlx5_1,mlx5_0 net-rep0-ffff,net-eth1 -1

注意：rep0-0 / rep1-0 是面向 x86 主机的 representor，而 rep0-ffff / rep1-ffff 是面向网络的 representor。

从 EMBEDDED 模式切换到 SEPERATED_HOST 模式

在 x86 主机上：

启用 SEPERATED_HOST 模式：

x86_host #mst start
x86_host #mlxconfig -d /dev/mst/mt41682_pciconf0 s INTERNAL_CPU_MODEL=0

重启。

验证：

x86_host#mst start
x86_host#mlxconfig -d /dev/mst/mt41682_pciconf0 q | grep -i model

在 EMBEDDED 模式下使用 DPDK 测试流量

在嵌入式模式下，从托管 DPU 的 x86 服务器到托管 ConnectX-5 的远程 x86 服务器的流量通过 DPU Arm 转发。DPDK 应用程序用于转发流量并绕过 Arm 操作系统内核。请按照以下配置步骤进行测试。

在 Arm 上

停止 OVS：
```
systemctl stop openvswitch
```

验证 hugetlbfs 是否已挂载：

mount | grep -i hugetlbfs

如果未挂载，则挂载：

mount -t hugetlbfs nodev /dev/hugepages

设置 Hugepages：

echo 4 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages

验证大页是否已分配（查看 HugePages_Total 和 Free）：
```
cat /proc/meminfo | grep -i huge
```
识别 DPU PCI 设备 ID：
```
lspci | grep -i mellanox
```
启动 testPMD 应用程序，同时探测 representor（PCI 设备为 03:00.0）：
```
cd /usr/bin/
./testpmd -w 03:00.0,representor=[0,65535] – -i
testpmd> start
```

在 x86 主机和 ConnectX-5 主机上设置 IP，并使用 ping 测试连通性（流量将由 Arm DPDK 转发）。
在 ConnectX-5 外部主机 上启动 iperf 服务器：
```
iperf -s
```
在 x86 主机 上启动 iperf 客户端：
```
iperf -c 172.16.0.2 -P12
```
测试运行时，可以在 Arm testPMD 上查看流量统计信息：
```
testpmd> show port stats all
```

作者


	Itai Levy过去几年，Itai Levy 一直担任解决方案架构师，是 NVIDIA 网络“解决方案实验室”团队的成员。他围绕云计算、软件定义网络、存储和安全设计和执行前沿解决方案。他的主要专业领域包括 NVIDIA BlueField 数据处理单元 (DPU) 解决方案以及加速的 K8s/OpenStack 平台。

相关文档

QSG: NVIDIA BlueField DPU with DPDK Acceleration.

Created on Jun 30, 2019

Introduction

BlueField's Ethernet DPU solutions combine Arm® processing power with advanced network offloads to accelerate a multitude

[IL][INTERNAL] Quick Start Guide: BlueField SNAP NVMe Emulation with SPDK Proxy for NVMe/TCP Transport
(xx.26.1040) ConnectX 网卡 Firmware Release Notes
(16.26.1502) ConnectX 网卡 Firmware Release Notes
(1.0) [IL][INTERNAL] Quick Start Guide: BlueField SNAP NVMe Emulation with SPDK Proxy for NVMe/RoCE Transport
[IL][INTERNAL] Quick Start Guide: BlueField SNAP NVMe Emulation with SPDK Proxy for NVMe/RoCE Transport
(xx.26.1040) PreBoot Drivers (FlexBoot/UEFI)
(16.26.1502) PreBoot Drivers (FlexBoot/UEFI)
(16.26.1602) PreBoot Drivers (FlexBoot/UEFI)
(20.99.9002) PreBoot Drivers (FlexBoot/UEFI)