超擎数智超擎数智

How-to: Build and launch VM over OVS-DPDK v18.02 using Nvidia ConnectX adapters with DL-open linkage

Created on Jul 11, 2019. This post shows how to launch a Virtual Machine (VM) over OVS-DPDK 18.02 using Nvidia ConnectX-4 or ConnectX-5 adapters.

文档目录

Created on Jul 11, 2019

Introduction

This post shows how to launch a Virtual Machine (VM) over OVS-DPDK 18.02 using Nvidia ConnectX-4 or ConnectX-5 adapters.

Compilation now allow DL-open dynamic linkage which benefits transparent use for none Nvidia users. In case Nvidia nics are not explicitly used, Nvidia code is not loaded.

In this example MLNX_OFED 4.2-1.0.0.0 is used.

References

Prerequisites

  1. Install MLNX_OFED and using ofed_info command verify the version is 4.2-1.0.0.0 or 4.3-0.1.6.0:

    OFED download link: http://www.mellanox.com/page/products_dyn?product_family=26

    Make sure OFED installation script done as follow:

    /mnt/mlnxofedinstall --dpdk --upstream-libs

    Check OFED version:

    ofed_info -s

  2. Check CPU support for 1G hugepages by checking for pdpe1gb flag:

    cat /proc/cpuinfo | grep pdpe1gb

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nxpdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology n_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid

    Warning: Make sure that the flag list includes pdpe1gb flag.

  3. Find the NUMA number per PCI slot:

    mst start

mst status -v

MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module is not loaded
PCI devices:
------------
DEVICE_TYPE MST PCI RDMA NET NUMA
ConnectX4LX(rev:0) NA 07:00.0 mlx5_1 net-ens2f0 0

  4. Check QEMU version information. It must be Rev. 2.1 or above.

    qemu-system-x86_64 --version
QEMU emulator version 2.7.50 (v2.7.0-456-gffd455a), Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers

    To download a new qemu version refer to the official qemu site:

    http://wiki.qemu.org/Download

  5. Download required packages:

    apt-get install unzip libnuma-dev python-six git

Configuration

Hugepages configuration

Configure the grub File and Mount hugepages

  1. Update the grub.conf file.

    Warning: Updating grub files is different for each Linux OS distribution. Refer to OS documentation.

    Edit the grub.conf file in the GRUB_CMDLINE_LINUX line as follows:

    "default_hugepagesz=1G hugepagesz=1G hugepages=8"

    The following line defines the hugepages size and quantity. For most Intel processors, 2Mb and 1G sizes are supported. It is recommended to leave the OS at least 2G RAM free.

  2. Update grub using OS grub script.

  3. Reboot the server. The configuration will apply after reboot.

  4. Check that hugepages are loaded correctly after reboot:

    cat /proc/meminfo | grep Hug

AnonHugePages: 2314240 kB
HugePages_Total: 8
HugePages_Free: 8
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB

    The output shows eight hugepages free, each at 1G size.

  5. Mount hugepages at size of 1G.

    mkdir -p /dev/hugepages

mount -t hugetlbfs -o pagesize=1G none /dev/hugepages

    Error: The mount of hugepages is not persistent. You must mount hugepages after each reboot.

DPDK Configuration

  1. Download and extract DPDK 18.02 package.

    cd /usr/src/
git clone git://dpdk.org/dpdk
cd /usr/src/dpdk
git checkout v18.02

  2. Set DPDK environment variables as follows:

    export DPDK_DIR=/usr/src/dpdk-stable-18.02.2
cd $DPDK_DIR
export DPDK_TARGET=x86_64-native-linuxapp-gcc
export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET

  3. Modify compilation settings to support ConnectX-4 and ConnectX-5 interfaces with Dlopen dynamic linkage.

    sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_PMD=\)n/\1y/g' $DPDK_DIR/config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE_MLX5_DLOPEN_DEPS=\)n/\1y/g' $DPDK_DIR/config/common_base

  4. Compile your DPDK code.

    make -j install T=$DPDK_TARGET DESTDIR=install

  5. Copy shared linked library to LD library path.

    cp x86_64-native-linuxapp-gcc/lib/librte_pmd_mlx5_glue* /usr/lib64/

OVS Configuration

  1. Download OVS version 2.9.0 or later.

    cd /usr/src/
git clone https://github.com/openvswitch/ovs.git -b branch-2.9

  2. Configure OVS to compile with DPDK 18.02.

    cd /usr/src/ovs
./boot.sh
./configure --with-dpdk=$DPDK_BUILD CFLAGS="-g -O2 -Wno-cast-align"

  3. Compile your code.

    make -j
make install

  4. Reset OVS environment.

    pkill -9 ovs
rm -rf /usr/local/var/run/openvswitch/
rm -rf /usr/local/etc/openvswitch/
rm -f /usr/local/etc/openvswitch/conf.db
mkdir -p /usr/local/var/run/openvswitch/
mkdir -p /usr/local/etc/openvswitch/
rm -f /tmp/conf.db
mkdir -p /usr/local/etc/openvswitch
mkdir -p /usr/local/var/run/openvswitch
ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschema
ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach

  5. Specify initial Open vSwitch (OVS) database to use:

    export PATH=$PATH:/usr/local/share/openvswitch/scripts
export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

  6. Configure OVS to support DPDK ports:

    ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

  7. Start OVS-DPDK service:

    ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start

    Warning: Each time you reboot or there is an OVS termination, you need to rebuild the OVS environment and repeat steps 4-7 of this section.

  8. Configure the source code analyzer (PMD) to work with 2G hugespages

前提条件

  • 已安装 DPDK 18.02 并编译。
  • 已安装 OVS 2.9.0 并支持 DPDK。
  • 已安装 QEMU 2.12.0 或更高版本。
  • 已安装 NVIDIA MLNX_OFED 驱动。
  • 已配置大页内存（Hugepages）。

配置步骤

1. 配置大页内存（Hugepages）

编辑 /etc/default/grub 文件，在 GRUB_CMDLINE_LINUX 行中添加以下参数：

default_hugepagesz=1G hugepagesz=1G hugepages=8

然后更新 GRUB 并重启：

grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

2. 挂载大页文件系统

mount -t hugetlbfs hugetlbfs /dev/hugepages

3. 检查网卡 PCI 插槽

lspci | grep Mellanox

输出示例：

07:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]

4. 加载 DPDK 驱动

modprobe uio
insmod /path/to/dpdk/build/kmod/igb_uio.ko

5. 将网卡绑定到 DPDK 驱动

/path/to/dpdk/usertools/dpdk-devbind.py --bind=igb_uio 0000:07:00.0

6. 验证绑定

/path/to/dpdk/usertools/dpdk-devbind.py --status

7. 启动 OVS DPDK

export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
ovsdb-server --remote=punix:$DB_SOCK --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach
ovs-vsctl --no-wait init
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

8. 设置 DPDK 套接字内存

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="2048,0"

注意： 根据前提条件中的 NUMA 节点编号调整。此示例使用 2G（=2048）大页和 NUMA 节点 0。

9. 设置 PMD 核心掩码

ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0xe

此示例使用核心 1-4（二进制 1110）。

10. 创建 OVS 网桥

ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev

11. 创建 DPDK 端口

ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:07:00.0,n_rxq_desc=1024,n_txq_desc=1024,n_rxq=1,pmd-rxq-affinity="0:1" ofport_request=1
  • PMD 0 绑定到核心 1。
  • RXQ/TXQ 描述符设置为 1024。

12. 创建 vhost-user 端口

ovs-vsctl add-port br0 vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuserclient options:n_rxq=2,pmd-rxq-affinity="0:2,1:3"
  • PMD 0 和 1 分别绑定到核心 2 和 3。

注意：

  • 确保有足够的空闲核心用于主机和客户机的 PMD。上述配置需要 3 个主机 PMD 核心和 2 个客户机核心（共 5 个）。
  • 使用 top 命令查看两个 PMD 核心是否以 100% CPU 使用率运行。
  • 如果未使用 ofport_request，OVS 将随机选择端口 ID。

13. 设置环境变量以标识端口 ID

DPDK0_INDEX=$(echo `ovs-ofctl show br1 | grep dpdk0 | cut -d '(' -f 1`)
VHOST_USER1_INDEX=$(echo `ovs-ofctl show br1 | grep vhost-user1 | cut -d '(' -f 1`)

验证分配的 ID：

echo $DPDK0_INDEX
1
echo $VHOST_USER1_INDEX
2

14. 设置 OVS 流规则

ovs-ofctl add-flow br1 in_port=$DPDK0_INDEX,action=output:$VHOST_USER1_INDEX
ovs-ofctl add-flow br1 in_port=$VHOST_USER1_INDEX,action=output:$DPDK0_INDEX

验证流规则：

ovs-ofctl dump-flows br1

输出示例：

NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=4.903s, table=0, n_packets=8035688, n_bytes=482141280, idle_age=0, in_port=1 actions=output:2
cookie=0x0, duration=3.622s, table=0, n_packets=0, n_bytes=0, idle_age=3, in_port=2 actions=output:1
cookie=0x0, duration=353.725s, table=0, n_packets=284039649, n_bytes=17042378940, idle_age=5, priority=0 actions=NORMAL

虚拟机配置

1. 启动客户机

numactl --cpunodebind 0 --membind 0 -- \
echo 'info cpus' | \
/usr/src/qemu/x86_64-softmmu/qemu-system-x86_64 \
-enable-kvm \
-name gen-l-vrt-019-006-Ubuntu-15.10 \
-cpu host -m 6G \
-realtime mlock=off \
-smp 8,sockets=8,cores=1,threads=1 \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/gen-l-vrt-019-006-Ubuntu-15.10.monitor,server,nowait \
-drive file=/images/gen-l-vrt-019-006/gen-l-vrt-019-006.img,if=none,id=drive-ide0-0-0,format=qcow2 \
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-netdev tap,id=hostnet0,script=no,downscript=no \
-device e1000,netdev=hostnet0,id=net0,mac=00:50:56:1b:b2:05,bus=pci.0,addr=0x3 \
-chardev socket,id=char1,path=/usr/local/var/run/openvswitch/vhost-user1 \
-netdev type=vhost-user,id=iface1,chardev=char1,vhostforce,queues=2 \
-device virtio-net-pci,netdev=iface1,mac=12:34:00:00:50:2c,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,mrg_rxbuf=off,mq=on,vectors=6 \
-object memory-backend-file,id=mem,size=6144M,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem \
-mem-prealloc \
-monitor stdio \
> /tmp/qemu_cpu_info.txt&

注意：

  • 选择正确的 vhost-user 队列数以匹配 RX 队列配置。此示例中 queues=2。
  • 如果外部 e1000 接口无法启动，添加 script=no,downscript=no。在客户机启动时，手动启用接口： 
    brctl addif br0 tap0
ifconfig tap0 up
  • 共享内存大小（size）必须等于 -m 参数。内存量不能超过主机上空闲大页内存总量。

2. 设置 CPU 亲和性

a=( $(cat /tmp/qemu_cpu_info.txt | grep thread_id | cut -d '=' -f 3 | tr -d '\r' ) )
taskset -p 0x004 ${a[0]}
taskset -p 0x008 ${a[1]}
taskset -p 0x010 ${a[2]}
taskset -p 0x020 ${a[3]}
taskset -p 0x040 ${a[4]}
taskset -p 0x080 ${a[5]}
taskset -p 0x100 ${a[6]}
taskset -p 0x200 ${a[7]}

注意： 确保所选核心不与主机 PMD 核心重叠。此示例将核心 3-10 绑定到客户机。

3. 在客户机中配置大页内存

参考 GRUB 和大页配置部分。

4. 加载客户机 DPDK 驱动

假设客户机镜像已包含编译好的 DPDK 驱动。

有关如何在客户机中编译 DPDK 18.02 的更多信息，请参阅：

Compiling the DPDK Target from Source — Data Plane Development Kit 18.02.0-rc0 documentation

5. 查找 virtio 接口总线号

lspci -nn | grep -i virtio
00:04.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]

6. 加载 DPDK 驱动

modprobe uio
insmod /usr/src/dpdk-18.02/x86_64-ivshmem-linuxapp-gcc/kmod/igb_uio.ko

7. 绑定 DPDK 驱动到 virtio 接口

/usr/src/dpdk-18.02/tools/dpdk-devbind.py --bind=igb_uio 0000:00:04.0

验证

1. 运行 testpmd 进行环路测试

此示例使用 2G 大页和两个 PMD 核心。--rxq 和 --txq 队列数必须与之前定义的队列数一致。

/usr/src/dpdk-18.02/x86_64-ivshmem-linuxapp-gcc/app/testpmd -v -c 0x1f -n 4 -m 2048 -- --burst=64 --rxq=2 --txq=2 --nb-cores=4 -a -i --mbcache=256 --rss-udp --port-topology=chained

注意： UDP RSS 仅在注入不同源 UDP 端口时生效。

性能调优建议

1. 隔离 PMD CPU

在 /etc/default/grub 的 GRUB_CMDLINE_LINUX 行中添加：

"isolcpus=1-8 nohz_full=1-8 rcu_nocbs=1-8"

2. 设置性能模式

for (( i=0; i<$(cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | wc -l); i++ )); do echo performance > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done

3. 停止 irqbalance 服务

service irqbalance stop

irqbalance stop


4. 禁用内核内存压缩：

echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag sysctl -w vm.zone_reclaim_mode=0 sysctl -w vm.swappiness=0


5. 在虚拟机内部，禁用 ksm/run。

echo 0 > /sys/kernel/mm/ksm/run


**相关文档**

- [RDG for VMware vSphere 8.0 with NSX 4.x Accelerated by NVIDIA Networking](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471345533/RDG+for+VMware+vSphere+8.0+with+NSX+4.x+Accelerated+by+NVIDIA+Networking) (解决方案) 标签：connectx, solution_platforms, enterprise, dpdk, firmware, esxi, nsx, vmware, market_verticals, telecom_and_isp, external, internal
- [RDG for Red Hat OpenStack Platform 16.1 Versatile Cloud for Packet and Data Processing with NVIDIA Network Hardware Acceleration](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471340193/RDG+for+Red+Hat+OpenStack+Platform+16.1+Versatile+Cloud+for+Packet+and+Data+Processing+with+NVIDIA+Network+Hardware+Acceleration) (解决方案) 标签：virtual_machine, openstack, redhat, virtualization, dpdk, asap², telecom_and_isp, gpu, external, internal
- [RDG for DPDK Applications on SR-IOV Enabled Kubernetes Cluster with NVIDIA Network Operator](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471330218/RDG+for+DPDK+Applications+on+SR-IOV+Enabled+Kubernetes+Cluster+with+NVIDIA+Network+Operator) (解决方案) 标签：k8s_and_containers, dpdk, k8s, cumulus, sr-iov, external, internal
- [RDG for Canonical Charmed OpenStack with NVIDIA Networking and Accelerated OVN for High Performance Workloads](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471329293/RDG+for+Canonical+Charmed+OpenStack+with+NVIDIA+Networking+and+Accelerated+OVN+for+High+Performance+Workloads) (解决方案) 标签：virtual_machine, openstack, redhat, virtualization, dpdk, asap², telecom_and_isp, gpu, unrestored-unknown-attachment, external

# 操作指南：使用支持 DL-open 链接的 NVIDIA ConnectX 网卡在 OVS-DPDK v18.02 上构建并启动虚拟机

创建于 2019 年 7 月 11 日

## 简介

本文介绍如何使用 NVIDIA ConnectX-4 或 ConnectX-5 网卡在 OVS-DPDK 18.02 上启动虚拟机 (VM)。

## 相关页面

- [RDG: Red Hat OpenStack Platform 13 with NVIDIA Network OVS Offload.](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471326747/RDG+Red+Hat+OpenStack+Platform+13+with+NVIDIA+Network+OVS+Offload.) (解决方案) 标签：`spectrum`, `virtual_machine`, `ospf`, `ovs`, `openstack`, `vnf`, `virtualization`, `rhosp`, `nfv`, `solution_platforms`, `asap²`, `vxlan`, `external`, `internal`
- [RDG: OpenStack SMPTE 2110 Media Streaming Cloud with NVIDIA Network Hardware Offload.](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471326551/RDG+OpenStack+SMPTE+2110+Media+Streaming+Cloud+with+NVIDIA+Network+Hardware+Offload.) (解决方案) 标签：`spectrum`, `virtual_machine`, `rivermax`, `media_and_entertainment`, `ovs`, `openstack`, `virtualization`, `rhosp`, `solution_platforms`, `asap²`, `vxlan`, `external`, `internal`
- [RDG: NFVi Platform Based on Red Hat OSP13, Nuage Accelerated SDN and NVIDIA ConnectX-5 SmartNIC.](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471325767/RDG+NFVi+Platform+Based+on+Red+Hat+OSP13+Nuage+Accelerated+SDN+and+NVIDIA+ConnectX-5+SmartNIC.) (解决方案) 标签：`spectrum`, `nuage`, `openstack`, `redhat`, `virtualization`, `rhosp`, `nfv`, `connectx`, `ethernet`, `asap²`, `telecom_and_isp`, `sdn`, `vxlan`, `linux`, `external`, `internal`
- [QSG of Building Docker Image with Compiled Nvidia Network DPDK PMD](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471325304/QSG+of+Building+Docker+Image+with+Compiled+Nvidia+Network+DPDK+PMD) (解决方案) 标签：...

# How-to: Build and launch VM over OVS-DPDK v18.02 using Nvidia ConnectX adapters with DL-open linkage

Created on Jul 11, 2019

## Introduction

This post shows how to launch a Virtual Machine (VM) over OVS-DPDK 18.02 using Nvidia ConnectX-4 or ConnectX-5 adapters.

## Labels

- [connectx](https://nvidia.atlassian.net/wiki/label/connectx)
- [k8s_and_containers](https://nvidia.atlassian.net/wiki/labels/viewlabel.action?ids=1773176333)
- [dpdk](https://nvidia.atlassian.net/wiki/label/dpdk)
- [ethernet](https://nvidia.atlassian.net/wiki/label/ethernet)
- [docker](https://nvidia.atlassian.net/wiki/label/docker)
- [external](https://nvidia.atlassian.net/wiki/label/external)
- [internal](https://nvidia.atlassian.net/wiki/label/internal)

## Related Pages

- [QSG for Configuring TRex in a few steps using Nvidia ConnectX adapters](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471325256/QSG+for+Configuring+TRex+in+a+few+steps+using+Nvidia+ConnectX+adapters) (解决方案)
  - Labels: connectx, dpdk, ethernet, market_verticals, telecom_and_isp, external, internal
- [QSG for Build and Run Vector Packet Processing (VPP) Applications Over NVIDIA SmartNICs](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471325008/QSG+for+Build+and+Run+Vector+Packet+Processing+VPP+Applications+Over+NVIDIA+SmartNICs) (解决方案)
  - Labels: nfv, connectx, dpdk, market_verticals, telecom_and_isp, vpp, external, internal
- [QSG: NVIDIA BlueField DPU with OVS Hardware Offload.](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471324988/QSG+NVIDIA+BlueField+DPU+with+OVS+Hardware+Offload.) (解决方案)
  - Labels: smartnic_and_soc, ovs, asap², bluefield, external, internal
- [QSG: NVIDIA BlueField DPU with DPDK Acceleration.](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471324968/QSG+NVIDIA+BlueField+DPU+with+DPDK+Acceleration.) (解决方案)
  - Labels: smartnic_and_soc, dpdk, bluefield, external, internal
- [QSG: High Availability with Mellanox ASAP2 Enhanced SR-IOV (VF-LAG).](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471324329/QSG+High+Availability+with+Mellanox+ASAP2+Enhanced+SR-IOV+VF-LAG+.) (解决方案)
  - Labels: redhat, virtualization, nfv, connectx, ethernet, asap², telecom_and_isp, connectivity

# How-to: Build and launch VM over OVS-DPDK v18.02 using Nvidia ConnectX adapters with DL-open linkage

Created on Jul 11, 2019

This post shows how to launch a Virtual Machine (VM) over OVS-DPDK 18.02 using Nvidia ConnectX-4 or ConnectX-5 adapters.

## Related Pages

- [QSG: ASAP² technology performance evaluation on Red Hat OpenStack Platform 13.](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471324241/QSG+ASAP+technology+performance+evaluation+on+Red+Hat+OpenStack+Platform+13.) (解决方案)
  - Labels: ovs, openstack, nfv, solution_platforms, asap², rhel, external, internal
- [How-to: Build and launch VM over OVS-DPDK v18.02 using Nvidia ConnectX adapters with DL-open linkage](https://nvidia.atlassian.net/wiki/spaces/SOL/pages/3471320341/How-to+Build+and+launch+VM+over+OVS-DPDK+v18.02+using+Nvidia+ConnectX+adapters+with+DL-open+linkage) (解决方案)
  - Labels: ovs, dpdk, market_verticals, telecom_and_isp, external, internal