超擎数智超擎数智

在启用 SR-IOV 的 Kubernetes 集群上使用 NVIDIA Network Operator 运行 DPDK 应用的参考部署指南

创建于 2021 年 7 月 7 日。范围：以下参考部署指南 (RDG) 说明了如何构建一个高性能的 Kubernetes (K8s) 集群，使用 containerd 容器运行时，能够在 NVIDIA 端到端以太网基础设施上运行基于 DPDK 的应用。

文档目录

创建于 2021 年 7 月 7 日。

范围

以下参考部署指南 (RDG) 说明了如何构建一个高性能的 Kubernetes (K8s) 集群，使用 containerd 容器运行时，能够在 NVIDIA 端到端以太网基础设施上运行基于 DPDK 的应用。

本 RDG 描述了一个多台服务器连接到单台交换机的解决方案，该交换机为 Kubernetes 集群提供辅助网络。更复杂的多 L2 域横向扩展网络拓扑不在本文档的讨论范围内。

缩写和缩略语

术语 定义 术语 定义
CNI 容器网络接口 LLDP 链路层发现协议
CR 自定义资源 NFD 节点特性发现
CRD 自定义资源定义 OCI 开放容器倡议
CRI 容器运行时接口 PF 物理功能
DHCP 动态主机配置协议 QSG 快速入门指南
DNS 域名系统 RDG 参考部署指南
DP 设备插件 RDMA 远程直接内存访问
DPDK 数据平面开发套件 RoCE 融合以太网上的 RDMA
EVPN 以太网 VPN SR-IOV 单根输入输出虚拟化
HWE 硬件启用 VF 虚拟功能
IPAM IP 地址管理 VPN 虚拟专用网络
K8s Kubernetes VXLAN 虚拟可扩展局域网

引言

使用 containerd 容器运行时配置 Kubernetes 集群以运行基于 DPDK 的工作负载可能变得极其复杂。 正确的设计以及软硬件组件的选择可能是成功部署的关键。 本指南提供了完整的解决方案周期，包括技术概述、设计、组件选择和部署步骤。 该解决方案将在标准服务器上基于 NVIDIA 端到端以太网基础设施交付。

在本文档中，我们将使用新的 NVIDIA Network Operator，它负责部署和配置 SRIOV 设备插件和 SRIOV CNI。这些组件允许在 Kubernetes 工作节点上运行 DPDK 工作负载。

参考文献

解决方案架构

关键组件和技术

  • NVIDIA ConnectX 智能网卡 10/25/40/50/100/200 和 400G 以太网网卡 业界领先的 NVIDIA® ConnectX® 系列智能网卡提供先进的硬件卸载和加速。 NVIDIA 以太网网卡为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供最高的 ROI 和最低的总拥有成本。

  • NVIDIA LinkX 线缆 NVIDIA® LinkX® 线缆和收发器产品系列提供业界最完整的 10、25、40、50、100、200 和 400GbE 以太网以及 100、200 和 400Gb/s InfiniBand 产品，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。

  • NVIDIA Spectrum 以太网交换机 灵活的外形，支持 16 到 128 个物理端口，支持 1GbE 到 400GbE 速度。 基于为性能和可扩展性优化的突破性硅技术，NVIDIA Spectrum 交换机非常适合构建高性能、高性价比和高效的云数据中心网络、以太网存储结构和深度学习互连。 NVIDIA 结合了基于业界领先的专用集成电路 (ASIC) 技术的 NVIDIA Spectrum™ 交换机的优势，以及多种现代网络操作系统选择，包括 NVIDIA Cumulus® Linux、SONiC 和 NVIDIA Onyx®。

  • NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux 是业界最具创新性的开放网络操作系统，允许您像其他系统一样自动化、定制和扩展数据中心网络。

  • RDMA RDMA 是一种允许网络中的计算机在不涉及任一计算机的处理器、缓存或操作系统的情况下交换数据的技术。 与本地 DMA 类似，RDMA 提高了吞吐量和性能，并释放了计算资源。

  • Kubernetes Kubernetes 是一个开源容器编排平台，用于...

deployment automation, scaling, and management of containerized applications.

  • Kubespray Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks and provides:

    • A highly available cluster
    • Composable attributes
    • Support for most popular Linux distributions

  • NVIDIA Network Operator The NVIDIA Network Operator simplifies the provisioning and management of NVIDIA networking resources in a Kubernetes cluster. The operator automatically installs the required host networking software - bringing together all the needed components to provide high-speed network connectivity. These components include the NVIDIA networking driver, Kubernetes device plugin, CNI plugins, IP address management (IPAM) plugin and others. The NVIDIA Network Operator works in conjunction with the NVIDIA GPU Operator to deliver high-throughput, low-latency networking for scale-out, GPU computing clusters.

  • What is containerd? An industry-standard container runtime with an emphasis on simplicity, robustness and portability. containerd is available as a daemon for Linux and Windows. It manages the complete container lifecycle of its host system, from image transfer and storage to container execution and supervision to low-level storage to network attachments and beyond.

  • NVIDIA PMDs NVIDIA Poll Mode Driver (PMD) is an open-source upstream driver, embedded within dpdk.org releases, designed for fast packet processing and low latency by providing kernel bypass for receive and send and by avoiding the interrupt processing performance overhead.

  • TRex—Realistic Traffic Generator TRex is an open source stateful and stateless traffic generator fueled by DPDK. It generates L3-7 traffic and provides in one tool capabilities provided by commercial tools. TRex can scale up to 200Gb/sec with one server.

Logical Design

The logical design includes the following parts:

  1. Deployment node running Kubespray that deploys Kubernetes clusters.
  2. K8s master node running all Kubernetes management components.
  3. K8s worker nodes.
  4. TRex server.
  5. High-speed Ethernet fabric for DPDK tenant network
  6. Deployment and K8s management network.

image2021-8-10_9-28-26.png

Fabric Design

The high-performance network is a secondary network for Kubernetes cluster and required the L2 network topology.

This RDG describes a solution with multiple servers connected to a single switch that provides secondary network for the Kubernetes cluster.

A more complex scale-out network topology of multiple L2 domains is beyond the scope of this document.

Software Stack Components

image2021-6-22_9-58-52.png

Bill of Materials

The following hardware setup is utilized in this guide.

image2021-8-10_9-34-31.png

Note: The above table does not contain the management network connectivity components.

Deployment and Configuration

Wiring

On each K8s worker node and TRex server, the first port of each NVIDIA 网卡 is wired to the NVIDIA switch in high-performance fabric using NVIDIA LinkX DAC cables.

image2021-8-10_9-36-6.png

Warning: Deployment and Management network is part of IT infrastructure and is not covered in this guide.

Fabric

Prerequisites

  • High-performance Ethernet fabric
    • Single switch NVIDIA SN2100
    • Switch OS Cumulus Linux v4.2.1
  • Deployment and management network DNS and DHCP network services and network topology are part of the IT infrastructure. The component installation and configuration are not covered in this guide.

Network Configuration

Below are the server names with their relevant network configurations.

Server/Switch type Server/Switch name High-speed network Management network 1/25 GbE
Deployment node depserver ens4f0: DHCP 192.168.222.110
Master node node1 ens4f0: DHCP 192.168.222.111
Worker node node2 ens2f0: no IP set ens4f0: DHCP 192.168.222.101
Worker node node3 ens2f0: no IP set ens4f0: DHCP 192.168.222.102
TRex server node4 ens2f0: no IP set, ens2f1: no IP set ens4f0: DHCP 192.168.222.103
High-speed switch leaf01 mgmt0: From DHCP 192.168.222.201

Warning

  • ensXf0 high-speed network interfaces do not require additional configuration.

Fabric Configuration

This solution is based on Cumulus Linux v4.2.1 switch operation system.

Intermediate-level Linux knowledge is assumed for this guide. Familiarity with basic text editing, Linux file permissions, and process monitoring is required. A variety of text editors are pre-installed, including vi and nano.

Networking engineers who are unfamiliar with Linux concepts should refer to this reference guide to compare the Cumulus Linux CLI and configuration options and their equivalent Cisco Nexus 3000 NX-OS commands and settings. There is also a series of short videos with an introduction to Linux and Cumulus-Linux-specific concepts.

A Greenfield deployment is assumed for this guide. Please refer to the following guide for Upgrading Cumulus Linux.

Fabric configuration steps:

  1. Administratively enable all physical ports.
  2. Create a bridge and configure one or more front panel ports as members of the bridge.
  3. Commit configuration.

Switch configuration steps.

Switch console
Linux swx-mld-l03 4.19.0-cl-1-amd64 #1 SMP Cumulus 4.19.94-1+cl4.2.1u1 (2020-08-28) x86_64

Welcome to NVIDIA Cumulus (R) Linux (R)

For support and online technical documentation, visit
http://www.cumulusnetworks.com/support

The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.

cumulus@leaf01:mgmt:~$ net add interface swp1-16
cumulus@leaf01:mgmt:~$ net add bridge bridge ports swp1-16
cumulus@leaf01:mgmt:~$ net commit

To view link status, use the net show interface all command. The following examples show the output of ports in admin down, down, and up modes.

Switch console
cumulus@leaf01:mgmt:~$ net show interface all
State  Name    Spd   MTU    Mode        LLDP                             Summary
-----  ------  ----  -----  ----------  -------------------------------  ------------------------
UP     lo      N/A   65536  Loopback                                     IP: 127.0.0.1/8
       lo                                                                IP: ::1/128
UP     eth0    1G    1500   Mgmt        mgmt-xxx-xxx-xxx-xxx (8)         Master: mgmt(UP)
       eth0                                                              IP: 192.168.222.201/24(DHCP)
UP     swp1    100G  9216   Access/L2                                    Master: bridge(UP)
UP     swp2    100G  9216   Access/L2   node2 (0c:42:a1:2b:74:ae)        Master: bridge(UP)
UP     swp3    100G  9216   Access/L2                                    Master: bridge(UP)
UP     swp4    100G  9216   Access/L2   node3 (0c:42:a1:24:05:4a)        Master: bridge(UP)
UP     swp5    100G  9216   Access/L2                                    Master: bridge(UP)
UP     swp6    100G  9216   Access/L2   node4 (0c:42:a1:24:05:1a)        Master: bridge(UP)
UP     swp7    100G  9216   Access/L2                                    Master: bridge(UP)
UP     swp8    100G  9216   Access/L2   node4 (0c:42:a1:24:05:1b)        Master: bridge(UP)
DN     swp9    N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp10   N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp11   N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp12   N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp13   N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp14   N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp15   N/A   9216   Access/L2                                    Master: bridge(UP)
DN     swp16   N/A   9216   Access/L2                                    Master: bridge(UP)
UP     bridge  N/A   9216   Bridge/L2
UP     mgmt    N/A   65536  VRF                                          IP: 127.0.0.1/8
       mgmt                                                              IP: ::1/128

Nodes Configuration

General Prerequisites:

  • Hardware

    All the K8s worker nodes have the same hardware specification (see BoM for details).

  • Host BIOS

    Verify that SR-IOV supported server platform is being used and review the BIOS settings in the server platform vendor documentation to enable SR-IOV in the BIOS.

  • Host OS

    Ubuntu Server 20.04 operating system should be installed on all servers with OpenSSH server packages.

  • Experience with Kubernetes

    Familiarization with the Kubernetes Cluster architecture is essential.

Error

Make sure that the BIOS settings on the worker nodes servers have SR-IOV enabled and that the servers are tuned for maximum performance.

All worker nodes must have the same PCIe placement for the NIC and expose the same interface name.

Host OS Prerequisites

Make sure Ubuntu Server 20.04 operating system is installed on all servers with OpenSSH server packages and create a non-root depuser account with sudo privileges without password.

Update the Ubuntu software packages by running the following commands:

Server console
$ sudo apt-get update
$ sudo apt-get upgrade -y
$ sudo reboot

In this solution we added the following line to the EOF /etc/sudoers:

Server console
$ sudo vim /etc/sudoers

#includedir /etc/sudoers.d

#K8s cluster deployment user with sudo privileges without password
depuser ALL=(ALL) NOPASSWD:ALL

NIC Firmware Upgrade

It is recommended to upgrade the NIC firmware on the worker nodes to the latest released version.

Download mlxup firmware update and query utility to each worker node and update the NIC firmware.

The most recent version of mlxup can be downloaded from the official download page. mlxup can download and update the NIC firmware to the latest firmware over the Internet.

The utility execution required sudo privileges:

Worker Node console
# wget http://www.mellanox.com/downloads/firmware/mlxup/4.15.2/SFX/linux_x64/mlxup
# chmod +x mlxup
# ./mlxup -online -u

RDMA Subsystem Configuration

RDMA subsystem configuration is required on each worker node.

  1. Instal LLDP Daemon and RDMA Core Userspace Libraries and Daemons.

    Worker Node console

    # apt install -y lldpd rdma-core

    LLDPD is a daemon able to receive and send LLDP frames. The Link Layer Discovery Protocol (LLDP) is a vendor-neutral Layer 2 protocol that allows a network device to advertise its identity and capabilities on the local network.

  2. Identify the name of the RDMA-capable interface for

high-performance K8s network.

In this guide, ens2f0 network interface for high-performance K8s network was chosen and will be activated by NVIDIA Network Operator deployment:

Worker Node console

# rdma link
link rocep7s0f0/1 state DOWN physical_state DISABLED netdev ens2f0
link rocep7s0f1/1 state DOWN physical_state DISABLED netdev ens2f1
link rocep131s0f0/1 state ACTIVE physical_state LINK_UP netdev ens4f0
link rocep131s0f1/1 state DOWN physical_state DISABLED netdev ens4f1

  1. Set RDMA subsystem network namespace mode to exclusive mode. RDMA subsystem network namespace mode (netns parameter in ib_core module) in exclusive mode allows network namespace isolation for RDMA workloads on the worker node servers. Please create /etc/modprobe.d/ib_core.conf configuration file to change ib_core module parameters:

    /etc/modprobe.d/ib_core.conf

    # Set netns to exclusive mode for namespace isolation
options ib_core netns_mode=0

    Then re-generate the initial RAM disks and reboot servers:

    Worker Node console

    # update-initramfs -u
# reboot

    After the server comes back, check netns mode:

    Worker Node console

    # rdma system

netns exclusive

K8s Cluster Deployment and Configuration

The Kubernetes cluster in this solution will be installed using Kubespray with a non-root depuser account from the deployment node.

SSH Private Key and SSH Passwordless Login

Log in to the deployment node as a deployment user (in this case, depuser) and create an SSH private key for configuring the passwordless authentication on your computer by running the following commands:

Deployment Node console
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/depuser/.ssh/id_rsa):
Created directory '/home/depuser/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/depuser/.ssh/id_rsa
Your public key has been saved in /home/depuser/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:IfcjdT/spXVHVd3n6wm1OmaWUXGuHnPmvqoXZ6WZYl0 depuser@depserver
The key's randomart image is:
+---[RSA 3072]----+
|                *|
|               .*|
|      . o . .  o=|
|       o + . o +E|
|        S o  .**O|
|         . .o=OX=|
|           . o%*.|
|             O.o.|
|           .*.ooo|
+----[SHA256]-----+

Copy your SSH private key, such as ~/.ssh/id_rsa, to all nodes in the deployment by running the following command (example):

Deployment Node console
$ ssh-copy-id depuser@192.168.222.111
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/depuser/.ssh/id_rsa.pub"
The authenticity of host '192.168.222.111 (192.168.222.111)' can't be established.
ECDSA key fingerprint is SHA256:6nhUgRlt9gY2Y2ofukUqE0ltH+derQuLsI39dFHe0Ag.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
depuser@192.168.222.111's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'depuser@192.168.222.111'"
and check to make sure that only the key(s) you wanted were added.

Verify that you have passwordless SSH connectivity to all nodes in your deployment by running the following command (example):

Deployment Node console
$ ssh depuser@192.168.222.111

Kubespray Deployment and Configuration

General Setting

To install dependencies for running Kubespray with Ansible on the deployment node, please run following commands:

Deployment Node console
$ cd ~
$ sudo apt -y install python3-pip jq
$ wget https://github.com/kubernetes-sigs/kubespray/archive/v2.15.0.tar.gz
$ tar -zxf v2.15.0.tar.gz
$ cd kubespray-2.15.0
$ sudo pip3 install -r requirements.txt

Note: The default folder for subsequent commands is ~/kubespray-2.15.0.

Deployment Customization

Create a new cluster configuration and host configuration file. Replace the IP addresses below with your nodes' IP addresses:

Deployment Node console
$ cp -rfp inventory/sample inventory/mycluster
$ declare -a IPS=(192.168.222.111 192.168.222.101 192.168.222.102)
$ CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

As a result, the inventory/mycluster/hosts.yaml file will be created. Review and change the host configuration in the file. Below is an example for this deployment:

inventory/mycluster/hosts.yaml
all:
  hosts:
    node1:
      ansible_host: 192.168.222.111
      ip: 192.168.222.111
      access_ip: 192.168.222.111
    node2:
      ansible_host: 192.168.222.101
      ip: 192.168.222.101
      access_ip: 192.168.222.101
    node3:
      ansible_host: 192.168.222.102
      ip: 192.168.222.102
      access_ip: 192.168.222.102
  children:
    kube-master:
      hosts:
        node1:
    kube-node:
      hosts:
        node2:
        node3:
    etcd:
      hosts:
        node1:
    k8s-cluster:
      children:
        kube-master:
        kube-node:
    calico-rr:
      hosts: {}

Review and change cluster installation parameters in the files:

  • inventory/mycluster/group_vars/all/all.yml
  • inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

In inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml set a default Kubernetes CNI by setting the desired kube_network_plugin value (default: calico) parameter.

inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
...

# Choose network plugin (cilium, calico, contiv, weave or flannel. Use cni for generic cni plugin)
# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
kube_network_plugin: calico

# Setting multi_networking to true will install Multus: https://github.com/intel/multus-cni
kube_network_plugin_multus: false

...

选择容器运行时

在本指南中，选择 containerd 作为 K8s 集群部署的默认容器运行时，因为 docker 即将被弃用。

要使用 containerd 容器运行时，请设置以下变量：

  1. 在 inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml 中：

    inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

    ...

## Container runtime
## docker for docker, crio for cri-o and containerd for containerd.
container_manager: containerd

...

  2. 在 inventory/mycluster/group_vars/all/all.yml 中：

    inventory/mycluster/group_vars/all/all.yml

    ...

## Experimental kubeadm etcd deployment mode. Available only for new deployment
etcd_kubeadm_enabled: true

...

  3. 在 inventory/mycluster/group_vars/etcd.yml 中：

    inventory/mycluster/group_vars/etcd.yml

    ...

## Settings for etcd deployment type
etcd_deployment_type: host

...

使用 KubeSpray Ansible Playbook 部署集群

运行以下命令启动部署过程：

部署节点控制台

$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

部署需要一些时间才能完成，请确保没有遇到错误。

成功的结果应类似于以下内容：

...
PLAY RECAP ***********************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
node1                      : ok=554  changed=81   unreachable=0    failed=0    skipped=1152 rescued=0    ignored=2
node2                      : ok=360  changed=42   unreachable=0    failed=0    skipped=633  rescued=0    ignored=1
node3                      : ok=360  changed=42   unreachable=0    failed=0    skipped=632  rescued=0    ignored=1

Sunday 11 July 2021  22:36:04 +0000 (0:00:00.053)      0:06:51.785 ************
===============================================================================
kubernetes/kubeadm : Join to cluster ------------------------------------------------------------------------------------------------------------------------------------------------- 37.24s
kubernetes/control-plane : kubeadm | Initialize first master ------------------------------------------------------------------------------------------------------------------------- 28.29s
download_file | Download item -------------------------------------------------------------------------------------------------------------------------------------------------------- 16.57s
kubernetes/control-plane : Master | wait for kube-scheduler -------------------------------------------------------------------------------------------------------------------------- 14.23s
download_container | Download image if required -------------------------------------------------------------------------------------------------------------------------------------- 11.06s
download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------- 9.18s
download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------- 8.61s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources --------------------------------------------------------------------------------------------------------------------------- 7.02s
container-engine/crictl : download_file | Download item ------------------------------------------------------------------------------------------------------------------------------- 5.78s
download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------- 5.52s
Configure | Check if etcd cluster is healthy ------------------------------------------------------------------------------------------------------------------------------------------ 5.24s
download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------- 4.89s
download_container | Download image if required --------------------------------------------------------------------------------------------------------------------------------------- 4.81s
kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS templates ---------------------------------------------------------------------------------------------------------------- 4.68s
reload etcd --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.65s
download_file | Download item --------------------------------------------------------------------------------------------------------------------------------------------------------- 4.24s
kubernetes/preinstall : Get current calico cluster version ---------------------------------------------------------------------------------------------------------------------------- 3.70s
network_plugin/calico : Start Calico resources ---------------------------------------------------------------------------------------------------------------------------------------- 3.42s
container-engine/crictl : extract_file | Unpacking archive ---------------------------------------------------------------------------------------------------------------------------- 3.35s
kubernetes-apps/cluster_roles : Apply workaround to allow all nodes with cert O=system:nodes to register ------------------------------------------------------------------------------ 3.32s

K8s 集群定制

现在 K8s 集群已部署，请使用 root 用户账户 连接到 K8s 主节点 以定制部署。

  1. 标记工作节点。

    主节点控制台

    # kubectl label nodes node2 node-role.kubernetes.io/worker=
# kubectl label nodes node3 node-role.kubernetes.io/worker=

K8s 集群部署验证

以下是使用 Calico CNI 插件的 K8s 集群部署信息输出示例。

为确保 Kubernetes 集群安装正确，请运行以下命令：

主节点控制台

# kubectl get nodes -o wide
NAME    STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
node1   Ready    master   44m   v1.19.7   192.168.222.111   <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   containerd://1.4.4
node2   Ready    worker   42m   v1.19.7   192.168.222.101   <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   containerd://1.4.4
node3   Ready    worker   42m   v1.19.7   192.168.222.102   <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   containerd://1.4.4

# kubectl -n kube-system get pods -o wide
NAME                                      READY   STATUS    RESTARTS   AGE   IP                NODE    NOMINATED NODE   READINESS GATES
calico-kube-controllers-8b5ff5d58-ph86x   1/1     Running   0          43m   192.168.222.101   node2   <none>           <none>
calico-node-l48qg                         1/1     Running   0          43m   192.168.222.102   node3   <none>           <none>
calico-node-ldx7w                         1/1     Running   0          43m   192.168.222.111   node1   <none>           <none>
calico-node-x9bh5                         1/1     Running   0          43m   192.168.222.101   node2   <none>           <none>
coredns-85967d65-pslmm                    1/1     Running   0          27m   10.233.96.1       node2   <none>           <none>
coredns-85967d65-qp2rl                    1/1     Running   0          43m   10.233.90.230     node1   <none>           <none>
dns-autoscaler-5b7b5c9b6f-8wb67           1/1     Running   0

43m 10.233.90.229 node1 etcd-node1 1/1 Running 0 45m 192.168.222.111 node1 kube-apiserver-node1 1/1 Running 0 45m 192.168.222.111 node1 kube-controller-manager-node1 1/1 Running 0 45m 192.168.222.111 node1 kube-proxy-6p4rm 1/1 Running 0 44m 192.168.222.101 node2 kube-proxy-8bj6s 1/1 Running 0 44m 192.168.222.111 node1 kube-proxy-dj4l8 1/1 Running 0 44m 192.168.222.102 node3 kube-scheduler-node1 1/1 Running 0 45m 192.168.222.111 node1 nginx-proxy-node2 1/1 Running 0 44m 192.168.222.101 node2 nginx-proxy-node3 1/1 Running 0 44m 192.168.222.102 node3 nodelocaldns-8b6kf 1/1 Running 0 43m 192.168.222.102 node3 nodelocaldns-kzmmh 1/1 Running 0 43m 192.168.222.101 node2 nodelocaldns-zh9fz 1/1 Running 0 43m 192.168.222.111 node1 


### NVIDIA Network Operator 安装（用于 K8S 集群）

NVIDIA Network Operator 利用 Kubernetes CRD 和 Operator SDK 管理网络相关组件，以便在 K8s 集群中为工作负载启用高速网络和 RDMA。高速网络是 K8s 集群的辅助网络，适用于需要高带宽或低延迟的应用。

要使其正常工作，需要配置和部署多个组件。所有 Operator 配置和安装步骤应在 **K8S 主节点**上使用 **root 用户**账户执行。

#### 先决条件

1.  安装 Helm。

    **主节点控制台**

    ```bash
    # snap install helm --classic
    ```

2.  安装额外的 RDMA CNI 插件
    RDMA CNI 插件允许在容器化环境中为 RDMA 工作负载提供网络命名空间隔离。
    使用以下 YAML 文件部署 CNI：

    **主节点控制台**

    ```bash
    # kubectl apply -f https://raw.githubusercontent.com/Mellanox/rdma-cni/master/deployment/rdma-cni-daemonset.yaml
    ```

    为确保插件正确安装，请运行以下命令：

    **主节点控制台**

    ```bash
    # kubectl -n kube-system get pods -o wide | egrep  "rdma"

    kube-rdma-cni-ds-5zl8d                    1/1     Running   0          11m    192.168.222.102   node3   <none>           <none>
    kube-rdma-cni-ds-q74n5                    1/1     Running   0          11m    192.168.222.101   node2   <none>           <none>
    kube-rdma-cni-ds-rnqkr                    1/1     Running   0          11m    192.168.222.111   node1   <none>           <none>
    ```

#### 部署

添加 NVIDIA Network Operator Helm 仓库：

**主节点控制台**

```bash
# helm repo add mellanox https://mellanox.github.io/network-operator
# helm repo update

在用户主目录下创建 values.yaml 文件（示例）：

nfd:
  enabled: true

sriovNetworkOperator:
  enabled: true

# NicClusterPolicy CR values:
deployCR: true
ofedDriver:
  deploy: false

nvPeerDriver:
  deploy: false

rdmaSharedDevicePlugin:
  deploy: false

sriovDevicePlugin:
  deploy: false

secondaryNetwork:
  deploy: true
  cniPlugins:
    deploy: true
    image: containernetworking-plugins
    repository: mellanox
    version: v0.8.7
    imagePullSecrets: []
  multus:
    deploy: true
    image: multus
    repository: nfvpe
    version: v3.6
    imagePullSecrets: []
    config: ''
  ipamPlugin:
    deploy: true
    image: whereabouts
    repository: mellanox
    version: v0.3
    imagePullSecrets: []

部署 Operator：

主节点控制台

# helm install -f ./values.yaml -n network-operator --create-namespace --wait mellanox/network-operator --generate-name

NAME: network-operator
LAST DEPLOYED: Sun Jul 11 23:06:54 2021
NAMESPACE: network-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Get Network Operator deployed resources by running the following commands:

$ kubectl -n network-operator get pods
$ kubectl -n mlnx-network-operator-resources get pods

为确保 Operator 正确部署，请运行以下命令：

主节点控制台

# kubectl -n network-operator get pods -o wide
NAME                                                            READY   STATUS    RESTARTS   AGE   IP                NODE    NOMINATED NODE   READINESS GATES
network-operator-1627211751-5bd467cbd9-2hwqx                      1/1     Running   0          29h   10.233.90.5      node1   <none>           <none>
network-operator-1627211751-node-feature-discovery-master-dgs69   1/1     Running   0          29h   10.233.90.6      node1   <none>           <none>
network-operator-1627211751-node-feature-discovery-worker-7n6gs   1/1     Running   0          29h   10.233.90.3      node1   <none>           <none>
network-operator-1627211751-node-feature-discovery-worker-sjdxw   1/1     Running   1          29h   10.233.96.7      node2   <none>           <none>
network-operator-1627211751-node-feature-discovery-worker-vzpvg   1/1     Running   1          29h   10.233.92.5      node3   <none>           <none>
network-operator-1627211751-sriov-network-operator-5f869696sdzp   1/1     Running   0          29h   10.233.90.4      node1   <none>           <none>

高速网络配置

安装 Operator 后，请检查 SriovNetworkNodeState CR 以查看节点中所有支持 SR-IOV 的设备。 在我们的部署中，选择了名为 ens2f0 的网络接口。要查看接口状态，请使用以下命令：

主节点控制台

# kubectl -n network-operator get sriovnetworknodestates.sriovnetwork.openshift.io node2 -o yaml

...

status:
  interfaces:
  - deviceID: 101d
    driver: mlx5_core
    linkSpeed: 100000 Mb/s
    linkType: ETH
    mac: 0c:42:a1:2b:74:ae
    mtu: 1500
    name: ens2f0
    pciAddress: "0000:07:00.0"
    totalvfs: 8
    vendor: 15b3
  - deviceID: 101d
    driver: mlx5_core
    linkType: ETH
    mac: 0c:42:a1:2b:74:af
    mtu: 1500
    name: ens2f1
    pciAddress: "0000:07:00.1"
    totalvfs: 8
    vendor: 15b3

...

创建 SriovNetworkNodePolicy CR 文件 policy.yaml，在 nicSelector 中指定所选接口（本例中为 ens2f0 接口）：

policy.yaml

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: mlnxnics
  namespace: network-operator
spec:
  nodeSelector:
    feature.node.kubernetes.io/network-sriov.capable: "true"

resourceName: mlnx2f0
  priority: 98
  mtu: 9000
  numVfs: 8
  nicSelector:
    vendor: "15b3"
    pfNames: [ "ens2f0" ]
  deviceType: netdevice
  isRdma: true

部署 `policy.yaml`：

#### Master Node console

```bash
# kubectl apply -f policy.yaml

创建一个SriovNetwork CR文件 network.yaml，该文件引用SriovNetworkNodePolicy中定义的'resourceName'（在此示例中，引用 mlnx2f0 资源，并将192.168.101.0/24设置为高速网络的CIDR范围）：

network.yaml

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: "netmlnx2f0"
  namespace: network-operator
spec:
  ipam: |
    {
      "datastore": "kubernetes",
      "kubernetes": {
         "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
      },
      "log_file": "/tmp/whereabouts.log",
      "log_level": "debug",
      "type": "whereabouts",
      "range": "192.168.101.0/24"
    }
  vlan: 0
  networkNamespace: "default"
  spoofChk: "off"
  resourceName: "mlnx2f0"
  linkState: "enable"
  metaPlugins: |
    {
      "type": "rdma"
    }

部署 network.yaml：

Master Node console

# kubectl apply -f network.yaml

验证部署

检查部署是否成功完成：

Master Node console

# kubectl -n nvidia-network-operator-resources get pods -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
cni-plugins-ds-f548q         1/1     Running   1          30m   192.168.222.101   node2   <none>           <none>
cni-plugins-ds-qw7hx         1/1     Running   1          30m   192.168.222.102   node3   <none>           <none>
kube-multus-ds-cjbf9         1/1     Running   1          30m   192.168.222.102   node3   <none>           <none>
kube-multus-ds-rgc95         1/1     Running   1          30m   192.168.222.101   node2   <none>           <none>
whereabouts-gwr7p            1/1     Running   1          30m   192.168.222.101   node2   <none>           <none>
whereabouts-n29nq            1/1     Running   1          30m   192.168.222.102   node3   <none>           <none>

检查已部署的网络：

Master Node console

# kubectl get network-attachment-definitions.k8s.cni.cncf.io
NAME         AGE
netmlnx2f0   4m56s

检查工作节点资源：

Master Node console

# kubectl describe nodes node2

...

Addresses:
  InternalIP:  192.168.222.101
  Hostname:    node2
Capacity:
  cpu:                 24
  ephemeral-storage:   229698892Ki
  hugepages-1Gi:       0
  hugepages-2Mi:       0
  memory:              264030604Ki
  nvidia.com/mlnx2f0:  8
  pods:                110
Allocatable:
  cpu:                 23900m
  ephemeral-storage:   211690498517
  hugepages-1Gi:       0
  hugepages-2Mi:       0
  memory:              242694540Ki
  nvidia.com/mlnx2f0:  8
  pods:                110

...

管理HugePages

Kubernetes支持Pod中的应用程序分配和消耗预分配的HugePages。节点将自动发现并报告所有HugePages资源作为可调度资源。有关K8s HugePages管理的更多信息，请参考此处。

为了分配HugePages，需要在 /etc/default/grub 中修改 GRUB_CMDLINE_LINUX_DEFAULT 参数。以下设置在启动时分配1GB * 16页 = 16GB和2MB * 2048页 = 4GB的HugePages：

/etc/default/grub

...

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048"

...

运行 update-grub 将配置应用到grub并重启服务器：

Worker Node console

# update-grub
# reboot

服务器重启后，从主节点检查hugepages分配情况：

Master Node console

# kubectl describe nodes node2
...
Capacity:
  cpu:                 24
  ephemeral-storage:   229698892Ki
  hugepages-1Gi:       16Gi
  hugepages-2Mi:       4Gi
  memory:              264030604Ki
  nvidia.com/mlnx2f0:  8
  pods:                110
Allocatable:
  cpu:                 23900m
  ephemeral-storage:   211690498517
  hugepages-1Gi:       16Gi
  hugepages-2Mi:       4Gi
  memory:              242694540Ki
  nvidia.com/mlnx2f0:  8
  pods:                110
...

启用CPU和拓扑管理

CPU管理器管理CPU组并将工作负载约束到特定CPU。

CPU管理器对于具有以下某些属性的工作负载非常有用：

  • 需要尽可能多的CPU时间
  • 对处理器缓存未命中敏感
  • 低延迟网络应用程序
  • 与其他进程协调并受益于共享单个处理器缓存

拓扑管理器使用从收集的提示中获取的拓扑信息，根据配置的拓扑管理器策略和Pod请求的资源，决定Pod是否可以在节点上被接受或拒绝。为了获得最佳性能，需要与CPU隔离以及内存和设备局部性相关的优化。

拓扑管理器对于使用硬件加速器支持延迟关键型执行和高吞吐量并行计算的工作负载非常有用。

注意： 要使用拓扑管理器，必须使用具有 static 策略的CPU管理器。

有关更多信息，请参考控制节点上的CPU管理策略和控制节点上的拓扑管理策略。

为了启用CPU管理器和拓扑管理器，请将以下行添加到kubelet配置文件 /etc/kubernetes/kubelet-config.yaml：

/etc/kubernetes/kubelet-config.yaml

...
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 10s
topologyManagerPolicy: single-numa-node
featureGates:
  CPUManager: true
  TopologyManager: true

由于对

cpuManagerPolicy，删除/var/lib/kubelet/cpu_manager_state并重启每个受影响的K8s工作节点上的kubelet服务。

Worker Node console
# rm -f /var/lib/kubelet/cpu_manager_state
# service kubelet restart

Application

DPDK流量仿真如下面的测试床流程图所示。流量将从Trex服务器通过ens2f0接口推送到TestPMD POD的SRIOV网络接口net1。TestPMD POD将交换MAC地址并通过同一接口net1将入站流量重新路由回Trex服务器的同一接口。

image2021-8-10_9-47-56.png

Verification

  1. 创建一个示例部署test-deployment.yaml（容器镜像应包含InfiniBand用户空间驱动程序和性能工具）：

    test-deployment.yaml

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: mlnx-inbox-pod
  labels:
    app: sriov
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sriov
  template:
    metadata:
      labels:
        app: sriov
      annotations:
        k8s.v1.cni.cncf.io/networks: netmlnx2f0
    spec:
      containers:
      - image: < Container image >
        name: mlnx-inbox-ctr
        securityContext:
          capabilities:
            add: [ "IPC_LOCK" ]
        resources:
          requests:
            cpu: 4
            nvidia.com/mlnx2f0: 1
          limits:
            cpu: 4
            nvidia.com/mlnx2f0: 1
        command:
        - sh
        - -c
        - sleep inf

  2. 部署示例部署。

    Master Node console

    # kubectl apply -f test-deployment.yaml

  3. 验证部署正在运行。

    Master Node console

    # kubectl get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE    NOMINATED NODE   READINESS GATES
mlnx-inbox-pod-599dc445c8-72x6g   1/1     Running   0          12s   10.233.96.5   node2   <none>           <none>
mlnx-inbox-pod-599dc445c8-v5lnx   1/1     Running   0          12s   10.233.92.4   node3   <none>           <none>

  4. 检查POD中的可用网络接口。

    Master Node console

    # kubectl exec -it mlnx-inbox-pod-599dc445c8-72x6g -- bash

root@mlnx-inbox-pod-599dc445c8-72x6g:/tmp# rdma link
link rocep7s0f0v2/1 state ACTIVE physical_state LINK_UP netdev net1

root@mlnx-inbox-pod-599dc445c8-72x6g:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if208: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 12:51:ab:b3:ef:26 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.233.96.5/32 brd 10.233.96.5 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::1051:abff:feb3:ef26/64 scope link
       valid_lft forever preferred_lft forever
201: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 02:40:7d:5e:5f:af brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.2/24 brd 192.168.101.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::40:7dff:fe5e:5faf/64 scope link
       valid_lft forever preferred_lft forever

  5. 使用ib_write_bw带宽和延迟测试（RDMA写事务）运行合成RDMA基准测试。

    Server ib_write_bw -F -d $IB_DEV_NAME --report_gbits
    Client ib_write_bw -F $SERVER_IP -d $IB_DEV_NAME --report_gbits

    请打开两个控制台连接到K8s主节点——一个用于服务器应用端，另一个用于客户端应用端。

    在第一个控制台（服务器端）连接到K8s主节点，运行以下命令：

    Master Node console

    # kubectl exec -it mlnx-inbox-pod-599dc445c8-72x6g -- bash
root@mlnx-inbox-pod-599dc445c8-72x6g:/tmp# ip a s net1 | grep inet
    inet 192.168.101.2/24 brd 192.168.101.255 scope global net1
    inet6 fe80::40:7dff:fe5e:5faf/64 scope link
root@mlnx-inbox-pod-599dc445c8-72x6g:/tmp# rdma link
link rocep7s0f0v2/1 state ACTIVE physical_state LINK_UP netdev net1
root@mlnx-inbox-pod-599dc445c8-72x6g:/tmp# ib_write_bw -F -d rocep7s0f0v2 --report_gbits

************************************
* Waiting for client to connect... *
************************************

    在第二个控制台（客户端）连接到K8s主节点，运行以下命令：

    Master Node console

    # kubectl exec -it mlnx-inbox-pod-599dc445c8-v5lnx -- bash
root@mlnx-inbox-pod-599dc445c8-v5lnx:/tmp# rdma link
link rocep7s0f0v3/1 state ACTIVE physical_state LINK_UP netdev net1
root@mlnx-inbox-pod-599dc445c8-v5lnx:/tmp# ib_write_bw  -F -d rocep7s0f0v3 192.168.101.2 --report_gbits
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF		Device         : rocep7s0f0v3
 Number of qps   : 1		Transport type : IB
 Connection type : RC		Using SRQ      : OFF
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 2
 Max inline data : 0[B]
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01f2 PSN 0x75e7cf RKey 0x050e26 VAddr 0x007f51e51b9000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:101:01
 remote address: LID 0000 QPN 0x00f2 PSN 0x13427f RKey 0x010e26 VAddr 0x007f1ecaac8000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:101:02
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 65536      5000             94.26              92.87  		         0.169509
---------------------------------------------------------------------------------------

TRex Server Deployment

在本指南中，我们使用了TRex包v2.87。 有关详细的TRex安装和配置指南，请参阅TRex文档。

TRex安装和配置步骤使用root用户账户完成。

Prerequisites

对于TRex服务器，使用了安装了RDMA子系统的标准服务器。

使用netplan激活TRex应用程序使用的网络接口。 在我们的部署中，

使用接口 ens2f0 和 ens2f1：

/etc/netplan/00-installer-config.yaml

# This is the network config written by 'subiquity'
network:
  ethernets:
    ens4f0:
      dhcp4: true
      dhcp-identifier: mac
    ens2f0: {}
    ens2f1: {}
  version: 2

然后重新应用 netplan 并检查 ens2f0/ens2f1 网络接口的链路状态。

TRex server console

# netplan apply
# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev ens2f0
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev ens2f1
link mlx5_2/1 state ACTIVE physical_state LINK_UP netdev ens4f0
link mlx5_3/1 state DOWN physical_state DISABLED netdev ens4f1

更新接口 ens2f0 和 ens2f1 的MTU大小。

TRex server console

# ip link set ens2f0 mtu 9000
# ip link set ens2f1 mtu 9000

Installation

创建TRex工作目录并获取TRex包。

TRex server console

# cd /tmp
# wget https://trex-tgn.cisco.com/trex/release/v2.87.tar.gz --no-check-certificate
# mkdir /scratch
# cd /scratch
# tar -zxf /tmp/v2.87.tar.gz
# chmod 777 -R /scratch

First-Time Scripts

下一步将从文件夹 /scratch/v2.87 继续。

以交互模式运行TRex配置脚本。按照屏幕上的说明创建基本配置文件 /etc/trex_cfg.yaml：

TRex server console

# ./dpdk_setup_ports.py -i

将创建 /etc/trex_cfg.yaml 配置文件。稍后我们将修改它以适合我们的设置。

Appendix

Performance Testing

下面展示了TRex流量生成器与K8s工作节点上运行的TESTPMD应用之间的DPDK流量仿真性能测试，符合上述测试平台图。

Prerequisites

在开始测试之前，使用TESTPMD pod中高性能接口的MAC地址更新TRex配置文件 /etc/trex_cfg.yaml。以下是完成此更新的步骤。

  1. 根据以下YAML配置文件 testpmd-inbox.yaml 在K8s集群上运行带有TESTPMD应用的pod（容器镜像应包含InfiniBand用户空间驱动程序和TESTPMD应用）：

    testpmd-inbox.yaml

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  labels:
    app: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
      annotations:
        k8s.v1.cni.cncf.io/networks: netmlnx2f0
    spec:
      containers:
      - image: <container image>
        name: test-pod
        securityContext:
          capabilities:
            add: [ "IPC_LOCK" ]
        volumeMounts:
        - mountPath: /hugepages
          name: hugepage
        resources:
          requests:
            hugepages-1Gi: 2Gi
            memory: 16Gi
            cpu: 8
            nvidia.com/mlnx2f0: 1
          limits:
            hugepages-1Gi: 2Gi
            memory: 16Gi
            cpu: 8
            nvidia.com/mlnx2f0: 1
        command:
        - sh
        - -c
        - sleep inf
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages

    使用以下命令部署deployment：

    Master Node console

    # kubectl apply -f testpmd-inbox.yaml

  2. 通过运行以下命令获取已部署pod的网络信息：

    Master Node console

    # kubectl get pod -o wide
NAME                               READY   STATUS        RESTARTS   AGE    IP            NODE    NOMINATED NODE   READINESS GATES
test-deployment-676476c78d-glbfs   1/1     Running       0          30s    10.233.92.5   node3   <none>           <none>

# kubectl exec -it test-deployment-676476c78d-glbfs -- ip a s net1
193: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 32:f9:3f:e3:dc:89 brd ff:ff:ff:ff:ff:ff
    inet 192.168.101.3/24 brd 192.168.101.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::30f9:3fff:fee3:dc89/64 scope link
       valid_lft forever preferred_lft forever

  3. 使用NET1网络接口的MAC地址 32:f9:3f:e3:dc:89 更新TRex配置文件 /etc/trex_cfg.yaml：

    /etc/trex_cfg.yaml

    ### Config file generated by dpdk_setup_ports.py ###

- version: 2
  interfaces: ['07:00.0', '0d:00.0']
  port_info:
      - dest_mac: 32:f9:3f:e3:dc:89 # MAC OF NET1 INTERFACE
        src_mac:  0c:42:a1:24:05:1a
      - dest_mac: 32:f9:3f:e3:dc:89 # MAC OF NET1 INTERFACE
        src_mac:  0c:42:a1:24:05:1b

  platform:
      master_thread_id: 0
      latency_thread_id: 12
      dual_if:
        - socket: 0
          threads: [1,2,3,4,5,6,7,8,9,10,11]

DPDK Emulation Test

  1. 在容器中运行TESTPMD应用：

    Master Node console

    # kubectl exec -it test-deployment-676476c78d-glbfs -- bash
root@test-deployment-676476c78d-glbfs:/tmp# dpdk-testpmd -c 0x1fe  -m 1024 -w $PCIDEVICE_NVIDIA_COM_MLNX2F0 -- --burst=64 --txd=1024 --rxd=1024 --mbcache=512 --rxq=8 --txq=8 --nb-cores=4  --rss-udp --forward-mode=macswap  -a -i
...
testpmd>

    注意： 特定TESTPMD参数：

    • $PCIDEVICE_NVIDIA_COM_MLNX2F0 - 系统变量，NET1的PCI地址

    更多关于其他TESTPMD参数的信息：

  2. 在TRex服务器上运行TRex流量生成器：

    TRex server console

    # cd /scratch/v2.87/
# ./t-rex-64 -v 7 -i -c 11 --no-ofed-check

    打开第二个TRex服务器屏幕，在文件夹 /scratch/v2.87 中创建流量生成文件 mlnx-trex.py：

    mlnx-trex.py

    from trex_stl_lib.api import *

class STLS1(object):

    def create_stream (self):

        pkt = Ether()/IP(src="https://networking-docs.nvidia.com/sol/16.0.0.1",dst="48.0.0.1")/UDP(dport=12)/(22*'x')

        vm = STLScVmRaw( [
                        STLVmFlowVar(name="v_port",

在启用SR-IOV的Kubernetes集群上使用NVIDIA Network Operator运行DPDK应用的参考部署指南

创建于2021年7月7日。

范围

以下参考部署指南（RDG）说明了如何构建一个高性能的Kubernetes（K8s）集群，使用containerd。

...

min_value=4337,
max_value=5337,
size=2, op="inc"),
STLVmWrFlowVar(fv_name="v_port",
    pkt_offset= "UDP.sport" ),
STLVmFixChecksumHw(l3_offset="IP",l4_offset="UDP",l4_type=CTRexVmInsFixHwCs.L4_TYPE_UDP),
]
)
return STLStream(packet = STLPktBuilder(pkt = pkt ,vm = vm ) ,
    mode = STLTXCont(pps = 8000000) )

def get_streams (self, direction = 0, **kwargs):
    # create 1 stream
    return [ self.create_stream() ]

# dynamic load - used for trex console or simulator
def register():
    return STLS1()

运行TRex控制台并向TESTPMD Pod生成流量后：

TRex服务器控制台

# cd /scratch/v2.87/
# ./trex-console
Using 'python3' as Python interpeter

Connecting to RPC server on localhost:4501                   [SUCCESS]

Connecting to publisher server on localhost:4500             [SUCCESS]

Acquiring ports [0, 1]:                                      [SUCCESS]

Server Info:
Server version:   v2.87 @ STL
Server mode:      Stateless
Server CPU:       11 x Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
Ports count:      2 x 100Gbps @ MT2892 Family [ConnectX-6 Dx]

-=TRex Console v3.0=-

Type 'help' or '?' for supported actions

trex> tui<enter>
...
tui> start -f mlnx-trex.py -m 45mpps -p 0
...
Global Statistitcs

connection   : localhost, Port 4501                       total_tx_L2  : 23.9 Gbps
version      : STL @ v2.87                                total_tx_L1  : 30.93 Gbps
cpu_util.    : 82.88% @ 11 cores (11 per dual port)       total_rx     : 25.31 Gbps
rx_cpu_util. : 0.0% / 0 pps                               total_pps    : 44.84 Mpps
async_util.  : 0.05% / 11.22 Kbps                         drop_rate    : 0 bps
total_cps.   : 0 cps                                      queue_full   : 0 pkts
...

总结

从上述测试可以看出，在Pod中使用SR-IOV网络端口时，期望流量为45mpps。

警告： 为了获得更好的结果，需要对Trex和TESTPMD进行额外的应用调优。

完成！

作者
VR.jpg Vitaliy RazinkovVitaliy Razinkov是NVIDIA网络团队的解决方案架构师，专注于复杂的Kubernetes、OpenShift和Microsoft解决方案。凭借超过25年的高级技术职位经验，他在设计和实施先进基础设施方面拥有深厚的专业知识。Vitaliy撰写了多份关于Microsoft技术、Kubernetes/OpenShift中RoCE/RDMA加速机器学习以及容器化解决方案的参考设计指南——所有这些都可以在NVIDIA网络文档网站上找到。
AZ.jpg Amir Zeidner多年来，Amir主要作为电信领域的解决方案架构师，领导先进解决方案以满足5G、NFV和SDN网络基础设施需求。Amir在数据平面加速技术（如加速交换和网络处理（ASAP²）和DPDK）方面的专业知识，加上对开源云基础设施的深入了解，使他能够在电信领域推广和交付独特的端到端NVIDIA网络解决方案。