基于NVIDIA DGX A100服务器和200Gbps以太网网络的加速K8s集群参考部署指南
创建于2022年5月15日。本参考部署指南（RDG）指导您如何在200Gb/s NVIDIA网络上设置高可用性的GPU和网络加速Kubernetes（K8s）集群。
文档目录
创建于2022年5月15日。
范围
以下**参考部署指南（RDG）**指导您如何在200Gb/s NVIDIA网络上设置高可用性的GPU和网络加速Kubernetes（K8s）集群。高可用性集群由多个控制平面节点（K8s主节点）、多个工作节点（DGX A100服务器）和负载均衡器应用程序（HAProxy）组成。本指南提供了如何在NVIDIA DGX A100服务器平台上使用Kubeflow训练运算符运行ML/DL应用程序的示例。
缩写和首字母缩略词
| 术语 | 定义 | 术语 | 定义 |
|---|---|---|---|
| CNI | 容器网络接口 | NFD | 节点特性发现 |
| CR | 自定义资源 | NCCL | NVIDIA集合通信库 |
| CRD | 自定义资源定义 | OCI | 开放容器倡议 |
| CRI | 容器运行时接口 | PF | 物理功能 |
| DHCP | 动态主机配置协议 | QSG | 快速入门指南 |
| DNS | 域名系统 | RDG | 参考部署指南 |
| DL | 深度学习 | RDMA | 远程直接内存访问 |
| DP | 设备插件 | RoCE | 融合以太网上的RDMA |
| IPAM | IP地址管理 | SR-IOV | 单根输入输出虚拟化 |
| K8s | Kubernetes | TF | TensorFlow |
| LLDP | 链路层发现协议 | VF | 虚拟功能 |
| ML | 机器学习 |
引言
配置高可用性Kubernetes集群以运行ML/DL应用程序工作负载可能变得极其复杂。 本指南提供了K8s集群部署的完整解决方案周期，包括技术概述、设计、组件选择、部署步骤和ML/DL工作负载示例。该解决方案将在标准服务器（控制平面）和DGX A100服务器（K8s工作节点）上交付。NVIDIA 200Gb/s端到端以太网基础设施用于处理工作负载，而100Gb/s网络用作主网络。 在本指南中，我们使用NVIDIA GPU运算符和NVIDIA网络运算符，它们负责在K8s集群中部署和配置GPU和网络组件。这些组件允许您使用CUDA、RDMA和GPUDirect技术加速ML/DL任务。
假设本指南为全新部署。 本指南展示了具有2到8个工作节点的K8s集群设计，并提供了部署4个K8s工作节点集群的详细说明。
参考文献
解决方案架构
关键组件和技术
-
NVIDIA DGX A100 NVIDIA DGX™ A100是适用于所有AI工作负载的通用系统，在世界上第一个5 petaFLOPS AI系统中提供前所未有的计算密度、性能和灵活性。NVIDIA DGX A100采用世界上最先进的加速器NVIDIA A100 Tensor Core GPU，使企业能够将训练、推理和分析整合到一个统一、易于部署的AI基础设施中，并可直接访问NVIDIA AI专家。
-
NVIDIA ConnectX智能网卡 10/25/40/50/100/200和400G以太网网卡 业界领先的NVIDIA® ConnectX®系列智能网卡提供先进的硬件卸载和加速。 NVIDIA以太网网卡为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供最高的ROI和最低的总拥有成本。
-
NVIDIA LinkX线缆 NVIDIA® LinkX®线缆和收发器产品系列提供业界最完整的10、25、40、50、100、200和400GbE以太网以及100、200和400Gb/s InfiniBand产品，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。
-
NVIDIA Spectrum以太网交换机 灵活的外形，支持16到128个物理端口，速度从1GbE到400GbE。 基于针对性能和可扩展性优化的突破性硅技术，NVIDIA Spectrum交换机非常适合构建高性能、经济高效且高效的云数据中心网络、以太网存储结构和深度学习互连。 NVIDIA将基于业界领先的专用集成电路（ASIC）技术的NVIDIA Spectrum™交换机的优势与多种现代网络操作系统选择相结合，包括NVIDIA Cumulus® Linux、SONiC和NVIDIA Onyx®。
-
NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux是业界最具创新性的开放网络操作系统，允许您像其他系统一样自动化、定制和扩展数据中心网络。
Kubernetes is an open-source container orchestration platform for deployment automation, scaling, and management of containerized applications.
Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks and provides:
- A highly available cluster
- Composable attributes
- Support for most popular Linux distributions
The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPU. These components include the NVIDIA drivers (to enable CUDA), Kubernetes device plugin for GPUs, the NVIDIA Container Runtime, automatic node labelling, DCGM-based monitoring, and more.
The NVIDIA Network Operator simplifies the provisioning and management of NVIDIA networking resources in a Kubernetes cluster. The operator automatically installs the required host networking software - bringing together all the needed components to provide high-speed network connectivity. These components include the NVIDIA networking driver, Kubernetes device plugin, CNI plugins, IP address management (IPAM) plugin and others. The NVIDIA Network Operator works in conjunction with the NVIDIA GPU Operator to deliver high-throughput, low-latency networking for scale-out, GPU computing clusters.
CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs). With CUDA, developers can dramatically speed up computing applications by harnessing the power of GPUs. In GPU-accelerated applications, the sequential part of the workload runs on the CPU – which is optimized for single-threaded performance – while the compute-intensive portion of the application runs on thousands of GPU cores in parallel.
RDMA is a technology that allows computers in a network to exchange data without involving the processor, cache or operating system of either computer. Like locally based DMA, RDMA improves throughput and performance and frees up compute resources.
GPUDirect (GDR) RDMA provides a direct P2P (Peer-to-Peer) data path between the GPU memory directly to and from NVIDIA host networking devices. This reduces GPU-to-GPU communication latency and completely offloads the CPU, removing it from all GPU-to-GPU communications across the network.
Unknown Attachment
Logical Design
The logical design includes the following parts:
- Deployment node running Kubespray that deploys Kubernetes clusters and HAProxy load-balancer
- K8s Master nodes running all Kubernetes management components
- NVIDIA DGX A100 K8s Worker nodes
- High-speed Ethernet fabric (Secondary K8s network with RoCE support)
- Deployment and K8s Management networks
Network / Fabric Design
The high-performance network is a secondary network for Kubernetes cluster and requires the L2 network topology.
This RDG describes two options with multiple K8s Worker Nodes:
- Design for 2-4 Worker Nodes In this solution, all K8s Worker Nodes are connected to a single switch, which provides a K8s secondary network.
- Design for 5-8 Worker Nodes In this solution, all K8s Worker Nodes are connected to two independent switches, which provide a K8s secondary network.
The Deployment and Kubernetes Management networks are parts of the IT infrastructure and are beyond the scope of this document.
Design for K8s Cluster of 2-4 Worker Nodes
All Nodes are connected to the MGMT switch by a single 100GbE cable, and all Data port from the K8s worker nodes are connected to Data switch by 200GbE cables. All server remote management ports and switch management ports are connected to 1GbE switch.
Design for K8s Cluster of 5-8 Worker Nodes
All Nodes are connected to the MGMT switch by a single 100GbE cable, and all Data port from the K8s worker nodes are connected to both Data switches by 200GbE cables: the first four data ports are connected to Data Switch1, and the remaining four data ports are connected to Data Switch2. See the Worker Node4 connections as an example. All server remote management ports and switch management ports are connected to 1GbE switch.
Software Stack Components
Bill of Materials
The following hardware setup is utilized in this guide to build K8s cluster with 4 K8s Worker nodes.
The following hardware setup is utilized in this guide to build a K8s cluster with 8 K8s Worker nodes.
Note: Server remote management and switch management BOM for 1GbE network is beyond the scope of this document.
Deployment and Configuration
Wiring
On each K8s Worker Node, all the networking ports of each NVIDIA 网卡 is wired to an NVIDIA switch in high-performance fabric using NVIDIA LinkX DAC cables.
The below figure illustrates the required wiring for building a K8s cluster with 4 K8s Worker nodes.
The below figure illustrates the required wiring for building a K8s cluster with 8 K8s Worker
网络/架构
通用前提条件
部署/管理网络拓扑和DNS/DHCP网络服务属于IT基础设施的一部分。本指南不涵盖组件的安装步骤和配置。
适用于最多4个DGX A100工作节点的网络和架构配置
前提条件
- 高性能以太网架构
- 单交换机 - NVIDIA SN3700
- 交换机操作系统 - Cumulus Linux v4.3及以上版本
网络配置
以下是服务器名称及其相关网络配置。
| 服务器/交换机类型 | 服务器/交换机名称 | 高速网络200GbE | 管理网络100GbE |
|---|---|---|---|
| 部署节点 | depserver | N/A | eth0: DHCP 192.168.222.110 |
| 主节点1 | Node1 | N/A | eth0: DHCP 192.168.222.111 |
| 主节点2 | Node2 | N/A | eth0: DHCP 192.168.222.112 |
| 主节点3 | Node3 | N/A | eth0: DHCP 192.168.222.113 |
| 工作节点1 | clx-host-081 | enp12s0: 无IP enp18s0: 无IP enp75s0: 无IP enp84s0: 无IP enp141s0: 无IP enp148s0: 无IP enp186s0: 无IP enp204s0: 无IP | enp225s0f0: DHCP 192.168.222.101 |
| 工作节点2 | clx-host-082 | enp12s0: 无IP enp18s0: 无IP enp75s0: 无IP enp84s0: 无IP enp141s0: 无IP enp148s0: 无IP enp186s0: 无IP enp204s0: 无IP | enp225s0f0: DHCP 192.168.222.102 |
| 工作节点3 | clx-host-083 | enp12s0: 无IP enp18s0: 无IP enp75s0: 无IP enp84s0: 无IP enp141s0: 无IP enp148s0: 无IP enp186s0: 无IP enp204s0: 无IP | enp225s0f0: DHCP 192.168.222.103 |
| 工作节点4 | clx-host-084 | enp12s0: 无IP enp18s0: 无IP enp75s0: 无IP enp84s0: 无IP enp141s0: 无IP enp148s0: 无IP enp186s0: 无IP enp204s0: 无IP | enp225s0f0: DHCP 192.168.222.104 |
| 高速交换机 | hs-sw01 | N/A | mgmt0: DHCP 192.168.222.201 |
enpXXXs0 高速网络接口无需额外配置。
架构配置
本解决方案基于Cumulus Linux v4.3交换机操作系统。
本指南假设为全新部署。
最佳实践：确保使用最新发布的Cumulus Linux NOS版本。请参阅如何升级Cumulus Linux指南。
确保您的Cumulus Linux交换机已完成初始配置阶段（有关更多信息，请参阅版本4.3的快速入门指南）。
架构配置步骤：
- 管理启用所有物理端口
- 创建网桥并将前面板端口配置为网桥成员
- 创建VLAN
- 将VLAN添加到网桥
- 提交配置
交换机配置步骤：
Linux hs-sw01 4.19.0-cl-1-amd64 #1 SMP Cumulus 4.19.149-1+cl4.3u1 (2021-01-28) x86_64
Welcome to NVIDIA Cumulus (R) Linux (R)
For support and online technical documentation, visit
http://www.cumulusnetworks.com/support
The registered trademark Linux (R) is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide
basis.
cumulus@hs-sw01:mgmt:~$ net show version
NCLU_VERSION=1.0-cl4.3.0u4
DISTRIB_ID="Cumulus Linux"
DISTRIB_RELEASE=4.3.0
DISTRIB_DESCRIPTION="Cumulus Linux 4.3.0"
cumulus@hs-sw01:mgmt:~$ net add interface swp1-32
cumulus@hs-sw01:mgmt:~$ net add bridge bridge ports swp1-32
cumulus@hs-sw01:mgmt:~$ net add vlan 11 vlan-id 11
cumulus@hs-sw01:mgmt:~$ net add vlan 12 vlan-id 12
cumulus@hs-sw01:mgmt:~$ net add vlan 13 vlan-id 13
cumulus@hs-sw01:mgmt:~$ net add vlan 14 vlan-id 14
cumulus@hs-sw01:mgmt:~$ net add vlan 15 vlan-id 15
cumulus@hs-sw01:mgmt:~$ net add vlan 16 vlan-id 16
cumulus@hs-sw01:mgmt:~$ net add vlan 17 vlan-id 17
cumulus@hs-sw01:mgmt:~$ net add vlan 18 vlan-id 18
cumulus@hs-sw01:mgmt:~$ net add bridge bridge vids 11-18
cumulus@hs-sw01:mgmt:~$ net commit
要查看链路状态，请使用 net show interface all 命令。以下示例显示了端口在管理关闭、关闭和开启模式下的输出。
cumulus@hs-sw01:mgmt:~$ net show interface
State Name Spd MTU Mode LLDP Summary
----- ------- ---- ----- --------- ---------------------------------- ------------------------
UP lo N/A 65536 Loopback IP: 127.0.0.1/8
lo IP: ::1/128
UP eth0 1G 1500 Mgmt Master: mgmt(UP)
eth0 IP: 192.168.222.201/24(DHCP)
UP swp1 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP)
UP swp2 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP)
UP swp3 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP)
UP swp4 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP)
UP ...
swp5 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP) UP swp6 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP) UP swp7 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP) UP swp8 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP) UP swp9 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp10 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp11 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp12 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp13 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp14 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp15 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp16 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp17 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp18 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp19 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp20 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp21 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp22 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp23 200G 9216 Trunk/L2 clx-host-083 Master: bridge(UP) UP swp24 200G 9216 Trunk/L2 clx-host-084 Master: bridge(UP) UP swp25 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP) UP swp26 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP) UP swp27 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP) UP swp28 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP) UP swp29 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP) UP swp30 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP) UP swp31 200G 9216 Trunk/L2 clx-host-081 Master: bridge(UP) UP swp32 200G 9216 Trunk/L2 clx-host-082 Master: bridge(UP) UP bridge N/A 9216 Bridge/L2 UP mgmt N/A 65536 VRF IP: 127.0.0.1/8 mgmt IP: ::1/128 UP vlan11 N/A 9216 Default UP vlan12 N/A 9216 Default UP vlan13 N/A 9216 Default UP vlan14 N/A 9216 Default UP vlan15 N/A 9216 Default UP vlan16 N/A 9216 Default UP vlan17 N/A 9216 Default UP vlan18 N/A 9216 Default
节点配置
通用前提条件
-
部署服务器和K8s主节点
所有服务器应安装Ubuntu Server 20.04操作系统，并包含OpenSSH服务器软件包。
-
K8s工作节点
- 所有K8s工作节点具有相同的硬件规格（详见BoM）。
- 确认使用支持SR-IOV的服务器平台，并查阅服务器平台供应商文档中的BIOS设置，以在BIOS中启用SR-IOV。
- 对于AMD处理器，应将每个插槽的NUMA节点（NPS）配置为NPS1。
- 所有高速200Gb/s ConnectX-6单端口网卡应配置为以太网模式。
主机操作系统前提条件
确保所有服务器上安装了Ubuntu Server 20.04操作系统，并包含OpenSSH服务器软件包，创建一个非root的depuser账户，具有无密码的sudo权限。
通过运行以下命令更新Ubuntu软件包：
sudo apt-get update
sudo apt-get upgrade -y
sudo reboot
在本解决方案中，我们在**/etc/sudoers**文件末尾添加了以下行：
sudo vim /etc/sudoers
#includedir /etc/sudoers.d
#K8s集群部署用户，具有无密码的sudo权限
depuser ALL=(ALL) NOPASSWD:ALL
NVIDIA DGX A100服务器固件更新
建议将DGX A100服务器固件更新到最新的GA版本。 如果您不熟悉服务器固件更新流程，请联系NVIDIA支持团队或访问DGX系统文档页面。
部署外部负载均衡器
在本部署中，高可用（HA）Kubernetes集群的拓扑采用堆叠控制平面节点，其中ETCD节点与控制平面节点位于同一位置。有关Kubernetes集群部署中可用的HA拓扑选项的更多信息，请参见此处。
高可用集群跨多个K8s控制平面节点（K8s主节点）、多个工作节点和一个负载均衡器构建。 为K8s集群部署添加负载均衡器使系统更加健壮，因为任何K8s主节点发生故障时，应用程序不会离线或数据不会丢失。 下图展示了此设置。
ETCD集群确保所有数据在主节点之间同步，负载均衡器调节流量分布。因此，可以通过一个单一入口点（负载均衡器）访问集群，请求被传递到任意节点。
参考：https://kubernetes.io/docs/setup/independent/ha-topology/#stacked-etcd-topology
使用HAProxy标准软件包。
在部署节点上使用root用户账户进行安装：
apt-get -y install haproxy
使用以下内容更新/etc/haproxy/haproxy.cfg：
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend stats
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats admin if LOCALHOST
listen kubernetes-apiserver-https
bind 192.168.222.110:6443
mode tcp
option log-health-checks
timeout client 3h
timeout server 3h
server node1 192.168.222.111:6443 check check-ssl verify none inter 10000
server node2 192.168.222.112:6443 check check-ssl verify none inter 10000
server node3 192.168.222.113:6443 check check-ssl verify none inter 10000
balance roundrobin
更新配置文件后，重启haproxy服务。
service haproxy restart
K8s集群部署与配置
本解决方案中的Kubernetes集群使用Kubespray安装，
从部署节点使用非root的depuser账户。
SSH私钥与SSH免密登录
以部署用户（本例中为depuser）登录到部署节点，并创建SSH私钥以配置免密码认证，运行以下命令：
ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/depuser/.ssh/id_rsa):
Created directory '/home/depuser/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/depuser/.ssh/id_rsa
Your public key has been saved in /home/depuser/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:IfcjdT/spXVHVd3n6wm1OmaWUXGuHnPmvqoXZ6WZYl0 depuser@depserver
The key's randomart image is:
+---[RSA 3072]----+
| *|
| .*|
| . o . . o=|
| o + . o +E|
| S o .**O|
| . .o=OX=|
| . o%*.|
| O.o.|
| .*.ooo|
+----[SHA256]-----+
将您的SSH私钥（例如~/.ssh/id_rsa）复制到部署中的所有节点，运行以下命令（示例）：
ssh-copy-id depuser@192.168.222.111
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/depuser/.ssh/id_rsa.pub"
The authenticity of host '192.168.222.111 (192.168.222.111)' can't be established.
ECDSA key fingerprint is SHA256:6nhUgRlt9gY2Y2ofukUqE0ltH+derQuLsI39dFHe0Ag.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
depuser@192.168.222.111's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'depuser@192.168.222.111'"
and check to make sure that only the key(s) you wanted were added.
验证您是否能够免密码SSH连接到部署中的所有节点，运行以下命令（示例）：
$ ssh depuser@192.168.222.111
Kubespray部署与配置
通用设置
在部署节点上安装运行Kubespray与Ansible所需的依赖，运行以下命令：
cd ~
sudo apt -y install python3-pip jq
wget https://github.com/kubernetes-sigs/kubespray/archive/refs/tags/v2.18.0.tar.gz
tar -zxf v2.18.0.tar.gz
cd kubespray-2.18.0
sudo pip3 install -r requirements.txt
注意：后续命令的默认目录为
~/kubespray-2.18.0。
部署定制
创建新的集群配置和主机配置文件。将下面的IP地址替换为您节点的IP地址：
cp -rfp inventory/sample inventory/mycluster
declare -a IPS=(192.168.222.111 192.168.222.112 192.168.222.113 192.168.222.101 192.168.222.102 192.168.222.103 192.168.222.104)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
结果将创建inventory/mycluster/hosts.yaml文件。检查并修改文件中的主机配置。以下是本部署的示例：
inventory/mycluster/hosts.yaml
all:
hosts:
node1:
ansible_host: 192.168.222.111
ip: 192.168.222.111
access_ip: 192.168.222.111
node2:
ansible_host: 192.168.222.112
ip: 192.168.222.112
access_ip: 192.168.222.112
node3:
ansible_host: 192.168.222.113
ip: 192.168.222.113
access_ip: 192.168.222.113
clx-host-081:
ansible_host: 192.168.222.101
ip: 192.168.222.101
access_ip: 192.168.222.101
clx-host-082:
ansible_host: 192.168.222.102
ip: 192.168.222.102
access_ip: 192.168.222.102
clx-host-083:
ansible_host: 192.168.222.103
ip: 192.168.222.103
access_ip: 192.168.222.103
clx-host-084:
ansible_host: 192.168.222.104
ip: 192.168.222.104
access_ip: 192.168.222.104
children:
kube_control_plane:
hosts:
node1:
node2:
node3:
kube_node:
hosts:
clx-host-081:
clx-host-082:
clx-host-083:
clx-host-084:
etcd:
hosts:
node1:
node2:
node3:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}
检查并修改以下文件中的集群安装参数：
inventory/mycluster/group_vars/all/all.yml
在inventory/mycluster/group_vars/all/all.yml中，设置以下参数以使用外部负载均衡器并禁用内部负载均衡：
inventory/mycluster/group_vars/all/all.yml
...
## External LB example config
apiserver_loadbalancer_domain_name: "ha-k8s.clx.labs.mlnx"
loadbalancer_apiserver:
address: 192.168.222.110
port: 6443
## Internal loadbalancers for apiservers
loadbalancer_apiserver_localhost: false
...
使用KubeSpray Ansible Playbook部署集群
运行以下命令启动部署过程：
ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
部署需要一些时间才能完成，请确保没有遇到错误。
成功的结果应类似于以下内容：
... PLAY RECAP ***********************************************************************************************************************************************************************************
clx-host-081 : ok=401 changed=31 unreachable=0 failed=0 skipped=718 rescued=0 ignored=1
clx-host-082 : ok=401 changed=31 unreachable=0 failed=0 skipped=718 rescued=0 ignored=1
clx-host-083 : ok=401 changed=31 unreachable=0 failed=0 skipped=718 rescued=0 ignored=1
clx-host-084 : ok=401 changed=30 unreachable=0 failed=0 skipped=718 rescued=0 ignored=1
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
node1 : ok=556 changed=62 unreachable=0 failed=0 skipped=1235 rescued=0 ignored=3
node2 : ok=505 changed=74 unreachable=0 failed=0 skipped=1080 rescued=0 ignored=2
node3 : ok=507 changed=53 unreachable=0 failed=0 skipped=1078 rescued=0 ignored=2
Thursday 17 February 2022 23:11:54 +0000 (0:00:00.265) 0:29:39.691 *****
===============================================================================
kubernetes/control-plane : Joining control plane node to the cluster. --------------------------------------------------------------------------------------------------------------- 810.38s
kubernetes/control-plane : kubeadm | Initialize first master ------------------------------------------------------------------------------------------------------------------------- 41.98s
kubernetes/control-plane : Master | wait for kube-scheduler -------------------------------------------------------------------------------------------------------------------------- 21.27s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources -------------------------------------------------------------------------------------------------------------------------- 15.54s
policy_controller/calico : Start of Calico kube controllers -------------------------------------------------------------------------------------------------------------------------- 14.76s
kubernetes/control-plane : Master | Remove controller manager container containerd/crio ---------------------------------------------------------------------------------------------- 11.30s
kubernetes/control-plane : Master | Remove scheduler container containerd/crio ------------------------------------------------------------------------------------------------------- 11.25s
kubernetes/preinstall : Update package management cache (APT) ------------------------------------------------------------------------------------------------------------------------ 10.33s
kubernetes/node : install | Copy kubelet binary from "
K8s集群自定义和验证
现在K8s集群已部署，可以从任何K8s主节点使用root用户账户或从安装了KUBECTL命令并配置了KUBECONFIG=<path-to-config-file>的其他服务器连接到K8s集群，以自定义部署。
在本指南中，我们继续从depserver使用root用户账户进行部署：
## 安装KUBECTL
snap install kubectl --channel=1.22/stable --classic
要开始使用集群，需要以普通用户身份运行以下命令：
mkdir -p $HOME/.kube
scp -i depuser@node1:/etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
标记工作节点：
主节点控制台
kubectl label nodes clx-host-081 node-role.kubernetes.io/worker=
kubectl label nodes clx-host-082 node-role.kubernetes.io/worker=
kubectl label nodes clx-host-083 node-role.kubernetes.io/worker=
kubectl label nodes clx-host-084 node-role.kubernetes.io/worker=
注意： K8s工作节点标记对于正确安装NVIDIA网络操作符是必需的。
以下是使用Calico CNI插件的K8s集群部署信息输出示例。
为确保Kubernetes集群正确安装，请运行以下命令：
## 获取集群节点状态
kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
clx-host-081 Ready worker 26h v1.22.5 192.168.222.101 <none> Ubuntu 20.04.4 LTS 5.4.0-100-generic containerd://1.5.8
clx-host-082 Ready worker 26h v1.22.5 192.168.222.102 <none> Ubuntu 20.04.4 LTS 5.4.0-100-generic containerd://1.5.8
clx-host-083 Ready worker 26h v1.22.5 192.168.222.103 <none> Ubuntu 20.04.4 LTS 5.4.0-100-generic containerd://1.5.8
clx-host-084 Ready worker 26h v1.22.5 192.168.222.104 <none> Ubuntu 20.04.4 LTS 5.4.0-100-generic containerd://1.5.8
node1 Ready control-plane,master 26h v1.22.5 192.168.222.111 <none> Ubuntu 20.04.4 LTS 5.4.0-100-generic containerd://1.5.8
node2 Ready control-plane,master 26h v1.22.5 192.168.222.112 <none> Ubuntu 20.04.3 LTS 5.4.0-100-generic containerd://1.5.8
node3 Ready control-plane,master 26h v1.22.5 192.168.222.113 <none> Ubuntu 20.04.3 LTS 5.4.0-100-generic containerd://1.5.8
## 获取系统Pod状态
kubectl -n kube-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-5788f6558-d9zcd 1/1 Running 6 26h 192.168.222.103 clx-host-083 <none> <none>
calico-node-7gdzm 1/1 Running 1 26h 192.168.222.104 clx-host-084 <none> <none>
calico-node-f6wz4 1/1 Running 1 26h 192.168.222.103 clx-host-083 <none> <none>
calico-node-fgtl7 1/1 Running 1 26h 192.168.222.102 clx-host-082 <none> <none>
calico-node-tb7hg 1/1 Running 1 26h 192.168.222.113 node3 <none> <none>
calico-node-v2hwz 1/1 Running 1 26h 192.168.222.101 clx-host-081 <none> <none>
calico-node-v7w7m 1/1 Running 0 26h 192.168.222.111 node1 <none> <none>
calico-node-vh984 1/1 Running 1 26h 192.168.222.112 node2 <none> <none>
coredns-8474476ff8-5rkrd 1/1 Running 0 26h 10.233.74.1 clx-host-082 <none> <none>
coredns-8474476ff8-crqh5 1/1 Running 0 26h 10.233.112.1 clx-host-084 <none> <none>
coredns-8474476ff8-n567s 1/1 Running 0 26h 10.233.111.1 clx-host-081 <none> <none>
coredns-8474476ff8-vr2ls 1/1 Running 0 26h 10.233.90.1 node1 <none> <none>
coredns-8474476ff8-wmcgv 1/1 Running 0 26h 10.233.78.1 clx-host-083 <none> <none>
dns-autoscaler-5ffdc7f89d-7fx8d 1/1 Running 0 26h 10.233.90.2 node1 <none> <none>
etcd-node1 1/1 Running 2 26h 192.168.222.111 node1 <none> <none>
etcd-node2 1/1 Running 1 26h 192.168.222.112 node2 <none> <none>
etcd-node3 1/1 Running 1 26h 192.168.222.113 node3 <none> <none>
kube-apiserver-node1 1/1 Running 4 26h 192.168.222.111 node1 <none> <none>
kube-apiserver-node2 1/1 Running 1 26h 192.168.222.112 node2 <none> <none>
kube-apiserver-node3 1/1 Running 1 26h 192.168.222.113 node3 <none> <none>
kube-controller-manager-node1 1/1 Running 4 26h 192.168.222.111 node1 <none> <none>
kube-controller-manager-node2 1/1 Running 3 26h 192.168.222.112 node2 <none> <none>
kube-controller-manager-node3 1/1 Running 3 26h 192.168.222.113 node3 <none> <none>
kube-proxy-7hrqw 1/1 Running 0 26h 192.168.222.101 clx-host-081 <none> <none>
kube-proxy-9n5lh 1/1 Running 0 26h 192.168.222.111 node1 <none> <none>
kube-proxy-b8mxv 1/1 Running 1 26h 192.168.222.113 node3 <none> <none>
kube-proxy-bq6zs 1/1 Running 1 26h 192.168.222.112 node2 <none> <none>
kube-proxy-cz7pz 1/1 Running 0 26h 192.168.222.104 clx-host-084 <none> <none>
kube-proxy-jrrw2 1/1 Running 0 26h 192.168.222.103 clx-host-083 <none> <none>
kube-proxy-rnt6g 1/1 Running 0 26h 192.168.222.102 clx-host-082 <none> <none>
kube-scheduler-node1 1/1 Running 2 26h 192.168.222.111 node1 <none> <none>
kube-scheduler-node2 1/1 Running 2 26h 192.168.222.112 node2 <none> <none>
kube-scheduler-node3 1/1 Running 2 26h 192.168.222.113 node3 <none> <none>
nodelocaldns-jf62n 1/1 Running 0 26h 192.168.222.104 clx-host-084 <none> <none>
nodelocaldns-lpmn7 1/1 Running 1 26h 192.168.222.113 node3 <none> <none>
nodelocaldns-pkhht 1/1 Running 0 26h 192.168.222.103 clx-host-083 nodelocaldns-rr6b2 1/1 Running 1 26h 192.168.222.112 node2 nodelocaldns-s2vnx 1/1 Running 0 26h 192.168.222.102 clx-host-082 nodelocaldns-sngtb 1/1 Running 0 26h 192.168.222.111 node1 nodelocaldns-x8nsf 1/1 Running 0 26h 192.168.222.101 clx-host-081
### NVIDIA GPU Operator 安装
NVIDIA GPU Operator 利用 Kubernetes 中的 Operator 框架来自动管理配置 GPU 所需的所有 NVIDIA 软件组件。这些组件包括 NVIDIA 驱动程序（用于启用 CUDA）、Kubernetes 设备插件（用于 GPU）、NVIDIA 容器运行时、自动节点标记、基于 DCGM 的监控等。有关平台支持和入门信息，请访问官方文档[仓库](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/overview.html)。
GPU Operator 部署需要 Helm：
```bash
## 安装 HELM
snap install helm --classic
添加 NVIDIA Helm 仓库：
## 添加 REPO
helm repo add nvidia https://nvidia.github.io/gpu-operator \
&& helm repo update
在 DGX 服务器平台上的 K8s 集群中安装 GPU Operator 的命令：
## 安装 GPU Operator
helm install --wait --generate-name -n gpu-operator --create-namespace nvidia/gpu-operator --set driver.enabled=false --set dcgm.enabled=false
## 查看安装
helm ls -n gpu-operator
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
gpu-operator-1646920855 gpu-operator 1 2022-03-10 14:01:05.942790618 +0000 UTC deployed gpu-operator-v1.9.1 v1.9.1
安装 Helm chart 后，检查 Pod 状态以确保所有容器正在运行且验证完成：
kubectl -n gpu-operator get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpu-feature-discovery-25csp 1/1 Running 0 2m14s 10.233.74.5 clx-host-082 <none> <none>
gpu-feature-discovery-4j5x2 1/1 Running 0 2m14s 10.233.78.7 clx-host-083 <none> <none>
gpu-feature-discovery-dthsq 1/1 Running 0 2m14s 10.233.112.4 clx-host-084 <none> <none>
gpu-feature-discovery-p7spz 1/1 Running 0 2m14s 10.233.111.4 clx-host-081 <none> <none>
gpu-operator-1646920855-node-feature-discovery-master-58cdc4vsk 1/1 Running 0 4m2s 10.233.96.2 node2 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-24ws8 1/1 Running 0 4m2s 10.233.92.4 node3 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-4xhkb 1/1 Running 0 4m2s 10.233.78.3 clx-host-083 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-ct6r7 1/1 Running 0 4m2s 10.233.111.2 clx-host-081 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-pf2bx 1/1 Running 0 4m2s 10.233.74.2 clx-host-082 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-ppwq7 1/1 Running 0 4m2s 10.233.90.3 node1 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-qv8k9 1/1 Running 0 4m2s 10.233.96.3 node2 <none> <none>
gpu-operator-1646920855-node-feature-discovery-worker-sqgww 1/1 Running 0 4m3s 10.233.112.2 clx-host-084 <none> <none>
gpu-operator-84b88fc49c-98wb7 1/1 Running 0 4m2s 10.233.92.3 node3 <none> <none>
nvidia-container-toolkit-daemonset-4mtwz 1/1 Running 0 2m13s 10.233.74.3 clx-host-082 <none> <none>
nvidia-container-toolkit-daemonset-h9xzm 1/1 Running 0 2m13s 10.233.112.3 clx-host-084 <none> <none>
nvidia-container-toolkit-daemonset-kqnsr 1/1 Running 0 2m13s 10.233.78.4 clx-host-083 <none> <none>
nvidia-container-toolkit-daemonset-zwvd9 1/1 Running 0 2m12s 10.233.111.3 clx-host-081 <none> <none>
nvidia-cuda-validator-c5lmr 0/1 Completed 0 110s 10.233.112.8 clx-host-084 <none> <none>
nvidia-cuda-validator-qlj4z 0/1 Completed 0 100s 10.233.78.9 clx-host-083 <none> <none>
nvidia-cuda-validator-rfdsd 0/1 Completed 0 98s 10.233.111.8 clx-host-081 <none> <none>
nvidia-cuda-validator-xqh28 0/1 Completed 0 104s 10.233.74.8 clx-host-082 <none> <none>
nvidia-dcgm-exporter-9rjqv 1/1 Running 0 2m16s 10.233.111.5 clx-host-081 <none> <none>
nvidia-dcgm-exporter-bl24c 1/1 Running 0 2m16s 10.233.112.6 clx-host-084 <none> <none>
nvidia-dcgm-exporter-nbn8z 1/1 Running 0 2m15s 10.233.74.7 clx-host-082 <none> <none>
nvidia-dcgm-exporter-trclg 1/1 Running 0 2m16s 10.233.78.5 clx-host-083 <none> <none>
nvidia-device-plugin-daemonset-72b9c 1/1 Running 0 2m14s 10.233.112.7 clx-host-084 <none> <none>
nvidia-device-plugin-daemonset-cz89s 1/1 Running 0 2m15s 10.233.111.6 clx-host-081 <none> <none>
nvidia-device-plugin-daemonset-nfrsr 1/1 Running 0 2m14s 10.233.78.8 clx-host-083 <none> <none>
nvidia-device-plugin-daemonset-rrpxg 1/1 Running 0 2m14s 10.233.74.4 clx-host-082 <none> <none>
nvidia-device-plugin-validator-2n686 0/1 Completed 0 89s 10.233.78.10 clx-host-083 <none> <none>
nvidia-device-plugin-validator-bt55c 0/1 Completed 0 87s 10.233.111.9 clx-host-081 <none> <none>
nvidia-device-plugin-validator-dczfx 0/1 Completed 0 103s 10.233.112.9 clx-host-084 <none> <none>
nvidia-device-plugin-validator-kssds 0/1 Completed 0 93s 10.233.74.9 clx-host-082 <none> <none>
nvidia-mig-manager-2wtr9 1/1 Running 0 79s 10.233.78.11 clx-host-083 <none> <none>
nvidia-mig-manager-49vpk 1/1 Running 0 83s 10.233.74.10 clx-host-082 <none> <none>
nvidia-mig-manager-4dktw 1/1 Running 0 79s 10.233.112.10 clx-host-084 <none> <none>
nvidia-mig-manager-kh8qd 1/1 Running 0 80s 10.233.111.10 clx-host-081 <none> <none>
nvidia-operator-validator-6dnpw 1/1 Running 0 2m16s 10.233.74.6 clx-host-082 <none> <none>
nvidia-operator-validator-gztcz 1/1 Running 0 2m15s 10.233.112.5 clx-host-084 <none> <none>
nvidia-operator-validator-vk98p 1/1 Running 0 2m16s 10.233.111.7 clx-host-081 <none> <none>
nvidia-operator-validator-wdz79 1/1 Running 0 2m16s 10.233.78.6 clx-host-083 <none> <none>
NVIDIA Network Operator 安装
NVIDIA Network Operator 利用 Kubernetes CRD 和 Operator SDK 来管理网络相关组件，以便在 K8s 集群中为工作负载启用快速网络和 RDMA。快速网络是 K8s 集群的辅助网络，适用于需要高带宽或低延迟的应用程序。
要使其工作，需要配置多个组件。Network Operator 部署需要 Helm。
添加 NVIDIA Network Operator Helm 仓库：
## 添加 REPO
helm repo add mellanox https://mellanox.github.io/network-operator \
&& helm repo update
创建 values.yaml 文件以自定义 Network Operator 部署（示例）：
nfd:
enabled: true
sriovNetworkOperator:
enabled: true
deployCR: true
ofedDriver:
deploy: false
nvPeerDriver:
deploy: false
rdmaSharedDevicePlugin:
deploy: false
sriovDevicePlugin:
deploy: false
secondaryNetwork:
deploy: true
cniPlugins:
deploy: true
multus:
deploy: true
ipamPlugin:
deploy: true
部署 operator：
helm install -f ./values.yaml -n network-operator --create-namespace --wait mellanox/network-operator --generate-name
NAME: network-operator-1646925670
LAST DEPLOYED: Thu Mar 10 15:21:22 2022
NAMESPACE: network-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Get Network Operator deployed resources by running the following commands:
$ kubectl -n network-operator get pods
$ kubectl -n nvidia-network-operator-resources get pods
安装 Helm chart 后，检查 pod 状态以确保所有容器都在运行：
## POD status in namespace - network-operator
kubectl -n network-operator get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
network-operator-1646925670-68d8f875f9-bzl4t 1/1 Running 0 3m36s 10.233.90.5 node1 <none> <none>
network-operator-1646925670-node-feature-discovery-master-mrzvc 1/1 Running 0 3m36s 10.233.96.5 node2 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-2hszv 1/1 Running 0 3m36s 10.233.78.12 clx-host-083 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-4xtct 1/1 Running 0 3m36s 10.233.96.4 node2 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-62lhk 1/1 Running 0 3m36s 10.233.112.11 clx-host-084 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-8vbhk 1/1 Running 0 3m36s 10.233.74.11 clx-host-082 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-8vrqt 1/1 Running 0 3m36s 10.233.111.11 clx-host-081 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-cv9rc 1/1 Running 0 3m36s 10.233.90.4 node1 <none> <none>
network-operator-1646925670-node-feature-discovery-worker-hbr7k 1/1 Running 0 3m36s 10.233.92.5 node3 <none> <none>
network-operator-1646925670-sriov-network-operator-6b75fd8ng66c 1/1 Running 0 3m36s 10.233.90.6 node1 <none> <none>
sriov-network-config-daemon-85dq5 3/3 Running 0 3m30s 192.168.222.103 clx-host-083 <none> <none>
sriov-network-config-daemon-8hn6g 3/3 Running 0 3m20s 192.168.222.104 clx-host-084 <none> <none>
sriov-network-config-daemon-9jb2j 3/3 Running 0 3m20s 192.168.222.101 clx-host-081 <none> <none>
sriov-network-config-daemon-kd6bp 3/3 Running 0 3m10s 192.168.222.102 clx-host-082 <none> <none>
## POD status in namespace - nvidia-network-operator-resources
kubectl -n nvidia-network-operator-resources get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cni-plugins-ds-9mg2g 1/1 Running 0 3m27s 192.168.222.101 clx-host-081 <none> <none>
cni-plugins-ds-lwzkn 1/1 Running 0 3m26s 192.168.222.103 clx-host-083 <none> <none>
cni-plugins-ds-w4pvx 1/1 Running 0 3m26s 192.168.222.104 clx-host-084 <none> <none>
cni-plugins-ds-w5hm8 1/1 Running 0 3m26s 192.168.222.102 clx-host-082 <none> <none>
kube-multus-ds-2xwws 1/1 Running 0 3m26s 192.168.222.102 clx-host-082 <none> <none>
kube-multus-ds-85cxw 1/1 Running 0 3m27s 192.168.222.101 clx-host-081 <none> <none>
kube-multus-ds-vk6hq 1/1 Running 0 3m26s 192.168.222.103 clx-host-083 <none> <none>
kube-multus-ds-xjx6x 1/1 Running 0 3m26s 192.168.222.104 clx-host-084 <none> <none>
whereabouts-6ftfb 1/1 Running 0 3m25s 192.168.222.103 clx-host-083 <none> <none>
whereabouts-89f2h 1/1 Running 0 3m25s 192.168.222.101 clx-host-081 <none> <none>
whereabouts-k6w4s 1/1 Running 0 3m24s 192.168.222.102 clx-host-082 <none> <none>
whereabouts-nqlb9 1/1 Running 0 3m25s 192.168.222.104 clx-host-084 <none> <none>
高速网络配置
安装 operator 后，请检查 SriovNetworkNodeState CR 以查看节点中所有支持 SR-IOV 的设备。
在此部署中，选择的网络接口名称如下：enp12s0、enp18s0、enp75s0、enp84s0、enp141s0、enp148s0、enp186s0 和 enp204s0。
要查看接口状态，请使用以下命令：
## NIC status
kubectl -n network-operator get sriovnetworknodestates.sriovnetwork.openshift.io clx-host-081 -o yaml
...
status:
interfaces:
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:b1:f4:fc
mtu: 1500
name: enp12s0
pciAddress: 0000:0c:00.0
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:c0:02:b2
mtu: 1500
name: enp18s0
pciAddress: "0000:12:00.0"
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:b1:f6:c8
mtu: 1500
name: enp75s0
pciAddress: 0000:4b:00.0
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:b1:f5:08
mtu: 1500
name: enp84s0
pciAddress: "0000:54:00.0"
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:b1:f2:d4
mtu: 1500
name: enp141s0
pciAddress: 0000:8d:00.0
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:c0:00:e2
mtu: 1500
name: enp148s0
pciAddress: 0000:94:00.0
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:b1:f6:f0
mtu: 1500
name: enp186s0
pciAddress: 0000:ba:00.0
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 200000 Mb/s
linkType: ETH
mac: 04:3f:72:b1:f6:bc
mtu: 1500
name: enp204s0
pciAddress: 0000:cc:00.0
totalvfs: 4
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkSpeed: 100000 Mb/s
linkType: ETH
mac: 04:3f:72:c1:cb:f0
mtu: 1500
name: enp225s0f0
pciAddress: 0000:e1:00.0
vendor: 15b3
- deviceID: 101b
driver: mlx5_core
linkType: ETH
mac: 04:3f:72:c1:cb:f1
mtu: 1500
name: enp225s0f1
pciAddress: 0000:e1:00.1
vendor: 15b3
- deviceID: "1533"
driver: igb
linkType: ETH
mac: 5c:ff:35:e2:1e:41
mtu: 1500
name: enp226s0
pciAddress: 0000:e2:00.0
vendor: "8086"
syncStatus: Succeeded
为每个选定的网络接口创建 SriovNetworkNodePolicy CR - policy.yaml 文件，通过在 'nicSelector' 中指定所选接口（本例中为 enp12s0 接口）：
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw1
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw1
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp12s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw2
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw2
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp18s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw3
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw3
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp75s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw4
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw4
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp84s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw5
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw5
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp141s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw6
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw6
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp148s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw7
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw7
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp186s0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw8
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: roce_sw8
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp204s0" ]
deviceType: netdevice
isRdma: true
部署 policy.yaml：
kubectl apply -f policy.yaml
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw1 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw2 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw3 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw4 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw5 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw6 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw7 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw8 created
注意： 此步骤需要一些时间。这取决于要应用配置的K8s工作节点数量以及每个所选网络接口的VF数量。
为每个选定的网络接口创建一个 SriovNetwork CR - network.yaml 文件，该文件引用在 SriovNetworkNodePolicy 中定义的 resourceName（在此示例中，引用 roce_swX 资源并为高速网络设置CIDR范围）：
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw1
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.101.0/24"
}
networkNamespace: default
resourceName: roce_sw1
vlan: 11
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw2
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.102.0/24"
}
networkNamespace: default
resourceName: roce_sw2
vlan: 12
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw3
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.103.0/24"
}
networkNamespace: default
resourceName: roce_sw3
vlan: 13
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw4
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.104.0/24"
}
networkNamespace: default
resourceName: roce_sw4
vlan: 14
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw5
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.105.0/24"
}
networkNamespace: default
resourceName: roce_sw5
vlan: 15
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw6
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.106.0/24"
}
networkNamespace: default
resourceName: roce_sw6
vlan: 16
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw7
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.107.0/24"
}
networkNamespace: default
resourceName: roce_sw7
vlan: 17
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: network-sw8
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.108.0/24"
}
networkNamespace: default
resourceName: roce_sw8
vlan: 18
部署 network.yaml：
kubectl apply -f network.yaml
sriovnetwork.sriovnetwork.openshift.io/network-sw1 created
sriovnetwork.sriovnetwork.openshift.io/network-sw2 created
sriovnetwork.sriovnetwork.openshift.io/network-sw3 created
sriovnetwork.sriovnetwork.openshift.io/network-sw4 created
sriovnetwork.sriovnetwork.openshift.io/network-sw5 created
sriovnetwork.sriovnetwork.openshift.io/network-sw6 created
sriovnetwork.sriovnetwork.openshift.io/network-sw7 created
sriovnetwork.sriovnetwork.openshift.io/network-sw8 created
验证部署
检查已部署的网络：
kubectl get network-attachment-definitions.k8s.cni.cncf.io
NAME AGE
network-sw1 33m
network-sw2 33m
network-sw3 33m
network-sw4 33m
network-sw5 33m
network-sw6 33m
network-sw7 33m
network-sw8 33m
检查工作节点资源：
kubectl get node clx-host-081 -o json | jq '.status.allocatable'
{
"cpu": "255900m",
"ephemeral-storage": "1698708802820",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "1056271380Ki",
"nvidia.com/gpu": "8",
...
}
"nvidia.com/roce_sw1": "8",
"nvidia.com/roce_sw2": "8",
"nvidia.com/roce_sw3": "8",
"nvidia.com/roce_sw4": "8",
"nvidia.com/roce_sw5": "8",
"nvidia.com/roce_sw6": "8",
"nvidia.com/roce_sw7": "8",
"nvidia.com/roce_sw8": "8",
"pods": "110"
}
kubectl get node clx-host-082 -o json | jq '.status.allocatable'
{
"cpu": "255900m",
"ephemeral-storage": "1698708802820",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "1056271428Ki",
"nvidia.com/gpu": "8",
"nvidia.com/roce_sw1": "8",
"nvidia.com/roce_sw2": "8",
"nvidia.com/roce_sw3": "8",
"nvidia.com/roce_sw4": "8",
"nvidia.com/roce_sw5": "8",
"nvidia.com/roce_sw6": "8",
"nvidia.com/roce_sw7": "8",
"nvidia.com/roce_sw8": "8",
"pods": "110"
}
kubectl get node clx-host-083 -o json | jq '.status.allocatable'
{
"cpu": "255900m",
"ephemeral-storage": "1698708802820",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "1056275120Ki",
"nvidia.com/gpu": "8",
"nvidia.com/roce_sw1": "8",
"nvidia.com/roce_sw2": "8",
"nvidia.com/roce_sw3": "8",
"nvidia.com/roce_sw4": "8",
"nvidia.com/roce_sw5": "8",
"nvidia.com/roce_sw6": "8",
"nvidia.com/roce_sw7": "8",
"nvidia.com/roce_sw8": "8",
"pods": "110"
}
kubectl get node clx-host-084 -o json | jq '.status.allocatable'
{
"cpu": "255900m",
"ephemeral-storage": "1698708802820",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "1056270348Ki",
"nvidia.com/gpu": "8",
"nvidia.com/roce_sw1": "8",
"nvidia.com/roce_sw2": "8",
"nvidia.com/roce_sw3": "8",
"nvidia.com/roce_sw4": "8",
"nvidia.com/roce_sw5": "8",
"nvidia.com/roce_sw6": "8",
"nvidia.com/roce_sw7": "8",
"nvidia.com/roce_sw8": "8",
"pods": "110"
}
在运行于不同K8s工作节点上的两个Pod之间，使用ib_write_bw运行合成RDMA基准测试。
此步骤包括以下内容：
- 创建容器镜像并推送到您的仓库
- 部署K8s部署应用
- 运行测试
RDMA基准测试Dockerfile：
FROM ubuntu:20.04
# Ubuntu 20.04 docker container with inbox Mellanox drivers
# LABEL about the custom image
LABEL maintainer=vitaliyra@nvidia.com
LABEL description="This is custom Container Image with inbox perftest package."
WORKDIR /tmp/
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get clean -y && apt-get -y update && apt-get install -y apt-utils udev vim bash && apt-get -y upgrade
RUN apt-get install -y iproute2 rdma-core libibmad5 ibutils ibverbs-utils infiniband-diags perftest \
mstflint strace iputils-ping
RUN ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
RUN dpkg-reconfigure --frontend noninteractive tzdata && apt-get clean all -y
CMD bash
请使用您喜欢的容器构建工具（docker、podman等）从上述Dockerfile创建容器镜像，以便在下面的部署中使用。创建镜像后，将其推送到容器仓库。
创建示例部署test-deployment.yaml（容器镜像应包含InfiniBand用户空间驱动程序和性能工具）：
test-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mlnx-inbox-pod
labels:
app: sriov
spec:
replicas: 2
selector:
matchLabels:
app: sriov
template:
metadata:
labels:
app: sriov
annotations:
k8s.v1.cni.cncf.io/networks: network-sw1
spec:
containers:
- image: < Container image >
name: mlnx-inbox-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
requests:
cpu: 4
nvidia.com/roce_sw1: 1
limits:
cpu: 4
nvidia.com/roce_sw1: 1
command:
- sh
- -c
- sleep inf
部署示例部署。
kubectl apply -f test-deployment.yaml
deployment.apps/mlnx-inbox-pod created
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mlnx-inbox-pod-6586dcc7b9-2b9nm 1/1 Running 0 2m14s 10.233.112.35 clx-host-084 <none> <none>
mlnx-inbox-pod-6586dcc7b9-xs7wx 1/1 Running 0 2m14s 10.233.111.34 clx-host-081 <none> <none>
检查每个Pod中的可用网络接口。
## First POD
kubectl exec -it mlnx-inbox-pod-6586dcc7b9-2b9nm -- ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether 26:1f:c8:a8:e2:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.112.35/32 brd 10.233.112.35 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::241f:c8ff:fea8:e28d/64 scope link
valid_lft forever preferred_lft forever
36: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether e6:5a:bd:85:35:15 brd ff:ff:ff:ff:ff:ff
inet 192.168.101.1/24 brd 192.168.101.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::e45a:bdff:fe85:3515/64 scope link
valid_lft forever preferred_lft forever
## Second POD
kubectl exec -it mlnx-inbox-pod-6586dcc7b9-xs7wx -- ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if94: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
link/ether 52:76:f4:e7:a2:9b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.111.34/32 brd 10.233.111.34 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5076:f4ff:fee7:a29b/64 scope link
valid_lft forever preferred_lft forever
28: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 72:72:6d:1d:84:5a brd ff:ff:ff:ff:ff:ff
inet 192.168.101.2/24 brd 192.168.101.255 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::7072:6dff:fe1d:845a/64 scope link
valid_lft forever preferred_lft forever
运行合成RDMA基准测试。
| 服务器 | ib_write_bw -a -F -d $IB_DEV_NAME --report_gbits |
|---|---|
| 客户端 | ib_write_bw -a -F $SERVER_IP -d $IB_DEV_NAME --report_gbits |
请为每个Pod打开控制台会话——一个用于服务器端，另一个用于客户端。
在第一个控制台（服务器端）中，运行以下命令：
kubectl exec -it mlnx-inbox-pod-6586dcc7b9-2b9nm -- bash
root@mlnx-inbox-pod-6586dcc7b9-2b9nm:/tmp# rdma link | grep net1
link mlx5_13/1 state ACTIVE physical_state LINK_UP netdev net1
root@mlnx-inbox-pod-6586dcc7b9-2b9nm:/tmp# ib_write_bw -a -F -d mlx5_13 --report_gbits
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_13
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0069 PSN 0xaa30eb RKey 0x010e00 VAddr 0x007fb3a9d52000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:101:01
remote address: LID 0000 QPN 0x00e9 PSN 0x32bd22 RKey 0x030e00 VAddr 0x007ff245361000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:101:02
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
8388608 5000 185.70 185.44 0.002763
---------------------------------------------------------------------------------------
在第二个控制台（客户端）中，运行以下命令：
side), 运行以下命令：
root@node1:~/YAMLs/8port/example# kubectl exec -it mlnx-inbox-pod-6586dcc7b9-xs7wx -- bash
root@mlnx-inbox-pod-6586dcc7b9-xs7wx:/tmp# rdma link | grep net1
link mlx5_15/1 state ACTIVE physical_state LINK_UP netdev net1
root@mlnx-inbox-pod-6586dcc7b9-xs7wx:/tmp# ib_write_bw -a -F 192.168.101.1 -d mlx5_15 --report_gbits
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_15
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x00e9 PSN 0x32bd22 RKey 0x030e00 VAddr 0x007ff245361000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:101:02
remote address: LID 0000 QPN 0x0069 PSN 0xaa30eb RKey 0x010e00 VAddr 0x007fb3a9d52000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:101:01
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 5000 0.044858 0.044364 2.772772
4 5000 0.089829 0.089793 2.806042
8 5000 0.18 0.18 2.788396
16 5000 0.36 0.36 2.801705
32 5000 0.72 0.72 2.801529
64 5000 1.10 1.05 2.056373
128 5000 2.17 2.16 2.107263
256 5000 4.32 4.32 2.110149
512 5000 8.65 8.64 2.110166
1024 5000 17.29 17.24 2.104959
2048 5000 34.32 34.23 2.089381
4096 5000 68.14 65.74 2.006262
8192 5000 170.15 139.82 2.133420
16384 5000 188.33 169.84 1.295812
32768 5000 190.95 180.36 0.688024
65536 5000 191.23 181.41 0.327763
131072 5000 192.34 190.78 0.181938
262144 5000 191.26 185.41 0.083644
524288 5000 191.15 183.44 0.043735
1048576 5000 190.31 187.27 0.022325
2097152 5000 187.04 185.88 0.011079
4194304 5000 189.42 185.82 0.005538
8388608 5000 185.70 185.44 0.002763
---------------------------------------------------------------------------------------
Kubeflow Training Operator
Kubeflow 是一个用于 Kubernetes 的机器学习工具包。
Kubeflow 训练操作符是 Kubeflow 的一部分，是一组 Kubernetes 操作符，为 Kubeflow 添加了对使用不同框架进行分布式训练机器学习模型的支持。
训练操作符提供了 Kubernetes CR，使得在 Kubernetes 上运行分布式或非分布式 TensorFlow/PyTorch/Apache MXNet/XGBoost/MPI 作业变得容易。
在下面的示例中，我们部署了 Kubeflow 训练操作符的稳定版本 v1.4.0：
kubectl apply -k "github.com/kubeflow/training-operator/manifests/overlays/standalone?ref=v1.4.0"
namespace/kubeflow created
customresourcedefinition.apiextensions.k8s.io/mpijobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/mxjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/xgboostjobs.kubeflow.org created
serviceaccount/training-operator created
clusterrole.rbac.authorization.k8s.io/training-operator created
clusterrolebinding.rbac.authorization.k8s.io/training-operator created
service/training-operator created
deployment.apps/training-operator created
Appendix
Job Testing Results
以下是不同网络配置的 Dockerfile 和 MPIJob 示例。
Dockerfile
用于 MPIJob 的 Dockerfile 示例：
FROM nvcr.io/nvidia/tensorflow:21.10-tf1-py3
RUN apt-get update && apt-get install -y --no-install-recommends openssh-client openssh-server && \
mkdir -p /var/run/sshd
# Allow OpenSSH to talk to containers without asking for confirmation
# by disabling StrictHostKeyChecking.
# mpi-operator mounts the .ssh folder from a Secret. For that to work, we need
# to disable UserKnownHostsFile to avoid write permissions.
# Disabling StrictModes avoids directory and files read permission checks.
RUN sed -i 's/[ #]\(.*StrictHostKeyChecking \).*/ \1no/g' /etc/ssh/ssh_config && \
echo " UserKnownHostsFile /dev/null" >> /etc/ssh/ssh_config && \
sed -i 's/#\(StrictModes \).*/\1no/g' /etc/ssh/sshd_config
RUN mkdir /tensorflow
WORKDIR "/tensorflow"
RUN git clone https://github.com/tensorflow/benchmarks
WORKDIR "/tensorflow/benchmarks"
CMD ["/bin/bash"]
此 Dockerfile 基于 TensorFlow NGC 容器镜像。TensorFlow NGC 容器针对 GPU 加速进行了优化，并包含一组经过验证的库，可启用和优化 GPU 性能。此容器还可能包含对 TensorFlow 源代码的修改，以最大化性能和兼容性。此容器还包含用于加速 ETL（DALI、RAPIDS）、训练（cuDNN、NCCL）和推理（TensorRT）工作负载的软件。
有关支持的版本，请参阅 Framework Containers Support Matrix 和 NVIDIA Container Toolkit 文档。
请使用您喜欢的容器构建工具（docker、podman 等）从 Dockerfile 创建容器镜像，以便在下面的部署中使用。创建镜像后，将其推送到容器注册表。
MPIJob Examples
以下是通过 K8s 管理网络进行网络配置的 MPIJob 示例：
# TF MPIJob over MGMT network
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: tensorflow-benchmarks
spec:
slotsPerWorker: 8
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: < Container image >
name: tensorflow-benchmarks
command:
- mpirun
- --allow-run-as-root
- -np
- "32"
- -bind-to
- none
- -map-by
- slot
- -x
- NCCL_DEBUG=INFO
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
- --batch_size=64
- --model=resnet152
- --variable_update=horovod
- --xla=true
- --use_fp16=true
Worker:
replicas: 4
template:
spec:
containers:
- image: < Container image >
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 8
以下是通过辅助 K8s 网络进行网络配置的 MPIJob 示例：
# TF MPIJob over high-perf TCP network
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: tensorflow-benchmarks
spec:
slotsPerWorker: 8
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: < Container image >
name: tensorflow-benchmarks
command:
- mpirun
- --allow-run-as-root
- -np
- "32"
- -bind-to
- none
- -map-by
- slot
- -x
- NCCL_DEBUG=INFO
- -x
- NCCL_IB_DISABLE=1
- -x
-
NCCL_NET_GDR_LEVEL=0
- -x
- NCCL_NET_PLUGIN=none
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
- --batch_size=64
- --model=resnet152
- --variable_update=horovod
- --xla=true
- --use_fp16=true
Worker:
replicas: 4
template:
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: network-sw1,network-sw2,network-sw3,network-sw4,network-sw5,network-sw6,network-sw7,network-sw8
spec:
containers:
- image: < Container image >
name: tensorflow-benchmarks
resources:
limits:
nvidia.com/gpu: 8
nvidia.com/roce_sw1: 1
nvidia.com/roce_sw2: 1
nvidia.com/roce_sw3: 1
nvidia.com/roce_sw4: 1
nvidia.com/roce_sw5: 1
nvidia.com/roce_sw6: 1
nvidia.com/roce_sw7: 1
nvidia.com/roce_sw8: 1
以下是通过RDMA启用的辅助K8s网络进行网络配置的MPIJob示例：
# TF MPIJob over RDMA network
apiVersion: kubeflow.org/v1
kind: MPIJob
metadata:
name: tensorflow-benchmarks
spec:
slotsPerWorker: 8
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: < Container image >
name: tensorflow-benchmarks
command:
- mpirun
- --allow-run-as-root
- -np
- "32"
- -bind-to
- none
- -map-by
- slot
- -x
- NCCL_DEBUG=INFO
- -x
- NCCL_IB_DISABLE=0
- -x
- NCCL_NET_GDR_LEVEL=2
- -x
- TF_ALLOW_IOLIBS=1
- -x
- LD_LIBRARY_PATH
- -x
- PATH
- -mca
- pml
- ob1
- -mca
- btl
- ^openib
- python
- scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py
- --batch_size=64
- --model=resnet152
- --variable_update=horovod
- --xla=true
- --use_fp16=true
Worker:
replicas: 4
template:
metadata:
annotations:
k8s.v1.cni.cncf.io/networks: network-sw1,network-sw2,network-sw3,network-sw4,network-sw5,network-sw6,network-sw7,network-sw8
spec:
containers:
- image: < Container image >
name: tensorflow-benchmarks
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
limits:
nvidia.com/gpu: 8
nvidia.com/roce_sw1: 1
nvidia.com/roce_sw2: 1
nvidia.com/roce_sw3: 1
nvidia.com/roce_sw4: 1
nvidia.com/roce_sw5: 1
nvidia.com/roce_sw6: 1
nvidia.com/roce_sw7: 1
nvidia.com/roce_sw8: 1
测试结果
警告：本文档中列出的性能结果仅供参考，不应视为NVIDIA产品的正式性能目标。
作者
 |
Vitaliy RazinkovVitaliy Razinkov是NVIDIA网络团队的解决方案架构师，专注于复杂的Kubernetes、OpenShift和Microsoft解决方案。拥有超过25年的高级技术职位经验，他在设计和实施先进基础设施方面拥有深厚的专业知识。Vitaliy撰写了多份关于Microsoft技术、RoCE/RDMA加速的Kubernetes/OpenShift机器学习以及容器化解决方案的参考设计指南——所有这些都可以在NVIDIA网络文档网站上找到。 |