RDG：基于InfiniBand网络的NVIDIA GPU Kubernetes集群部署
创建于2020年11月16日，作者：Boris Kovalev, Vitaliy Razinkov。范围：本参考部署指南（RDG）介绍了如何构建性能最高的Kubernetes
文档目录
创建于2020年11月16日，作者：Boris Kovalev, Vitaliy Razinkov
范围
本参考部署指南（RDG） 介绍了如何构建性能最高的Kubernetes（K8s）集群，该集群能够承载最苛刻的分布式工作负载，运行在NVIDIA GPU和NVIDIA端到端InfiniBand网络之上。
缩写和缩略语
| 术语 | 定义 | 术语 | 定义 |
|---|---|---|---|
| AOC | 有源光缆 | IB | InfiniBand |
| AI | 人工智能 | K8s | Kubernetes |
| CNI | 容器网络接口 | ML | 机器学习 |
| CR | 自定义资源 | MOFED | Mellanox OpenFabrics Enterprise Distribution |
| DAC | 直连铜缆 | PF | 物理功能 |
| DHCP | 动态主机配置协议 | RDMA | 远程直接内存访问 |
| EDR | 增强数据速率 - 100Gb/s | QSG | 快速入门指南 |
| GPU | 图形处理单元 | SR-IOV | 单根输入输出虚拟化 |
| HDR | 高数据速率 - 200Gb/s | VF | 虚拟功能 |
| HPC | 高性能计算 |
参考文献
- NVIDIA T4 GPU
- NVIDIA MLNX-OS (InfiniBand) - Mellanox Docs
- NVIDIA OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)
- What is Kubernetes?
- Kubespray
- NVIDIA GPU Operator
- Whereabouts-CNI
- SR-IOV Network Operator
引言
机器学习和高性能计算云解决方案的配置可能变得非常复杂。正确的设计以及软硬件组件的选择可能成为成功部署的关键任务。
本文档将指导您完成完整的解决方案周期，包括设计、组件选择、技术概述和部署步骤。
该解决方案将基于GPU服务器，在NVIDIA端到端InfiniBand网络上进行配置。 NVIDIA GPU和SR-IOV网络运营商允许在InfiniBand网络上运行GPU加速和原生RDMA工作负载，例如HPC、大数据、机器学习、人工智能和其他应用。
以下描述了以下过程：
- 使用Kubespray在运行Ubuntu 20.04操作系统的裸机节点上部署K8s集群。
- 部署NVIDIA GPU Operator。
- 配置InfiniBand网络。
- POD部署示例。
本文档涵盖单Kubernetes控制器部署场景。 对于高可用性集群部署，请参考https://github.com/kubernetes-sigs/kubespray/blob/master/docs/ha-mode.md
解决方案架构
关键组件和技术
-
NVIDIA® T4 GPU NVIDIA® T4 GPU基于NVIDIA Turing™架构，采用节能的70瓦小型PCIe外形。T4针对主流计算环境进行了优化，并具有多精度Turing Tensor Cores和RT Cores。结合NGC的加速容器化软件栈，T4在规模上提供了革命性的性能，以加速云工作负载，如高性能计算、深度学习训练和推理、机器学习、数据分析和图形处理。
-
NVIDIA MLNX-OS® NVIDIA MLNX-OS是Mellanox的InfiniBand/VPI交换机操作系统，适用于数据中心，包括存储、企业、高性能、机器学习、大数据计算和云网络。
-
NVIDIA ConnectX InfiniBand网卡 NVIDIA® ConnectX® InfiniBand智能网卡配备加速引擎，提供最佳的网络性能和效率，支持SDR、QDR、DDR、FDR、EDR和HDR InfiniBand速度下的低延迟、高吞吐量和高消息速率。
-
NVIDIA智能InfiniBand交换机系统 NVIDIA智能InfiniBand交换机系统为高性能计算、人工智能、Web 2.0、大数据、云和企业数据中心提供最高的性能和端口密度。支持高达36至800端口配置，速率最高可达
200Gb/s per port, allows compute clusters and converged data centers to operate at any scale, reducing operational costs and infrastructure complexity.
-
NVIDIA LinkX® InfiniBand Cables NVIDIA Mellanox LinkX cables and transceivers are designed to maximize the performance of High Performance Computing networks, requiring high-bandwidth, low-latency connections between compute nodes and switch nodes. DAC is available up to 7m. AOCs are available in <30m OM2 fiber lowest-cost lengths; OM3/OM4 multimode to 100m. DACs and AOCs data rates of QDR(40G), FDR10(40G), FDR(56G), EDR(100G), HDR100 (100G) and HDR (200G).
-
Kubernetes Kubernetes (K8s) is an open-source container orchestration platform for deployment automation, scaling, and management of containerized applications.
-
Kubespray (From Kubernetes.io) Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks and provides:
- A highly available cluster
- Composable attributes
- Support for most popular Linux distributions
-
NVIDIA GPU Operator NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPUs. These components include the NVIDIA drivers (to enable CUDA), Kubernetes device plugin for GPUs, the NVIDIA Container Runtime, automatic node labelling, DCGM based monitoring and others.
-
RDMA Remote Direct Memory Access (RDMA) is a technology that allows computers in a network to exchange data without involving the processor, cache or operating system of either computer. Like locally based Direct Memory Access (DMA), RDMA improves throughput and performance and frees up compute resources.
-
SR-IOV Network Operator SR-IOV Network Operator is designed to help the user to provision and configure SR-IOV CNI plugin and Device plugin in the Openshift and Kubernetes clusters.
Logical Design
The logical design includes the following layers:
-
One compute layer:
- Deployment node
- K8s Master node
- 2 x K8s Worker nodes with two NVIDIA T4 GPUs and one Mellanox ConnectX adapter.
-
Two separate networking layers:
- Management network
- High-speed InfiniBand (IB) fabric
Fabric Design
In this RDG we will describe a small scale solution with only one switch.
Simple Setup with One Switch
In a single switch case, by using an NVIDIA QM8700 InfiniBand HDR Switch System you can connect up to 40 servers with NVIDIA LinkX HDR 200Gb/s QSFP56 DAC cables.
Scaled Setup for InfiniBand Fabric
For assistance in designing the scaled InfiniBand topology, use the NVIDIA InfiniBand Topology Generator, an online cluster configuration tool that offers flexible cluster configurations and sizes. For a scaled setup we recommend using NVIDIA Unified Fabric Manager (UFM®).
Bill of Materials (BoM)
The following hardware setup is utilized in the distributed K8s configuration described in this guide:
Warning: The above table does not contain Kubernetes Management network connectivity components.
Deployment and Configuration
The deployment is validated using Ubuntu 20.04 OS and Kubespray v2.14.2.
Wiring
The first port of each NVIDIA HCA on each Worker node is wired to the NVIDIA switch using NVIDIA LinkX HDR 200Gb/s QSFP56 DAC cables.
Network
Prerequisites
- InfiniBand fabric
- Switch NVIDIA QM8700
- Switch OS NVIDIA MLNX-OS®
- Management Network DHCP and DNS services are part of the IT infrastructure. The component installation and configuration are not covered in this guide.
Network Configuration
Below are the server names with their relevant network configurations.
| Server/Switch type | Server/Switch name | High-speed network HDR | Management network 1 GigE |
|---|---|---|---|
| Master Node | node1 | eno0: DHCP 192.168.1.40 | |
| Worker Node | node2 | ibs6f0: none | eno0: DHCP 192.168.1.10 |
| Worker Node | node3 | ibs6f0: none | eno0: DHCP 192.168.1.11 |
RDG：基于InfiniBand网络的NVIDIA GPU Kubernetes集群部署
节点配置
部署节点
| 角色 | 主机名 | 网卡 | IP地址 |
|---|---|---|---|
| 部署节点 | sl-depl-node | eno0: DHCP 192.168.1.43 | |
| 高速交换机 | swx-mld-ib67 | 无 | mgmt0: 从DHCP获取 192.168.1.38 |
警告：
ibs6f0接口无需额外配置。
InfiniBand网络配置
以下是配置过程中的重要建议和前提条件：
- 请参考MLNX-OS用户手册以熟悉交换机软件（位于 enterprise-support.nvidia.com/s/）
- 将交换机软件升级到最新的MLNX-OS版本
- 需要InfiniBand子网管理器（SM）来正确配置InfiniBand网络
在InfiniBand网络中有三种运行InfiniBand SM的方式：
- 在一个或多个托管交换机上启动SM。这是一种非常方便快捷的操作，可实现InfiniBand的即插即用。
- 通过执行
/etc/init.d/opensmd命令在一个或多个服务器上运行OpenSM守护进程。如果节点数达到648个或更多，建议在服务器上运行SM。 - 使用统一结构管理（UFM®）。UFM是一个强大的可扩展计算平台，消除了结构管理的复杂性，提供对流量的深度可见性，并优化结构性能。
在本指南中，我们将在InfiniBand交换机上启动InfiniBand SM（方法1）。以下是所选方法的配置步骤。
在托管交换机上启用SM：
- 登录交换机并输入以下配置命令（
swx-mld-ib67是我们的交换机名称）：
Mellanox MLNX-OS Switch Management
switch login: admin
Password:
swx-mld-ib67 [standalone: master] > enable
swx-mld-ib67 [standalone: master] # configure terminal
swx-mld-ib67 [standalone: master] (config) # ib smnode swx-mld-ib67 enable
swx-mld-ib67 [standalone: master] (config) # ib smnode swx-mld-ib67 sm-priority 0
swx-mld-ib67 [standalone: master] (config) # ib sm virt enable
swx-mld-ib67 [standalone: master] (config) # write memory
swx-mld-ib67 [standalone: master] (config) # reload
- 交换机重启后，检查交换机配置。应如下所示：
Mellanox MLNX-OS Switch Management
switch login: admin
Password:
swx-mld-ib67 [standalone: master] > enable
swx-mld-ib67 [standalone: master] # configure terminal
swx-mld-ib67 [standalone: master] (config) # show running-config
##
## Running database "initial"
## Generated at 2020/12/16 17:40:41 +0000
## Hostname: swx-mld-ib67
## Product release: 3.9.1600
##
##
## Running-config temporary prefix mode setting
##
no cli default prefix-modes enable
##
## Subnet Manager configuration
##
ib sm virt enable
##
## Other IP configuration
##
hostname swx-mld-ib67
##
## Other IPv6 configuration
##
no ipv6 enable
##
## Local user account configuration
##
username admin password 7 $6$6GZ8Q0RF$FZW9pc23JJkwwOJTq85xZe1BJgqQV/m6APQNPkagZlTEUgKMWLr5X3Jq2hsUyB.K5nrGdDNUaSLiK2xupnIJo1
username monitor password 7 $6$z1.r4Kl7$TIwaNf7uXNxZ9UdGdUpOO9kVug0shRqGtu75s3dSrY/wY1v1mGjrqQLNPHvHYh5HAhVuUz5wKzD6H/beYeEqL.
##
## AAA remote server configuration
##
# ldap bind-password ********
# radius-server key ********
# tacacs-server key ********
##
## Network management configuration
##
# web proxy auth basic password ********
##
## X.509 certificates configuration
##
#
# Certificate name system-self-signed, ID 12d0989d8623825b71bc25f9bc02de813fc9fe2a
# (public-cert config omitted since private-key config is hidden)
##
## IB nodename to GUID mapping
##
ib smnode swx-mld-ib67 create
ib smnode swx-mld-ib67 enable
ib smnode swx-mld-ib67 sm-priority 0
##
## Persistent prefix mode setting
##
cli default prefix-modes enable
节点配置
通用前提条件
- 硬件：所有K8s工作节点具有相同的硬件规格（详见BoM）。
- 主机BIOS：确保使用支持SR-IOV的服务器平台作为K8s工作节点，并查阅硬件文档中的BIOS设置以在BIOS中启用SR-IOV。
- 主机操作系统：所有服务器应安装Ubuntu Server 20.04操作系统，并包含OpenSSH服务器软件包。
- Kubernetes经验：确保熟悉Kubernetes集群架构。
主机操作系统前提条件
确保所有服务器已安装Ubuntu Server 20.04操作系统并包含OpenSSH服务器软件包，创建一个具有sudo权限且无需密码的非root用户账户。
通过运行以下命令更新Ubuntu软件包：
$ sudo apt-get update
$ sudo apt-get upgrade -y
$ sudo reboot
非root用户账户前提条件
在本解决方案中，我们在 /etc/sudoers 文件末尾添加了以下行：
$ sudo vim /etc/sudoers
#includedir /etc/sudoers.d
#K8s cluster deployment user with sudo privileges without password
user ALL=(ALL) NOPASSWD:ALL
软件前提条件
- 在工作节点服务器上禁用/黑名单Nouveau NVIDIA驱动，运行以下命令或将每行粘贴到终端：
$ sudo su -
# lsmod |grep nouv
# bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
# bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf"
# update-initramfs -u
# reboot
$ lsmod |grep nouv
- 在工作节点服务器上安装NVIDIA MLNX_OFED并升级固件，运行以下命令或将每行粘贴到终端：
$ sudo su -
# apt-get install rdma-core
# wget -qO - https://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -
# curl https://linux.mellanox.com/public/repo/mlnx_ofed/latest/ubuntu20.04/mellanox_mlnx_ofed.list --output /etc/apt/sources.list.d/mellanox_mlnx_ofed.list
# apt update
# apt install -y mlnx-ofed-kernel-only
# wget http://www.mellanox.com/downloads/firmware/mlxup/4.15.2/SFX/linux_x64/mlxup
# chmod +x mlxup
# ./mlxup --online -u
# reboot
- 在工作节点服务器上设置IB端口链路：
root@node2:~# ibdev2netdev
...
mlx5_2 port 1 ==> ibs6f0 (Down)
mlx5_3 port 1 ==> ibs6f1 (Down)
...
root@node2:~# vim /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
ibs6f0: {}
eno1:
dhcp4: true
version: 2
root@node2:~# netplan apply
root@node2:~# ibdev2netdev
...
mlx5_2 port 1 ==> ibs6f0 (Up)
mlx5_3 port 1 ==> ibs6f1 (Down)
...
- 在工作节点服务器上将
netns设置为exclusive模式，以允许RDMA工作负载的网络命名空间隔离：
root@node2:~# vim /etc/modprobe.d/ib_core.conf
# Set netns to exclusive mode for namespace isolation
options ib_core netns_mode=0
root@node2:~# update-initramfs -u
root@node2:~# reboot
- 在工作节点服务器上检查
netns模式和InfiniBand设备：
$ rdma system
netns exclusive
$ ls -la /dev/infiniband/
total 0
drwxr-xr-x 2 root root 300 Jan 26 16:26 .
drwxr-xr-x 22 root root 5100 Jan 26 16:55 ..
crw------- 1 root root 231, 64 Jan 26 16:26 issm0
crw------- 1 root root 231, 65 Jan 26 16:26 issm1
crw------- 1 root root 231, 66 Jan 26 16:26 issm2
crw------- 1 root root 231, 67 Jan 26 16:26 issm3
crw-rw-rw- 1 root root 10, 57 Jan 26 16:26 rdma_cm
crw------- 1 root root 231, 0 Jan 26 16:26 umad0
crw------- 1 root root 231, 1 Jan 26 16:26 umad1
crw------- 1 root root 231, 2 Jan 26 16:26 umad2
crw------- 1 root root 231, 3 Jan 26 16:26 umad3
crw-rw-rw- 1 root root 231, 192 Jan 26 16:26 uverbs0
crw-rw-rw- 1 root root 231, 193 Jan 26 16:26 uverbs1
crw-rw-rw- 1 root root 231, 194 Jan 26 16:26 uverbs2
crw-rw-rw- 1 root root 231, 195 Jan 26 16:26 uverbs3
$ ls -la /sys/class/infiniband
total 0
drwxr-xr-x 2 root root 0 Jan 11 13:52 .
drwxr-xr-x 82 root root 0 Jan 11 13:52 ..
lrwxrwxrwx 1 root root 0 Jan 11 13:53 mlx5_0 -> ../../devices/pci0000:11/0000:11:02.0/0000:13:00.0/infiniband/mlx5_0
lrwxrwxrwx 1 root root 0 Jan 11 13:53 mlx5_1 -> ../../devices/pci0000:11/0000:11:02.0/0000:13:00.1/infiniband/mlx5_1
lrwxrwxrwx 1 root root 0 Jan 11 13:52 mlx5_2 -> ../../devices/pci0000:ae/0000:ae:00.0/0000:af:00.0/infiniband/mlx5_2
lrwxrwxrwx 1 root root 0 Jan 11 13:52 mlx5_3 -> ../../devices/pci0000:ae/0000:ae:00.0/0000:af:00.1/infiniband/mlx5_3
警告： 所有Worker节点必须具有相同的配置和相同的PCIe卡槽位。 检查IB接口是否已启用。
K8s集群部署与配置
本方案中的Kubernetes集群将使用Kubespray从部署节点以非root用户账户安装。
SSH私钥与SSH免密登录
-
登录到部署节点作为部署用户（本例中为user），并创建SSH私钥以配置免密码认证，运行以下命令：
$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/user/.ssh/id_rsa): Created directory '/home/user/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/user/.ssh/id_rsa. Your public key has been saved in /home/user/.ssh/id_rsa.pub. The key fingerprint is: SHA256:PaZkvxV4K/h8q32zPWdZhG1VS0DSisAlehXVuiseLgA user@sl-depl-node The key's randomart image is: +---[RSA 2048]----+ | ...+oo+o..o| | .oo .o. o| | . .. . o +.| | E . o + . +| | . S = + o | | . o = + o .| | . o.o + o| | ..+.*. o+o| | oo*ooo.++| +----[SHA256]-----+ -
将您的SSH私钥（例如
~/.ssh/id_rsa）复制到部署中的所有节点，运行以下命令。示例：Sample: $ ssh-copy-id -i ~/.ssh/id_rsa user@192.168.1.40 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/user/.ssh/id_rsa.pub" The authenticity of host '192.168.1.40 (192.168.1.40)' can't be established. ECDSA key fingerprint is SHA256:uyglY5g0CgPNGDm+XKuSkFAbx0RLaPijpktANgXRlD8. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys user@192.168.1.40's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'user@192.168.1.40'" and check to make sure that only the key(s) you wanted were added. -
检查与部署中所有节点的SSH连接，运行以下命令：
Sample: $ ssh user@192.168.1.40 Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-52-generic x86_64) * 文档: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Mon Jan 11 17:23:23 IST 2021 System load: 0.0 Processes: 216 Usage of /: 6.5% of 68.40GB Users logged in: 1 Memory usage: 2% IP address for ens160: 192.168.1.40 Swap usage: 0% * Introducing self-healing high availability clusters in MicroK8s. Simple, hardened, Kubernetes for production, from RaspberryPi to DC. https://microk8s.io/high-availability 8 packages can be updated. 8 of these updates are security updates. To see these additional updates run: apt list --upgradable New release '20.04.1 LTS' available. Run 'do-release-upgrade' to upgrade to it. Your Hardware Enablement Stack (HWE) is supported until April 2023. Last login: Mon Jan 11 17:04:04 2021 from 192.168.1.43 user@node1:~$ exit
Kubespray部署与配置
-
在部署服务器上安装运行Kubespray与Ansible所需的依赖。
$ cd ~ $ sudo apt -y install python3-pip jq $ wget https://github.com/kubernetes-sigs/kubespray/archive/v2.14.2.tar.gz $ tar -zxf v2.14.2.tar.gz $ cd kubespray-2.14.2 $ sudo pip3 install -r requirements.txt注意： 后续命令的默认文件夹为
~/kubespray-2.14.2。 -
创建新的集群配置。
$ cp -rfp inventory/sample inventory/mycluster $ declare -a IPS=(192.168.1.40 192.168.1.10 192.168.1.11) $ CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}结果将创建
inventory/mycluster/hosts.yaml文件。错误： 请检查并修改主机配置文件
inventory/mycluster/hosts.yaml。以下是本部署的示例。
$ sudo vim inventory/mycluster/hosts.yaml all: hosts: node1: ansible_host: 192.168.1.40 ip: 192.168.1.40 access_ip: 192.168.1.40 node2: ansible_host: 192.168.1.10 ip: 192.168.1.10 access_ip: 192.168.1.10 node3: ansible_host: 192.168.1.11 ip: 192.168.1.11 access_ip: 192.168.1.11 children: kube-master: hosts: node1: kube-node: hosts: node2: node3: etcd: hosts: node1: k8s-cluster: children: kube-master: kube-node: calico-rr: hosts: {} -
检查并修改以下文件中的集群安装参数：
inventory/mycluster/group_vars/all/all.ymlinventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
在
inventory/mycluster/group_vars/all/all.yml中取消注释以下行，以便metrics能够接收集群资源使用数据：$ sudo vim inventory/mycluster/group_vars/all/all.yml ## The read-only port for the Kubelet to serve on with no authentication/authorization. Uncomment to enable. kube_read_only_port: 10255在
inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml中设置默认的Kubernetes CNI，通过设置所需的kube_network_plugin值（默认：calico），并通过设置kube_network_plugin_multus: true启用多网络。$ sudo vim inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml ... # Choose network plugin (cilium, calico, contiv, weave or flannel. Use cni for generic cni plugin) # Can also be set to 'cloud', which lets the cloud provider setup appropriate routing kube_network_plugin: calico # Setting multi_networking to true will install Multus: https://github.com/intel/multus-cni kube_network_plugin_multus: true ...
通过Kubespray Ansible Playbook部署K8s集群
通过Kubespray Ansible Playbook部署K8s集群
$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
此步骤的执行时间可能较长。
Playbook成功完成的示例输出如下：
PLAY RECAP ***************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0
node1 : ok=617 changed=101 unreachable=0 failed=0
node2 : ok=453 changed=58 unreachable=0 failed=0
node3 : ok=410 changed=53 unreachable=0 failed=0
Monday 30 November 2020 10:48:14 +0300 (0:00:00.265) 0:13:49.321 **********
===============================================================================
kubernetes/master : kubeadm | Initialize first master ------------------------------------------------------------------------------------ 55.94s
kubernetes/kubeadm : Join to cluster ----------------------------------------------------------------------------------------------------- 37.65s
kubernetes/master : Master | wait for kube-scheduler ------------------------------------------------------------------------------------- 21.97s
download : download_container | Download image if required ------------------------------------------------------------------------------- 21.34s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ------------------------------------------------------------------------------ 14.85s
kubernetes/preinstall : Update package management cache (APT) ---------------------------------------------------------------------------- 12.49s
download : download_file | Download item ------------------------------------------------------------------------------------------------- 11.45s
etcd : Install | Copy etcdctl binary from docker container ------------------------------------------------------------------------------- 10.57s
download : download_file | Download item -------------------------------------------------------------------------------------------------- 9.37s
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------- 9.18s
etcd : wait for etcd up ------------------------------------------------------------------------------------------------------------------- 8.78s
etcd : Configure | Check if etcd cluster is healthy --------------------------------------------------------------------------------------- 8.62s
download : download_file | Download item -------------------------------------------------------------------------------------------------- 8.24s
kubernetes-apps/network_plugin/multus : Multus | Start resources -------------------------------------------------------------------------- 7.32s
download : download_container | Download image if required -------------------------------------------------------------------------------- 6.61s
policy_controller/calico : Start of Calico kube controllers ------------------------------------------------------------------------------- 4.92s
download : download_file | Download item -------------------------------------------------------------------------------------------------- 4.76s
kubernetes-apps/cluster_roles : Apply workaround to allow all nodes with cert O=system:nodes to register ---------------------------------- 4.56s
download : download_container | Download image if required -------------------------------------------------------------------------------- 4.48s
download : download | Download files / images --------------------------------------------------------------------------------------------- 4.28s
使用 node-role.kubernetes.io/worker 标签标记工作节点，在 K8s Master 节点上执行：
# kubectl label nodes node2 node-role.kubernetes.io/worker=
# kubectl label nodes node3 node-role.kubernetes.io/worker=
K8s部署验证
在 K8s Master 节点上使用 ROOT 用户账户验证Kubernetes集群部署。
以下是K8s集群部署信息的输出示例，使用默认Kubespray配置和Calico Kubernetes CNI插件。
运行以下命令确保Kubernetes集群正确安装：
root@node1:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready master 16d v1.19.2 192.168.1.40 <none> Ubuntu 18.04.5 LTS 5.4.0-52-generic docker://19.3.12
node2 Ready worker 16d v1.19.2 192.168.1.10 <none> Ubuntu 20.04.1 LTS 5.4.0-56-generic docker://19.3.12
node3 Ready worker 16d v1.19.2 192.168.1.11 <none> Ubuntu 20.04.1 LTS 5.4.0-56-generic docker://19.3.12
root@node1:~# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-b885f5f4-8cr8s 1/1 Running 0 23m 192.168.1.10 node2 <none> <none>
calico-node-8bb6p 1/1 Running 1 24m 192.168.1.10 node2 <none> <none>
calico-node-9hnd4 1/1 Running 0 24m 192.168.1.40 node1 <none> <none>
calico-node-qm7z9 1/1 Running 1 24m 192.168.1.11 node3 <none> <none>
coredns-dff8fc7d-5n645 1/1 Running 0 30s 10.233.92.4 node3 <none> <none>
coredns-dff8fc7d-6qqcc 1/1 Running 0 32s 10.233.96.1 node2 <none> <none>
dns-autoscaler-66498f5c5f-vhz22 1/1 Running 0 23m 10.233.90.2 node1 <none> <none>
kube-apiserver-node1 1/1 Running 0 25m 192.168.1.40 node1 <none> <none>
kube-controller-manager-node1 1/1 Running 0 25m 192.168.1.40 node1 <none> <none>
kube-multus-ds-amd64-cgz57 1/1 Running 0 50s 192.168.1.40 node1 <none> <none>
kube-multus-ds-amd64-jwhwj 1/1 Running 0 50s 192.168.1.10 node2 <none> <none>
kube-multus-ds-amd64-qj4dh 1/1 Running 0 50s 192.168.1.11 node3 <none> <none>
kube-proxy-ddjjm 1/1 Running 0 24m 192.168.1.11 node3 <none> <none>
kube-proxy-j4228 1/1 Running 0 24m 192.168.1.10 node2 <none> <none>
kube-proxy-qsb2g 1/1 Running 0 25m 192.168.1.40 node1 <none> <none>
kube-scheduler-node1 1/1 Running 0 25m 192.168.1.40 node1 <none> <none>
kubernetes-dashboard-667c4c65f8-7xdxf 1/1 Running 0 23m 10.233.92.1 node3 <none> <none>
kubernetes-metrics-scraper-54fbb4d595-6mtgd 1/1 Running 0 23m 10.233.92.2 node3 <none> <none>
nginx-proxy-node2 1/1 Running 0 23m 192.168.1.10 node2 <none> <none>
nginx-proxy-node3 1/1 Running 0 24m 192.168.1.11 node3 <none> <none>
nodelocaldns-67s2w 1/1 Running 0 23m 192.168.1.10 node2 <none> <none>
nodelocaldns-mmb2r 1/1 Running 0 23m 192.168.1.11 node3 <none> <none>
nodelocaldns-zxlzl 1/1 Running 0 23m 192.168.1.40 node1 <none> <none>
为K8s集群安装NVIDIA GPU Operator
-
部署设备插件的首选方法是使用 helm 从 K8s Master 节点以 daemonset 形式部署。从官方安装脚本安装Helm。
# curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 # chmod 700 get_helm.sh # ./get_helm.sh -
添加NVIDIA Helm仓库。
# helm repo add nvidia https://nvidia.github.io/gpu-operator # helm repo update -
部署NVIDIA GPU Operator。
# helm install --wait --generate-name nvidia/gpu-operator "nvidia" has been added to your repositories root@sl-k8s-master:~# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "nvidia" chart repository Update Complete. ⎈Happy Helming!⎈ root@sl-k8s-master:~# helm install --wait --generate-name nvidia/gpu-operator NAME: gpu-operator-1610381204 LAST DEPLOYED: Mon Jan 11 18:06:50 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None# helm ls NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION gpu-operator-1610381204 default 1 2021-01-11 18:06:50.465874914 +0200 IST deployed gpu-operator-1.4.0 1.4.0 -
验证NVIDIA GPU Operator安装（等待约5-10分钟，直到Operator安装完成）。
# kubectl get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default gpu-operator-1610455631-node-feature-discovery-master-c8dbgrnpf 1/1 Running 0 6m52s 10.233.90.5 node1 <none> <none> default gpu-operator-1610455631-node-feature-discovery-worker-24zlr 1/1 Running 0 6m52s 10.233.92.4 node3 <none> <none> default gpu-operator-1610455631-node-feature-discovery-worker-47mbw 1/1 Running 0 6m52s 10.233.90.4 node1 <none> <none> default gpu-operator-1610455631-node-feature-discovery-worker-qmnmj 1/1 Running 0 6m52s 10.233.96.1 node2 <none> <none> default gpu-operator-7d4649d96c-2d2xj 1/1 Running 4 6m52s 10.233.90.3 node1 <none> <none> gpu-operator-resources gpu-feature-discovery-4h8dh 1/1 Running 0 75s 10.233.92.11 node3 <none> <none> gpu-operator-resources gpu-feature-discovery-c4fzh 1/1 Running 0 75s 10.233.96.5 node2 <none> <none> gpu-operator-resources nvidia-container-toolkit-daemonset-5hpng 1/1 Running 0 4m19s 10.233.96.2 node2 <none> <none> gpu-operator-resources nvidia-container-toolkit-daemonset-n7mkv 1/1 Running 0 4m19s 10.233.92.5 node3 <none> <none> gpu-operator-resources nvidia-dcgm-exporter-mjpg7 1/1 Running 0 2m5s 10.233.92.10 node3 <none> <none> gpu-operator-resources nvidia-dcgm-exporter-smmpp 1/1 Running 0 2m5s 10.233.96.4 node2 <none> <none> gpu-operator-resources nvidia-device-plugin-daemonset-7tvqh 1/1 Running 0 3m5s 10.233.92.7 node3 <none> <none> gpu-operator-resources nvidia-device-plugin-daemonset-p9djf 1/1 Running 0 3m5s 10.233.96.3 node2 <none> <none> gpu-operator-resources nvidia-device-plugin-validation 0/1 Completed 0 2m8s 10.233.92.8 node3 <none> <none> gpu-operator-resources nvidia-driver-daemonset-5cxb7 1/1 Running 0 5m41s 192.168.1.10 node2 <none> <none> gpu-operator-resources nvidia-driver-daemonset-b5dlv 1/1 Running 0 5m41s 192.168.1.11 node3 <none> <none> gpu-operator-resources nvidia-driver-validation 0/1 Completed 2 3m54s 10.233.92.6 node3 <none> <none> ...警告： 运行示例GPU应用程序：https://github.com/NVIDIA/gpu-operator#running-a-sample-gpu-application
SR-IOV Network Operator Installation for K8s Cluster
SR-IOV网络是Kubernetes集群的一个附加功能。要使其工作，需要配置不同的组件。
SR-IOV Network Operator部署步骤
- 在选定节点上初始化支持的SR-IOV网卡类型。
- 在选定节点上配置SR-IOV设备插件可执行文件。
- 在选定节点上配置SR-IOV CNI插件可执行文件。
- 管理主机上SR-IOV设备插件的配置。
- 为SR-IOV CNI插件生成net-att-def CR。
先决条件
在Master节点服务器上安装通用依赖项，运行以下命令：
# apt-get install jq make gcc -y
# snap install skopeo --edge --devmode
# snap install go --classic
# export GOPATH=$HOME/go
# export PATH=$GOPATH/bin:$PATH
以下是SR-IOV Network Operator安装的详细步骤说明。
-
安装Whereabouts CNI。
您可以使用DaemonSet安装此插件，命令如下：
# kubectl apply -f https://raw.githubusercontent.com/openshift/whereabouts-cni/master/doc/daemonset-install.yaml # kubectl apply -f https://raw.githubusercontent.com/openshift/whereabouts-cni/master/doc/whereabouts.cni.cncf.io_ippools.yaml # kubectl apply -f https://raw.githubusercontent.com/openshift/whereabouts-cni/master/doc/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml要确保插件正确安装，运行以下命令：
# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE ....... kube-system whereabouts-nsw6x 1/1 Running 0 22d kube-system whereabouts-pnhvn 1/1 Running 1 27d kube-system whereabouts-pv694 1/1 Running 0 27d -
克隆此GitHub仓库。
# cd /root # go get github.com/k8snetworkplumbingwg/sriov-network-operator -
部署Operator。
默认情况下，Operator将部署在Kubernetes集群的命名空间
sriov-network-operator中。您可以检查部署是否成功完成。# cd go/src/github.com/k8snetworkplumbingwg/sriov-network-operator/ # make deploy-setup-k8s -
检查SriovNetworkNodeState CR的状态，以找出集群中所有支持SR-IOV的设备。
在我们的部署中，我们选择了名为
ibs6f0的IB接口。# kubectl -n sriov-network-operator get sriovnetworknodestates.sriovnetwork.openshift.io node2 -o yaml ... deviceID: 101b driver: mlx5_core linkType: IB mac: 00:00:03:87:fe:80:00:00:00:00:00:00:98:03:9b:03:00:9f:cd:b6 mtu: 4092 name: ibs6f0 numVfs: 8 pciAddress: 0000:af:00.0 totalvfs: 8 vendor: 15b3 - deviceID: 101b driver: mlx5_core linkType: IB mac: 00:00:0b:0f:fe:80:00:00:00:00:00:00:98:03:9b:03:00:9f:cd:b7 mtu: 4092 name: ibs6f1 pciAddress: 0000:af:00.1 totalvfs: 8 vendor: 15b3 ... -
使用选定的IB接口创建SriovNetworkNodePolicy CR。
# kubectl create -f - <<EOF apiVersion: sriovnetwork.openshift.io/v1 kind: SriovNetworkNodePolicy metadata: name: ib-policy namespace: sriov-network-operator spec: nodeSelector: feature.node.kubernetes.io/network-sriov.capable: "true" resourceName: ib_sriov priority: 99 numVfs: 8 nicSelector: pfNames: ["ibs6f0"] deviceType: netdevice isRdma: true EOF
cd /root
mkdir YAMLs
cd YAMLs/
vim policy.yaml
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: policy-ib0
namespace: sriov-network-operator
spec:
resourceName: "mlnx_ib0"
nodeSelector:
feature.node.kubernetes.io/custom-rdma.available: "true"
priority: 10
numVfs: 8
nicSelector:
vendor: "15b3"
deviceID: "101b"
pfNames: [ "ibs6f0" ]
isRdma: true
linkType: ib
应用 SriovNetworkNodePolicy。
# kubectl apply -f policy.yaml
检查策略激活后的Operator部署。
# kubectl -n sriov-network-operator get all
NAME READY STATUS RESTARTS AGE
pod/sriov-cni-bzdsv 2/2 Running 0 59s
pod/sriov-cni-vsjbt 2/2 Running 0 9m6s
pod/sriov-device-plugin-9ghjx 1/1 Running 0 9m6s
pod/sriov-device-plugin-hkzct 1/1 Running 0 12s
pod/sriov-network-config-daemon-8x749 1/1 Running 0 22m
pod/sriov-network-config-daemon-k7plr 1/1 Running 0 61s
pod/sriov-network-operator-79b8bb586f-ptgr6 1/1 Running 0 22m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/sriov-cni 2 2 2 2 2 beta.kubernetes.io/os=linux,node-role.kubernetes.io/worker= 9m6s
daemonset.apps/sriov-device-plugin 2 2 2 2 2 beta.kubernetes.io/os=linux,node-role.kubernetes.io/worker= 9m6s
daemonset.apps/sriov-network-config-daemon 2 2 2 2 2 beta.kubernetes.io/os=linux,node-role.kubernetes.io/worker= 22m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/sriov-network-operator 1/1 1 1 22m
NAME DESIRED CURRENT READY AGE
replicaset.apps/sriov-network-operator-79b8bb586f 1 1 1 22m
创建一个名为 sriov-ib0.yaml 的网络附加定义。
# vim sriov-ib0.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
annotations:
k8s.v1.cni.cncf.io/resourceName: openshift.io/mlnx_ib0
name: sriovib0
namespace: default
spec:
config: |-
{
"cniVersion": "0.3.1",
"name": "sriovib0",
"plugins": [
{
"type": "ib-sriov",
"link_state": "enable",
"rdmaIsolation": true,
"ibKubernetesEnabled": false,
"ipam": {
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.101.0/24"
}
}
]
}
应用网络附加定义。
# kubectl apply -f sriov-ib0.yaml
验证网络附加定义的安装。
# kubectl get network-attachment-definitions.k8s.cni.cncf.io
NAME AGE
sriovib0 28d
检查Worker节点2。
# kubectl describe nodes node2
Name: node2
Roles: worker
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
feature.node.kubernetes.io/cpu-cpuid.ADX=true
feature.node.kubernetes.io/cpu-cpuid.AESNI=true
feature.node.kubernetes.io/cpu-cpuid.AVX=true
feature.node.kubernetes.io/cpu-cpuid.AVX2=true
feature.node.kubernetes.io/cpu-cpuid.AVX512BW=true
feature.node.kubernetes.io/cpu-cpuid.AVX512CD=true
feature.node.kubernetes.io/cpu-cpuid.AVX512DQ=true
feature.node.kubernetes.io/cpu-cpuid.AVX512F=true
feature.node.kubernetes.io/cpu-cpuid.AVX512VL=true
feature.node.kubernetes.io/cpu-cpuid.FMA3=true
feature.node.kubernetes.io/cpu-cpuid.HLE=true
feature.node.kubernetes.io/cpu-cpuid.IBPB=true
feature.node.kubernetes.io/cpu-cpuid.MPX=true
feature.node.kubernetes.io/cpu-cpuid.RTM=true
feature.node.kubernetes.io/cpu-cpuid.STIBP=true
feature.node.kubernetes.io/cpu-cpuid.VMX=true
feature.node.kubernetes.io/cpu-rdt.RDTCMT=true
feature.node.kubernetes.io/cpu-rdt.RDTL3CA=true
feature.node.kubernetes.io/cpu-rdt.RDTMBA=true
feature.node.kubernetes.io/cpu-rdt.RDTMBM=true
feature.node.kubernetes.io/cpu-rdt.RDTMON=true
feature.node.kubernetes.io/custom-rdma.available=true
feature.node.kubernetes.io/custom-rdma.capable=true
feature.node.kubernetes.io/kernel-config.NO_HZ=true
feature.node.kubernetes.io/kernel-config.NO_HZ_IDLE=true
feature.node.kubernetes.io/kernel-version.full=5.4.0-56-generic
feature.node.kubernetes.io/kernel-version.major=5
feature.node.kubernetes.io/kernel-version.minor=4
feature.node.kubernetes.io/kernel-version.revision=0
feature.node.kubernetes.io/memory-numa=true
feature.node.kubernetes.io/pci-0300_102b.present=true
feature.node.kubernetes.io/pci-0302_10de.present=true
feature.node.kubernetes.io/pci-0302_10de.sriov.capable=true
feature.node.kubernetes.io/storage-nonrotationaldisk=true
feature.node.kubernetes.io/system-os_release.ID=ubuntu
feature.node.kubernetes.io/system-os_release.VERSION_ID=20.04
feature.node.kubernetes.io/system-os_release.VERSION_ID.major=20
feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04
kubernetes.io/arch=amd64
kubernetes.io/hostname=node2
kubernetes.io/os=linux
node-role.kubernetes.io/worker=
nvidia.com/gpu.present=true
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
nfd.node.kubernetes.io/extended-resources:
nfd.node.kubernetes.io/feature-labels:
cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.AVX512BW,cpu-cpuid.AVX512CD,cpu-cpuid.AVX512DQ,cpu-cpuid.AVX512F,cpu-...
nfd.node.kubernetes.io/worker.version: v0.6.0
node.alpha.kubernetes.io/ttl: 0
sriovnetwork.openshift.io/state: Idle
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 01 Dec 2020 17:22:46 +0200
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: node2
AcquireTime: <unset>
RenewTime: Wed, 30 Dec 2020 14:30:59 +0200
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Mon, 07 Dec 2020 16:00:20 +0200 Mon, 07 Dec 2020 16:00:20 +0200 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 30 Dec 2020 14:31:06 +0200 Mon, 07 Dec 2020 15:59:48 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 30 Dec 2020 14:31:06 +0200 Mon, 07 Dec 2020 15:59:48 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 30 Dec 2020 14:31:06 +0200 Mon, 07 Dec 2020 15:59:48 +0200 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 30 Dec 2020 14:31:06 +0200 Mon, 07 Dec 2020 15:59:53 +0200 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.1.10
Hostname: node2
Capacity:
cpu: 32
ephemeral-storage: 229700940Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 197754972Ki
nvidia.com/gpu: 2
openshift.io/mlnx_ib0: 8
pods: 110
Allocatable:
cpu: 31900m
ephemeral-storage: 211692385954
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 197402572Ki
nvidia.com/gpu: 2
openshift.io/mlnx_ib0: 8
pods: 110
System Info:
Machine ID: 646aa8cc13d14c47ac112babe9daf77c
System UUID: 37383638-3330-5a43-3238-3435304d3647
Boot ID: 049266f0-98a5-48a8-b225-e118d1508ae1
Kernel Version: 5.4.0-56-generic
OS Image: Ubuntu 20.04.1 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.12
Kubelet Version: v1.19.2
Kube-Proxy Version: v1.19.2
PodCIDR: 10.233.65.0/24
PodCIDRs: 10.233.65.0/24
Non-terminated Pods: (15 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default gpu-operator-1606837056-node-feature-discovery-worker-sjh9c 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d
gpu-operator-resources gpu-feature-discovery-lfzpz 0 (0%) 0
# RDG：基于InfiniBand网络的NVIDIA GPU Kubernetes集群部署
创建于2020年11月16日，作者：Boris Kovalev, Vitaliy Razinkov
## 范围
本参考部署指南（RDG）说明如何构建高性能的Kubernetes集群，结合NVIDIA GPU和InfiniBand网络。
## 检查Worker节点
### 检查Worker节点2
kubectl describe nodes node2
Name: node2 Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux feature.node.kubernetes.io/cpu-cpuid.ADX=true feature.node.kubernetes.io/cpu-cpuid.AESNI=true feature.node.kubernetes.io/cpu-cpuid.AVX=true feature.node.kubernetes.io/cpu-cpuid.AVX2=true feature.node.kubernetes.io/cpu-cpuid.AVX512BW=true feature.node.kubernetes.io/cpu-cpuid.AVX512CD=true feature.node.kubernetes.io/cpu-cpuid.AVX512DQ=true feature.node.kubernetes.io/cpu-cpuid.AVX512F=true feature.node.kubernetes.io/cpu-cpuid.AVX512VL=true feature.node.kubernetes.io/cpu-cpuid.FMA3=true feature.node.kubernetes.io/cpu-cpuid.HLE=true feature.node.kubernetes.io/cpu-cpuid.IBPB=true feature.node.kubernetes.io/cpu-cpuid.MPX=true feature.node.kubernetes.io/cpu-cpuid.RTM=true feature.node.kubernetes.io/cpu-cpuid.STIBP=true feature.node.kubernetes.io/cpu-cpuid.VMX=true feature.node.kubernetes.io/cpu-hardware_multithreading=true feature.node.kubernetes.io/cpu-rdt.RDTCMT=true feature.node.kubernetes.io/cpu-rdt.RDTL3CA=true feature.node.kubernetes.io/cpu-rdt.RDTMBA=true feature.node.kubernetes.io/cpu-rdt.RDTMBM=true feature.node.kubernetes.io/cpu-rdt.RDTMON=true feature.node.kubernetes.io/custom-rdma.available=true feature.node.kubernetes.io/custom-rdma.capable=true feature.node.kubernetes.io/kernel-config.NO_HZ=true feature.node.kubernetes.io/kernel-config.NO_HZ_IDLE=true feature.node.kubernetes.io/kernel-version.full=5.4.0-56-generic feature.node.kubernetes.io/kernel-version.major=5 feature.node.kubernetes.io/kernel-version.minor=4 feature.node.kubernetes.io/kernel-version.revision=0 feature.node.kubernetes.io/memory-numa=true feature.node.kubernetes.io/pci-0300_102b.present=true feature.node.kubernetes.io/pci-0302_10de.present=true feature.node.kubernetes.io/pci-0302_10de.sriov.capable=true feature.node.kubernetes.io/storage-nonrotationaldisk=true feature.node.kubernetes.io/system-os_release.ID=ubuntu feature.node.kubernetes.io/system-os_release.VERSION_ID=20.04 feature.node.kubernetes.io/system-os_release.VERSION_ID.major=20 feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04 kubernetes.io/arch=amd64 kubernetes.io/hostname=node2 kubernetes.io/os=linux node-role.kubernetes.io/worker= nvidia.com/gpu.present=true Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock nfd.node.kubernetes.io/extended-resources: nfd.node.kubernetes.io/feature-labels: cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.AVX512BW,cpu-cpuid.AVX512CD,cpu-cpuid.AVX512DQ,cpu-cpuid.AVX512F,cpu-... nfd.node.kubernetes.io/worker.version: v0.6.0 node.alpha.kubernetes.io/ttl: 0 sriovnetwork.openshift.io/state: Idle volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 01 Dec 2020 17:22:53 +0200 Taints: Unschedulable: false Lease: HolderIdentity: node2 AcquireTime: RenewTime: Wed, 30 Dec 2020 14:36:15 +0200 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message
NetworkUnavailable False Mon, 07 Dec 2020 15:40:51 +0200 Mon, 07 Dec 2020 15:40:51 +0200 CalicoIsUp Calico is running on this node MemoryPressure False Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:42 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:42 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:42 +0200 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:44 +0200 KubeletReady kubelet is posting ready status. AppArmor enabled Addresses: InternalIP: 192.168.1.11 Hostname: node2 Capacity: cpu: 64 ephemeral-storage: 229698892Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 197747532Ki nvidia.com/gpu: 2 openshift.io/mlnx_ib0: 7 pods: 110 Allocatable: cpu: 63900m ephemeral-storage: 211690498517 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 197395132Ki nvidia.com/gpu: 2 openshift.io/mlnx_ib0: 0 pods: 110 System Info: Machine ID: c9f34445383f445eb44cd27fb90634e8 System UUID: 37383638-3330-5a43-3238-3435304d3643 Boot ID: 20be7b74-ce7d-4180-b904-48135f823819 Kernel Version: 5.4.0-56-generic OS Image: Ubuntu 20.04.1 LTS Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.12 Kubelet Version: v1.19.2 Kube-Proxy Version: v1.19.2 PodCIDR: 10.233.66.0/24 PodCIDRs: 10.233.66.0/24 Non-terminated Pods: (17 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
default gpu-operator-1606837056-node-feature-discovery-worker-4mxl8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d default rdma-test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d gpu-operator-resources gpu-feature-discovery-kbkwt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-container-toolkit-daemonset-fmvgk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-dcgm-exporter-nlwhx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-device-plugin-daemonset-k7c99 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-driver-daemonset-sslgt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d kube-system calico-node-hmsml 150m (0%) 300m (0%) 64M (0%) 500M (0%) 28d kube-system coredns-84646c885d-zh86b 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 22d kube-system kube-multus-ds-amd64-4r25b 100m (0%) 100m (0%) 90Mi (0%) 90Mi (0%) 28d kube-system kube-proxy-bbd99 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d kube-system nginx-proxy-node3 25m (0%) 0 (0%) 32M (0%) 0 (0%) 28d kube-system nodelocaldns-xw9tq 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 28d kube-system whereabouts-pnhvn 100m (0%) 100m (0%) 50Mi (0%) 50Mi (0%) 28d sriov-network-operator sriov-cni-kcz44 0 (0%) 0 (0%) 0 (0%) 0 (0%) 10m sriov-network-operator sriov-device-plugin-gmmdr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4m50s sriov-network-operator sriov-network-config-daemon-7qtmt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits
cpu 475m (1%) 500m (1%) memory 316200960 (0%) 825058560 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) nvidia.com/gpu 0 0 openshift.io/mlnx_ib0 0 0 Events:
### 检查Worker节点3
kubectl describe nodes node3
Name: node3 Roles: worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux feature.node.kubernetes.io/cpu-cpuid.ADX=true feature.node.kubernetes.io/cpu-cpuid.AESNI=true feature.node.kubernetes.io/cpu-cpuid.AVX=true feature.node.kubernetes.io/cpu-cpuid.AVX2=true feature.node.kubernetes.io/cpu-cpuid.AVX512BW=true feature.node.kubernetes.io/cpu-cpuid.AVX512CD=true feature.node.kubernetes.io/cpu-cpuid.AVX512DQ=true feature.node.kubernetes.io/cpu-cpuid.AVX512F=true feature.node.kubernetes.io/cpu-cpuid.AVX512VL=true feature.node.kubernetes.io/cpu-cpuid.FMA3=true feature.node.kubernetes.io/cpu-cpuid.HLE=true feature.node.kubernetes.io/cpu-cpuid.IBPB=true feature.node.kubernetes.io/cpu-cpuid.MPX=true feature.node.kubernetes.io/cpu-cpuid.RTM=true feature.node.kubernetes.io/cpu-cpuid.STIBP=true feature.node.kubernetes.io/cpu-cpuid.VMX=true feature.node.kubernetes.io/cpu-hardware_multithreading=true feature.node.kubernetes.io/cpu-rdt.RDTCMT=true feature.node.kubernetes.io/cpu-rdt.RDTL3CA=true feature.node.kubernetes.io/cpu-rdt.RDTMBA=true feature.node.kubernetes.io/cpu-rdt.RDTMBM=true feature.node.kubernetes.io/cpu-rdt.RDTMON=true feature.node.kubernetes.io/custom-rdma.available=true feature.node.kubernetes.io/custom-rdma.capable=true feature.node.kubernetes.io/kernel-config.NO_HZ=true feature.node.kubernetes.io/kernel-config.NO_HZ_IDLE=true feature.node.kubernetes.io/kernel-version.full=5.4.0-56-generic feature.node.kubernetes.io/kernel-version.major=5 feature.node.kubernetes.io/kernel-version.minor=4 feature.node.kubernetes.io/kernel-version.revision=0 feature.node.kubernetes.io/memory-numa=true feature.node.kubernetes.io/pci-0300_102b.present=true feature.node.kubernetes.io/pci-0302_10de.present=true feature.node.kubernetes.io/pci-0302_10de.sriov.capable=true feature.node.kubernetes.io/storage-nonrotationaldisk=true feature.node.kubernetes.io/system-os_release.ID=ubuntu feature.node.kubernetes.io/system-os_release.VERSION_ID=20.04 feature.node.kubernetes.io/system-os_release.VERSION_ID.major=20 feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=04 kubernetes.io/arch=amd64 kubernetes.io/hostname=node3 kubernetes.io/os=linux node-role.kubernetes.io/worker= nvidia.com/gpu.present=true Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock nfd.node.kubernetes.io/extended-resources: nfd.node.kubernetes.io/feature-labels: cpu-cpuid.ADX,cpu-cpuid.AESNI,cpu-cpuid.AVX,cpu-cpuid.AVX2,cpu-cpuid.AVX512BW,cpu-cpuid.AVX512CD,cpu-cpuid.AVX512DQ,cpu-cpuid.AVX512F,cpu-... nfd.node.kubernetes.io/worker.version: v0.6.0 node.alpha.kubernetes.io/ttl: 0 sriovnetwork.openshift.io/state: Idle volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Tue, 01 Dec 2020 17:22:53 +0200 Taints: Unschedulable: false Lease: HolderIdentity: node3 AcquireTime: RenewTime: Wed, 30 Dec 2020 14:36:15 +0200 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message
NetworkUnavailable False Mon, 07 Dec 2020 15:40:51 +0200 Mon, 07 Dec 2020 15:40:51 +0200 CalicoIsUp Calico is running on this node MemoryPressure False Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:42 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:42 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:42 +0200 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Wed, 30 Dec 2020 14:36:19 +0200 Mon, 07 Dec 2020 15:40:44 +0200 KubeletReady kubelet is posting ready status. AppArmor enabled Addresses: InternalIP: 192.168.1.11 Hostname: node3 Capacity: cpu: 64 ephemeral-storage: 229698892Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 197747532Ki nvidia.com/gpu: 2 openshift.io/mlnx_ib0: 7 pods: 110 Allocatable: cpu: 63900m ephemeral-storage: 211690498517 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 197395132Ki nvidia.com/gpu: 2 openshift.io/mlnx_ib0: 0 pods: 110 System Info: Machine ID: c9f34445383f445eb44cd27fb90634e8 System UUID: 37383638-3330-5a43-3238-3435304d3643 Boot ID: 20be7b74-ce7d-4180-b904-48135f823819 Kernel Version: 5.4.0-56-generic OS Image: Ubuntu 20.04.1 LTS Operating System: linux Architecture: amd64 Container Runtime Version: docker://19.3.12 Kubelet Version: v1.19.2 Kube-Proxy Version: v1.19.2 PodCIDR: 10.233.66.0/24 PodCIDRs: 10.233.66.0/24 Non-terminated Pods: (17 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
default gpu-operator-1606837056-node-feature-discovery-worker-4mxl8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d default rdma-test-pod 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d gpu-operator-resources gpu-feature-discovery-kbkwt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-container-toolkit-daemonset-fmvgk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-dcgm-exporter-nlwhx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-device-plugin-daemonset-k7c99 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d gpu-operator-resources nvidia-driver-daemonset-sslgt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d kube-system calico-node-hmsml 150m (0%) 300m (0%) 64M (0%) 500M (0%) 28d kube-system coredns-84646c885d-zh86b 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 22d kube-system kube-multus-ds-amd64-4r25b 100m (0%) 100m (0%) 90Mi (0%) 90Mi (0%) 28d kube-system kube-proxy-bbd99 0 (0%) 0 (0%) 0 (0%) 0 (0%) 28d kube-system nginx-proxy-node3 25m (0%) 0 (0%) 32M (0%) 0 (0%) 28d kube-system nodelocaldns-xw9tq 100m (0%) 0 (0%) 70Mi (0%) 170Mi (0%) 28d kube-system whereabouts-pnhvn 100m (0%) 100m (0%) 50Mi (0%) 50Mi (0%) 28d sriov-network-operator sriov-cni-kcz44 0 (0%) 0 (0%) 0 (0%) 0 (0%) 10m sriov-network-operator sriov-device-plugin-gmmdr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4m50s sriov-network-operator sriov-network-config-daemon-7qtmt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 22d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits
cpu 475m (1%) 500m (1%) memory 316200960 (0%) 825058560 (0%) ephemeral-storage 0 (0%) 0 (0%) hugepages-1Gi 0 (0%) 0 (0%) hugepages-2Mi 0 (0%) 0 (0%) nvidia.com/gpu 0 0 openshift.io/mlnx_ib0 0 0 Events:
# RDG：基于InfiniBand网络的NVIDIA GPU Kubernetes集群部署
## 部署验证
1. 创建示例Deployment（容器镜像需包含CUDA和InfiniBand性能工具）：
```yaml
# vim sample-depl.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-pod
labels:
app: sriov
spec:
replicas: 2
selector:
matchLabels:
app: sriov
template:
metadata:
labels:
app: sriov
annotations:
k8s.v1.cni.cncf.io/networks: sriovib0
spec:
containers:
- image: <Container Image Name>
name: mlnx-inbox-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
requests:
openshift.io/mlnx_ib0: '1'
nvidia.com/gpu: 1
limits:
openshift.io/mlnx_ib0: '1'
nvidia.com/gpu: 1
command:
- sh
- -c
- sleep inf
-
部署示例Pod：
# kubectl apply -f sample-depl.yaml -
验证Pod是否运行：
# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES gpu-operator-1610455631-node-feature-discovery-master-c8dbgrnpf 1/1 Running 0 20h 10.233.90.5 node1 <none> <none> gpu-operator-1610455631-node-feature-discovery-worker-24zlr 1/1 Running 4 20h 10.233.92.31 node3 <none> <none> gpu-operator-1610455631-node-feature-discovery-worker-47mbw 1/1 Running 1 20h 10.233.90.4 node1 <none> <none> gpu-operator-1610455631-node-feature-discovery-worker-qmnmj 1/1 Running 2 20h 10.233.96.20 node2 <none> <none> gpu-operator-7d4649d96c-2d2xj 1/1 Running 4 20h 10.233.90.3 node1 <none> <none> sample-pod-65b94586b4-8k784 1/1 Running 0 17h 10.233.92.37 node3 <none> <none> sample-pod-65b94586b4-8xn6m 1/1 Running 0 17h 10.233.96.27 node2 <none> <none> -
检查容器中的GPU：
# kubectl exec -it sample-pod-65b94586b4-8k784 -- bash root@sample-pod-65b94586b4-8k784:/tmp# nvidia-smi Wed Jan 13 09:38:49 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: N/A | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla T4 On | 00000000:37:00.0 Off | 0 | | N/A 48C P8 16W / 70W | 0MiB / 15109MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ root@sample-pod-65b94586b4-8k784:/# exit exit -
检查网络适配器：
# kubectl exec -it sample-pod-65b94586b4-8k784 -- bash root@sample-pod-65b94586b4-8k784:/tmp# ip a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 4: eth0@if48: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 8a:87:13:3b:bd:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.233.92.37/32 scope global eth0 valid_lft forever preferred_lft forever 49: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 256 link/infiniband 00:00:0e:e3:fe:80:00:00:00:00:00:00:60:cc:fa:35:1d:14:a4:cc brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 192.168.101.1/24 brd 192.168.101.255 scope global net1 valid_lft forever preferred_lft forever root@sample-pod-65b94586b4-8k784:/tmp# ibdev2netdev mlx5_9 port 1 ==> net1 (Up) root@sample-pod-65b94586b4-8k784:/# exit exit -
运行RDMA Write - ib_write_bw 带宽压力测试（基于IB）：
Server Client ib_write_bw -a -d mlx5_0 &ib_write_bw -a -F $server_IP -d mlx5_0 --report_gbits打开两个终端连接到K8s Master节点。
-
在第一个终端（服务端）中运行以下命令：
# kubectl exec -it sample-pod-65b94586b4-8k784 -- bash root@sample-pod-65b94586b4-8k784:/tmp# ibdev2netdev mlx5_9 port 1 ==> net1 (Up) root@sample-pod-65b94586b4-8k784:/tmp# ib_write_bw -a -d mlx5_9 & [1] 1081 root@sample-pod-65b94586b4-8k784:/tmp# ************************************ * Waiting for client to connect... * ************************************ -
在第二个终端（客户端）中运行以下命令：
# kubectl exec -it sample-pod-65b94586b4-8xn6m -- bash root@sample-pod-65b94586b4-8xn6m:/tmp# ibdev2netdev mlx5_7 port 1 ==> net1 (Up) root@sample-pod-65b94586b4-8xn6m:/tmp# ib_write_bw -a -F 192.168.101.1 -d mlx5_7 --report_gbits -
结果：
Server: --------------------------------------------------------------------------------------- RDMA_Write BW Test Dual-port : OFF Device : mlx5_9 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF CQ Moderation : 100 Mtu : 4096[B] Link type : IB Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0x0d QPN 0x0ec6 PSN 0xa49cc3 RKey 0x0e0400 VAddr 0x007fa17b1ef000 remote address: LID 0x0c QPN 0x0bac PSN 0xa6c47a RKey 0x0a0400 VAddr 0x007f54c7554000 --------------------------------------------------------------------------------------- #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 8388608 5000 96.58 96.54 0.001439 --------------------------------------------------------------------------------------- Client: --------------------------------------------------------------------------------------- RDMA_Write BW Test Dual-port : OFF Device : mlx5_7 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF TX depth : 128 CQ Moderation : 100 Mtu : 4096[B] Link type : IB Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0x0c QPN 0x0ba9 PSN 0xf563c8 RKey 0x0a0400 VAddr 0x007fd3ff9cb000 remote address: LID 0x0d QPN 0x0ec3 PSN 0x9445de RKey 0x0e0400 VAddr 0x007fac5f879000
-
RDG：基于InfiniBand网络的NVIDIA GPU Kubernetes集群部署
创建于2020年11月16日，作者：Boris Kovalev, Vitaliy Razinkov
范围
本文档（参考部署指南）介绍了如何构建高性能的Kubernetes集群，利用InfiniBand网络和NVIDIA GPU加速机器学习与HPC工作负载。
性能测试结果
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
2 5000 0.11 0.11 6.680528
4 5000 0.23 0.22 6.961376
8 5000 0.48 0.43 6.746956
16 5000 0.96 0.86 6.703131
32 5000 1.90 1.80 7.021876
64 5000 3.82 3.59 7.014856
128 5000 7.45 7.02 6.853576
256 5000 14.69 14.38 7.019255
512 5000 28.51 27.75 6.774283
1024 5000 54.31 48.41 5.909477
2048 5000 82.91 80.73 4.927545
4096 5000 95.75 95.62 2.918237
8192 5000 95.88 95.88 1.462960
16384 5000 96.18 96.15 0.733546
32768 5000 96.49 96.37 0.367604
65536 5000 96.54 96.53 0.184124
131072 5000 96.56 96.55 0.092081
262144 5000 96.56 96.56 0.046041
524288 5000 96.57 96.57 0.023024
1048576 5000 96.58 96.57 0.011512
2097152 5000 96.58 96.52 0.005753
4194304 5000 96.58 96.56 0.002878
8388608 5000 96.58 96.56 0.001439
---------------------------------------------------------------------------------------
删除示例部署
- 退出容器并删除部署：
# root@sample-pod-65b94586b4-lz5x9:/tmp# exit
# kubectl delete -f sample-depl.yaml
完成！
作者
 |
Boris KovalevBoris Kovalev has worked for the past several years as a 解决方案 Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the NVIDIA Documents website. |
 |
Vitaliy RazinkovVitaliy Razinkov is a 解决方案 Architect on the NVIDIA Networking team, specializing in complex Kubernetes, OpenShift, and Microsoft solutions. With over 25 years of experience in senior technical roles, he brings deep expertise in designing and implementing advanced infrastructures. Vitaliy has authored several reference design guides on Microsoft technologies, RoCE/RDMA-accelerated machine learning in Kubernetes/OpenShift, and containerized solutions—all available on the NVIDIA Networking 文档 site. |
相关文档
-
Kubernetes on Docker in Docker (UNDER TRYING, INSIDE the NVIDIA)
-
How-to: Deploy Horovod framework with RDMA/RoCE acceleration in Docker container.
-
How-to: Deploy RDMA accelerated Linux container (LXD) over InfiniBand fabric.
-
How-to: Deploy RDMA accelerated Docker container over InfiniBand fabric.
-
RDG: RoCE accelerated K8s cluster deployment for ML and HPC workloads.
-
RDG: RDMA accelerated Horovod framework in Docker on HPE Apollo 6500 servers over InfiniBand fabric.
-
QSG of Building Docker Image with Compiled Nvidia Network DPDK PMD
-
QSG: Kubernetes Cluster Deployment on InfiniBand Fabric with RDMA Shared Device Plugin.
-
Cookbook for setting up Apollo-compatible microk8s (on Ubuntu18.04)
-
RDG: Storage Disaggregation for K8s ML workloads with Excelero NVMesh SDS and Bluefield NVMe SNAP.