使用Rivermax和DeepStream在加速K8s集群上部署媒体流应用的参考部署指南
创建于2022年6月15日。范围：以下参考部署指南（RDG）展示了在加速Kubernetes集群上部署Rivermax和DeepStream流媒体应用。
文档目录
创建于2022年6月15日。
范围
以下参考部署指南（RDG）展示了在加速Kubernetes集群上部署Rivermax和DeepStream流媒体应用。
缩写和首字母缩略词
| 术语 | 定义 | 术语 | 定义 |
|---|---|---|---|
| CDN | 内容分发网络 | LLDP | 链路层发现协议 |
| CNI | 容器网络接口 | NFD | 节点特性发现 |
| CR | 自定义资源 | NCCL | NVIDIA 集合通信库 |
| CRD | 自定义资源定义 | OCI | 开放容器倡议 |
| CRI | 容器运行时接口 | PF | 物理功能 |
| DHCP | 动态主机配置协议 | QSG | 快速入门指南 |
| DNS | 域名系统 | RDG | 参考部署指南 |
| DP | 设备插件 | RDMA | 远程直接内存访问 |
| DS | DeepStream | RoCE | 融合以太网上的RDMA |
| IPAM | IP地址管理 | SR-IOV | 单根输入输出虚拟化 |
| K8s | Kubernetes | VF | 虚拟功能 |
引言
本指南提供了K8s集群部署的完整解决方案周期，包括技术概述、设计、组件选择、部署步骤和应用工作负载示例。该解决方案将在标准服务器上交付。NVIDIA端到端以太网基础设施用于监督工作负载。
在本指南中，我们使用NVIDIA GPU Operator和NVIDIA Network Operator来管理K8s集群中GPU和网络组件的部署和配置。这些组件允许您使用CUDA、RDMA和GPUDirect技术加速工作负载。
本指南展示了具有两个K8s工作节点的K8s集群设计，并提供了部署K8s集群的详细说明。本指南假设为全新部署。
本文档面向需要为客户部署Rivermax流媒体应用的资深媒体和娱乐广播系统管理员、系统工程师和解决方案架构师。
参考文献
解决方案架构
关键组件和技术
-
NVIDIA DGX™ A100是所有AI工作负载的通用系统，提供前所未有的计算密度、性能和灵活性，是世界上第一个5 petaFLOPS AI系统。NVIDIA DGX A100采用世界上最先进的加速器NVIDIA A100 Tensor Core GPU，使企业能够将训练、推理和分析整合到一个统一、易于部署的AI基础设施中，并可直接访问NVIDIA AI专家。
-
10/25/40/50/100/200和400G以太网网卡
业界领先的NVIDIA® ConnectX®系列智能网卡提供先进的硬件卸载和加速。NVIDIA以太网网卡为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供最高的ROI和最低的总拥有成本。
-
NVIDIA® LinkX®线缆和收发器产品系列提供业界最完整的10、25、40、50、100、200和400GbE以太网以及100、200和400Gb/s InfiniBand产品，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。
-
灵活的外形，16到128个物理端口，支持1GbE到400GbE速度。
基于针对性能和可扩展性优化的突破性硅技术，NVIDIA Spectrum交换机非常适合构建高性能、高性价比、高效的云数据中心网络、以太网存储结构和深度学习互连。NVIDIA将基于业界领先的专用集成电路（ASIC）技术的NVIDIA Spectrum™交换机与多种现代网络操作系统选择相结合，包括NVIDIA Cumulus® Linux、SONiC和NVIDIA Onyx®。
-
Kubernetes是一个开源容器编排平台，用于容器化应用的部署自动化、扩展和管理。
-
Kubespray由Ansible剧本、清单、配置工具和通用OS/Kubernetes集群配置管理任务的领域知识组成，并提供：
- 高可用集群
- 可组合属性
- 支持大多数流行的Linux发行版
-
NVIDIA GPU Operator使用Kubernetes中的operator框架来自动化管理配置GPU所需的所有NVIDIA软件组件。这些组件包括NVIDIA驱动程序（以启用CUDA）、GPU的Kubernetes设备插件、NVIDIA容器运行时、自动节点标记、基于DCGM的监控等。
-
NVIDIA Network Operator使用Kubernetes中的operator框架来自动化管理配置网络所需的所有NVIDIA软件组件。这些组件包括NVIDIA网卡驱动程序、容器网络接口（CNI）插件、RDMA、SR-IOV等。
NVIDIA Network Operator简化了Kubernetes集群中NVIDIA网络资源的配置和管理。该操作符自动安装所需的主机网络软件，汇集所有必要组件以提供高速网络连接。这些组件包括NVIDIA网络驱动、Kubernetes设备插件、CNI插件、IP地址管理（IPAM）插件等。NVIDIA Network Operator与NVIDIA GPU Operator协同工作，为可扩展的GPU计算集群提供高吞吐量、低延迟的网络。
-
NVIDIA CUDA CUDA®是NVIDIA开发的并行计算平台和编程模型，用于在图形处理单元（GPU）上进行通用计算。借助CUDA，开发人员可以通过利用GPU的强大功能显著加速计算应用。在GPU加速的应用中，工作负载的顺序部分在CPU上运行（针对单线程性能优化），而计算密集型部分则在数千个GPU核心上并行运行。
-
NVIDIA Rivermax SDK NVIDIA Rivermax为任何媒体和数据流用例提供独特的基于IP的解决方案。Rivermax与NVIDIA GPU加速计算技术相结合，为媒体与娱乐（M&E）、广播、医疗、智慧城市等领域的广泛应用解锁创新。Rivermax利用NVIDIA ConnectX和BlueField DPU硬件流加速技术，实现直接数据传输到GPU和从GPU传输，为流工作负载提供最佳的吞吐量和延迟，同时最小化CPU利用率。
-
NVIDIA DeepStream SDK NVIDIA DeepStream允许快速开发和部署视觉AI应用和服务。DeepStream提供多平台、可扩展、TLS加密的安全性，可部署在本地、边缘和云端。它为基于AI的多传感器处理、视频、音频和图像理解提供了完整的流分析工具包。DeepStream主要面向视觉AI开发者、软件合作伙伴、初创公司和OEM厂商，用于构建IVA应用和服务。
-
网络化媒体开放规范（NMOS） NMOS规范是一系列开放、免费的规范，使IP基础设施上的媒体设备之间能够实现互操作性。核心规范IS-04注册与发现和IS-05设备连接管理提供了统一的机制，使媒体设备和服务能够向网络通告其能力，并让控制系统配置设备发送器和接收器之间的视频、音频和数据流。NMOS是可扩展的，例如包括音频通道映射、事件和提词信息交换以及API安全规范，利用IT最佳实践。有可用的开源NMOS实现，NVIDIA在DeepStream SDK中提供了免费的NMOS节点库。
逻辑设计
逻辑设计包括以下部分：
- 运行Kubespray的部署节点，用于部署Kubernetes集群
- K8s主节点，运行所有Kubernetes管理组件
- 带有NVIDIA GPU和NVIDIA ConnectX-6Dx网卡的K8s工作节点
- 高速以太网结构（辅助K8s网络）
- 部署和K8s管理网络
应用逻辑设计
在本指南中，我们部署了以下应用：
- Rivermax媒体节点
- NMOS注册控制器
- DeepStream网关
- 时间同步服务
- 用于内部GUI访问的VNC应用
软件栈组件
物料清单
本指南中使用以下硬件设置来构建包含两个K8s工作节点的K8s集群。
注意： 您可以根据网络拓扑和软件栈使用任何合适的硬件。
部署与配置
网络/结构
本RDG描述了具有多个K8s工作节点的K8s集群部署。高性能网络是Kubernetes集群的辅助网络，需要L2网络拓扑。部署/管理网络拓扑和DNS/DHCP网络服务是IT基础设施的一部分。本指南不涵盖组件安装过程和配置。
网络IP配置
以下是服务器名称及其相关网络配置。
| 服务器/交换机类型 | 服务器/交换机名称 | 高速网络 | 管理网络 |
|---|---|---|---|
| 部署节点 | depserver | N/A | eth0: DHCP 192.168.100.202 |
| K8s主节点 | node1 | N/A | eth0: DHCP |
| 节点 | 主机名 | 高速网络接口 | 管理网络接口 |
|---|---|---|---|
| K8s部署节点 | depserver | - | eth0: DHCP 192.168.100.29 |
| K8s工作节点1 | node2 | enp57s0f0: 无IP设置 | eth0: DHCP 192.168.100.34 |
| K8s工作节点2 | node3 | enp57s0f0: 无IP设置 | eth0: DHCP 192.168.100.39 |
| 高速交换机 | switch | - | mgmt0: DHCP 192.168.100.49 |
- enpXXs0f0 高速网络接口无需额外配置。
布线
每个K8s工作节点上，仅将NVIDIA网卡的第一个端口通过NVIDIA LinkX DAC线缆连接到NVIDIA交换机，构成高性能网络。
下图展示了构建K8s集群所需的布线。
网络配置
交换机配置如下：
##
## Running database "initial"
## Generated at 2022/05/10 15:49:25 +0200
## Hostname: switch
## Product release: 3.9.3202
##
##
## Running-config temporary prefix mode setting
##
no cli default prefix-modes enable
##
## Interface Ethernet configuration
##
interface ethernet 1/1-1/32 speed 100GxAuto force
interface ethernet 1/1-1/32 switchport mode hybrid
##
## VLAN configuration
##
vlan 2
vlan 1001
vlan 2 name "RiverData"
vlan 1001 name "PTP"
interface ethernet 1/1-1/32 switchport hybrid allowed-vlan all
interface ethernet 1/5 switchport access vlan 1001
interface ethernet 1/7 switchport access vlan 1001
interface ethernet 1/5 switchport hybrid allowed-vlan add 2
interface ethernet 1/7 switchport hybrid allowed-vlan add 2
##
## STP configuration
##
no spanning-tree
##
## L3 configuration
##
interface vlan 1001
interface vlan 1001 ip address 172.20.0.1/24 primary
##
## IGMP Snooping configuration
##
ip igmp snooping unregistered multicast forward-to-mrouter-ports
ip igmp snooping
vlan 1001 ip igmp snooping
vlan 1001 ip igmp snooping querier
interface ethernet 1/5 ip igmp snooping fast-leave
interface ethernet 1/7 ip igmp snooping fast-leave
##
## Local user account configuration
##
username admin password 7 $6$mSW1WwYI$M5xfvsphrTRht6J2ByfF.J475tq8YuGKR6K1FwSgvkdb1QQFZbx/PtqK.GVJEBoMcmXsnB57QycP7jSp.Hy/Q.
username monitor password 7 $6$V/Og9kzY$qc.oU2Ma9MPJClZlbvymOrb1wtE0N5yfQYPamhzRYeN2npVY/lOE5iisHUpxNqm3Ku8lIWDTPiO/bklyCMi2o.
##
## AAA remote server configuration
##
# ldap bind-password ********
ldap vrf default enable
radius-server vrf default enable
# radius-server key ********
tacacs-server vrf default enable
# tacacs-server key ********
##
## Password restriction configuration
##
no password hardening enable
##
## SNMP configuration
##
snmp-server vrf default enable
##
## Network management configuration
##
# web proxy auth basic password ********
clock timezone Asia Middle_East Jerusalem
ntp vrf default disable
terminal sysrq enable
web vrf default enable
##
## PTP protocol
##
protocol ptp
ptp priority1 1
ptp vrf default enable
interface ethernet 1/5 ptp enable
interface ethernet 1/7 ptp enable
interface vlan 1001 ptp enable
##
## X.509 certificates configuration
##
#
# Certificate name system-self-signed, ID ca9888a2ed650c5c4bd372c055bdc6b4da65eb1e
# (public-cert config omitted since private-key config is hidden)
##
## Persistent prefix mode setting
##
cli default prefix-modes enable
主机
通用配置
通用前提条件：
-
硬件 确保所有K8s工作节点具有完全相同的硬件规格（详见BoM）。
-
主机BIOS 确认使用支持SR-IOV的服务器平台，并查阅服务器平台供应商文档中的BIOS设置以启用SR-IOV。
-
主机操作系统 所有服务器应安装Ubuntu Server 20.04操作系统，并包含OpenSSH服务器软件包。
-
Kubernetes经验 必须熟悉Kubernetes集群架构。
注意： 确保K8s工作节点的BIOS设置为最大性能。 所有K8s工作节点必须具有完全相同的NIC PCIe位置，并暴露相同的接口名称。
主机操作系统前提条件
确保在部署Ubuntu Server 20.04操作系统期间创建了非root用户depuser。
运行以下命令更新Ubuntu软件包：
服务器控制台
$ sudo apt-get update
$ sudo apt-get install linux-image-lowlatency -y
$ sudo apt-get upgrade -y
$ sudo reboot
为非root用户depuser添加无密码sudo权限。
在本解决方案中，在**/etc/sudoers**文件末尾添加了以下行：
服务器控制台
$ sudo vim /etc/sudoers
#includedir /etc/sudoers.d
#K8s cluster deployment user with sudo privileges without password
depuser ALL=(ALL) NOPASSWD:ALL
OFED安装与配置
仅需在K8s工作节点上安装OFED。下载最新OFED版本请访问Linux Drivers (nvidia.com)。
下载和安装步骤如下，所有步骤需要root权限。
OFED安装后，请重启节点。
服务器控制台
wget https://content.mellanox.com/ofed/MLNX_OFED-5.5-1.0.3.2/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso
mount -o loop ./MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.iso /mnt/
/mnt/mlnxofedinstall --vma --without-fw-update
reboot
K8s集群部署
本解决方案中的Kubernetes集群使用Kubespray从部署节点以非root用户depuser身份安装。
SSH私钥与无密码登录
以部署用户（本例中为depuser）登录到部署节点，运行以下命令创建SSH私钥以配置无密码认证：
部署节点控制台
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/depuser/.ssh/id_rsa):
Created directory '/home/depuser/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/depuser/.ssh/id_rsa
Your public key has been saved in /home/depuser/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:IfcjdT/spXVHVd3n6wm1OmaWUXGuHnPmvqoXZ6WZYl0 depuser@depserver
The key's randomart image is:
+---[RSA 3072]----+
| *|
| .*|
| . o . . o=|
| o + . o +E|
| S o .**O|
| . .o=OX=|
| . o%*.|
| O.o.|
| .*.ooo|
+----[SHA256]-----+
复制您的SSH私钥
key，例如~/.ssh/id_rsa，复制到部署中的所有节点，运行以下命令（示例）：
部署节点控制台
$ ssh-copy-id depuser@192.168.100.29
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/depuser/.ssh/id_rsa.pub"
The authenticity of host '192.168.100.29 (192.168.100.29)' can't be established.
ECDSA key fingerprint is SHA256:6nhUgRlt9gY2Y2ofukUqE0ltH+derQuLsI39dFHe0Ag.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
depuser@192.168.100.29's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'depuser@192.168.100.29'"
and check to make sure that only the key(s) you wanted were added.
验证您可以通过无密码SSH连接到部署中的所有节点，运行以下命令（示例）：
部署节点控制台
$ ssh depuser@192.168.100.29
Kubespray部署与配置
通用设置
要在部署节点上安装运行Kubespray与Ansible所需的依赖，请运行以下命令：
部署节点控制台
$ cd ~
$ sudo apt -y install python3-pip jq
$ wget https://github.com/kubernetes-sigs/kubespray/archive/v2.18.1.tar.gz
$ tar -zxf v2.18.1.tar.gz
$ cd kubespray-2.18.1
$ sudo pip3 install -r requirements.txt
警告：后续命令的默认文件夹为~/kubespray-2.18.1。 要下载最新Kubespray版本，请访问 Releases · kubernetes-sigs/kubespray · GitHub。
部署定制
创建新的集群配置和主机配置文件。 将下面的IP地址替换为您的节点IP地址：
部署节点控制台
$ cp -rfp inventory/sample inventory/mycluster
$ declare -a IPS=(192.168.100.29 192.168.100.34 192.168.100.39)
$ CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
结果将创建inventory/mycluster/hosts.yaml文件。 检查并更改文件中的主机配置。以下是此部署的示例：
inventory/mycluster/hosts.yaml
all:
hosts:
node1:
ansible_host: 192.168.100.29
ip: 192.168.100.29
access_ip: 192.168.100.29
node2:
ansible_host: 192.168.100.34
ip: 192.168.100.34
access_ip: 192.168.100.34
node3:
ansible_host: 192.168.100.39
ip: 192.168.100.39
access_ip: 192.168.100.39
children:
kube_control_plane:
hosts:
node1:
kube_node:
hosts:
node2:
node3:
etcd:
hosts:
node1:
k8s_cluster:
children:
kube_control_plane:
kube_node:
calico_rr:
hosts: {}
使用KubeSpray Ansible Playbook部署集群
运行以下命令启动部署过程：
部署节点控制台
$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
K8s集群部署需要一些时间才能完成，请确保playbook日志中没有错误。
以下是成功结果的示例：
...
PLAY RECAP ***************************************************************************************************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
node1 : ok=501 changed=111 unreachable=0 failed=0 skipped=1131 rescued=0 ignored=2
node2 : ok=360 changed=40 unreachable=0 failed=0 skipped=661 rescued=0 ignored=1
node3 : ok=360 changed=40 unreachable=0 failed=0 skipped=660 rescued=0 ignored=1
Sunday 9 May 2021 19:39:17 +0000 (0:00:00.064) 0:06:54.711 ********
===============================================================================
kubernetes/control-plane : kubeadm | Initialize first master ----------------------------------------------------------------------------------------- 28.13s
kubernetes/control-plane : Master | wait for kube-scheduler ------------------------------------------------------------------------------------------ 12.78s
download : download_container | Download image if required ------------------------------------------------------------------------------------------- 10.56s
container-engine/containerd : ensure containerd packages are installed -------------------------------------------------------------------------------- 9.48s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 9.36s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 9.08s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 9.05s
download : download_file | Download item -------------------------------------------------------------------------------------------------------------- 8.91s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 8.47s
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------------------- 8.30s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 7.49s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 7.39s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ------------------------------------------------------------------------------------------- 7.07s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 5.99s
container-engine/containerd : ensure containerd repository is enabled --------------------------------------------------------------------------------- 5.59s
container-engine/crictl : download_file | Download item ----------------------------------------------------------------------------------------------- 5.45s
download : download_file | Download item -------------------------------------------------------------------------------------------------------------- 5.34s
kubernetes-apps/ansible : Kubernetes Apps | Lay Down CoreDNS templates -------------------------------------------------------------------------------- 5.00s
download : download_container | Download image if required -------------------------------------------------------------------------------------------- 4.95s
download : download_file | Download item -------------------------------------------------------------------------------------------------------------- 4.50s
K8s集群定制与验证
现在K8s集群已部署，可以从任何K8s主节点使用root用户连接到K8s集群。
账户或从安装了KUBECTL命令并配置了KUBECONFIG=的其他服务器进行自定义部署。
在我们的指南中，我们从K8s主节点使用root用户账户继续部署：
标记工作节点：
主节点控制台
$ kubectl label nodes node2 node-role.kubernetes.io/worker=
$ kubectl label nodes node3 node-role.kubernetes.io/worker=
注意： K8s工作节点标记是正确安装NVIDIA网络操作符所必需的。
以下是使用Calico CNI插件的K8s集群部署信息输出示例。
要确保Kubernetes集群正确安装，请运行以下命令：
主节点控制台
## 获取集群节点状态
kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready control-plane,master 9d v1.22.8 192.168.100.29 <none> Ubuntu 20.04.4 LTS 5.4.0-109-generic containerd://1.5.8
node2 Ready worker 9d v1.22.8 192.168.100.34 <none> Ubuntu 20.04.4 LTS 5.4.0-109-lowlatency containerd://1.5.8
node3 Ready worker 9d v1.22.8 192.168.100.39 <none> Ubuntu 20.04.4 LTS 5.4.0-109-lowlatency containerd://1.5.8
## 获取系统Pod状态
kubectl -n kube-system get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-5788f6558-bm5h9 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
calico-node-4f748 1/1 Running 0 9d 192.168.100.34 node2 <none> <none>
calico-node-jhbjh 1/1 Running 0 9d 192.168.100.39 node3 <none> <none>
calico-node-m78p6 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
coredns-8474476ff8-dczww 1/1 Running 0 9d 10.233.90.23 node1 <none> <none>
coredns-8474476ff8-ksvkd 1/1 Running 0 9d 10.233.96.234 node2 <none> <none>
dns-autoscaler-5ffdc7f89d-h6nc8 1/1 Running 0 9d 10.233.90.20 node1 <none> <none>
kube-apiserver-node1 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
kube-controller-manager-node1 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
kube-proxy-2bq45 1/1 Running 0 9d 192.168.100.34 node2 <none> <none>
kube-proxy-4c8p7 1/1 Running 0 9d 192.168.100.39 node3 <none> <none>
kube-proxy-j226w 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
kube-scheduler-node1 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
nginx-proxy-node2 1/1 Running 0 9d 192.168.100.34 node2 <none> <none>
nginx-proxy-node3 1/1 Running 0 9d 192.168.100.39 node3 <none> <none>
nodelocaldns-9rffq 1/1 Running 0 9d 192.168.100.39 node3 <none> <none>
nodelocaldns-fdnr7 1/1 Running 0 9d 192.168.100.34 node2 <none> <none>
nodelocaldns-qhpxk 1/1 Running 0 9d 192.168.100.29 node1 <none> <none>
为K8s集群安装NVIDIA GPU操作符
部署GPU操作符的首选方法是使用helm从K8s主节点进行。要安装helm，只需使用以下命令：
$ snap install helm --classic
添加NVIDIA GPU操作符Helm仓库。
$ helm repo add nvidia https://nvidia.github.io/gpu-operator
$ helm repo update
部署NVIDIA GPU操作符。
GPU操作符应启用GPUDirect内核模块进行部署 - driver.rdma.enabled=true。
$ helm install --wait --generate-name -n gpu-operator --create-namespace nvidia/gpu-operator --set driver.rdma.enabled=true --set driver.rdma.useHostMofed=true
$ helm ls -n gpu-operator
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
gpu-operator-1652190420 gpu-operator 1 2022-05-10 13:47:01.106147933 +0000 UTC deployed gpu-operator-v1.10.0 v1.10.0 NAME
安装Helm chart后，检查Pod状态以确保所有容器正在运行且验证完成：
$ kubectl get pod -n gpu-operator -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpu-feature-discovery-bcc22 1/1 Running 1 (3d8h ago) 5d18h 10.233.96.3 node2 <none> <none>
gpu-feature-discovery-vl68h 1/1 Running 0 5d18h 10.233.92.58 node3 <none> <none>
gpu-operator-1652190420-node-feature-discovery-master-5b5fx8zlx 1/1 Running 1 (4m5s ago) 5d18h 10.233.90.17 node1 <none> <none>
gpu-operator-1652190420-node-feature-discovery-worker-czsb4 1/1 Running 0 4s 10.233.92.75 node3 <none> <none>
gpu-operator-1652190420-node-feature-discovery-worker-fnlj6 1/1 Running 0 4s 10.233.96.253 node2 <none> <none>
gpu-operator-1652190420-node-feature-discovery-worker-r44r8 1/1 Running 1 (4m5s ago) 5d18h 10.233.90.22 node1 <none> <none>
gpu-operator-6497cbf9cd-vcsrg 1/1 Running 1 (4m6s ago) 5d18h 10.233.90.19 node1 <none> <none>
nvidia-container-toolkit-daemonset-4h9dr 1/1 Running 0 5d18h 10.233.96.246 node2 <none> <none>
nvidia-container-toolkit-daemonset-rv7sn 1/1 Running 1 (5d18h ago) 5d18h 10.233.92.50 node3 <none> <none>
nvidia-cuda-validator-kr6q9 0/1 Completed 0 5d18h 10.233.92.61 node3 <none> <none>
nvidia-cuda-validator-zb4p8 0/1 Completed 0 5d18h 10.233.96.4 node2 <none> <none>
nvidia-dcgm-exporter-5hdzh 1/1 Running 0 5d18h 10.233.96.198 node2 <none> <none>
nvidia-dcgm-exporter-lnqzb 1/1 Running 0 5d18h 10.233.92.57 node3 <none> <none>
nvidia-device-plugin-daemonset-dxgnz 1/1 Running 0 5d18h 10.233.92.62 node3 <none> <none>
nvidia-device-plugin-daemonset-w692b 1/1 Running 0 5d18h 10.233.96.9 node2 <none> <none>
nvidia-device-plugin-validator-pqns8 0/1 Completed 0 5d18h 10.233.92.64 node3 <none> <none>
nvidia-device-plugin-validator-sgtmt 0/1 Completed 0 5d18h 10.233.96.10 node2 <none> <none>
nvidia-driver-daemonset-l9x4n 2/2 Running 1 (2d19h ago) 5d18h 10.233.92.30 node3 <none> <none>
nvidia-driver-daemonset-tf2tl 2/2 Running 5 (2d21h ago) 5d18h 10.233.96.244 node2 <none> <none>
nvidia-operator-validator-p6794 1/1 Running 0 5d18h 10.233.96.6 node2 <none> <none>
nvidia-operator-validator-xjrg9 1/1 Running 0 5d18h 10.233.92.54 node3 <none> <none>
NVIDIA Network Operator Installation
The NVIDIA Network Operator leverages Kubernetes CRDs and Operator SDK to manage networking-related components in order to enable fast networking and RDMA for workloads in K8s cluster. The Fast Network is a secondary network of the K8s cluster for applications that require high bandwidth or low latency.
To make it work, several components need to be provisioned and configured. The Helm is required for the Network Operator deployment.
Add the NVIDIA Network Operator Helm repository:
## Add REPO
helm repo add mellanox https://mellanox.github.io/network-operator \
&& helm repo update
Create the values.yaml file to customize the Network Operator deployment (example):
nfd:
enabled: true
sriovNetworkOperator:
enabled: true
ofedDriver:
deploy: false
nvPeerDriver:
deploy: false
rdmaSharedDevicePlugin:
deploy: false
sriovDevicePlugin:
deploy: false
deployCR: true
secondaryNetwork:
deploy: true
cniPlugins:
deploy: true
multus:
deploy: true
ipamPlugin:
deploy: true
Deploy the operator:
helm install -f ./values.yaml -n network-operator --create-namespace --wait mellanox/network-operator --generate-name
helm ls -n network-operator
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
network-operator-1648457278 network-operator 1 2022-03-28 08:47:59.548667592 +0000 UTC deployed network-operator-1.1.0 v1.1.0
Once the Helm chart is installed, check the status of the pods to ensure all the containers are running:
## PODs status in namespace - network-operator
kubectl -n network-operator get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
network-operator-1648457278-5885dbfff5-wjgsc 1/1 Running 0 5m 10.233.90.15 node1 <none> <none>
network-operator-1648457278-node-feature-discovery-master-zbcx8 1/1 Running 0 5m 10.233.90.16 node1 <none> <none>
network-operator-1648457278-node-feature-discovery-worker-kk4qs 1/1 Running 0 5m 10.233.90.18 node1 <none> <none>
network-operator-1648457278-node-feature-discovery-worker-n44b6 1/1 Running 0 5m 10.233.92.221 node3 <none> <none>
network-operator-1648457278-node-feature-discovery-worker-xhzfw 1/1 Running 0 5m 10.233.96.233 node2 <none> <none>
network-operator-1648457278-sriov-network-operator-5cd4bdb6mm9f 1/1 Running 0 5m 10.233.90.21 node1 <none> <none>
sriov-device-plugin-cxnrl 1/1 Running 0 5m 192.168.100.34 node2 <none> <none>
sriov-device-plugin-djlmn 1/1 Running 0 5m 192.168.100.39 node3 <none> <none>
sriov-network-config-daemon-rgfvk 3/3 Running 0 5m 192.168.100.39 node3 <none> <none>
sriov-network-config-daemon-zzchs 3/3 Running 0 5m 192.168.100.34 node2 <none> <none>
## PODs status in namespace - nvidia-network-operator-resources
kubectl -n nvidia-network-operator-resources get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cni-plugins-ds-snf6x 1/1 Running 0 5m 192.168.100.39 node3 <none> <none>
cni-plugins-ds-zjb27 1/1 Running 0 5m 192.168.100.34 node2 <none> <none>
kube-multus-ds-mz7nd 1/1 Running 0 5m 192.168.100.39 node3 <none> <none>
kube-multus-ds-xjxgd 1/1 Running 0 5m 192.168.100.34 node2 <none> <none>
whereabouts-jgt24 1/1 Running 0 5m 192.168.100.34 node2 <none> <none>
whereabouts-sphx4 1/1 Running 0 5m 192.168.100.39 node3 <none> <none>
High-Speed Network Configuration
After installing the operator, please check the SriovNetworkNodeState CRs to see all SR-IOV-enabled devices in your node.
In this deployment, the network interface has been chosen with the following name: enp57s0f0.
To review the interface status please use the following command:
NICs status
## NIC status
kubectl -n network-operator get sriovnetworknodestates.sriovnetwork.openshift.io node2 -o yaml
...
status:
interfaces:
deviceID: 101d
driver: mlx5_core
eSwitchMode: legacy
linkSpeed: 100000 Mb/s
linkType: ETH
mac: 0c:42:a1:2b:73:fa
mtu: 9000
name: enp57s0f0
numVfs: 8
pciAddress: "0000:39:00.0"
totalvfs: 8
vendor: 15b3
- deviceID: 101d
driver: mlx5_core
...
Create SriovNetworkNodePolicy CR for chosen network interface - policy.yaml file, by specifying the chosen interface in the 'nicSelector'.
According to application design VF0 allotted into a separate pool from the rest of VFn:
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw1
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: timepool
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp57s0f0#0-0" ]
deviceType: netdevice
isRdma: true
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: mlnxnics-sw2
namespace: network-operator
spec:
nodeSelector:
feature.node.kubernetes.io/custom-rdma.capable: "true"
resourceName: rdmapool
priority: 99
mtu: 9000
numVfs: 8
nicSelector:
pfNames: [ "enp57s0f0#1-7" ]
deviceType: netdevice
isRdma: true
Deploy policy.yaml:
kubectl apply -f policy.yaml
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw1 created
sriovnetworknodepolicy.sriovnetwork.openshift.io/mlnxnics-sw2 created
Note: This step takes a while. This depends on the amount of K8s Worker Nodes to apply the configuration, and the number of VFs for each selected network interface.
Create an SriovNetwork CR for chosen network interface - network.yaml file which refers to the 'resourceName' defined in SriovNetworkNodePolicy.
In this example below created:
- timenet - K8s network name for PTP time sync
- rdmanet - K8s network name with dynamic IPAM
- rdma-static - K8s network name with static IPAM
apiVersion: "sriovnetwork.openshift.io/v1"
kind: SriovNetwork
metadata:
name: timenet
namespace: network-operator
spec:
resourceName: timepool
networkNamespace: default
ipam: |
{
"type": "whereabouts",
"datastore": "kubernetes",
"range": "192.168.10.0/24",
"exclude": [
"192.168.10.0/30"
]
}
---
apiVersion: "sriovnetwork.openshift.io/v1"
kind: SriovNetwork
metadata:
name: rdmanet
namespace: network-operator
spec:
resourceName: rdmapool
networkNamespace: default
ipam: |
{
"type": "whereabouts",
"datastore": "kubernetes",
"range": "192.168.20.0/24",
"exclude": [
"192.168.20.0/30"
]
}
---
apiVersion: "sriovnetwork.openshift.io/v1"
kind: SriovNetwork
metadata:
name: rdma-static
namespace: network-operator
spec:
resourceName: rdmapool
networkNamespace: default
ipam: |
{
"type": "static"
}
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: timenet
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": {"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"},
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "172.20.0.0/24",
"exclude": [ "172.20.0.1/32" ]
}
networkNamespace: default
resourceName: timepool
trust: "on"
vlan: 0
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: rdmanet
namespace: network-operator
spec:
ipam: |
{
"datastore": "kubernetes",
"kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
"log_file": "/tmp/whereabouts.log",
"log_level": "debug",
"type": "whereabouts",
"range": "192.168.102.0/24",
"exclude": [ "192.168.102.254/32", "192.168.102.253/32" ]
}
networkNamespace: default
resourceName: rdmapool
vlan: 2
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
name: rdmanet-static
namespace: network-operator
spec:
ipam: |
{
"type": "static"
}
networkNamespace: default
resourceName: rdmapool
vlan: 2
部署 network.yaml：
kubectl apply -f network.yaml
sriovnetwork.sriovnetwork.openshift.io/timenet created
sriovnetwork.sriovnetwork.openshift.io/rdmanet created
sriovnetwork.sriovnetwork.openshift.io/rdmanet-static created
管理HugePages
Kubernetes支持Pod中的应用分配和消费预分配的HugePages。节点会自动发现并报告所有HugePages资源作为可调度资源。有关K8s HugePages管理的更多信息，请参考此处。
为了分配HugePages，需要在/etc/default/grub中修改GRUB_CMDLINE_LINUX_DEFAULT参数。以下设置在启动时分配2MB * 8192页 = 16GB HugePages：
/etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=2M hugepagesz=2M hugepages=8192"
...
运行update-grub以应用配置到grub并重启服务器：
Worker Node console
# update-grub
# reboot
服务器重启后，从主节点通过命令检查hugepages分配：
Master Node console
# kubectl describe nodes node2
...
Capacity:
cpu: 48
ephemeral-storage: 459923528Ki
hugepages-1Gi: 0
hugepages-2Mi: 16Gi
memory: 264050900Ki
nvidia.com/gpu: 2
nvidia.com/rdmapool: 7
nvidia.com/timepool: 1
pods: 110
Allocatable:
cpu: 46
ephemeral-storage: 423865522704
hugepages-1Gi: 0
hugepages-2Mi: 16Gi
memory: 246909140Ki
nvidia.com/gpu: 2
nvidia.com/rdmapool: 7
nvidia.com/timepool: 1
pods: 110
...
启用CPU和拓扑管理
CPU Manager管理CPU组并将工作负载约束到特定CPU上。
CPU Manager对于具有以下某些属性的工作负载非常有用：
- 需要尽可能多的CPU时间
- 对处理器缓存未命中敏感
- 是低延迟网络应用
- 与其他进程协调并受益于共享单个处理器缓存
Topology Manager根据收集的拓扑信息提示，基于配置的Topology Manager策略和Pod请求的资源，决定Pod是否可以在节点上被接受或拒绝。为了获得最佳性能，需要与CPU隔离以及内存和设备局部性相关的优化。
Topology Manager对于使用硬件加速器支持延迟关键型执行和高吞吐量并行计算的工作负载非常有用。
注意： 要使用Topology Manager，必须使用具有static策略的CPU Manager。
有关更多信息，请参考控制节点上的CPU管理策略和控制节点上的拓扑管理策略。
为了启用CPU Manager和Topology Manager，请在kubelet配置文件/etc/kubernetes/kubelet-config.yaml中添加以下行：
/etc/kubernetes/kubelet-config.yaml
...
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 10s
topologyManagerPolicy: single-numa-node
featureGates:
CPUManager: true
TopologyManager: true
由于cpuManagerPolicy的更改，请删除/var/lib/kubelet/cpu_manager_state并在每个受影响的K8s工作节点上重启kubelet服务。
Worker Node console
# rm -f /var/lib/kubelet/cpu_manager_state
# service kubelet restart
应用
以下提供了K8s特定组件和K8s YAML配置文件，用于在K8s集群中部署Rivermax应用。
注意： 为了正确执行应用，需要Rivermax许可证。要获取许可证，请查看Rivermax许可证生成指南。
注意： 要从容器仓库下载Rivermax应用容器镜像和应用管道，您需要注册并登录Rivermax门户，点击“Get Started”。
Rivermax许可证
将Rivermax许可证作为configmap值上传到K8s集群。
kubectl create configmap rivermax-config --from-file=rivermax.lic=./rivermax.lic
媒体节点应用
此Pod定义包含AMWA网络化媒体开放规范（NMOS）的实现，使用NMOS Rivermax节点实现。有关AMWA、NMOS和网络化媒体孵化器的更多信息，请参考http://amwa.tv/。有关Rivermax SDK的更多信息，请参考https://developer.nvidia.com/networking/Rivermax。
以下是用于媒体节点部署的YAML配置文件。请填写您的容器文件名和注册密钥。
apiVersion: v1
kind: ConfigMap
metadata:
name: river-config
data:
container-config: |-
#media_node JSON file to run
config_json=/var/home/config.json
#Output registry stdout/stderr output to a log inside container
log_output=FALSE
#Update/insert label parameter with container hostname on entrypoint
script run
update_label=TRUE
#Allow these network interfaces in /etc/avahi/avahi-daemon.conf
allow_interfaces=net1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: "mnc"
labels:
apps: rivermax
spec:
replicas: 3
selector:
matchLabels:
app: rivermax
template:
metadata:
labels:
app: rivermax
annotations:
k8s.v1.cni.cncf.io/networks: rdmanet
spec:
containers:
- command:
image: < media node container image >
name: "medianode"
env:
- name: DISPLAY
value: "192.168.102.253:0.0"
resources:
requests:
nvidia.com/rdmapool: 1
hugepages-2Mi: 4Gi
memory: 8Gi
cpu: 4
limits:
nvidia.com/rdmapool: 1
hugepages-2Mi: 4Gi
memory: 8Gi
cpu: 4
securityContext:
capabilities:
add: [ "IPC_LOCK", "SYS_RESOURCE", "NET_RAW","NET_ADMIN" ]
volumeMounts:
- name: config
mountPath: /var/home/ext/
- name: licconfig
mountPath: /opt/mellanox/rivermax/
- mountPath: /hugepages
name: hugepage
- mountPath: /dev/shm
name: dshm
volumes:
- name: config
configMap:
name: river-config
- name: licconfig
configMap:
name: rivermax-config
- name: hugepage
emptyDir:
medium: HugePages
- name: dshm
emptyDir: {
medium: 'Memory',
sizeLimit: '4Gi'
}
imagePullSecrets:
- name: < Container registry secret >
NMOS控制器
AMWA NMOS控制器是一种能够与NMOS API交互的设备，NMOS API是一系列用于专业应用网络化媒体的开放规范。NMOS控制器可以使用通用方法和协议在IP基础设施上发现、注册、连接和管理媒体设备。NMOS控制器还可以处理事件和计数、音频通道映射、授权以及其他属于NMOS路线图的功能。更多信息，请参阅README.md。
apiVersion: v1
kind: Pod
metadata:
name: nmos-cpp
labels:
app.kubernetes.io/name: nmos
annotations:
k8s.v1.cni.cncf.io/networks: |
[
{ "name": "rdmanet-static",
"ips": [ "192.168.102.254/24" ]
}
]
spec:
containers:
- name: nmos-pod
image: docker.io/rhastie/nmos-cpp:latest
env:
- name: RUN_NODE
value: "true"
resources:
requests:
cpu: 2
memory: 1Gi
nvidia.com/rdmapool: 1
limits:
cpu: 2
memory: 1Gi
nvidia.com/rdmapool: 1
ports:
- containerPort: 8010
name: port-8010
- containerPort: 8011
name: port-8011
- containerPort: 11000
name: port-11000
- containerPort: 11001
name: port-11001
- containerPort: 1883
name: port-1883
- containerPort: 5353
name: port-5353
protocol: UDP
DeepStream媒体网关
DeepStream SDK的应用之一是将原始数据编码为SRT流。该应用程序可以从摄像头或文件捕获视频帧，使用H.264或H.265编解码器进行编码，并通过SRT协议通过网络发送。SRT代表安全可靠传输（Secure Reliable Transport），是一种低延迟、安全的流媒体技术。该应用程序可用于远程监控、直播或视频会议等场景。
以下是媒体网关部署的YAML配置文件。请填写您的容器文件名和注册表密钥。
apiVersion: v1
kind: Pod
metadata:
name: ds-rmax
labels:
name: dsrmax-app
annotations:
k8s.v1.cni.cncf.io/networks: rdmanet
spec:
containers:
- name: dsrmax
image: < DeepStream media gateway container image >
command:
- sh
- -c
- sleep inf
env:
- name: DISPLAY
value: "192.168.102.253:0.0"
ports:
- containerPort: 7001
name: udp-port
securityContext:
capabilities:
add: [ "IPC_LOCK", "SYS_RESOURCE", "NET_RAW","NET_ADMIN"]
resources:
requests:
nvidia.com/rdmapool: 1
nvidia.com/gpu: 1
hugepages-2Mi: 2Gi
memory: 8Gi
cpu: 8
limits:
nvidia.com/rdmapool: 1
nvidia.com/gpu: 1
hugepages-2Mi: 2Gi
memory: 8Gi
cpu: 8
volumeMounts:
- name: config
mountPath: /var/home/ext/
- name: licconfig
mountPath: /opt/mellanox/rivermax/
- mountPath: /hugepages
name: hugepage
- mountPath: /dev/shm
name: dshm
volumes:
- name: config
configMap:
name: river-config
- name: licconfig
configMap:
name: rivermax-config
- name: hugepage
emptyDir:
medium: HugePages
- name: dshm
emptyDir: {
medium: 'Memory',
sizeLimit: '4Gi'
}
imagePullSecrets:
- name: < Container registry secret >
---
apiVersion: v1
kind: Service
metadata:
name: rmax-service
spec:
type: NodePort
selector:
name: dsrmax-app
ports:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
- port: 7001
name: udp-port
protocol: UDP
targetPort: 7001
带GUI的VNC容器
此Pod定义允许您通过Kubernetes集群内的Web VNC界面访问Ubuntu LXDE/LXQT桌面环境。它使用K8s辅助网络的接口，通过GUI管理集群节点上的应用程序。
以下是VNC部署的YAML配置文件。请填写您的容器文件名。
此应用程序的示例可在GitHub - theasp/docker-novnc找到，但您可以创建自己的容器镜像。
apiVersion: v1
kind: Pod
metadata:
name: ub-vnc
labels:
name: ubuntu-vnc
annotations:
k8s.v1.cni.cncf.io/networks: |
[
{ "name": "rdmanet-static",
"ips": [ "192.168.102.253/24" ]
}
]
spec:
volumes:
- name: dshm
emptyDir:
medium: Memory
containers:
- image: < NOVNC container image >
name: vnc-container
resources:
limits:
cpu: 4
memory: 8Gi
nvidia.com/rdmapool: 1
env:
- name: DISPLAY_WIDTH
value: "1920"
- name: DISPLAY_HEIGHT
value: "1080"
- name: RUN_XTERM
value: "yes"
- name: RUN_FLUXBOX
value: "yes"
ports:
- containerPort: 8080
name: http-port
volumeMounts:
- mountPath: /dev/shm
name: dshm
---
apiVersion: v1
kind: Service
metadata:
name: vnc-service
spec:
type: NodePort
selector:
name: ubuntu-vnc
ports:
- port: 8080
name: http-port
targetPort: 8080
作者
 |
Vitaliy Razinkov Vitaliy Razinkov是NVIDIA网络团队的解决方案架构师，专注于复杂的Kubernetes、OpenShift和Microsoft解决方案。凭借超过25年的高级技术职位经验，他在设计和实施先进基础设施方面拥有深厚的专业知识。Vitaliy撰写了多份关于Microsoft技术、RoCE/RDMA加速的Kubernetes/OpenShift机器学习以及容器化解决方案的参考设计指南——所有这些都可在NVIDIA网络文档网站上找到。 |
 |
Gareth Sylvester-Bradley Gareth Sylvester-Bradley是NVIDIA的首席工程师，目前担任高级媒体工作流协会（AMWA）网络化媒体开放规范（NMOS）架构审查组主席。他专注于构建软件工具包和敏捷、协作的行业规范，为广播、直播制作提供开放、软件定义、硬件加速的媒体工作流。 |
医学影像、工业视频等。