RDG for NVIDIA Network Accelerated Red Hat OpenShift Container Platform over VMware vSphere
创建于2022年8月14日。本参考部署指南（RDG）演示了在VMware vSphere 7.0.3集群上，使用NVIDIA加速的端到端25/100 Gb/s以太网解决方案，部署和配置Red Hat OpenShift Container Platform (OCP) 4.10.18集群的过程。该设置能够运行基于RDMA和DPDK的应用。VMware的vSAN over RDMA将用作Red Hat OpenShift Container Platform工作负载的共享存储。
文档目录
创建于2022年8月14日
范围
以下参考部署指南（RDG）演示了使用NVIDIA加速网络部署和配置Red Hat OpenShift Container Platform (OCP) 4.10.18集群的过程。部署在单个VMware vSphere 7.0.3集群上，基于NVIDIA®加速的端到端25/100 Gb/s以太网解决方案。该设置能够运行基于RDMA和DPDK的应用。VMware的vSAN over RDMA将用作Red Hat OpenShift Container Platform工作负载的共享存储。
缩写和首字母缩略词
| 术语 | 定义 | 术语 | 定义 |
|---|---|---|---|
| DAC | 直接连接电缆 | RDMA | 远程直接内存访问 |
| DHCP | 动态主机配置协议 | RoCE | 融合以太网上的RDMA |
| DPDK | 数据平面开发套件 | SDN | 软件定义网络 |
| CNI | 容器网络接口 | SDS | 软件定义存储 |
| IB | InfiniBand | SR-IOV | 单根输入/输出虚拟化 |
| NFD | 节点特性发现操作器 | VDS | vSphere分布式交换机 |
| NOS | 网络操作系统 | VF | 虚拟功能 |
| OCP | Red Hat® OpenShift Container Platform® | VM | 虚拟机 |
引言
在VMware vSphere环境中配置Red Hat OCP集群以运行基于RDMA和DPDK的工作负载可能具有挑战性。正确的设计、软件和硬件组件选择可能是成功部署的关键任务。
本指南提供了使用OCP Assisted Installer 在单个vSphere集群上部署Red Hat OCP集群的逐步说明，包括技术概述、设计、组件选择和部署步骤。Assisted Installer 是Red Hat Hybrid Cloud Console上提供的用户友好型安装解决方案。Assisted Installer专注于裸机和vSphere基础设施。
VMware的vSANoRDMA现已完全合格，并自ESXi 7.0 U2版本起可用，已准备好部署。
在本文档中，我们将使用NVIDIA Network Operator，它负责以Host Device Network模式部署和配置。这允许在OCP集群工作节点上运行RDMA和DPDK工作负载。
参考资料
- 什么是RDMA？
- 融合以太网上的RDMA (RoCE)
- ConnectX® VMware® ESXi服务器以太网驱动程序
- NVIDIA Cumulus
- OpenShift Container Platform 4.10产品文档
- VMware vSAN 7设计指南
- 如何：使用NVIDIA网络结构配置vSphere分布式交换机
- RDG：VMware vSphere 7.0 U3上的VMware vSAN over RoCE
- NVIDIA Network Operator
解决方案架构
关键组件和技术
-
灵活的外形，16到128个物理端口，支持1GbE到400GbE速度。
基于突破性的硅技术，针对性能和可扩展性进行了优化，NVIDIA Spectrum交换机非常适合构建高性能、高性价比、高效的云数据中心网络、以太网存储结构和深度学习互连。
NVIDIA将基于行业领先的专用集成电路（ASIC）技术的NVIDIA Spectrum™交换机的优势与多种现代网络操作系统选择相结合，包括NVIDIA Cumulus® Linux、SONiC和NVIDIA Onyx®。
-
NVIDIA® Cumulus® Linux是业界最具创新性的开放网络操作系统，允许您像其他系统一样自动化、定制和扩展数据中心网络。
-
10/25/40/50/100/200和400G以太网网卡
业界领先的NVIDIA® ConnectX®系列智能网卡提供先进的硬件卸载和加速。
NVIDIA以太网网卡为超大规模、公有和私有云、存储、机器学习、AI、大数据和电信平台提供最高的ROI和最低的总拥有成本。
-
NVIDIA® LinkX®线缆和收发器产品系列提供业界最完整的10、25、40、50、100、200和400GbE以太网以及100、200和400Gb/s InfiniBand产品线，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。
RDMA是一种允许网络中的计算机在不涉及处理器、缓存或操作系统的情况下交换数据的技术。与本地DMA类似，RDMA提高了吞吐量和性能，并释放了计算资源。
RDMA over Converged Ethernet（RoCE）或InfiniBand over Ethernet（IBoE）是一种网络协议，允许通过以太网网络进行远程直接内存访问（RDMA）。它通过将InfiniBand（IB）传输数据包封装在以太网上实现。
Red Hat® OpenShift Container Platform®是一个企业级Kubernetes容器平台，专为开放的混合云策略构建。它提供了一个一致的应用程序平台来管理混合云、多云和边缘部署。
NVIDIA Network Operator简化了Kubernetes集群中NVIDIA网络资源的配置和管理。该操作符自动安装所需的主机网络软件，汇集所有必要组件以提供高速网络连接。这些组件包括NVIDIA网络驱动程序、Kubernetes设备插件、CNI插件、IP地址管理（IPAM）插件等。NVIDIA Network Operator与NVIDIA GPU Operator协同工作，为可扩展的GPU计算集群提供高吞吐量、低延迟的网络。
Logical Design
所述设置使用一个vSphere集群，包含4台ESXi服务器，连接到两台NVIDIA® Spectrum® SN2010以太网交换机（管理、入口和出口流量）和一台NVIDIA® Spectrum® SN2100以太网交换机（高速vSAN、RDMA和DPDK流量）。
vCenter、基础架构节点和路由器虚拟机将放置在同一集群中。
警告： 对于生产设计，建议将vCenter、基础架构节点和路由器虚拟机放置在单独的管理集群中。此建议适用于VMware Validated Design（VVD）和基于VVD的VMware Cloud Foundation（VCF）。
Network Design
集群网络配置了以下网络：
-
企业（管理）网络（VLAN 1610 - 192.168.1.x/24） 负责ESXi VMkernel管理接口、vCenter和其他管理虚拟机之间的连接。
警告： 需要DHCP和DNS服务。本指南不涵盖组件的安装和配置。
-
vMotion网络（VLAN 1611 – 192.168.11.0/24） ESXi vMotion VMkernel接口将位于此网络。
-
OCP网络（VLAN 1677 - 192.168.77.0/24） – OpenShift管理网络。
-
vSAN网络（VLAN 1630 – 192.168.30.0/24） ESXi vSAN VMkernel接口将位于此网络。
-
RDMA网络（VLAN 1631 – 192.168.3.0/24） ESXi vSAN VMkernel接口将位于此网络。
vSphere Distributed 交换机 Design
OCP的互联网访问
在本指南中，OCP集群部署在非Air Gap环境中，需要互联网访问。
OpenShift Container Platform Networking
Software Stack Components
本指南假定已安装以下软件和驱动程序：
- VMware ESXi 7.0.3，构建版本17630552
- VMware vCenter 7.0.3，构建版本17694817
- Distributed Switch 7.0.3
- RedHat OCP 4.10.18
- NVIDIA® ConnectX® Driver for VMware ESXi Server v4.21.71.101
- NVIDIA® ConnectX®-6DX FW版本22.32.2004
- NVIDIA® ConnectX®-6LX FW版本26.32.1010
- 网络操作系统（NOS）：NVIDIA Cumulus™ v5.1
Bill of Materials
本指南中vSphere环境使用的硬件设置如下：
集群计算/存储：
| VM | CPU | MEM | DISK |
|---|---|---|---|
| Compute Cluster vCenter (基于小型) | 4 | 20GB | 48GB |
| Infra Node | 2 | 4GB | 50GB |
| Control Node x 3 | 8 | 32GB | 120GB |
| Worker Node x 3 | 8 | 32GB | 120GB |
Deployment and Configuration
警告： 以下主题不在本指南范围内。
本文档范围：
- ESXi安装
- vCenter安装
- 虚拟数据中心配置
- vSphere集群配置
- 将ESXi主机添加到集群
实验室环境示例：
Wiring
本文档涵盖高可用性VMware vSphere集群部署。
Network/Fabric
Prerequisites
-
Switch OS
NVIDIA® Cumulus® Linux v5.1
-
Network adapter
NVIDIA ConnectX-6 Lx and ConnectX-6 Dx 网卡.
最新固件和驱动版本。更多信息，请参阅：
How-to: Firmware update for NVIDIA ConnectX-5/6 adapter on VMware ESXi 6.5 and above.
How-to: NVIDIA ConnectX driver upgrade on VMware ESXi 6.7/7.0 and above.
Hosts Network Configuration
下表提供了ESXi服务器、交换机名称及其网络配置的详细信息。
| Server | Server Name | IP and NICs | |
|---|---|---|---|
| High-Speed Ethernet Network | Management Network | ||
| ESXi-01 | sl01w01esx11 | vmk1: 192.168.11.111 (vMotion)vmk2: 192.168.130.111 (vSAN) | vmk0: 192.168.1.111From Corporate DHCP (reserved) |
| ESXi-02 | sl01w01esx12 | vmk1: 192.168.11.112 (vMotion)vmk2: 192.168.130.112 (vSAN) | vmk0: 192.168.1.112From Corporate DHCP (reserved) |
| ESXi-03 | sl01w01esx13 | vmk1: 192.168.11.113(vMotion)vmk2: 192.168.130.113 (vSAN) | vmk0: 192.168.1.113From Corporate DHCP (reserved) |
| ESXi-04 | sl01w01esx14 | vmk1: 192.168.11.114(vMotion)vmk2: 192.168.130.114 (vSAN) | vmk0: 192.168.1.114From Corporate DHCP (reserved) |
| Leaf-01 | clx-swx-033 | - | 192.168.1.233 |
| Leaf-02 | clx-swx-034 | - | 192.168.1.234 |
| Higt-Speed-01 | clx-swx-035 | - | 192.168.1.235 |
| vCenter (VM) | sl01w01vc01 | - | 192.168.1.25From Corporate DHCP (reserved) |
| Infra Node (VM) | sl01w01depl | - | 192.168.77.201 |
| Router Node (VM) | sl01w01pfsense | - | WAN - 192.168.1.202LAN - 192.168.77.254 |
| Control Node 01 (VM) | control01 | - | 192.168.77.11From Infra Node DHCP (reserved) |
| Control Node 01 (VM) | control02 | - | 192.168.77.12From Infra Node DHCP (reserved) |
| Control Node 01 (VM) | control03 | - | 192.168.77.13From Infra Node DHCP (reserved) |
| Worker Node 01 (VM) | worker01 | - | 192.168.77.21From Infra Node DHCP (reserved) |
| Worker Node 02 (VM) | worker02 | - | 192.168.77.22From Infra Node DHCP (reserved) |
| Worker Node 03 (VM) | worker03 | - | 192.168.77.23From Infra Node DHCP (reserved) |
Network Switch Configuration
ESXi to Leaf Connection
Port Channel and VLAN Configuration
在Supervisor Cluster中的两个Leaf NVIDIA SN2010交换机上运行以下命令，以配置端口通道和VLAN。
Leaf-01交换机配置示例（名为clx-swx-033）：
Switch console
cumulus@clx-swx-033:mgmt:~$sudo nv set system hostname clx-swx-033
cumulus@clx-swx-033:mgmt:~$sudo nv set interface lo ip address 10.10.10.1/32
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-22 type swp
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-4 bridge domain br_default
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1610
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1611
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1677
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 0.0.0.0/0 address-family ipv4-unicast
cumulus@clx-swx-033:mgmt:~$sudo nv set interface peerlink bond member swp21-22
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag backup 10.10.10.2
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag peer-ip linklocal
cumulus@clx-swx-033:mgmt:~$sudo nv config apply
cumulus@clx-swx-033:mgmt:~$sudo nv config save
Leaf-02 交换机配置示例（命名为 clx-swx-034）：
Switch console
cumulus@clx-swx-034:mgmt:~$sudo nv set system hostname clx-swx-034
cumulus@clx-swx-034:mgmt:~$sudo nv set interface lo ip address 10.10.10.2/32
cumulus@clx-swx-034:mgmt:~$sudo nv set interface swp1-22 type swp
cumulus@clx-swx-034:mgmt:~$sudo nv set interface swp1-4 bridge domain br_default
cumulus@clx-swx-034:mgmt:~$sudo nv set bridge domain br_default vlan 1610
cumulus@clx-swx-034:mgmt:~$sudo nv set bridge domain br_default vlan 1611
cumulus@clx-swx-034:mgmt:~$sudo nv set bridge domain br_default vlan 1677
cumulus@clx-swx-034:mgmt:~$sudo nv set vrf default router static 0.0.0.0/0 address-family ipv4-unicast
cumulus@clx-swx-034:mgmt:~$sudo nv set interface peerlink bond member swp21-22
cumulus@clx-swx-034:mgmt:~$sudo nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@clx-swx-034:mgmt:~$sudo nv set mlag backup 10.10.10.1
cumulus@clx-swx-034:mgmt:~$sudo nv set mlag peer-ip linklocal
cumulus@clx-swx-034:mgmt:~$sudo nv config apply
cumulus@clx-swx-034:mgmt:~$sudo nv config save
高速 NVIDIA SN2100 交换机上的端口通道和 VLAN 配置
在 vSphere 集群 中的高速交换机上运行以下命令以配置端口通道和 VLAN。
高速交换机 Higt-Speed-01（clx-swx-035）示例：
Switch console
cumulus@clx-swx-035:mgmt:~$sudo nv set system hostname clx-swx-035
cumulus@clx-swx-035:mgmt:~$sudo nv set interface swp1-16 type swp
cumulus@clx-swx-035:mgmt:~$sudo nv set interface swp1-16 bridge domain br_default
cumulus@clx-swx-035:mgmt:~$sudo nv set bridge domain br_default vlan 1630
cumulus@clx-swx-035:mgmt:~$sudo nv set bridge domain br_default vlan 1631
cumulus@clx-swx-035:mgmt:~$sudo nv set qos roce enable on
cumulus@clx-swx-035:mgmt:~$sudo nv config apply
cumulus@clx-swx-035:mgmt:~$sudo nv config save
在高速 SN2100 交换机上启用 RDMA over Converged Ethernet 无损（带 PFC 和 ETS）
RoCE 传输用于加速 vSAN 网络。为获得最佳结果，网络配置为无损。
在高速交换机上运行以下命令以配置无损网络（适用于 NVIDIA Cumulus）：
Switch console
cumulus@clx-swx-035:mgmt:~$sudo nv set qos roce enable on
cumulus@clx-swx-035:mgmt:~$sudo nv config apply
cumulus@clx-swx-035:mgmt:~$sudo nv config save
要检查 RoCE 配置，请运行以下命令：
cumulus@leaf-01:mgmt:~$sudo nv show qos roce
operational applied description
------------------ ----------- -------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mode lossless lossless Roce Mode
cable-length 100 100 Cable Length(in meters) for Roce Lossless Config
congestion-control
congestion-mode ECN Congestion config mode
enabled-tc 0,3 Congestion config enabled Traffic Class
max-threshold 1.43 MB Congestion config max-threshold
min-threshold 146.48 KB Congestion config min-threshold
pfc
pfc-priority 3 switch-prio on which PFC is enabled
rx-enabled enabled PFC Rx Enabled status
tx-enabled enabled PFC Tx Enabled status
trust
trust-mode pcp,dscp Trust Setting on the port for packet classification
RoCE PCP/DSCP->SP mapping configurations
===========================================
pcp dscp switch-prio
-- --- ----------------------- -----------
0 0 0,1,2,3,4,5,6,7 0
1 1 8,9,10,11,12,13,14,15 1
2 2 16,17,18,19,20,21,22,23 2
3 3 24,25,26,27,28,29,30,31 3
4 4 32,33,34,35,36,37,38,39 4
5 5 40,41,42,43,44,45,46,47 5
6 6 48,49,50,51,52,53,54,55 6
7 7 56,57,58,59,60,61,62,63 7
RoCE SP->TC mapping and ETS configurations
=============================================
switch-prio traffic-class scheduler-weight
-- ----------- ------------- ----------------
0 0 0 DWRR-50%
1 1 0 DWRR-50%
2 2 0 DWRR-50%
3 3 3 DWRR-50%
4 4 0 DWRR-50%
5 5 0 DWRR-50%
6 6 6 strict-priority
7 7 0 DWRR-50%
RoCE pool config
===================
name mode size switch-priorities traffic-class
-- --------------------- ------- ----- ----------------- -------------
0 lossy-default-ingress Dynamic 50.0% 0,1,2,4,5,6,7 -
1 roce-reserved-ingress Dynamic 50.0% 3 -
2 lossy-default-egress Dynamic 50.0% - 0,6
3 roce-reserved-egress Dynamic inf - 3
Exception List
=================
description
ESXi 主机和 vSphere 集群配置
前提条件
-
主机 BIOS
确认使用的是支持 SR-IOV 的服务器平台，并查阅服务器平台供应商文档中的 BIOS 设置，以在 BIOS 中启用 SR-IOV。
确认在 BIOS 中启用了 VT-d/IOMMU。
-
物理服务器配置
所有 ESXi 服务器的网卡 PCIe 插槽位置必须相同，并暴露相同的接口名称。
-
Kubernetes/OpenShift 经验
熟悉 Kubernetes 集群架构至关重要。
-
权限
确认您拥有读取和创建所需资源的账户权限。使用具有全局管理权限的账户是获取所有必要权限的最简单方法。有关更多信息，请参阅 Chapter 20. Installing on vSphere OpenShift Container Platform Container Platform 4.10 | Red Hat Customer Portal。
-
确认 NTP 已在您的环境中正确配置并正常工作。
-
创建并配置 2 个 VMware VDS，使用以下文档：How-to: Configure a vSphere Distributed Switch with NVIDIA network fabric.
环境中将使用两个 VDS：
-
SL-WL01-DS01，包含以下端口组：
- SL-WL01-MGMT-VLAN1610-DPG
- SL-WL01-vMotion-VLAN611-DPG
- SL-WL01-OCP-VLAN1677-DPG
-
SL-WL01-DS02，包含以下端口组：
- SL-WL01-vSAN-VLAN1630-DPG
- SL-WL01-RDMA-VLAN1631-DPG
-
-
示例：
-
创建并配置 VMware vSAN RDMA集群，使用以下文档 - RDG: VMware vSAN over RoCE on VMware vSphere 7.0 U3.
-
在ESXi主机上启用SR-IOV NVIDIA Network Operator利用Kubernetes CRD和Operator SDK管理网络相关组件，为OCP集群中的工作负载启用快速网络和RDMA。快速网络是K8s集群的辅助网络，适用于需要高带宽或低延迟的应用程序。 在OCP集群中，可以使用Dynamic DirectPath I/O将多个PCI直通或SR-IOV设备分配给Kubernetes工作负载VM。 为此，需要在ConnectX-6 Dx网卡上启用SR-IOV功能。 启用SR-IOV的步骤：
- 启动 vSphere Web Client，并连接到vCenter Server实例。
- 导航到ESXi主机，选择 Configure → Hardware → PCI Devices。点击 ALL PCI DEVICES。点击 Filter。
- 输入 Mellanox 并点击 Vendor Name。
- 选择 ConnectX-6 Dx NIC。
- 点击 CONFIGURE SR-IOV。
- 启用 SR-IOV，并设置 虚拟功能（VF） 的数量。点击 OK。
- 点击 PASSTHROUGH-ENABLED DEVICES 以验证已启用 8个VF。
-
为3个控制节点和3个工作节点OpenShift集群创建6个VM，配置如下：
在 自定义硬件 选项卡中，点击 VM Options → Advanced。- 点击 Boot Options。将 Boot Delay 设置为 9999 毫秒。
- 向下滚动并点击 Edit Configuration。在配置参数窗口中，点击 Add Configuration Params。定义以下参数名称和值：
- disk.EnableUUID：指定 TRUE。
点击 OK 两次以完成 控制VM 的配置。
对于 工作节点VM，点击 OK。返回虚拟硬件选项卡，通过点击 ADD NEW DEVICE 并选择 网卡 添加额外的SR-IOV网卡。
在 New Network 上点击 Browse...
选择一个 RDMA网络 端口组。点击 OK。
打开 New Network 并选择 Adapter Type：SR-IOV passthrough 和 Physical Function（例如，之前配置了VF的 vmnic4）。点击 OK。
- disk.EnableUUID：指定 TRUE。
- 点击 Boot Options。将 Boot Delay 设置为 9999 毫秒。
OCP集群安装
基础设施节点先决条件
OCP安装需要DHCP和DNS服务。本例中，使用基础设施节点运行这些服务。 OCP管理网络上的DHCP服务需要额外的ntp-servers配置。
警告： 本指南不涵盖组件的安装和配置。 更多信息，请参见 Networking requirements for user-provisioned infrastructure。
OCP
集群安装需要两个VIP地址：
- API 地址用于访问集群API。
- Ingress 地址用于集群入口流量。
安装OCP集群时，必须将这些IP地址提供给安装程序。 在我们的环境中，我们使用以下IP和DNS记录：
| 组件 | IP | 记录 | 描述 |
|---|---|---|---|
| Kubernetes API | 192.168.77.210 | api.ocp4.vwd.clx | 用于标识API负载均衡器的DNS A/AAAA或CNAME记录以及DNS PTR记录。这些记录必须能被集群外部客户端和集群内所有节点解析。 |
| Ingress | 192.168.77.211 | *.apps.ocp4.vwd.clx | 指向应用程序入口负载均衡器的通配符DNS A/AAAA或CNAME记录。应用程序入口负载均衡器以运行Ingress Controller POD的机器为目标。默认情况下，Ingress Controller POD在计算节点上运行。这些记录必须能被集群外部客户端和集群内所有节点解析。例如，console-openshift-console.apps.ocp4.vwd.clx用作OCP控制台的通配符路由。 |
| control01 | 192.168.77.11 | control01.ocp4.vwd.clx | 用于标识控制平面节点每台机器的DNS A/AAAA或CNAME记录和DNS PTR记录。这些记录必须能被集群内节点解析。 |
| control02 | 192.168.77.12 | control02.ocp4.vwd.clx | 用于标识控制平面节点每台机器的DNS A/AAAA或CNAME记录和DNS PTR记录。这些记录必须能被集群内节点解析。 |
| control03 | 192.168.77.13 | control03.ocp4.vwd.clx | 用于标识控制平面节点每台机器的DNS A/AAAA或CNAME记录和DNS PTR记录。这些记录必须能被集群内节点解析。 |
| worker01 | 192.168.77.21 | worker01.ocp4.vwd.clx | 用于标识工作节点每台机器的DNS A/AAAA或CNAME记录和DNS PTR记录。这些记录必须能被集群内节点解析。 |
| worker02 | 192.168.77.22 | worker02.ocp4.vwd.clx | 用于标识工作节点每台机器的DNS A/AAAA或CNAME记录和DNS PTR记录。这些记录必须能被集群内节点解析。 |
| worker03 | 192.168.77.23 | worker03.ocp4.vwd.clx | 用于标识工作节点每台机器的DNS A/AAAA或CNAME记录和DNS PTR记录。这些记录必须能被集群内节点解析。 |
生成SSH公钥
要生成密钥，请在基础设施节点上运行以下命令：
基础设施节点控制台
# ssh-keygen
OCP集群部署
您可以通过Assisted Installer、安装程序置备或用户置备的基础架构在vSphere上安装OpenShift Container Platform。
本文档描述了使用Assisted Installer在vSphere上安装OCP虚拟机。
按照本节中的步骤在vSphere上安装OpenShift。
-
在基础设施节点（本例中使用带UI的AlmaLinux）上打开Web浏览器，连接到Red Hat Hybrid Cloud网站并使用您的凭据登录控制台。
-
使用左侧菜单栏，选择OpenShift：
-
点击创建集群。
-
选择数据中心并向下滚动窗口。
-
选择vSphere。
-
选择Assisted Installer安装类型。
-
在打开的集群详细信息窗口中，提供集群名称和基础域名。从下拉列表中选择要安装的OpenShift版本，然后点击下一步。
#545454">示例：
-
复制文件到Infra Node上的
/root/.kube/config文件。Infra Node Console
[root@sl01lw01depl ~]# cp kubeconfig /root/.kube/config [root@sl01lw01depl ~]# export KUBECONFIG=/root/.kube/config -
使用以下提供的信息连接到已安装的集群：
要验证OCP集群是否正确安装，打开Infra Node控制台，并使用提供的Web Console URL、Username和Password打开Web浏览器。
确保集群状态正常。
使用左侧菜单栏，展开Compute部分并选择Nodes状态。
点击每个节点以查看该节点上的所有POD是否正常运行且无错误。
要从ssh控制台检查集群状态：
转到Infra Node的ssh控制台，运行以下命令查看节点状态：
Infra Node Console
[root@sl01lw01depl ocp4-helpernode]# oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
control01.ocp4.vwd.clx Ready master 2d v1.23.5+3afdacb 192.168.77.11 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
control02.ocp4.vwd.clx Ready master 2d v1.23.5+3afdacb 192.168.77.12 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
control03.ocp4.vwd.clx Ready master 2d v1.23.5+3afdacb 192.168.77.13 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
worker01.ocp4.vwd.clx Ready worker 2d v1.23.5+3afdacb 192.168.77.21 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
worker02.ocp4.vwd.clx Ready worker 2d v1.23.5+3afdacb 192.168.77.22 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
worker03.ocp4.vwd.clx Ready worker 2d v1.23.5+3afdacb 192.168.77.23 <none> Red Hat Enterprise Linux CoreOS 410.84.202206080346-0 (Ootpa) 4.18.0-305.49.1.el8_4.x86_64 cri-o://1.23.3-3.rhaos4.10.git5fe1720.el8
要查看所有正在运行的集群操作器状态，运行以下命令：
Infra Node Console
[root@sl01lw01depl ocp4-helpernode]# oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.10.18 True False False 135m
baremetal 4.10.18 True False False 2d
cloud-controller-manager 4.10.18 True False False 2d
cloud-credential 4.10.18 True False False 2d
cluster-autoscaler 4.10.18 True False False 2d
config-operator 4.10.18 True False False 2d
console 4.10.18 True False False 2d
csi-snapshot-controller 4.10.18 True False False 2d
dns 4.10.18 True False False 2d
etcd 4.10.18 True False False 2d
image-registry 4.10.18 True False False 2d
ingress 4.10.18 True False False 2d
insights 4.10.18 True False False 2d
kube-apiserver 4.10.18 True False False 2d
kube-controller-manager 4.10.18 True False False 2d
kube-scheduler 4.10.18 True False False 2d
kube-storage-version-migrator 4.10.18 True False False 2d
machine-api 4.10.18 True False False 2d
machine-approver 4.10.18 True False False 2d
machine-config 4.10.18 True False False 2d
marketplace 4.10.18 True False False 2d
monitoring 4.10.18 True False False 2d
network 4.10.18 True False False 2d
node-tuning 4.10.18 True False False 2d
openshift-apiserver 4.10.18 True False False 2d
openshift-controller-manager 4.10.18 True False False 24h
openshift-samples 4.10.18 True False False 2d
operator-lifecycle-manager 4.10.18 True False False 2d
operator-lifecycle-manager-catalog 4.10.18 True False False 2d
operator-lifecycle-manager-packageserver 4.10.18 True False False 2d
service-ca 4.10.18 True False False 2d
storage 4.10.18 True False False 2d
要查看所有正在运行的POD状态，运行以下命令：
[root@sl01lw01depl ocp4-helpernode]# oc get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
assisted-installer
assisted-installer-controller-5zh8w 0/1 Completed 0 2d openshift-apiserver-operator openshift-apiserver-operator-6756c64b88-njtk8 1/1 Running 2 (2d ago) 2d openshift-apiserver apiserver-7c48bcb965-6cqp5 2/2 Running 0 2d openshift-apiserver apiserver-7c48bcb965-mg7t5 2/2 Running 0 2d openshift-apiserver apiserver-7c48bcb965-rvt8j 2/2 Running 0 2d openshift-authentication-operator authentication-operator-6ddbbdbf46-w5n2w 1/1 Running 2 (2d ago) 2d openshift-authentication oauth-openshift-6c775fd8f6-6cxxd 1/1 Running 0 2d openshift-authentication oauth-openshift-6c775fd8f6-k6nxq 1/1 Running 0 2d openshift-authentication oauth-openshift-6c775fd8f6-pd4cm 1/1 Running 0 2d openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-7db699cb4f-wdh7d 2/2 Running 4 (2d ago) 2d openshift-cloud-credential-operator cloud-credential-operator-67f75477db-mczvd 2/2 Running 0 2d openshift-cluster-machine-approver machine-approver-697c6c67f-6thpm 2/2 Running 2 (2d ago) 2d openshift-cluster-node-tuning-operator cluster-node-tuning-operator-59cdb46c9-s82qc 1/1 Running 1 (2d ago) 2d openshift-cluster-node-tuning-operator tuned-92q5b 1/1 Running 0 2d openshift-cluster-node-tuning-operator tuned-jsgls 1/1 Running 0 2d openshift-cluster-node-tuning-operator tuned-lg5rc 1/1 Running 0 2d openshift-cluster-node-tuning-operator tuned-nvrg6 1/1 Running 0 2d openshift-cluster-node-tuning-operator tuned-pf6fh 1/1 Running 0 2d openshift-cluster-node-tuning-operator tuned-wz9s6 1/1 Running 0 2d openshift-cluster-samples-operator cluster-samples-operator-b6fc87b94-hslfp 2/2 Running 0 2d openshift-cluster-storage-operator cluster-storage-operator-7857f978b8-6q6jt 1/1 Running 1 (2d ago) 2d openshift-cluster-storage-operator csi-snapshot-controller-5b97674c78-m7s9b 1/1 Running 0 2d openshift-cluster-storage-operator csi-snapshot-controller-5b97674c78-n9ffn 1/1 Running 1 (2d ago) 2d openshift-cluster-storage-operator csi-snapshot-controller-operator-856bb56b5d-zj2gx 1/1 Running 1 (2d ago) 2d openshift-cluster-storage-operator csi-snapshot-webhook-7f447d57f7-4g8l4 1/1 Running 0 2d openshift-cluster-storage-operator csi-snapshot-webhook-7f447d57f7-xtldm 1/1 Running 0 2d openshift-cluster-version cluster-version-operator-688999f8cd-nm9xm 1/1 Running 0 2d openshift-config-operator openshift-config-operator-57c79476d7-r6nfq 1/1 Running 2 (2d ago) 2d openshift-console-operator console-operator-c74b55d67-m9g7w 1/1 Running 0 2d openshift-console console-5846db579c-6z5f5 1/1 Running 1 (2d ago) 2d openshift-console console-5846db579c-xczc5 1/1 Running 1 (2d ago) 2d openshift-console downloads-6cdbd64fbb-89dt4 1/1 Running 0 2d openshift-console downloads-6cdbd64fbb-8v942 1/1 Running 0 2d openshift-controller-manager-operator openshift-controller-manager-operator-b4c944d9f-r7x7w 1/1 Running 2 (2d ago) 2d openshift-controller-manager controller-manager-lwc2w 1/1 Running 0 24h openshift-controller-manager controller-manager-p62j5 1/1 Running 0 24h openshift-controller-manager controller-manager-x2qt9 1/1 Running 0 24h openshift-dns-operator dns-operator-5f65cf6746-g52zd 2/2 Running 0 2d openshift-dns dns-default-4l48f 2/2 Running 0 2d openshift-dns dns-default-8fdhq 2/2 Running 0 2d openshift-dns dns-default-gqd5h 2/2 Running 0 2d openshift-dns dns-default-j6bv5 2/2 Running 0 2d openshift-dns dns-default-lqbv4 2/2 Running 0 2d openshift-dns dns-default-nhdxz 2/2 Running 0 2d openshift-dns node-resolver-2p8cv 1/1 Running 0 2d openshift-dns node-resolver-5522q 1/1 Running 0 2d openshift-dns node-resolver-5gw9c 1/1 Running 0 2d openshift-dns node-resolver-7c2sf 1/1 Running 0 2d openshift-dns node-resolver-b68qq 1/1 Running 0 2d openshift-dns node-resolver-njm5b 1/1 Running 0 2d openshift-etcd-operator etcd-operator-6d6ff5499d-6fnts 1/1 Running 2 (2d ago) 2d openshift-etcd etcd-control01.ocp4.vwd.clx 4/4 Running 0 2d openshift-etcd etcd-control02.ocp4.vwd.clx 4/4 Running 0 2d openshift-etcd etcd-control03.ocp4.vwd.clx 4/4 Running 0 2d openshift-etcd etcd-quorum-guard-54c99fcc95-4dl45 1/1 Running 0 2d openshift-etcd etcd-quorum-guard-54c99fcc95-gsbjq 1/1 Running 0 2d openshift-etcd etcd-quorum-guard-54c99fcc95-zs8cw 1/1 Running 0 2d openshift-etcd installer-2-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-3-retry-1-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-5-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-6-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-6-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-6-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-7-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-7-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-etcd installer-7-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-image-registry cluster-image-registry-operator-5959746b9-mw5x9 1/1 Running 1 (2d ago) 2d openshift-image-registry image-pruner-27688320-g48f6 0/1 Completed 0 38h openshift-image-registry image-pruner-27689760-kp96s 0/1 Completed 0 14h openshift-image-registry node-ca-7z6nj 1/1 Running 0 2d openshift-image-registry node-ca-9m8sx 1/1 Running 0 2d openshift-image-registry node-ca-jmq9w 1/1 Running 0 2d openshift-image-registry node-ca-sl87m 1/1 Running 0 2d openshift-image-registry node-ca-z7clg 1/1 Running 0 2d openshift-image-registry node-ca-zs99s 1/1 Running 0 2d openshift-ingress-canary ingress-canary-5r6qh 1/1 Running 0 2d openshift-ingress-canary ingress-canary-l8q6j 1/1 Running 0 2d openshift-ingress-canary ingress-canary-zpbcd 1/1 Running 0 2d openshift-ingress-operator ingress-operator-7899578f6-tvbzr 2/2 Running 4 (2d ago) 2d openshift-ingress router-default-547bcf454b-hn5gh 1/1 Running 0 2d openshift-ingress router-default-547bcf454b-z2lk9 1/1 Running 0 2d openshift-insights
insights-operator-7d4cc4cf88-vn9px 1/1 Running 1 (2d ago) 2d openshift-kni-infra coredns-control01.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra coredns-control02.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra coredns-control03.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra coredns-worker01.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra coredns-worker02.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra coredns-worker03.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra haproxy-control01.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra haproxy-control02.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra haproxy-control03.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra keepalived-control01.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra keepalived-control02.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra keepalived-control03.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra keepalived-worker01.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra keepalived-worker02.ocp4.vwd.clx 2/2 Running 0 2d openshift-kni-infra keepalived-worker03.ocp4.vwd.clx 2/2 Running 0 2d openshift-kube-apiserver-operator kube-apiserver-operator-5f76847f99-84k6k 1/1 Running 2 (2d ago) 2d openshift-kube-apiserver installer-10-control02.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver installer-11-control01.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver installer-11-control02.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver installer-11-control03.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver installer-12-control01.ocp4.vwd.clx 0/1 Completed 0 25h openshift-kube-apiserver installer-12-control02.ocp4.vwd.clx 0/1 Completed 0 25h openshift-kube-apiserver installer-12-control03.ocp4.vwd.clx 0/1 Completed 0 25h openshift-kube-apiserver installer-13-control01.ocp4.vwd.clx 0/1 Completed 0 24h openshift-kube-apiserver installer-13-control02.ocp4.vwd.clx 0/1 Completed 0 24h openshift-kube-apiserver installer-13-control03.ocp4.vwd.clx 0/1 Completed 0 24h openshift-kube-apiserver installer-9-control01.ocp4.vwd.clx 0/1 Completed 0 30h openshift-kube-apiserver installer-9-control02.ocp4.vwd.clx 0/1 Completed 0 30h openshift-kube-apiserver installer-9-control03.ocp4.vwd.clx 0/1 Completed 0 30h openshift-kube-apiserver kube-apiserver-control01.ocp4.vwd.clx 5/5 Running 0 24h openshift-kube-apiserver kube-apiserver-control02.ocp4.vwd.clx 5/5 Running 0 24h openshift-kube-apiserver kube-apiserver-control03.ocp4.vwd.clx 5/5 Running 0 24h openshift-kube-apiserver kube-apiserver-guard-control01.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-apiserver kube-apiserver-guard-control02.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-apiserver kube-apiserver-guard-control03.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-apiserver revision-pruner-10-control01.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver revision-pruner-10-control02.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver revision-pruner-10-control03.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver revision-pruner-11-control01.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver revision-pruner-11-control02.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver revision-pruner-11-control03.ocp4.vwd.clx 0/1 Completed 0 29h openshift-kube-apiserver revision-pruner-12-control01.ocp4.vwd.clx 0/1 Completed 0 25h openshift-kube-apiserver revision-pruner-12-control02.ocp4.vwd.clx 0/1 Completed 0 25h openshift-kube-apiserver revision-pruner-12-control03.ocp4.vwd.clx 0/1 Completed 0 25h openshift-kube-apiserver revision-pruner-13-control01.ocp4.vwd.clx 0/1 Completed 0 24h openshift-kube-apiserver revision-pruner-13-control02.ocp4.vwd.clx 0/1 Completed 0 24h openshift-kube-apiserver revision-pruner-13-control03.ocp4.vwd.clx 0/1 Completed 0 24h openshift-kube-apiserver revision-pruner-9-control01.ocp4.vwd.clx 0/1 Completed 0 30h openshift-kube-apiserver revision-pruner-9-control02.ocp4.vwd.clx 0/1 Completed 0 30h openshift-kube-apiserver revision-pruner-9-control03.ocp4.vwd.clx 0/1 Completed 0 30h openshift-kube-controller-manager-operator kube-controller-manager-operator-8569749977-7bdf2 1/1 Running 2 (2d ago) 2d openshift-kube-controller-manager installer-5-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager installer-6-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager installer-6-retry-1-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager installer-6-retry-1-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager installer-7-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager installer-7-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager installer-7-retry-1-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager kube-controller-manager-control01.ocp4.vwd.clx 4/4 Running 0 2d openshift-kube-controller-manager kube-controller-manager-control02.ocp4.vwd.clx 4/4 Running 1 (2d ago) 2d openshift-kube-controller-manager kube-controller-manager-control03.ocp4.vwd.clx 4/4 Running 2 (2d ago) 2d openshift-kube-controller-manager kube-controller-manager-guard-control01.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-controller-manager kube-controller-manager-guard-control02.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-controller-manager kube-controller-manager-guard-control03.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-controller-manager revision-pruner-7-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager revision-pruner-7-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-controller-manager revision-pruner-7-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler-operator openshift-kube-scheduler-operator-78bdb8d5df-jj7mc 1/1 Running 2 (2d ago) 2d openshift-kube-scheduler installer-6-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler installer-7-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler installer-7-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler installer-7-retry-1-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler openshift-kube-scheduler-control01.ocp4.vwd.clx 3/3 Running 0 2d openshift-kube-scheduler openshift-kube-scheduler-control02.ocp4.vwd.clx 3/3 Running 0 2d openshift-kube-scheduler openshift-kube-scheduler-control03.ocp4.vwd.clx 3/3 Running 0 2d openshift-kube-scheduler openshift-kube-scheduler-guard-control01.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-scheduler openshift-kube-scheduler-guard-control02.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-scheduler openshift-kube-scheduler-guard-control03.ocp4.vwd.clx 1/1 Running 0 2d openshift-kube-scheduler revision-pruner-6-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler revision-pruner-6-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler
revision-pruner-7-control01.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler revision-pruner-7-control02.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-scheduler revision-pruner-7-control03.ocp4.vwd.clx 0/1 Completed 0 2d openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-858f98bf69-gmhmk 1/1 Running 2 (2d ago) 2d openshift-kube-storage-version-migrator migrator-8f5cb494f-cv2sw 1/1 Running 0 2d openshift-machine-api cluster-autoscaler-operator-67d67f467c-ftjnd 2/2 Running 2 (2d ago) 2d openshift-machine-api cluster-baremetal-operator-9f7c58876-7gfcf 2/2 Running 0 2d openshift-machine-api machine-api-controllers-5d48848dc-f2lbn 7/7 Running 8 (2d ago) 2d openshift-machine-api machine-api-operator-5869dbb6f9-jmd9d 2/2 Running 1 (2d ago) 2d openshift-machine-api metal3-596fff4b5b-s45j9 7/7 Running 0 2d openshift-machine-api metal3-image-cache-bjmnd 1/1 Running 0 2d openshift-machine-api metal3-image-cache-g7dcc 1/1 Running 0 2d openshift-machine-api metal3-image-cache-nxdlr 1/1 Running 0 2d openshift-machine-api metal3-image-customization-95866c848-6dgvd 1/1 Running 0 2d openshift-machine-config-operator machine-config-controller-77b5755d57-mqlk9 1/1 Running 1 (2d ago) 2d openshift-machine-config-operator machine-config-daemon-7l7bd 2/2 Running 0 2d openshift-machine-config-operator machine-config-daemon-c5n52 2/2 Running 0 2d openshift-machine-config-operator machine-config-daemon-dhhtc 2/2 Running 0 2d openshift-machine-config-operator machine-config-daemon-fl96h 2/2 Running 0 2d openshift-machine-config-operator machine-config-daemon-jmmbg 2/2 Running 0 2d openshift-machine-config-operator machine-config-daemon-n456s 2/2 Running 0 2d openshift-machine-config-operator machine-config-operator-6b86784cb5-v7zf4 1/1 Running 1 (2d ago) 2d openshift-machine-config-operator machine-config-server-l4sm6 1/1 Running 0 2d openshift-machine-config-operator machine-config-server-rdngb 1/1 Running 0 2d openshift-machine-config-operator machine-config-server-wn8z2 1/1 Running 0 2d openshift-marketplace certified-operators-9tdzr 1/1 Running 0 75m openshift-marketplace community-operators-57zzr 1/1 Running 0 28m openshift-marketplace marketplace-operator-5bd7c86474-w47th 1/1 Running 3 (2d ago) 2d openshift-marketplace redhat-marketplace-p92tb 1/1 Running 0 4h30m openshift-marketplace redhat-operators-f98h8 1/1 Running 0 29h openshift-monitoring alertmanager-main-0 6/6 Running 0 2d openshift-monitoring alertmanager-main-1 6/6 Running 0 2d openshift-monitoring cluster-monitoring-operator-77f85687f5-nj662 2/2 Running 0 2d openshift-monitoring grafana-5487f6d9cf-jk5sv 3/3 Running 0 2d openshift-monitoring kube-state-metrics-6d685b8687-wbfj6 3/3 Running 0 2d openshift-monitoring node-exporter-7lcqv 2/2 Running 0 2d openshift-monitoring node-exporter-9rcbv 2/2 Running 0 2d openshift-monitoring node-exporter-cbhhd 2/2 Running 0 2d openshift-monitoring node-exporter-hk5f4 2/2 Running 0 2d openshift-monitoring node-exporter-rsw5t 2/2 Running 0 2d openshift-monitoring node-exporter-wntwr 2/2 Running 0 2d openshift-monitoring openshift-state-metrics-65f58d4c67-jpp6b 3/3 Running 0 2d openshift-monitoring prometheus-adapter-b954595b-qc2nw 1/1 Running 0 24h openshift-monitoring prometheus-adapter-b954595b-z6wrp 1/1 Running 0 24h openshift-monitoring prometheus-k8s-0 6/6 Running 0 2d openshift-monitoring prometheus-k8s-1 6/6 Running 0 2d openshift-monitoring prometheus-operator-76f79c8d85-f2wqq 2/2 Running 0 2d openshift-monitoring telemeter-client-7f76f65fd4-gndjp 3/3 Running 0 2d openshift-monitoring thanos-querier-7b7cf74976-hm8qm 6/6 Running 0 2d openshift-monitoring thanos-querier-7b7cf74976-zh66s 6/6 Running 0 2d openshift-multus multus-66mmt 1/1 Running 0 2d openshift-multus multus-8zpj9 1/1 Running 0 2d openshift-multus multus-additional-cni-plugins-5xfgt 1/1 Running 0 2d openshift-multus multus-additional-cni-plugins-8ntxc 1/1 Running 0 2d openshift-multus multus-additional-cni-plugins-cf4pl 1/1 Running 0 2d openshift-multus multus-additional-cni-plugins-kgzcv 1/1 Running 0 2d openshift-multus multus-additional-cni-plugins-lzntc 1/1 Running 0 2d openshift-multus multus-additional-cni-plugins-mkvqn 1/1 Running 0 2d openshift-multus multus-admission-controller-2x94g 2/2 Running 0 2d openshift-multus multus-admission-controller-54fzg 2/2 Running 0 2d openshift-multus multus-admission-controller-rpq6d 2/2 Running 0 2d openshift-multus multus-gbxp4 1/1 Running 0 2d openshift-multus multus-h8mjv 1/1 Running 0 2d openshift-multus multus-lbv7j 1/1 Running 0 2d openshift-multus multus-wbqr8 1/1 Running 0 2d openshift-multus network-metrics-daemon-dkv45 2/2 Running 0 2d openshift-multus network-metrics-daemon-gnxjt 2/2 Running 0 2d openshift-multus network-metrics-daemon-j5ng9 2/2 Running 0 2d openshift-multus network-metrics-daemon-sc6cd 2/2 Running 0 2d openshift-multus network-metrics-daemon-sddb9 2/2 Running 0 2d openshift-multus network-metrics-daemon-vkhmw 2/2 Running 0 2d openshift-network-diagnostics network-check-source-695cdb7747-nh98w 1/1 Running 0 2d openshift-network-diagnostics network-check-target-24wqk 1/1 Running 0 2d openshift-network-diagnostics network-check-target-jfpbl 1/1 Running 0 2d openshift-network-diagnostics network-check-target-jsdtb 1/1 Running 0 2d openshift-network-diagnostics network-check-target-rlpxk 1/1 Running 0 2d openshift-network-diagnostics network-check-target-t6zqv 1/1 Running 0 2d openshift-network-diagnostics network-check-target-w5nc5 1/1 Running 0 2d openshift-network-operator network-operator-7d77dffff9-54tbt 1/1 Running 1 (2d ago) 2d openshift-oauth-apiserver apiserver-5cc46c79bc-f2d6n 1/1 Running 0 2d openshift-oauth-apiserver apiserver-5cc46c79bc-l5hjx 1/1 Running 0 2d openshift-oauth-apiserver
apiserver-5cc46c79bc-tjbw4 1/1 Running 0 2d
openshift-operator-lifecycle-manager catalog-operator-74c9986ff-xvnj8 1/1 Running 0 2d
openshift-operator-lifecycle-manager collect-profiles-27690600-8mwnv 0/1 Completed 0 40m
openshift-operator-lifecycle-manager collect-profiles-27690615-gd57g 0/1 Completed 0 25m
openshift-operator-lifecycle-manager collect-profiles-27690630-kkz28 0/1 Completed 0 10m
openshift-operator-lifecycle-manager olm-operator-67fcdb489c-m4cvc 1/1 Running 0 2d
openshift-operator-lifecycle-manager package-server-manager-765bb5f5-xrf8v 1/1 Running 2 (2d ago) 2d
openshift-operator-lifecycle-manager packageserver-fdd697dc6-c5b7v 1/1 Running 0 2d
openshift-operator-lifecycle-manager packageserver-fdd697dc6-qtqf2 1/1 Running 0 2d
openshift-sdn sdn-4j62m 2/2 Running 0 2d
openshift-sdn sdn-4q6jf 2/2 Running 0 2d
openshift-sdn sdn-7ct7v 2/2 Running 0 2d
openshift-sdn sdn-controller-29cvs 2/2 Running 0 2d
openshift-sdn sdn-controller-lm7g9 2/2 Running 0 2d
openshift-sdn sdn-controller-mrssb 2/2 Running 1 (2d ago) 2d
openshift-sdn sdn-fcv7m 2/2 Running 0 2d
openshift-sdn sdn-pq7pb 2/2 Running 0 2d
openshift-sdn sdn-rzz7l 2/2 Running 0 2d
openshift-service-ca-operator service-ca-operator-74997d4678-6dg7d 1/1 Running 2 (2d ago) 2d
openshift-service-ca service-ca-7567987bc8-cm9j7 1/1 Running 1 (2d ago) 2d
安装节点特性发现（NFD）Operator
OpenShift节点特性发现（NFD）管理硬件特性的检测及其在OpenShift容器平台中的配置。NFD为节点打上特定属性的标签。因此，NFD是NVIDIA GPU Operator的先决条件。作为集群管理员，您可以通过OpenShift容器平台Web控制台中的Red Hat OperatorHub目录安装NFD Operator。
-
使用左侧菜单栏，展开 Operators 部分，选择 OperatorHub。
使用搜索栏搜索“Node Feature Discovery”。应找到两个结果。 选择标记为Red Hat Operator的版本。这是Red Hat支持的版本。
-
点击 Install。
-
下一个菜单允许您确定Operator的安装位置和方式。通常无需更改任何选项。 点击 Install 继续使用默认设置。
-
等待NFD Operator安装完成。当看到“Installed operator - ready for use”状态时，点击 View Operator。注意，您现在位于Operator安装过程中创建的 openshift-nfd 项目中。
-
点击 Create Instance 按钮创建新的 NFD 对象。
-
下一个菜单提供配置NFD Operator扫描集群的选项。通常无需更改任何默认设置。点击 Create 在集群上实例化NFD资源。
-
您可以在Infra Node ssh控制台中查看安装进度。NFD安装可能需要5分钟。
-
在水平菜单栏中，选择 NodeFeatureDiscovery。您应该能看到一个名为 nfd-instance 的实例。等待状态显示部署完成。
启用集群范围的授权
NVIDIA Network Operator部署多个POD，用于管理和启用OpenShift容器平台中的NVIDIA高速网络。其中一些POD需要默认通用基础镜像（UBI）中不可用的软件包。要使软件包可用，您必须在OpenShift中启用集群范围的授权容器构建。为此，请按照启用集群范围的授权中的说明操作。
通过运行 oc get mcp/worker 命令，您可以查看 MachineConfigPool 对象，监控MCP是否已更新、未更新或降级，从而确保所有 MachineConfig 资源已成功应用到节点，然后可以继续验证集群。
安装NVIDIA Network Operator
请参阅以下使用Web UI部署Network Operator的步骤。
警告：开始之前，通过运行
[root@sl01lw01depl ocp4-helpernode]# oc create ns nvidia-operator-resources为Network Operator资源创建命名空间。
-
使用左侧菜单栏，展开 Operators 部分，选择 OperatorHub。
使用搜索栏搜索“NVIDIA Network Operator”。
-
点击 Install。
-
选择 A specific namespace on the cluster，然后从下拉菜单中选择 nvidia-operator-resources。点击 Install。
-
等待Operator安装完成。当看到“Installed operator - ready for use”状态时，点击 View Operator。
-
在 Provided APIs 下，点击 NicClusterPolicy 磁贴中的 Create instance。
-
在YAML编辑器中，将
spec部分替换为以下内容：apiVersion: nvidia.com/v1 kind: NicClusterPolicy metadata: name: nic-cluster-policy spec: deployment: enable: true sriovNetworkNodePolicy: enable: true ofedDriver: enable: true version: "5.8-1.1.1.0" rdmaSharedDevicePlugin: enable: true sriovNetworkOperator: enable: true点击 Create。
-
等待所有POD启动并运行。您可以在 Workloads > Pods 下查看状态。
-
验证所有POD都处于 Running 状态。
the Operators section, and select the OperatorHub.
Use the search bar to search for NVIDIA. Two items should be found.
Select the NVIDIA Network Operator.
-
Click Install.
-
The next menu allows you to define how and where the operator will be installed. You would rarely have to change any of these options. Click Install to continue with the default values.
-
Wait while the NVIDIA Network operator is being installed. Once you see the "Installed operator - ready for use" status, click View Operator.
-
Click the Create Instance button to create a new NicClusterPolicy object.
-
Click on YAML view, and change to the following:
NicClusterPolicy sample
apiVersion: mellanox.com/v1alpha1 kind: NicClusterPolicy metadata: name: nic-cluster-policy spec: ofedDriver: image: mofed livenessProbe: initialDelaySeconds: 30 periodSeconds: 30 readinessProbe: initialDelaySeconds: 10 periodSeconds: 30 repository: nvcr.io/nvidia/mellanox startupProbe: initialDelaySeconds: 10 periodSeconds: 20 upgradePolicy: autoUpgrade: false drain: deleteEmptyDir: true enable: true force: true podSelector: '' timeoutSeconds: 300 maxParallelUpgrades: 1 version: 5.7-1.0.2.0 sriovDevicePlugin: image: sriov-network-device-plugin repository: ghcr.io/k8snetworkplumbingwg version: a765300344368efbf43f71016e9641c58ec1241b config: | { "resourceList": [ { "resourcePrefix": "nvidia.com", "resourceName": "host_dev", "selectors": { "vendors": ["15b3"], "isRdma": true } } ] }Click Create.
-
Wait for the status to show State ready.
-
In the horizontal menu bar, select HostDeviceNetwork, and click on Create HostDeviceNetwork to create a new object.
-
Click on YAML view, and change to the following:
HostDeviceNetwork sample
apiVersion: mellanox.com/v1alpha1 kind: HostDeviceNetwork metadata: name: hostdev-net spec: networkNamespace: "default" resourceName: "host_dev" ipam: | { "type": "whereabouts", "range": "192.168.3.225/28" } -
Click Create.
-
Wait for the status to show State ready.
Running the Application
Now it is possible to deploy a sample POD.
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# vim podrdma.yaml
K8s CLI VM console
apiVersion: v1
kind: Pod
metadata:
name: rdma-test-pod-1
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-net
spec:
restartPolicy: OnFailure
containers:
- image: harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04
name: rdma-test-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
limits:
nvidia.com/host_dev: '1'
requests:
nvidia.com/host_dev: '1'
command:
- sh
- -c
- sleep infinity
A sample of harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04 docker file:
Docker Sample
FROM ubuntu:20.04
# Ubuntu 20.04 docker container with inbox Mellanox drivers
WORKDIR /tmp/
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get clean -y && apt-get -y update && apt-get install -y apt-utils udev vim bash && apt-get -y upgrade
RUN apt-get install -y iproute2 rdma-core libibmad5 ibutils ibverbs-utils infiniband-diags perftest \
mstflint strace iputils-ping
RUN ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
RUN dpkg-reconfigure --frontend noninteractive tzdata && apt-get clean all -y
CMD bash
Run the following command to create the POD:
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# oc apply -f podrdma.yaml
pod/hostdev-test-pod created
To verify that the POD is running, run:
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rdma-test-pod-1 1/1 Running 0 10s 10.129.3.80 worker02.ocp4.vwd.clx <none> <none>
Get a shell for the running
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# oc exec -it rdma-test-pod-1 -- bash
检查POD中的可用网络接口：
K8s CLI VM console
root@rdma-test-pod-1:/tmp# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@rdma-test-pod-1:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if2890: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:81:03:50 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.129.3.80/23 brd 10.129.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::c05d:6ff:fe08:84ad/64 scope link
valid_lft forever preferred_lft forever
161: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a2:9b:7f brd ff:ff:ff:ff:ff:ff
inet 192.168.3.225/28 brd 192.168.3.239 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea2:9b7f/64 scope link
valid_lft forever preferred_lft forever
Verification
检查RDMA
要检查RDMA，请部署第二个POD。打开另一个控制台窗口，并运行以下命令：
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# vim podrdma2.yaml
K8s CLI VM console
apiVersion: v1
kind: Pod
metadata:
name: rdma-test-pod-2
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-net
spec:
restartPolicy: OnFailure
containers:
- image: harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04
name: rdma-test-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
limits:
nvidia.com/host_dev: '1'
requests:
nvidia.com/host_dev: '1'
command:
- sh
- -c
- sleep infinity
运行以下命令以运行第二个POD：
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# oc apply -f podrdma2.yaml
pod/hostdev-test-pod-2 created
验证两个POD正在不同的Worker节点上运行。
K8s CLI VM console
[root@sl01lw01depl ocp4-helpernode]# oc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
rdma-test-pod-1 1/1 Running 0 102s 10.129.3.80 worker02.ocp4.vwd.clx <none> <none>
rdma-test-pod-2 1/1 Running 0 83s 10.128.2.18 worker03.ocp4.vwd.clx <none> <none>
如您所见，第一个POD rdma-test-pod-1 运行在 worker02 上，第二个POD rdma-test-pod-2 运行在 worker03 上。
获取第二个运行中容器的shell。
K8s CLI VM console
[root@sl01lw01depl ocp4]# oc exec -it rdma-test-pod-2 -- bash
检查POD中的可用网络接口。
K8s CLI VM console
root@rdma-test-pod-1:/tmp# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@hostdev-test-pod-2:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 0a:58:0a:80:02:12 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.128.2.18/23 brd 10.128.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::3415:caff:fed9:b936/64 scope link
valid_lft forever preferred_lft forever
26: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:a2:2e:a2 brd ff:ff:ff:ff:ff:ff
inet 192.168.3.226/28 brd 192.168.3.239 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:fea2:2ea2/64 scope link
valid_lft forever preferred_lft forever
现在您可以运行 ib_write_bw（InfiniBand写带宽）工具，该工具是 Perftest Package 的一部分，运行以下命令：
返回第一个运行中的容器shell窗口，并运行：
K8s CLI VM console
root@rdma-test-pod-1:/tmp# ib_write_bw -F -d mlx5_0 --report_gbits
************************************
* Waiting for client to connect... *
************************************
返回另一个运行中的容器shell窗口，并运行：
K8s CLI VM console
root@rdma-test-pod-2:/tmp# ib_write_bw -F 192.168.3.225 -d mlx5_0 --report_gbits
结果：
K8s CLI VM console
On Client side.
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x00c8 PSN 0xae9396 RKey 0x020b05 VAddr 0x007f1fd458a000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:226
remote address: LID 0000 QPN 0x00c8 PSN 0x5fc1b8 RKey 0x020b05 VAddr 0x007f1c19487000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:225
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 90.52 90.52 0.172651
---------------------------------------------------------------------------------------
On Server side.
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x00c8 PSN 0x5fc1b8 RKey 0x020b05 VAddr 0x007f1c19487000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:225
remote address: LID 0000 QPN 0x00c8 PSN 0xae9396 RKey 0x020b05 VAddr 0x007f1fd458a000
GID:
00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:226
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 65536 5000 90.52 90.52 0.172651
警告： 本文档中列出的性能结果仅供参考，不应视为NVIDIA产品的正式性能目标。
要运行DPDK应用程序，请参阅以下文档：RDG: DPDK Applications on SR-IOV Enabled Kubernetes Cluster with NVIDIA Network Operator.
完成！
作者
 |
Boris KovalevBoris Kovalev过去几年一直担任解决方案架构师，专注于NVIDIA Networking/Mellanox技术，负责复杂的机器学习、大数据和基于VMware的高级云研究与设计。此前，他在多家公司担任高级顾问和解决方案架构师超过20年，最近在VMware工作。他撰写了多份涵盖VMware、机器学习、Kubernetes和容器解决方案的参考设计，可在NVIDIA文档网站上获取。 |
 |
Vitaliy RazinkovVitaliy Razinkov是NVIDIA Networking团队的解决方案架构师，专注于复杂的Kubernetes、OpenShift和Microsoft解决方案。凭借超过25年的高级技术职位经验，他在设计和实施高级基础设施方面拥有深厚的专业知识。他撰写了多份关于Microsoft技术、RoCE/RDMA加速机器学习在Kubernetes/OpenShift以及容器化解决方案的参考设计指南，所有这些都可在NVIDIA Networking文档网站上获取。 |