RDG for NVIDIA network Accelerated VMware vSphere with Tanzu Cluster
Created on Jan 16, 2022 Updated on May 24, 2022 Introduction The following Reference Deployment Guide (RDG) explains how to install and configure VMware vSphere Tanzu version 7.0 Update 3d with NSX-Data Center 3.2 version on a single vSphere cluster over NVIDIA accelerated end-to-end 25/100 Gbps Ethernet solution. This setup is capable of running RDMA and DPDK-based applications. VMware's vSAN over RDMA will be used as a share storage for the vSphere Tanzu Workloads.
文档目录
Created on Jan 16, 2022
Updated on May 24, 2022
Introduction
The following Reference Deployment Guide (RDG) explains how to install and configure VMware vSphere Tanzu version 7.0 Update 3d with NSX-Data Center 3.2 version on a single vSphere cluster over NVIDIA® accelerated end-to-end 25/100 Gbps Ethernet solution. This setup is capable of running RDMA and DPDK-based applications. VMware’s vSAN over RDMA will be used as a share storage for the vSphere Tanzu Workloads.
Abbreviations and Acronyms
| Term | Definition | Term | Definition |
|---|---|---|---|
| DAC | Direct Attached Cable | RDMA | Remote Direct Memory Access |
| DHCP | Dynamic Host Configuration Protocol | RoCE | RDMA over Converged Ethernet |
| DPDK | Data Plane Development Kit | SDS | Software-Defined Storage |
| CNI | Container Network Interface | VDS | vSphere Distributed Switch |
| NOS | Network Operation System | VM | Virtual Machine |
Introduction
Provisioning Tanzu Kubernetes cluster for running RDMA and DPDK-based workloads may become an extremely complicated task. Proper design, and software and hardware component selection may become a gating task toward successful deployment.
This guide provides a step-by-step instructions to deploy vSphere with Tanzu with combined Management, Edge and Workload functions on a single vSphere cluster including technology overview, design, component selection, and deployment steps.
vSphere with Tanzu requires specific networking configuration to enable connectivity to the Supervisor Clusters, vSphere Namespaces, and all objects that run inside the namespaces, such as vSphere Pods, VMs, and Tanzu Kubernetes clusters. We are going to configure the networking manually by deploying a new instance of NSX-T Data Center and vSphere VDS.
VMware’s vSANoRDMA is now fully qualified and available as of the ESXi 7.0 U2 release, making it ready for deployments.
In this document, we will be using the NVIDIA Network Operator which is in responsible for deploying and configuring with a Host Device Network mode. This allow to run RDMA and DPDK workloads on a Tanzu Kubernetes Cluster Worker Node.
References
- What is RDMA?
- RDMA over Converged Ethernet (RoCE)
- ConnectX® Ethernet Driver for VMware® ESXi Server
- NVIDIA Cumulus
- vSphere with Tanzu
- Supervisor Cluster Networking
- VMware vSAN 7 Design guide
- How-to: Configure a vSphere Distributed Switch with NVIDIA network fabric.
- RDG: VMware vSAN over RoCE on VMware vSphere 7.0 U3.
- NVIDIA Network Operator
Solution Architecture
Key Components and Technologies
-
NVIDIA Spectrum 以太网交换机 Flexible form-factors with 16 to 128 physical ports, supporting 1GbE through 400GbE speeds. Based on a ground-breaking silicon technology optimized for performance and scalability, NVIDIA Spectrum switches are ideal for building high-performance, cost-effective, and efficient Cloud Data Center Networks, Ethernet Storage Fabric, and Deep Learning Interconnects. NVIDIA combines the benefits of NVIDIA Spectrum™ switches, based on an industry-leading application-specific integrated circuit (ASIC) technology, with a wide variety of modern network operating system choices, including NVIDIA Cumulus® Linux, SONiC and NVIDIA Onyx®.
-
NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.
-
NVIDIA ConnectX SmartNICs 10/25/40/50/100/200 and 400G Ethernet Network 网卡 The industry-leading NVIDIA® ConnectX® family of smart network interface cards (SmartNICs) offer advanced hardware offloads and accelerations. NVIDIA Ethernet adapters enable the highest ROI and lowest Total Cost of Ownership for hyperscale, public and private clouds, storage, machine learning, AI, big data, and telco platforms.
-
NVIDIA LinkX Cables The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and 100, 200 and 400Gb/s InfiniBand products for Cloud, HPC, hyperscale, Enterprise, telco, storage and artificial intelligence, data center applications.
-
vSphere with Tanzu Run Kubernetes workloads using your existing IT infrastructure. vSphere with Tanzu bridges the gap between IT and developers for cloud-native apps on-premises and in the cloud.
-
VMware vSphere Distributed Switch (VDS) provides a centralized interface from which you can configure, monitor and administer virtual machine access switching for the entire data center. The VDS provides simplified Virtual Machine network configuration, enhanced network monitoring and troubleshooting.
功能
- VMware NSX-T Data Center 提供敏捷的软件定义基础设施，用于构建云原生应用环境。NSX-T Data Center 专注于为新兴应用框架和架构提供网络、安全、自动化及运维简化，支持异构端点环境和技术栈，涵盖云原生应用、裸机工作负载、多虚拟机管理程序环境、公有云及多云。NSX-T Data Center 专为开发组织的管理、运维和消费而设计，允许IT团队和开发团队选择最适合其应用的技术。
- vSAN over RoCE Support for VMware (vSAN RDMA) 为vSAN提供增强性能。
- RDMA (远程直接内存访问) 是一种创新技术，可提升数据通信性能和效率。RDMA使数据传输更高效，支持服务器与存储之间快速移动数据，无需使用操作系统或占用服务器CPU，从而提高吞吐量、降低延迟并释放CPU以运行应用程序。
- RDMA over Converged Ethernet (RoCE) 或 InfiniBand over Ethernet (IBoE) 是一种网络协议，允许通过以太网网络进行远程直接内存访问（RDMA），通过将InfiniBand（IB）传输数据包封装在以太网上实现。
- NVIDIA Network Operator 利用Kubernetes CRDs和Operator SDK管理网络相关组件，以在Kubernetes集群中实现快速网络、RDMA和GPUDirect。
逻辑设计
该设置使用一个vSphere集群，包含4台ESXi服务器，连接到两台NVIDIA® Spectrum® SN2010以太网交换机（管理、入口和出口流量）和一台NVIDIA® Spectrum® 2 SN3700以太网交换机（高速vSAN、RDMA和DPDK流量）。vCenter和NSX-T Manager虚拟机将放置在同一集群中。
警告： 对于生产设计，建议将vCenter和NSX-T Manager放置在单独的管理集群中。这是VMware Validated Design (VVD) 和基于VVD的VMware Cloud Foundation (VCF) 的建议。
网络设计
vSphere Tanzu网络
vSphere with Tanzu需要特定的网络配置，以启用与Supervisor Cluster、vSphere Namespaces以及命名空间内所有对象（如vSphere Pods、VM和Tanzu Kubernetes集群）的连接。我们将通过部署新的NSX-T Data Center实例手动配置Supervisor Cluster网络。
SL-WL01-DS-01 VDS：
- 管理网络 (VLAN 1 - 192.168.1.x/24) – ESXi VMkernel接口和管理VM（如NSX-T Manager和vCenter）将位于此网络。
警告： 需要DHCP和DNS服务。本指南不涵盖组件的安装和配置。
- vMotion网络 (VLAN 1611 – 192.168.11.0/24) – ESXi vMotion VMkernel接口将位于此网络。
- NSX-T Geneve Overlay网络（ESXi主机） (VLAN 1624 - 192.168.24.0/24) – 此网络将用于ESXi主机上的Geneve Overlay隧道端点VMkernel接口。
- NSX-T Geneve Overlay网络（Edge VM） (VLAN 1625 - 192.168.25.0/24) – 此网络将用于Edge VM上的Geneve Overlay隧道端点VMkernel接口。
- NSX-T Edge VM上行链路网络 (VLAN 1630 -192.168.25.0/24) – 此网络将用于...
SL-WL01-DS-02 VDS：
- vSAN网络 (VLAN 1612 – 192.168.12.0/24) – ESXi vSAN VMkernel接口将位于此网络。
- RDMA网络 (VLAN 1614 – 192.168.14.0/24) – ESXi vSAN VMkernel接口将位于此网络。
互联网网络 – 环境（包括TKG集群）可以访问互联网。
Supervisor Cluster网络
工作负载管理将在NSX-T中实现。
- Pod CIDR (默认 – 10.244.0.0/20) – 此网络用于Kubernetes Pods。它将进一步划分为每个命名空间或每个TKG集群的/28子网。这是一个内部地址池，无需从物理路由器路由。
- Services CIDR (默认 10.96.0.0/23) – 此网络池将在创建Kubernetes服务时使用。这是一个内部地址池，无需从物理路由器路由。
- Ingress CIDR (192.168.100.0/24) – 此网络池将为应用部署所需的负载均衡服务提供地址。例如，一个NSX-T负载均衡器VIP将分配给Supervisor Cluster的控制平面IP地址。
- Egress CIDR (192.168.200.0/24) – 此网络池将在Pods需要与NSX-T环境外部通信（如访问互联网）时使用。例如，一个IP将分配给T0路由器作为Pods访问互联网时的源NAT（SNAT）源地址。
警告： Ingress和Egress CIDR网络都需要在物理路由器上路由到T0的下一跳。
软件栈组件
本指南假定已安装以下软件和驱动程序：
- VMware ESXi 7.0.3，构建版本 17630552
- VMware vCenter 7.0.3，构建版本 17694817
- Distributed Switch 7.0.3
- NSX-T 3.2
- NVIDIA® ConnectX® Driver for VMware ESXi Server v4.21.71.101
- NVIDIA® ConnectX®-6DX FW版本 22.32.2004
- NVIDIA® ConnectX®-6LX FW版本 26.32.1010
- 网络操作系统 (NOS)：NVIDIA Cumulus™ v5.1
物料清单
物料清单
以下硬件配置用于本指南所述的 vSphere 环境。
Supervisor 集群硬件：
Supervisor 集群计算/存储：
| VM | CPU | MEM | DISK |
|---|---|---|---|
| 计算集群 vCenter（基于 Small） | 4 | 20GB | 48GB |
| NSX-T Manager x 3 | 6 | 24GB | 300GB |
| NSX-T Edge VM（至少 Large）x 2 | 8 | 32GB | 200GB |
| Supervisor 集群 VM（基于 Tiny） | 2 | 8GB | 22GB |
| 来宾集群（基于 x-small）x 3 | 2 | 2GB | 16GB |
部署和配置
警告： 安装 ESXi、vCenter、ESXi 主机、配置虚拟数据中心、集群、vSphere 集群以及将主机添加到集群不在本文档的讨论范围内。
布线
本文档涵盖高可用性 VMware vSphere 集群部署。
Supervisor 集群：
vSphere 分布式交换机设计
网络
前提条件
-
交换机操作系统 NVIDIA® Cumulus® Linux v5.1
-
网卡 NVIDIA ConnectX-6 Lx 和 ConnectX-6 Dx 网卡。 最新固件和驱动程序版本。更多信息，请参阅： 如何：在 VMware ESXi 6.5 及以上版本上更新 NVIDIA ConnectX-5/6 网卡固件 如何：在 VMware ESXi 6.7/7.0 及以上版本上升级 NVIDIA ConnectX 驱动程序
-
vSAN over RDMA 需要 vSAN RDMA Ready 网卡，可在 vSAN VCG 上找到。
-
管理网络 需要 DHCP 和 DNS 服务。
警告： 本指南不涵盖组件的安装和配置。
主机网络配置
下表提供了 ESXi 服务器、交换机名称及其网络配置的详细信息。
SL-WL01-Cluster01 Supervisor 集群
| 服务器 | 服务器名称 | IP 和网卡 |
|---|---|---|
| 高速以太网网络 | ||
| ESXi-01 | clx-host-51 | vmk1: 192.168.11.111 (vMotion)vmk2: 192.168.12.111 (vSAN)vmk10: 来自 IP 池 192.168.24.0/24 (NSX Host TEP) |
| ESXi-02 | clx-host-52 | vmk1: 192.168.11.112 (vMotion)vmk2: 192.168.12.112 (vSAN)vmk10: 来自 IP 池 192.168.24.0/24 (NSX Host TEP) |
| ESXi-03 | clx-host-53 | vmk1: 192.168.11.113 (vMotion)vmk2: 192.168.12.113 (vSAN)vmk10: 来自 IP 池 192.168.24.0/24 (NSX Host TEP) |
| ESXi-04 | clx-host-54 | vmk1: 192.168.11.114 (vMotion)vmk2: 192.168.12.114 (vSAN)vmk10: 来自 IP 池 192.168.24.0/24 (NSX Host TEP) |
| Leaf-01 | clx-swx-033 | |
| Leaf-02 | clx-swx-034 | |
| Leaf-03 | clx-swx-035 | |
| vCenter (VM) | sl01w01vc01 | |
| NSX-T Manager 01 (VM) | sl01w01nsx01 | |
| NSX-T Edge 01 |
网络交换机配置
ESXi与Leaf交换机的连接
端口通道和VLAN配置
在Supervisor集群中的两个Leaf NVIDIA SN2010交换机上运行以下命令，以配置端口通道和VLAN。
clx-swx-033交换机示例：
cumulus@clx-swx-033:mgmt:~$sudo nv set system hostname clx-swx-033
cumulus@clx-swx-033:mgmt:~$sudo nv set interface lo ip address 10.10.10.1/32
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-22 type swp
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp7 link speed 1G
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp7 link mtu 1500
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-4 bridge domain br_default
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp7 bridge domain br_default
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1611
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1624-1625
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1630
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1 ip address 192.168.1.254/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1624 ip address 192.168.24.1/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1625 ip address 192.168.25.1/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1630 ip address 192.168.30.1/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1 link mtu 1500
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 0.0.0.0/0 via 192.168.1.21
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 192.168.100.0/24 via 192.168.30.4
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 192.168.200.0/24 via 192.168.30.4
cumulus@clx-swx-033:mgmt:~$sudo nv set interface peerlink bond member swp21-22
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag backup 10.10.10.2
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag peer-ip linklocal
cumulus@clx-swx-033:mgmt:~$sudo nv config apply
cumulus@clx-swx-033:mgmt:~$sudo nv config save
clx-swx-034交换机示例：
cumulus@clx-swx-033:mgmt:~$sudo nv set system hostname clx-swx-034
cumulus@clx-swx-033:mgmt:~$sudo nv set interface lo ip address 10.10.10.2/32
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-22 type swp
cumulus@clx-swx-033:mgmt:~$sudo nv set interface swp1-4 bridge domain br_default
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1611
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1624-1625
cumulus@clx-swx-033:mgmt:~$sudo nv set bridge domain br_default vlan 1630
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1 ip address 192.168.1.254/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1624 ip address 192.168.24.1/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1625 ip address 192.168.25.1/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1630 ip address 192.168.30.1/24
cumulus@clx-swx-033:mgmt:~$sudo nv set interface vlan1 link mtu 1500
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 0.0.0.0/0 via 192.168.1.21
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 192.168.100.0/24 via 192.168.30.4
cumulus@clx-swx-033:mgmt:~$sudo nv set vrf default router static 192.168.200.0/24 via 192.168.30.4
cumulus@clx-swx-033:mgmt:~$sudo nv set interface peerlink bond member swp21-22
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag mac-address 44:38:39:BE:EF:AA
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag backup 10.10.10.1
cumulus@clx-swx-033:mgmt:~$sudo nv set mlag peer-ip linklocal
cumulus@clx-swx-033:mgmt:~$sudo nv config apply
cumulus@clx-swx-033:mgmt:~$sudo nv config save
在高速NVIDIA SN2100交换机上配置端口通道和VLAN
在vSphere集群中的高速交换机上运行以下命令，以配置端口通道和VLAN。
clx-swx-035示例：
cumulus@clx-swx-035:mgmt:~$sudo nv set interface swp9-16 link mtu 1500
cumulus@clx-swx-035:mgmt:~$sudo nv set interface swp1-16 bridge domain br_default
cumulus@clx-swx-035:mgmt:~$sudo nv set bridge domain br_default vlan 1612
cumulus@clx-swx-035:mgmt:~$sudo nv set bridge domain br_default vlan 1614
cumulus@clx-swx-035:mgmt:~$sudo set interface swp1-16 bridge domain br_default untagged 1614
cumulus@clx-swx-035:mgmt:~$sudo nv config apply
cumulus@clx-swx-035:mgmt:~$sudo nv config save
在高速SN2100交换机上启用RDMA over Converged Ethernet无损模式（含PFC和ETS）
RoCE传输用于加速vSAN网络。为获得最佳性能，网络配置为无损模式。
在所有Leaf交换机上运行以下命令，以配置NVIDIA Cumulus的无损网络。
cumulus@clx-swx-035:mgmt:~$sudo nv set qos roce
cumulus@clx-swx-035:mgmt:~$sudo nv config apply
cumulus@clx-swx-035:mgmt:~$sudo nv config save
要检查RoCE配置，请运行以下命令：
cumulus@leaf-01:mgmt:~$sudo nv show qos roce
operational applied description
------------------ ----------- -------- ------------------------------------------------------
enable on Turn the feature 'on' or 'off'. The default is 'off'.
mode lossless lossless Roce Mode
cable-length 100 100 Cable Length(in meters) for Roce Lossless Config
congestion-control
congestion-mode ECN Congestion config mode
enabled-tc 0,3 Congestion config enabled Traffic Class
max-threshold 1.43 MB Congestion config max-threshold
min-threshold 146.48 KB Congestion config min-threshold
pfc
pfc-priority 3 switch-prio on which PFC is enabled
rx-enabled enabled PFC Rx Enabled status
tx-enabled enabled PFC Tx Enabled status
trust
trust-mode pcp,dscp Trust Setting on the port for packet classification
RoCE PCP/DSCP->SP mapping configurations
===========================================
pcp dscp switch-prio
-- --- ----------------------- -----------
0 0 0,1,2,3,4,5,6,7 0
1 1 8,9,10,11,12,13,14,15 1
2 2 16,17,18,19,20,21,22,23 2
3 3 24,25,26,27,28,29,30,31 3
4 4 32,33,34,35,36,37,38,39 4
5 5 40,41,42,43,44,45,46,47 5
6 6 48,49,50,51,52,53,54,55 6
7 7 56,57,58,59,60,61,62,63 7
RoCE SP->TC mapping and ETS configurations
=============================================
switch-prio traffic-class scheduler-weight
-- ----------- ------------- ----------------
0 0 0 DWRR-50%
1 1 0 DWRR-50%
2 2
0 DWRR-50% 3 3 3 DWRR-50% 4 4 0 DWRR-50% 5 5 0 DWRR-50% 6 6 6 strict-priority 7 7 0 DWRR-50%
RoCE pool config
name mode size switch-priorities traffic-class
0 lossy-default-ingress Dynamic 50.0% 0,1,2,4,5,6,7 - 1 roce-reserved-ingress Dynamic 50.0% 3 - 2 lossy-default-egress Dynamic 50.0% - 0,6 3 roce-reserved-egress Dynamic inf - 3
Exception List
description
Supervisor Cluster Configuration
Prerequisites
-
Host BIOS
Verify that an SR-IOV supported server platform is being used and review the BIOS settings in the server platform vendor documentation to enable SR-IOV in the BIOS.
-
Physical server configuration
All ESXi servers must have the same PCIe placement for the NIC and expose the same interface name.
-
Experience with Kubernetes
Familiarization with the Kubernetes Cluster architecture is essential.
-
Verify that your environment meets the system requirements for configuring a vSphere cluster as a Supervisor Cluster. For information about requirements, see System Requirements for Setting Up vSphere with Tanzu with NSX-T Data Center.
-
Assign the VMware vSphere 7 Enterprise Plus with an Add-on for Kubernetes license to all ESXi hosts that will be part of the Supervisor Cluster.
-
Verify that you have the Modify cluster-wide configuration privilege on the cluster.
-
Verify that in your environment NTP configured and works properly.
-
Create and configure 2 VMware VDS by using following document - How-to: Configure a vSphere Distributed Switch with NVIDIA network fabric.
Two VDS will be used in the environment:
- SL-WL01-DS01 with following port groups:
- SL-WL01-MGMT-VLAN1
- SL-WL01-vMotion-VLAN611
- SL-WL01-Trunk-PG
- SL-WL01-DS02 with following port groups:
- SL-WL01-vSAN-VLAN1612
- SL-WL01-RDMA-VLAN1614
- SL-WL01-DS01 with following port groups:
-
Create and configure a VMware vSAN RDMA cluster by using following document - RDG: VMware vSAN over RoCE on VMware vSphere 7.0 U3.
Warning: As one of prerequisites for Supervisor Cluster configuration, you need to Create the VM Storage Policies. We will use in our case the vSAN Storage Policy.
-
Enable DRS and HA on the SL-WL01-Cluster01 vSphere Cluster.
-
Enable SR-IOV.
NVIDIA Network Operator leverages Kubernetes CRDs and Operator SDK to manage networking-related components to enable fast networking and RDMA for workloads in TKG cluster. The fast network is a secondary network of the K8s cluster for applications that require high bandwidth or low latency.
In Tanzu Kubernetes Cluster we can use Dynamic DirectPath I/O to assign multiple PCI passthrough or SR-IOV devices to a Kubernetes Workload VM.
To make it work, we need to enable SR-IOV capability on a ConnectX-6 Dx network adapter.
To Enable SR-IOV:
-
Launch the vSphere Web Client and connect to a vCenter Server instance.
-
Navigate to a ESXi host and select Configure → Hardware → PCI Devices. Click on ALL PCI DEVICES. Click on Filter.
-
Type Mellanox and click on Vendor Name.
-
Select a ConnectX-6 Dx NIC.
-
Click on CONFIGURE SR-IOV.
-
Enable SR-IOV and set the number of Virtual functions (VF).
-
Click OK.
-
Click on PASSTHROUGH-ENABLED DEVICES to verify that 8 VFs were enabled.
-
-
Enable Content Library.
To enable Content Library:
-
Launch the vSphere Web Client and connect to a vCenter Server instance.
-
Navigate to vCenter → Menu → Content Libraries.
-
Click CREATE.
-
Fill Name → Tanzu.
-
Click NEXT.
-
Select Subscribed content library. Fill the Subscription URL
-
style="--color: #212529"> → https://wp-content.vmware.com/v2/latest/lib.json.
The following shows that both interfaces for the Tier-0 Gateway are created correctly.
-
Click Set under HA VIP Configuration.
-
Click ADD HA VIP CONFIGURATION. Fill IP Address / Mask → 192.168.30.4/24, Interface → T0-Uplink1-Int1, T0-Uplink1-Int2.
-
Click ADD.
The following shows that the HA VIP configuration has been successfully created.
-
To ensure that the Tier-0 Gateway Uplink is configured correctly, we shall login to the next hop device, in my case is the SN2010, to do a ping test.
Firstly ping yourself ie. 192.168.30.1 which is configured on the switch then follow be the HA VIP configured on the Tier-0 Gateway.
-
Lastly we need to configured a default route out so that the containers can communicate back to IP addresses outside the NSX-T domain.
Click Set under Static Routes in the Routing option.
Warning: If you are using BGP, then probably this step would differ.
-
Click ADD STATIC ROUTE. Fill Name → Default, Network → 0.0.0.0/0.
-
Click Set under the Next Hops option.
-
Click SET NEXT HOP. IP Address → 192.168.30.1.
-
Click ADD.
-
Click SAVE.
-
Click CLOSE.
-
Click SAVE.
-
Once the static route has been added, one way is to test is from outside the NSX-T domain. In our case, we have the DG VM which is outside the NSX-T domain and the gateway of the VM is pointing to the SN2010 as well. A ping test was done from the VM to the Tier-0 Gateway VIP. If the ping test is successful, it means the static route we added to the Tier-0 gateway is successfully configured.
-
Validate whether NSX-T has been successfully set up for vSphere with Kubernetes.
As all the configuration on the NSX-T, vSphere VDS and physical network are set up, now go back to Workload Management to see whether we are ready to deploy Workload Management Clusters.
Enabling Workload Management and Creating a Supervisor Cluster.
To enable Workload Management and create a Supervisor Cluster:
-
Launch the vSphere Web Client and connect to a vCenter Server instance.
-
Navigate to vCenter → Menu → Workload Management.
-
Click GET STARTED.
-
Select NSX under Select a networking stack.
-
Select vSphere cluster → Sl-WL01-Cluster01.
-
Click NEXT.
-
Select a storage policy → vSAN Default Storage Policy.
-
Click NEXT.
-
Configure Management Network. Network Mode → DHCP, Network → SL-WL01-MGMT-VLAN1.
-
Click NEXT.
Error: NTP is very important. Thus, when you see authentication errors in the wcpsvc logs, usually
this has to do with NTP not working correctly.
-
Configure Workload Network. vSphere Distributed Switch → SL-WL01-DS01, Edge Cluster → EdgeCluster1, DNS Server(s) → 192.168.1.21, Tier-0 Gateway → T0-EdgeCluster1, NAT Mode → Enabled (Default),** Subnet Prefix → /28 (Default), Namespace Network → 10.244.0.0./20 (Default), Service CIDR → 10.96.0.0./23 (Default), Ingress CIDRs → 192.168.100.0/24, Egress CIDRs → 192.168.200.0/24.
-
Click NEXT.
-
Click Add to select the Content Library.
-
Select Tanzu content library.
-
Click OK.
-
Click NEXT.
-
Click FINISH.
-
The Supervisor Cluster Control VMs is being deployed.
-
Come back in about 25 mins and see the Supervisor Cluster being deployed.
You can view the Network configuration here.
Create New VM Class
To create New VM Class included second high speed network:
-
Launch the vSphere Web Client and connect to a vCenter Server instance.
-
Navigate to vCenter → Menu → Workload Management → Services.
-
Click GOT IT.
-
Click CREATE VM CLASS.
Fill following data:
VM Class Name → best-effort-2xlarge-pci
vCPU Count → 8
Memory → 64 GB
Add Advanced Configuration → Select PCI Device.
Click NEXT.
-
Click ADD PCI DEVICE and select Dynamic DirectPath IO.
-
Select ConnectX Family nmlx5Gen Virtual Function. And click NEXT and FINISH in case you don't want to add another PCI Devices.
-
Click FINISH.
Create Namespace, Set up Permissions, Storage, Add Content Library and VM Classes
To create a Namespace:
-
Launch the vSphere Web Client and connect to a vCenter Server instance.
-
Navigate to vCenter → Menu → Workload Management → Namespaces.
-
Click Create Namespace.
-
Provide a name and select the Supervisor Cluster.
-
Add permissions, storage, content library, and VM classes as needed.
-
Click OK to finish.
-
Click on Workload Management.
-
Click on the Namespaces tab.
-
Click CREATE NAMESPACE.
-
Select Cluster → SL-WL01-Cluster01 where you want to create the namespace and give a Name → sl-wl01-ns01 to the namespace.
-
Click CREATE.
-
The namespace has been created successfully.
-
Click ADD PERMISSIONS.
-
Give permissions to Administrator@vsphere.local with edit role.
-
Click ADD STORAGE to add a storage to the Namespace.
-
Add Storage Policies → vSAN Default Storage Policy.
-
Click ADD CONTENT LIBRARY to add a Content Library.
-
Select the Tanzu Content Library.
-
Click OK.
-
Click ADD VM CLASS to add VM CLASSES.
-
Select the best-effort-2xlarge-pci VM class created before. We are going to use the VM class as a TKC Worker VM template as we need a second high speed network. In additional select the best-effort-small. We are going to use the VM class as a TKC control VM template.
This is how it looks like.
Download and Install the Kubernetes CLI Tools for vSphere
You can use Kubernetes CLI tools for vSphere to view and control vSphere with Tanzu namespaces and clusters.
The Kubernetes CLI tools download package includes two executables: the standard open-source kubectl and the vSphere Plugin for kubectl.
-
Launch the vSphere Web Client and connect to a vCenter Server instance.
-
Navigate to vCenter → Menu → Workload Management. Select the Namespace ns-01.
-
Select the Summary tab and locate the Status area on this page.
-
Select Open underneath the Link to CLI Tools heading to open the download page.
-
Using a browser, navigate to the Kubernetes CLI Tools download URL for your environment. Referee to the prerequisites section above for guidance on how to locate the download URL.
-
Select the operating system. Depends on your K8s CLI client VM OS.
-
Download the
vsphere-plugin.zipfile. -
Extract the contents of the ZIP file to a working directory. The vsphere-plugin.zip package contains two executable files: kubectl and vSphere Plugin for kubectl.
kubectlis the standard Kubernetes CLI.kubectl-vsphereis the vSphere Plugin for kubectl to help you authenticate with the Supervisor Cluster and Tanzu Kubernetes clusters using your vCenter Single Sign-On credentials. -
Add the location of both executables to your system's PATH variable.
-
To verify the installation of the kubectl CLI, start a shell, terminal, or command prompt session and run the command
kubectl. You see the kubectl banner message, and the list of command-line options for the CLI.
-
To verify the installation of the vSphere Plugin for kubectl, run the command
kubectl vsphere. You see the vSphere Plugin for kubectl banner message, and the list of command-line options for the plugin.
Create TKG Clusters
Start a shell, terminal, or command prompt session on Kubernetes Client VM. In our lab this is a Ubuntu 20.04 VM.
To begin, we are login, to the Supervisor Cluster.
K8s CLI VM console
root@user:~# kubectl-vsphere login --vsphere-username administrator@vsphere.local --server=192.168.100.2 --insecure-skip-tls-verify
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
Logged in successfully.
You have access to the following contexts:
192.168.100.2
sl-wl01-ns01
If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.
To change context, use `kubectl config use-context <workload name>`
root@user:~#
获取节点列表、命名空间列表，并将上下文设置为我们之前创建的新命名空间。
K8s CLI VM console
root@user:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
422c84eaa32359de85bf2c23da755530 Ready control-plane,master 11d v1.21.0+vmware.wcp.2
422cbcc4e5e9327986c2d05773175a6b Ready control-plane,master 11d v1.21.0+vmware.wcp.2
422cfca0e639bb91581ee525ae08813b Ready control-plane,master 11d v1.21.0+vmware.wcp.2
sl01w01esx11.vwd.clx Ready agent 11d v1.21.0-sph-fc0747b
sl01w01esx12.vwd.clx Ready agent 11d v1.21.0-sph-fc0747b
sl01w01esx13.vwd.clx Ready agent 11d v1.21.0-sph-fc0747b
sl01w01esx14.vwd.clx Ready agent 11d v1.21.0-sph-fc0747b
root@user:~# kubectl get ns
NAME STATUS AGE
default Active 11d
kube-node-lease Active 11d
kube-public Active 11d
kube-system Active 11d
sl-wl01-ns01 Active 51m
svc-tmc-c8 Active 11d
vmware-system-appplatform-operator-system Active 11d
vmware-system-capw Active 11d
vmware-system-cert-manager Active 11d
vmware-system-csi Active 11d
vmware-system-kubeimage Active 11d
vmware-system-license-operator Active 11d
vmware-system-logging Active 11d
vmware-system-nsop Active 11d
vmware-system-nsx Active 11d
vmware-system-registry Active 11d
vmware-system-supervisor-services Active 11d
vmware-system-tkg Active 11d
vmware-system-ucs Active 11d
vmware-system-vmop Active 11d
root@user:~# kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
192.168.100.2 192.168.100.2 wcp:192.168.100.2:administrator@vsphere.local
* sl-wl01-ns01 192.168.100.2 wcp:192.168.100.2:administrator@vsphere.local sl-wl01-ns01
root@user:~# kubectl config use-context sl-wl01-ns01
Switched to context "sl-wl01-ns01".
确保 StorageClass 可用，并且 TKG 来宾集群虚拟机镜像已同步并在内容库中可用。镜像用于创建 TKG 来宾集群中的控制平面 VM 和工作节点 VM。
K8s CLI VM console
root@user:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
vsan-default-storage-policy csi.vsphere.vmware.com Delete Immediate true 11d
root@user:~# kubectl get virtualmachineimages
NAME CONTENTSOURCENAME VERSION OSTYPE FORMAT AGE
ob-15957779-photon-3-k8s-v1.16.8---vmware.1-tkg.3.60d2ffd f636d81a-96b1-4861-8516-39e4c032c589 v1.16.8+vmware.1-tkg.3.60d2ffd vmwarePhoton64Guest ovf 11d
ob-16466772-photon-3-k8s-v1.17.7---vmware.1-tkg.1.154236c f636d81a-96b1-4861-8516-39e4c032c589 v1.17.7+vmware.1-tkg.1.154236c vmwarePhoton64Guest ovf 11d
ob-16545581-photon-3-k8s-v1.16.12---vmware.1-tkg.1.da7afe7 f636d81a-96b1-4861-8516-39e4c032c589 v1.16.12+vmware.1-tkg.1.da7afe7 vmwarePhoton64Guest ovf 11d
ob-16551547-photon-3-k8s-v1.17.8---vmware.1-tkg.1.5417466 f636d81a-96b1-4861-8516-39e4c032c589 v1.17.8+vmware.1-tkg.1.5417466 vmwarePhoton64Guest ovf 11d
ob-16897056-photon-3-k8s-v1.16.14---vmware.1-tkg.1.ada4837 f636d81a-96b1-4861-8516-39e4c032c589 v1.16.14+vmware.1-tkg.1.ada4837 vmwarePhoton64Guest ovf 11d
ob-16924026-photon-3-k8s-v1.18.5---vmware.1-tkg.1.c40d30d f636d81a-96b1-4861-8516-39e4c032c589 v1.18.5+vmware.1-tkg.1.c40d30d vmwarePhoton64Guest ovf 11d
ob-16924027-photon-3-k8s-v1.17.11---vmware.1-tkg.1.15f1e18 f636d81a-96b1-4861-8516-39e4c032c589 v1.17.11+vmware.1-tkg.1.15f1e18 vmwarePhoton64Guest ovf 11d
ob-17010758-photon-3-k8s-v1.17.11---vmware.1-tkg.2.ad3d374 f636d81a-96b1-4861-8516-39e4c032c589 v1.17.11+vmware.1-tkg.2.ad3d374 vmwarePhoton64Guest ovf 11d
ob-17332787-photon-3-k8s-v1.17.13---vmware.1-tkg.2.2c133ed f636d81a-96b1-4861-8516-39e4c032c589 v1.17.13+vmware.1-tkg.2.2c133ed vmwarePhoton64Guest ovf 11d
ob-17419070-photon-3-k8s-v1.18.10---vmware.1-tkg.1.3a6cd48 f636d81a-96b1-4861-8516-39e4c032c589 v1.18.10+vmware.1-tkg.1.3a6cd48 vmwarePhoton64Guest ovf 11d
ob-17654937-photon-3-k8s-v1.18.15---vmware.1-tkg.1.600e412 f636d81a-96b1-4861-8516-39e4c032c589 v1.18.15+vmware.1-tkg.1.600e412 vmwarePhoton64Guest ovf 11d
ob-17658793-photon-3-k8s-v1.17.17---vmware.1-tkg.1.d44d45a f636d81a-96b1-4861-8516-39e4c032c589 v1.17.17+vmware.1-tkg.1.d44d45a vmwarePhoton64Guest ovf 11d
ob-17660956-photon-3-k8s-v1.19.7---vmware.1-tkg.1.fc82c41 f636d81a-96b1-4861-8516-39e4c032c589 v1.19.7+vmware.1-tkg.1.fc82c41 vmwarePhoton64Guest ovf 11d
ob-17861429-photon-3-k8s-v1.20.2---vmware.1-tkg.1.1d4f79a f636d81a-96b1-4861-8516-39e4c032c589 v1.20.2+vmware.1-tkg.1.1d4f79a vmwarePhoton64Guest ovf 11d
ob-18035533-photon-3-k8s-v1.18.15---vmware.1-tkg.2.ebf6117 f636d81a-96b1-4861-8516-39e4c032c589 v1.18.15+vmware.1-tkg.2.ebf6117 vmwarePhoton64Guest ovf 11d
ob-18035534-photon-3-k8s-v1.19.7---vmware.1-tkg.2.f52f85a f636d81a-96b1-4861-8516-39e4c032c589 v1.19.7+vmware.1-tkg.2.f52f85a vmwarePhoton64Guest ovf 11d
ob-18037317-photon-3-k8s-v1.20.2---vmware.1-tkg.2.3e10706 f636d81a-96b1-4861-8516-39e4c032c589 v1.20.2+vmware.1-tkg.2.3e10706 vmwarePhoton64Guest ovf 11d
ob-18186591-photon-3-k8s-v1.20.7---vmware.1-tkg.1.7fb9067 f636d81a-96b1-4861-8516-39e4c032c589 v1.20.7+vmware.1-tkg.1.7fb9067 vmwarePhoton64Guest ovf 11d
ob-18284400-photon-3-k8s-v1.18.19---vmware.1-tkg.1.17af790 f636d81a-96b1-4861-8516-39e4c032c589 v1.18.19+vmware.1-tkg.1.17af790 vmwarePhoton64Guest ovf 11d
ob-18324108-photon-3-k8s-v1.19.11---vmware.1-tkg.1.9d9b236 f636d81a-96b1-4861-8516-39e4c032c589 v1.19.11+vmware.1-tkg.1.9d9b236 vmwarePhoton64Guest ovf 11d
ob-18461281-photon-3-k8s-v1.20.9---vmware.1-tkg.1.a4cee5b f636d81a-96b1-4861-8516-39e4c032c589 v1.20.9+vmware.1-tkg.1.a4cee5b vmwarePhoton64Guest ovf 11d
ob-18532793-photon-3-k8s-v1.19.14---vmware.1-tkg.1.8753786 f636d81a-96b1-4861-8516-39e4c032c589 v1.19.14+vmware.1-tkg.1.8753786 vmwarePhoton64Guest ovf 11d
ob-18592554-photon-3-k8s-v1.21.2---vmware.1-tkg.1.ee25d55 f636d81a-96b1-4861-8516-39e4c032c589 v1.21.2+vmware.1-tkg.1.ee25d55 vmwarePhoton64Guest ovf 11d
ob-18807685-tkgs-ova-ubuntu-2004-v1.20.8---vmware.1-tkg.2 f636d81a-96b1-4861-8516-39e4c032c589 v1.20.8+vmware.1-tkg.2 ubuntu64Guest ovf 11d
ob-18895415-photon-3-k8s-v1.19.16---vmware.1-tkg.1.df910e2 f636d81a-96b1-4861-8516-39e4c032c589 v1.19.16+vmware.1-tkg.1.df910e2 vmwarePhoton64Guest ovf 11d
ob-18900476-photon-3-k8s-v1.21.6---vmware.1-tkg.1.b3d708a f636d81a-96b1-4861-8516-39e4c032c589 v1.21.6+vmware.1-tkg.1.b3d708a vmwarePhoton64Guest ovf 11d
ob-18903450-photon-3-k8s-v1.20.12---vmware.1-tkg.1.b9a42f3 f636d81a-96b1-4861-8516-39e4c032c589 v1.20.12+vmware.1-tkg.1.b9a42f3 vmwarePhoton64Guest ovf 11d
root@user:~# kubectl get virtualmachineclasses
NAME CPU MEMORY AGE
best-effort-2xlarge-pci 8 64Gi 165m
best-effort-small 2 4Gi 165m
以上输出显示一切正常。我们已经切换到新的命名空间，并验证了 Storage Class、虚拟机镜像和 VM 类可用。现在可以继续部署 TKG 来宾集群。以下是用于部署集群的清单。
我们创建了以下清单文件 sl-wl01-tkc01.yaml。在此清单中，我们请求了单个控制平面节点和3个工作节点。
我们将使用 vsan-default-storage-policy 作为 Storage Class，因为这是我们在此命名空间中配置的唯一一个。
节点大小设置为：控制平面节点使用 best-effort-small，工作节点使用 best-effort-2xlarge-pci。
v1.20.8---vmware.1-tkg.2 虚拟机镜像将用于两者。
每个工作节点将添加两个卷：200GB 和 50GB。
将使用自定义 Antrea CNI。
要创建清单文件 sl-wl01-tkc01.yaml，请运行：
K8s CLI VM console
# 创建清单文件的命令未提供，请参考相关文档
root@user:~# vim sl-wl01-tkc01.yaml
示例 sl-wl01-tkc01.yaml：
K8s CLI VM console
apiVersion: run.tanzu.vmware.com/v1alpha2 #TKGS API endpoint
kind: TanzuKubernetesCluster #required parameter
metadata:
name: sl-wl01-tkc01 #cluster name, user defined
namespace: sl-wl01-ns01 #vsphere namespace
spec:
distribution:
fullVersion: v1.20.8+vmware.1-tkg.2
topology:
controlPlane:
replicas: 1 #number of control plane nodes
storageClass: vsan-default-storage-policy #storageclass for control plane
tkr:
reference:
name: v1.20.8---vmware.1-tkg.2 #vm image for control plane nodes
vmClass: best-effort-small #vmclass for control plane nodes
nodePools:
- name: workercx6dx
replicas: 3 #number of worker nodes
storageClass: vsan-default-storage-policy #storageclass for worker nodes
tkr:
reference:
name: v1.20.8---vmware.1-tkg.2 #vm image for worker nodes
vmClass: best-effort-2xlarge-pci #vmclass for worker nodes
volumes:
- capacity:
storage: 200Gi
mountPath: /var/lib/containerd
name: containerd
- capacity:
storage: 50Gi
mountPath: /var/lib/kubelet
name: kubelet
settings:
network:
cni:
name: antrea #Use Antrea CNI
pods:
cidrBlocks:
- 193.0.2.0/16 #Must not overlap with SVC
services:
cidrBlocks:
- 195.51.100.0/12 #Must not overlap with SVC
serviceDomain: managedcluster.local
要构建 TKG 集群，请运行：
K8s CLI VM console
root@user:~#kubectl apply -f sl-wl01-tkc01.yaml
要查看部署进度，首先查看集群（5-10 分钟后）。
K8s CLI VM console
root@user:~#kubectl get TanzuKubernetesCluster
NAME CONTROL PLANE WORKER TKR NAME AGE READY TKR COMPATIBLE UPDATES AVAILABLE
sl-wl01-tkc01 1 3 v1.20.8---vmware.1-tkg.2 137m True True
查询支持控制平面和节点的虚拟机。
K8s CLI VM console
root@user:~#kubectl get VirtualMachines
NAME POWERSTATE AGE
sl-wl01-tkc01-control-plane-crtld poweredOn 138m
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq poweredOn 134m
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv poweredOn 134m
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 poweredOn 134m
非常有趣的是对集群执行 describe 命令。
root@user:~# kubectl describe TanzuKubernetesCluster sl-wl01-tkc01
Name: sl-wl01-tkc01
Namespace: sl-wl01-ns01
Labels: run.tanzu.vmware.com/tkr=v1.20.8---vmware.1-tkg.2
Annotations: <none>
API Version: run.tanzu.vmware.com/v1alpha2
Kind: TanzuKubernetesCluster
Metadata:
Creation Timestamp: 2022-02-20T08:07:28Z
Finalizers:
tanzukubernetescluster.run.tanzu.vmware.com
Generation: 1
Managed Fields:
API Version: run.tanzu.vmware.com/v1alpha2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:distribution:
.:
f:fullVersion:
f:settings:
.:
f:network:
.:
f:cni:
.:
f:name:
f:pods:
.:
f:cidrBlocks:
f:serviceDomain:
f:services:
.:
f:cidrBlocks:
f:topology:
.:
f:controlPlane:
.:
f:replicas:
f:storageClass:
f:tkr:
.:
f:reference:
.:
f:name:
f:vmClass:
f:nodePools:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-02-20T08:07:28Z
API Version: run.tanzu.vmware.com/v1alpha2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"tanzukubernetescluster.run.tanzu.vmware.com":
f:labels:
.:
f:run.tanzu.vmware.com/tkr:
f:status:
.:
f:addons:
f:apiEndpoints:
f:conditions:
f:phase:
f:totalWorkerReplicas:
Manager: manager
Operation: Update
Time: 2022-02-20T08:11:31Z
Resource Version: 9909758
Self Link: /apis/run.tanzu.vmware.com/v1alpha2/namespaces/sl-wl01-ns01/tanzukubernetesclusters/sl-wl01-tkc01
UID: a11347b8-79ea-4d41-9b13-e448deb18522
Spec:
Distribution:
Full Version: v1.20.8+vmware.1-tkg.2
Settings:
Network:
Cni:
Name: antrea
Pods:
Cidr Blocks:
193.0.2.0/16
Service Domain: managedcluster.local
Services:
Cidr Blocks:
195.51.100.0/12
Topology:
Control Plane:
Replicas: 1
Storage Class: vsan-default-storage-policy
Tkr:
Reference:
Name: v1.20.8---vmware.1-tkg.2
Vm Class: best-effort-small
Node Pools:
Name: workercx6dx
Replicas: 3
Storage Class: vsan-default-storage-policy
Tkr:
Reference:
Name: v1.20.8---vmware.1-tkg.2
Vm Class: best-effort-2xlarge-pci
Volumes:
Capacity:
Storage: 200Gi
Mount Path: /var/lib/containerd
Name: containerd
Capacity:
Storage: 50Gi
Mount Path: /var/lib/kubelet
Name: kubelet
Status:
Addons:
Conditions:
Last Transition Time: 2022-02-20T08:11:36Z
Status: True
Type: Provisioned
Name: CoreDNS
Type: DNS
Version: v1.7.0_vmware.12
Conditions:
Last Transition Time: 2022-02-20T08:11:40Z
Status: True
Type: Provisioned
Name: antrea
Type: CNI
Version: v0.13.5+vmware.3
Conditions:
Last Transition Time: 2022-02-20T08:11:34Z
Status: True
Type: Provisioned
Name: pvcsi
Type: CSI
Version: vsphere70u2-f665008-8a37f95
Conditions:
Last Transition Time: 2022-02-20T08:11:33Z
Status: True
Type: Provisioned
Name: vmware-guest-cluster
Type: CPI
Version: v0.1-87-gb6bb261
Conditions:
Last Transition Time: 2022-02-20T08:11:42Z
Status: True
Type: Provisioned
Name: authsvc
Type: AuthService
Version: v0.1-71-g64e1c73
Conditions:
Last Transition Time: 2022-02-20T08:11:36Z
Status: True
Type: Provisioned
Name: kube-proxy
Type: Proxy
Version: v1.20.8+vmware.1
Conditions:
Last Transition Time: 2022-02-20T08:11:31Z
Status: True
Type: Provisioned
Name: defaultpsp
Type: PSP
Version: v1.20.8+vmware.1-tkg.2
Conditions:
Last Transition Time: 2022-02-20T08:11:42Z
Status: True
Type: Provisioned
Name: metrics-server
Type: MetricsServer
Version: v0.4.0+vmware.2
API Endpoints:
Host: 192.168.100.3
Port: 6443
Conditions:
Last Transition Time: 2022-02-20T08:20:29Z
Status: True
Type: Ready
Last Transition Time: 2022-02-20T08:11:42Z
Status: True
Type: AddonsReady
Last Transition Time: 2022-02-20T08:11:33Z
Status: True
Type: ControlPlaneReady
Last Transition Time: 2022-02-20T08:20:29Z
Status: True
Type: NodePoolsReady
Last Transition Time: 2022-02-20T08:20:28Z
Message: 1/1 Control Plane Node(s) healthy. 3/3 Worker Node(s) healthy
Status: True
Type: NodesHealthy
Last Transition Time: 2022-02-20T08:11:31Z
Status: True
Type: ProviderServiceAccountsReady
Last Transition Time: 2022-02-20T08:11:31Z
Status: True
Type: RoleBindingSynced
Last Transition Time: 2022-02-20T08:11:33Z
Status: True
Type: ServiceDiscoveryReady
Last Transition Time: 2022-02-20T08:11:31Z
Status: True
Type: StorageClassSynced
Last Transition Time: 2022-02-20T08:11:33Z
Status: True
Type: TanzuKubernetesReleaseCompatible
Last Transition Time: 2022-02-08T15:20:53Z
Reason: NoUpdates
Status: False
Type: UpdatesAvailable
Phase: running
Total Worker Replicas: 3
Events: <none>
从 UI 角度来看，我们现在可以看到 TKG 集群已部署在 tkg-guest-01 命名空间中。我们还可以看到控制平面节点和三个工作节点。
选择 sl-wl01-tkc01 命名空间。导航至 Compute > VMware Resources > Tanzu Kubernetes clusters。
在此处可以查看TKG集群的更多详细信息。请注意，API服务器的负载均衡器IP地址（192.168.100.3）来自我们在启用工作负载管理和创建Supervisor集群过程中提供的Ingress范围。
在 VMware Resources > Virtual Machines 中，您可以查看TKG集群节点VM的详细信息，包括VM类的清单。
可以查看VM类以了解节点的配置详情，包括其资源保证。
Network Operator Deployment with a Host Device Network
Network operator deployment with:
- SR-IOV device plugin, single SR-IOV resource pool
- Secondary network
- Multus CNI
- Container networking-plugins CNI plugins
- Whereabouts IPAM CNI plugin
在此模式下，Network Operator也可以部署在虚拟化环境中。它支持以太网和InfiniBand模式。从Network Operator的角度来看，部署过程没有区别。要在VM（虚拟机）上工作，必须为SR-IOV设备配置PCI直通。Network Operator在VM内部同时支持VF（虚拟功能）和PF（物理功能）。
启动shell、终端或命令提示符会话。
要部署Network Operator，请切换上下文。我们不使用命名空间上下文，而是切换到TKG集群上下文。这使我们能够在访客集群的上下文中运行操作。为此，请注销并重新登录，在登录命令中指定TKG集群命名空间和集群名称。登录命令相当长，如下所示。
K8s CLI VM console
root@user:~# kubectl-vsphere logout
Your KUBECONFIG context has changed.
The current KUBECONFIG context is unset.
To change context, use `kubectl config use-context <workload name>`
Logged out of all vSphere namespaces.
root@user:~# kubectl-vsphere login --vsphere-username administrator@vsphere.local --server=192.168.100.2 --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace=sl-wl01-ns01 --tanzu-kubernetes-cluster-name=sl-wl01-tkc01
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
Logged in successfully.
You have access to the following contexts:
192.168.100.2
sl-wl01-ns01
sl-wl01-tkc01
If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.
To change context, use `kubectl config use-context <workload name>`
root@user:~# kubectl config use-context sl-wl01-tkc01
Switched to context "sl-wl01-tkc01".
要显示TKG的K8s节点，请运行：
K8s CLI VM console
root@user:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
sl-wl01-tkc01-control-plane-crtld Ready control-plane,master 3h29m v1.20.8+vmware.1 10.244.0.34 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq Ready <none> 3h21m v1.20.8+vmware.1 10.244.0.35 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv Ready <none> 3h20m v1.20.8+vmware.1 10.244.0.36 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 Ready <none> 3h20m v1.20.8+vmware.1 10.244.0.37 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
现在需要为我们的工作节点手动添加一个角色"worker"：
K8s CLI VM console
root@user:~# kubectl label node sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq node-role.kubernetes.io/worker=worker
node/sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq labeled
root@user:~# kubectl label node sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv node-role.kubernetes.io/worker=worker
node/sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv labeled
root@user:~# kubectl label node sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 node-role.kubernetes.io/worker=worker
node/sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 labeled
root@user:~# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
sl-wl01-tkc01-control-plane-crtld Ready control-plane,master 3h36m v1.20.8+vmware.1 10.244.0.34 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq Ready worker 3h28m v1.20.8+vmware.1 10.244.0.35 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv Ready worker 3h28m v1.20.8+vmware.1 10.244.0.36 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 Ready worker 3h28m v1.20.8+vmware.1 10.244.0.37 <none> Ubuntu 20.04.3 LTS 5.4.0-88-generic containerd://1.4.6
我们需要安装Helm，运行：
K8s CLI VM console
root@user:~# snap install helm --classic
要使用chart默认值安装operator，请运行：
K8s CLI VM console
root@user:~# helm repo add mellanox https://mellanox.github.io/network-operator
'mellanox" has been added to your repository
root@user:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "mellanox" chart repository
Update Complete. *Happy Helming!*
创建values.yaml文件。
K8s CLI VM console
root@user:~# vim values.yaml
K8s CLI VM console
nfd:
enabled: true
sriovNetworkOperator:
enabled: false
NicClusterPolicy CR 值：
deployCR: true ofedDriver: deploy: true
rdmaSharedDevicePlugin: deploy: false
sriovDevicePlugin: deploy: true resources: - name: hostdev vendors: [15b3] secondaryNetwork: deploy: true multus: deploy: true cniPlugins: deploy: true ipamPlugin: deploy: true
以下是部署示例，这些示例是安装网络运营商时提供给 Helm 的 values.yaml 文件。通过运行以下命令实现。
警告：默认情况下，NVIDIA 网络运营商不会部署 Pod 安全策略。要启用，请通过设置
psp.enabled=true覆盖 psp 图表参数。
K8s CLI VM 控制台
root@user:~# helm install network-operator -f ./values.yaml -n network-operator --create-namespace --wait mellanox/network-operator --set psp.enabled=true
验证部署
通过运行以下命令获取网络运营商部署的资源。需要等待安装完成约 10-15 分钟。
K8s CLI VM 控制台
root@user:~# kubectl -n network-operator get pods -o wide
network-operator-6688d556cb-ccmfw 1/1 Running 0 2m11s 193.0.3.3 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
network-operator-node-feature-discovery-master-596fb8b7cb-cx99m 1/1 Running 0 2m11s 193.0.1.4 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
network-operator-node-feature-discovery-worker-6c2bk 1/1 Running 0 2m11s 193.0.2.3 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
network-operator-node-feature-discovery-worker-8rfpb 1/1 Running 0 2m11s 193.0.1.3 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
network-operator-node-feature-discovery-worker-rs694 1/1 Running 0 2m11s 193.0.3.4 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
network-operator-node-feature-discovery-worker-wprgw 1/1 Running 0 2m11s 193.0.0.8 sl-wl01-tkc01-control-plane-crtld <none> <none>
root@user:~# kubectl -n nvidia-operator-resources get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cni-plugins-ds-55t2r 1/1 Running 0 14m 10.244.0.37 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
cni-plugins-ds-gvj9l 1/1 Running 0 14m 10.244.0.36 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
cni-plugins-ds-tf9kz 1/1 Running 0 14m 10.244.0.35 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
kube-multus-ds-jp7mq 1/1 Running 0 14m 10.244.0.37 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
kube-multus-ds-qv2gr 1/1 Running 0 14m 10.244.0.35 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
kube-multus-ds-rbqlp 1/1 Running 0 14m 10.244.0.36 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
mofed-ubuntu20.04-ds-lh8rf 1/1 Running 0 14m 10.244.0.37 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
mofed-ubuntu20.04-ds-ntwct 1/1 Running 0 14m 10.244.0.35 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
mofed-ubuntu20.04-ds-stjhk 1/1 Running 0 14m 10.244.0.36 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
sriov-device-plugin-fkn5w 1/1 Running 0 3m56s 10.244.0.36 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
sriov-device-plugin-n8k5q 1/1 Running 0 5m28s 10.244.0.35 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
sriov-device-plugin-ppqpl 1/1 Running 0 64s 10.244.0.37 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
whereabouts-5726c 1/1 Running 0 14m 10.244.0.35 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq <none> <none>
whereabouts-7m5wr 1/1 Running 0 14m 10.244.0.37 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
whereabouts-c8flr 1/1 Running 0 14m 10.244.0.36 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
要显示具有 nvidia.com/hostdev: 1 的 TKG K8s 工作节点，请运行：
K8s CLI VM 控制台
root@user:~# kubectl describe node sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-drmlq
...
Capacity:
cpu: 8
ephemeral-storage: 205374420Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65868016Ki
nvidia.com/hostdev: 1
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 189273065159
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 65765616Ki
nvidia.com/hostdev: 1
pods: 110
...
部署后，应配置网络运营商，并部署 K8s 网络以便在 Pod 配置中使用。
host-device-net.yaml 是此类部署的配置文件。
K8s CLI VM 控制台
root@user:~# vim host-device-net.yaml
K8s CLI VM 控制台
apiVersion: mellanox.com/v1alpha1
kind: HostDeviceNetwork
metadata:
name: hostdev-net
spec:
networkNamespace: "default"
resourceName: "nvidia.com/hostdev"
ipam: |
{
"type": "whereabouts",
"datastore": "kubernetes",
"kubernetes": {
"kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig"
},
"range": "192.168.3.225/28",
"exclude": [
"192.168.3.229/30",
"192.168.3.236/32"
],
"log_file" : "/var/log/whereabouts.log",
"log_level" : "info"
}
并运行以下命令：
K8s CLI VM 控制台
root@user:~# kubectl apply -f host-device-net.yaml
hostdevicenetwork.mellanox.com/hostdev-net created
应用程序
现在我们可以部署一个示例 Pod。
K8s CLI VM 控制台
root@user:~# vim pod.yaml
K8s CLI VM 控制台
apiVersion: v1
kind: Pod
metadata:
name: hostdev-test-pod
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-net
spec:
restartPolicy: OnFailure
containers:
- image: harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04
name: mofed-test-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
requests:
nvidia.com/hostdev: 1
limits:
nvidia.com/hostdev: 1
command:
- sh
- -c
- sleep inf
并运行以下命令：
K8s CLI VM 控制台
root@user:~# kubectl apply -f pod.yaml
pod/hostdev-test-pod created
检查 RDMA
要检查 RDMA，我们需要部署第二个 Pod。
K8s CLI VM 控制台
root@user:~# vim pod2.yaml
apiVersion: v1
kind: Pod
metadata:
name: hostdev-test-pod-2
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-net
spec:
restartPolicy: OnFailure
containers:
- image: harbor.mellanox.com/nbu-solutions-labs/ubuntu-mlnx-inbox:20.04
name: mofed-test-ctr
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
resources:
requests:
nvidia.com/hostdev: 1
limits:
nvidia.com/hostdev: 1
command:
- sh
- -c
- sleep inf
并运行以下命令。
K8s CLI VM console
root@user:~# kubectl apply -f pod2.yaml
pod/hostdev-test-pod-2 created
验证两个Pod正在运行。
K8s CLI VM console
root@user:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hostdev-test-pod 1/1 Running 0 102s 193.0.2.4 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 <none> <none>
hostdev-test-pod-2 1/1 Running 0 83s 193.0.3.5 sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv <none> <none>
可以看到第一个 hostdev-test-pod Pod运行在worker sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-rsz75 上，第二个 hostdev-test-pod-2 Pod运行在worker sl-wl01-tkc01-workercx6dx-jgfgf-7875fd7f9-q2prv 上。
现在我们可以运行 ib_write_bw（InfiniBand写带宽工具，属于 Perftest Package），通过以下步骤。
获取第一个运行容器的shell。
K8s CLI VM console
root@user:~# kubectl exec -it hostdev-test-pod -- bash
检查Pod中的可用网络接口。
K8s CLI VM console
root@hostdev-test-pod:/tmp# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@hostdev-test-pod:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether a2:8d:77:4f:68:70 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 193.0.2.4/24 brd 193.0.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a08d:77ff:fe4f:6870/64 scope link
valid_lft forever preferred_lft forever
10: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:70:e2:e7 brd ff:ff:ff:ff:ff:ff
inet 192.168.3.225/28 brd 192.168.3.239 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe70:e2e7/64 scope link
valid_lft forever preferred_lft forever
并运行。
K8s CLI VM console
root@hostdev-test-pod:/tmp# ib_write_bw -F -d mlx5_0 --report_gbits
打开另一个控制台窗口，获取第二个运行容器的shell。
K8s CLI VM console
root@user:~# kubectl exec -it hostdev-test-pod-2 -- bash
检查Pod中的可用网络接口。
K8s CLI VM console
root@hostdev-test-pod-2:/tmp# rdma link
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev net1
root@hostdev-test-pod-2:/tmp# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether a2:8d:77:4f:68:70 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 193.0.3.5/24 brd 193.0.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::a08d:77ff:fe4f:6870/64 scope link
valid_lft forever preferred_lft forever
10: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:70:e2:e7 brd ff:ff:ff:ff:ff:ff
inet 192.168.3.226/28 brd 192.168.3.239 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe70:e2e7/64 scope link
valid_lft forever preferred_lft forever
并运行。
K8s CLI VM console
root@hostdev-test-pod-2:/tmp# ib_write_bw -F 192.168.3.225 -d mlx5_0 --report_gbits
结果。
K8s CLI VM console
On Server side.
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0127 PSN 0x6e0491 RKey 0x038b04 VAddr 0x007f23bd877000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:225
remote address: LID 0000 QPN 0x0127 PSN 0xcdfca6 RKey 0x038b04 VAddr 0x007fdb2dbd7000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:226
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 91.87 91.85 0.174290
---------------------------------------------------------------------------------------
On Client side.
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : mlx5_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 128
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0127 PSN 0xcdfca6 RKey 0x038b04 VAddr 0x007fdb2dbd7000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:226
remote address: LID 0000 QPN 0x0127 PSN 0x6e0491 RKey 0x038b04 VAddr 0x007f23bd877000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:03:225
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536 5000 91.87 91.85 0.174290
---------------------------------------------------------------------------------------
要运行DPDK应用程序，请参阅以下文档：RDG: DPDK Applications on SR-IOV Enabled Kubernetes Cluster with NVIDIA Network Operator。
完成！
Authors
| 作者信息 |
Boris Kovalev
Boris Kovalev has worked for the past several years as a 解决方案 Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the NVIDIA Documents website.
Vitaliy Razinkov
Vitaliy Razinkov is a 解决方案 Architect on the NVIDIA Networking team, specializing in complex Kubernetes, OpenShift, and Microsoft solutions. With over 25 years of experience in senior technical roles, he brings deep expertise in designing and implementing advanced infrastructures. Vitaliy has authored several reference design guides on Microsoft technologies, RoCE/RDMA-accelerated machine learning in Kubernetes/OpenShift, and containerized solutions—all available on the NVIDIA Networking 文档 site.