RDG for Redis on vSphere Distributed Services Engine (Project Monterey) deployment over NVIDIA BlueField-2 DPU

Created on Dec 18,2023

On This Page

• Scope
• Abbreviations and Acronyms
• Introduction
• References
• Solution Architecture
  • Key Components and Technologies
  • Logical Design
  • Host Network Design
  • Bill of Materials
    • vSphere Cluster Compute/Storage
• Deployment and Configuration
  • Wiring
  • Setup Configuration
  • Network
    • Prerequisites
    • Network Switch Configuration
  • Hosts Preparation
    • Hosts Network Configuration
  • ESXi/ESXio Installation using a Unified Image
  • vCenter Deployment
  • vSphere Configuration
    • Create vSphere Datacenter, Cluster and add the ESXi Host to the Cluster
    • Add Additional MFT Depot to the Cluster Image
    • SR-IOV Enabling on the ESXi Host
    • NTP Configuration
    • Create and Configure a Regular (non-offloaded) Distributed Virtual Switch for vMotion and vSAN
    • Create and Configure a VMware vSAN ESA
    • Enable DRS and HA on Cluster
    • Create and Configure an Offloaded Distributed Virtual Switch
  • NSX Manager Deployment and Configuration
    • NSX Manager Deployment

NSX Manager Configuration

• Create a Linux VM.
  • UPTv2 Interface Validation
• Authors

Scope

The following Reference Deployment Guide (RDG) explains how to install and configure Redis VMware vSphere 8.0.x Distributed Services Engine (DSE) with VMware NSX 4.1 version on a single vSphere cluster over Dell NVIDIA® BlueField Data Processing Units (DPU) and end-to-end NVIDIA 100Gbps Ethernet solution. VMware’s vSAN Express Storage Architecture (ESA) will be used as a share storage for the environment.

Abbreviations and Acronyms

Term	Definition	Term	Definition
DAC	Direct Attached Cable	MFT	NVIDIA Mellanox Firmware Tools
DCUI	Direct Console User Interface	NOS	Network Operation System
DHCP	Dynamic Host Configuration Protocol	OOB	Out-of-band
DPDK	Data Plane Development Kit	OSA	Original Storage Architecture
DPU	Data Processing Unit	RDG	Reference Deployment Guide
DSE	Distributed Services Engine	SR-IOV	Single Root Input/Output Virtualization
EDP	Enhanced Data Path	VF	Virtual Function
ESA	Express Storage Architecture	VM	Virtual Machine

Introduction

Redis is one of the most popular in-memory, key-value NoSQL databases. It is favored by web-based applications due to its strength in managing unstructured data and its simple data model compared to a traditional relational database. Redis popularized the notion that data always resides in memory; in some ways, it is built on the premise that it is faster to access data in remote memory than going to a local disk. In a distributed environment, having many clients accessing a common Redis database puts significant stress on the network system, which requires many concurrent connections with very low latency. This application is a showcase to demonstrate the strengths and benefits of vSphere 8 with NVIDIA BlueField DPU. By offloading and accelerating network operations to the DPU, we free up CPU cycles that can now be used to run more Redis virtual machines (VMs) in isolation from network processing tasks and deliver higher transactions per second at a lower latency, compared to the same servers using a standard network card. Using the DPU also improves energy efficiency by supporting more Redis transactions per Watt (or lower Watt per Redis transaction).

Starting with vSphere 8, VMware and NVIDIA introduce a VMware vSphere Distributed Switch and VMware NSX networking running on the NVIDIA BlueField-2 DPU.

The NVIDIA BlueField-2 DPU provides a new control point for scaling infrastructure functions and enables security controls that are separate from the workload domain.

This guide provides step-by-step instructions to deploy and configure Redis, VMware ESXi and NVIDIA BlueField-2 DPU, vCenter, and NSX manager to support a DSE environment on a single vSphere cluster including technology overview, design, component selection, and deployment steps.

Our testing with Memtier shows that we can execute larger Redis workloads on the same hardware by using DPU offload and acceleration that maximizes transactions per second and lowers transaction latency. CPU cores are freed up by using the DPU, so a particular Redis workload could be supported by a smaller hardware footprint with reduced power consumption, reducing both capital and operating costs.

References

• NVIDIA Cumulus
• NVIDIA BlueField Data Processing Units
• VMware ESXi 8.0b server, build 21203435
• VMware vCenter Server 8.0b
• VMware NSX 4.1
• vSphere on DPU
• VMware vSAN Express Storage Architecture (ESA)
• How-to: Configure a vSphere Distributed Switch with NVIDIA network fabric.

RDG for VMware vSAN ESA over NVIDIA RoCE on VMware vSphere 8.0

Solution Architecture

Key Components and Technologies

• NVIDIA Spectrum 以太网交换机 Flexible form-factors with 16 to 128 physical ports, supporting 1GbE through 400GbE speeds. Based on a ground-breaking silicon technology optimized for performance and scalability, NVIDIA Spectrum switches are ideal for building high-performance, cost-effective, and efficient Cloud Data Center Networks, Ethernet Storage Fabric, and Deep Learning Interconnects. NVIDIA combines the benefits of NVIDIA Spectrum™ switches, based on an industry-leading application-specific integrated circuit (ASIC) technology, with a wide variety of modern network operating system choices, including NVIDIA Cumulus® Linux, SONiC and NVIDIA Onyx®.

• NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.

• NVIDIA BlueField® Data Processing Unit (DPU) The NVIDIA® BlueField® data processing unit (DPU) ignites unprecedented innovation for modern data centers and supercomputing clusters. With its robust compute power and integrated software-defined hardware accelerators for networking, storage, and security, BlueField creates a secure and accelerated infrastructure for any workload in any environment, ushering in a new era of accelerated computing and AI.

• NVIDIA LinkX Cables The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and 100, 200 and 400Gb/s InfiniBand products for Cloud, HPC, hyperscale, Enterprise, telco, storage and artificial intelligence, data center applications.

• VMware vSphere on DPU, enabled by vSphere Distributed Services Engine modernizes virtual infrastructure by offloading functions from CPU to DPU. It enables modern distributed workloads to take advantage of resource savings, accelerated networking, and enhanced workload security, while also improving DPU lifecycle management with workflows integrated into vSphere.

• VMware vSphere Distributed Switch (VDS) provides a centralized interface from which you can configure, monitor and administer virtual machine access switching for the entire data center. The VDS provides simplified Virtual Machine network configuration, enhanced network monitoring and troubleshooting capabilities.

• VMware NSX provides an agile software-defined infrastructure to build cloud-native application environments. NSX focuses on providing networking, security, automation, and operational simplicity for emerging application frameworks and architectures that have heterogeneous endpoint environments and technology stacks. NSX Data Center supports cloud-native applications, bare metal workloads, multi-hypervisor environments, public clouds, and multiple clouds. NSX Data Center is designed for management, operation, and consumption by development organizations. NSX Data Center allows IT teams and development teams to select the technologies best suited for their applications.

• VMware vSAN Express Storage Architecture (ESA) represents a massive step forward in the capabilities of the solution. This is an optional, alternative storage architecture (OSA) to the vSAN original storage architecture also found in vSAN 8. The best way to think of the vSAN Express Storage Architecture is as a new way to process and store data. It is an optional, alternative architecture in vSAN that is designed to achieve all-new levels of efficiency, scalability, and performance. The ESA is optimized to exploit the full potential of the very latest in hardware, and unlocks new capabilities.

• Distributed Services Engine (DSE) introduces a new data path that is only available with DSE. With vSphere Distributed Services Engine, you can use Uniform pass-through (UPTv2) and/or MUX mode to improve performance and reduce network hops and CPU resources on x86 servers. The administrator can fine-tune the offloading and acceleration behavior using two modes: MUX Mode and UPTv2. MUX Mode is the default mode and does some processing on the x86 host. It provides higher flexibility and has fewer requirements than UPTv2. UPTv2 completely offloads all network and security processing to the DPU and provides higher performance. It requires Guest Memory Reservation and a specific VMXNET3 driver version. UPTv2 supported guest operating systems for DSE - see Distributed Services Engine (DSE) UPTv2 VMXNET3 driver requirements. Both modes provide accelerated networking services for VLAN and overlay networking, as well as offloading TCP Segment Offload (TSO), Checksum, and Receive Side Scaling (RSS). To configure the new EDP with either MUX Mode or UPTv2, you need to deploy NSX Manager. UPTv2 requires publishing virtualized device functions (VFs), like SR-IOV. The number of VMs that can be connected depends on the number of VFs published by the device vendor. UPTv2 supports core vSphere features like vMotion for VMs that use it.

Logical Design

The setup uses one vSphere cluster that includes 3 ESXi servers connected to single high speed ethernet switch for VM Applications, vMotion traffics.

The Management traffic uses dedicated 1GbE switch.

This setup will explain how to configure VMs (Virtual Machine) network connectivity UPT types.

VMware vCenter and NSX Manager VMs will be placed on a separate Management cluster which is not in the scope of the document.

In our deployment example we use NVIDIA® Spectrum® SN3700 Ethernet switch and NVIDIA 100GbE QSFP28 Direct Attach Copper Cables for high speed ethernet network. Management traffic is served by NVIDIA® SN2201 Ethernet switch.

Currently, only one Dell NVIDIA BlueField-2 DPU may be used per host. To provide High Availability solution you can add NVIDIA ConnectX-6DX network card for vMotion and High speed vSAN traffics. Please follow the RDG for VMware vSAN ESA over NVIDIA RoCE on VMware vSphere 8.0 - 解决方案 - NVIDIA Networking Docs guide.

Host Network Design

Bill of Materials

vSphere Cluster Compute/Storage

VM	CPU	MEM	DISK
vCenter	4	21GB	50GB
NSX Manager	12	48GB	300GB

Deployment and Configuration

Wiring

Setup Configuration

Before starting the configuration process, make sure you are familiar with VMware vSphere, vCenter and NSX deployment and management procedures.

The installation process requires administrator privileges on the target machines.

In addition, the following components are required:

• ESXi server platform with Dell NVIDIA® BlueField-2 100GbE network adapter with inbox ESXi driver and firmware version 22.34.1002
• NVIDIA® Spectrum®-2 SN3000 Ethernet switch
• NVIDIA® SN2201 Ethernet switch
• VMware vSphere 8.0.x Cluster installed and configured
• VMware vCenter 8.0.x installed and configured
• VMware NSX 4.1

Network

Prerequisites

交换机 OS

NVIDIA Cumulus 5.5.

Management Network

DHCP and DNS services are required.

Warning: The components' installation and configuration are not covered by this guide.

Network Switch Configuration

In the document we will use the following networks.

• Management Network (VLAN 10 – 192.168.1.0/24) – Management VMs such as the vCenter and NSX Manager location
• ESXi Management Network (VLAN 10 – 192.168.1.0/24) – ESXi VMkernel interfaces location
• vMotion Network (VLAN 1620 – 192.168.20.0/24) – ESXi vMotion VMkernel interfaces location
• vSAN Network (VLAN 1630 – 192.168.30.0/24) – ESXi vSAN VMkernel interfaces location
• NSX Geneve Overlay Network for ESXi Hosts (VLAN 1640- 192.168.50.0/24) – Used by the Geneve Overlay Tunnel endpoints VMkernel interfaces on the ESXi Hosts aka vmk10.

交换机 Configuration

Run the following commands on both switches in the vSphere Cluster to configure the VLAN.

Sample on SN2201 switch.

nv set interface swp1-48 bridge domain br_default
nv set bridge domain br_default vlan 10
nv set bridge domain br_default untagged 10
nv config apply
nv config save

Sample on SN3700 switch.

nv set interface swp1-32 bridge domain br_default
nv set bridge domain br_default vlan 1620,1630,1640
nv config apply
nv config save

Hosts Preparation

Warning: Hosts in the vSphere Cluster must be configured before a vSphere Distributed Services Engine can be configured.

Host setup preparation:

1. 3 x Dell R750 physical server installed with Dell NVIDIA BlueField-2 DPU in slot 2. To make sure, perform the following:

   1. Access iDRAC via the web browser.

      

   2. From the System → Overview → PCI Slots, select PCIe Slot 2.

      Sample.

2. Each NVIDIA BlueField-2 DPU is connected to server through Paige card, which is connected to server by UART connection and to NVIDIA BlueField-2 DPU by NC-SI cable.

3. Installer privileges: The installation requires administrator privileges on the target machine.

4. Management network connectivity to host and Dell NVIDIA BlueField-2 DPU.

5. High speed network connectivity to 2 Dell NVIDIA BlueField-2 DPU ports.

6. Ensure that your server is configured with boot mode: UEFI boot mode.

7. VERY IMPORTANT!: Ensure BIOS, iDRAC, CPLD, Dell NVIDIA BlueField-2 BlueField-2 DPU BIOS and NIC firmware versions and update to the required ones.

   To make sure, perform the following:

   1. Access the VMware Compatibility Guide via the web browser and select your server.

      Sample.

• Choose ESXi 8.0 in the VMware Product Name option to see the certified BIOS/iDRAC/CPLD and DPU NIC firmware versions. Sample.

• Access iDRAC via the web browser.

• From the System → Inventory, select Firmware Inventory and check BIOS, iDRAC, CPLD, BlueField-2 DPU BIOS and NIC FW versions. Sample. ...

• To update the component/s.

  1. Enter to Dell Support portal via the web browser.

  2. Select your server → Drivers&下载 → OS → Category.

  3. Select the components to download and click on Download Selected.

  4. Access iDRAC via the web browser.

  5. Navigate to the Maintenance → System Update → Manual Update, select Local in the Location Type and click on Choose File.

     Note: Recommended order to update the components:

     1. BIOS
     2. iDRAC
     3. CPLD
     4. BIOS of BlueField-2 DPU
     5. BlueField-2 DPU NIC firmware

     

  6. Select the downloaded DUP (.exe) file and click Open.

  7. Click on Upload.

  8. Select the package and click on Install/Install and Reboot.

  9. Wait for the update to complete and recheck the component version.

  10. Repeat the process for other updated components.

• Ensure OS to iDRAC Pass-through is enabled and works. Otherwise, BlueField-2 DPU will not be detected during ESXi/ESXio install.

  1. Access iDRAC via the web browser.
  2. Navigate to the iDRAC Settings → Connectivity → OS to iDRAC Pass-through.
     1. Select:
        • Enabled in the State
        • USB NIC in the Pass-through Mode Sample.
     2. Click on Test network connection.

• Ensure optimal performance:

  1. Access iDRAC via the web browser.
  2. From the Configuration → BIOS Settings, select System Profile Settings.

• Set the System Profile to Performance Per Watt (OS).

• Set Workload Profile to Telco Optimized Profile.

  

1. On iDRAC, check and enable DPUBootSynchronization, DPUTrust.

   1. Login to the iDRAC console by running #ssh root@<hostname>-ilo or by any other method you use, and press Enter.

   2. Check DPUBootSynchronization and DPUTrust. Expected values for both are Enabled.

      racadm>>get system.PCISlot.2
      ...
      DPUBootSynchronization=Enabled
      ...
      DPUTrust=Enabled
      ...
      

   3. Enable DPUBootSynchronization, run:

      racadm>>set system.pcislot.2.DPUBootSynchronization 1
      [Key=system.Embedded.1#PCISlot.2]
      Object value modified successfully
      

   4. Enable DPUTrust, run:

      racadm>>set system.pcislot.2.DPUTrust 1
      [Key=system.Embedded.1#PCISlot.2]
      Object value modified successfully
      

   5. Check and confirm it is enabled.

      racadm>>get system.PCISlot.2
      ...
      DPUBootSynchronization=Enabled
      ...
      DPUTrust=Enabled
      ...
      

   6. COLD BOOT the x86 host.

2. Ensure access to the BlueField-2 DPU console through iDRAC, and check BMC and OOB NAC addresses.

   Note: These addresses you need to provide to your System Administrators for DHCP records creating.

   1. Login to the iDRAC console by running #ssh root@<hostname>-ilo or by any other method you use and press Enter.

   2. Use iDRAC for the SmartNOC console access.

      racadm>>console dpu1
      dpu-bmc login: root
      Password:
      root@dpu-bmc:~#
      

      Warning: First-time login credentials are: root/0penBmc. When prompted to change the password, change it to a new one.

      

   3. Check the MAC/IP addresses on BMC.

      Sample.

      root@dpu-bmc:~# ip a s
      1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
             valid_lft forever preferred_lft forever
          inet6 ::1/128 scope host
             valid_lft forever preferred_lft forever
      2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
          link/ether 94:6d:ae:2f:61:c0 brd ff:ff:ff:ff:ff:ff
          inet 169.254.20.165/16 brd 169.254.255.255 scope link eth0
             valid_lft forever preferred_lft forever
          inet 10.236.153.19/16 brd 10.236.255.255 scope global dynamic eth0
             valid_lft 41432sec preferred_lft 41432sec
          inet6 fdfd:fdfd:10:236:966d:aeff:fe2f:61c0/64 scope global dynamic
             valid_lft 2591999sec preferred_lft 604799sec
          inet6 fe80::966d:aeff:fe2f:61c0/64 scope link
             valid_lft forever preferred_lft forever=
      

      In our case the eth0 interface has MAC - 94:6d:ae:2f:61:c0 and IP - 10.236.153.19 addresses.

   4. Connect to OOB by running the command below and press Enter.

      root@dpu-bmc:~# obmc-console-client
      ubuntu@clx-host-153-oob:~$
      

      Warning: First-time OOB login credentials are: ubuntu/ubuntu. When prompted to change the password, change it to a new one.

   5. Check the MAC/IP addresses on BMC.

      Sample.

      ubuntu@clx-host-153-oob:~$ ip a s
      ...
      3: oob_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
          link/ether 94:6d:ae:2f:61:cc brd ff:ff:ff:ff:ff:ff
          inet 10.236.153.18/16 brd 10.236.255.255 scope global dynamic noprefixroute oob_net0
             valid_lft 72374sec preferred_lft 72374sec
          inet6 fe80::966d:aeff:fe2f:61bc/64 scope link
             valid_lft forever preferred_lft forever
      ...
      

      In our case, the oob_net0 interface has MAC - 94:6d:ae:2f:61:cc and IP - 10.236.153.18 addresses.

      Warning: To quit the iDRAC console, press the following key combinations: Ctrl+6 and Ctrl+\ or Ctrl+^ and Ctrl+
      To quit the iDRAC console, press the following key combination: ENTER and ~ and . To quit the BMC console, press the following key combination:~ and ~ and .

Hosts Network Configuration

This table provides details of the ESXi server, switches names and their network configuration.

SL-WL01-Cluster01 vSphere Cluster

Server	Server Name	IP and NICs
		High-Speed Ethernet Network
ESXi-01	clx-host-153	vmk0: 169.254.0.2 (BMC_Network)vmk2: 192.168.20.153 (vMotion)vmk3: 192.168.30.153 (vSAN)vmk10: From IP Pool 192.168.40.0/24 (NSX Host TEP)
ESXi-01-BF	clx-host-153-oobclx-host-153-bmc
ESXi-02	clx-host-154	vmk0: 169.254.0.2 (BMC_Network)vmk2: 192.168.20.154 (vMotion)vmk3: 192.168.30.154 (vSAN)vmk10: From IP Pool 192.168.40.0/24 (NSX Host TEP)
ESXi-02-BF	clx-host-154-oobclx-host-154-bmc

		vmk0: 10.236.154.128 (OOB, ESXio) BMC: 10.236.154.129 From DHCP (reserved)
ESXi-03	clx-host-155	vmk0: 169.254.0.2 (BMC_Network) vmk2: 192.168.20.155 (vMotion) vmk3: 192.168.30.155 (vSAN) vmk10: From IP Pool 192.168.40.0/24 (NSX Host TEP)	vmk0: 10.236.155.1 From DHCP (reserved)
ESXi-03-BF	clx-host-155-oob clx-host-155-bmc		vmk0: 10.236.155.18 (OOB, ESXio) BMC: 10.236.155.19 From DHCP (reserved)
Leaf-01	clx-swx-033		10.130.250.233
Leaf-03	clx-swx-034		10.130.250.234
vCenter (VM)	sl01w01vc01		10.130.250.115
NSX Manager 01 (VM)	sl01w01nsx01		10.130.250.140
PV-VM-01	clx-vm-pv-01	10.10.10.2	10.236.153.2
PV-VM-02	clx-vm-pv-02	10.10.10.3	10.236.153.3
SRIOV-VM-01	clx-vm-sriov-01	10.10.10.4	10.236.153.4
SRIOV-VM-02	clx-vm-sriov-02	10.10.10.5	10.236.153.5
UPT-VM-01	clx-vm-upt-01	10.10.10.6	10.236.153.6
UPT-VM-02	clx-vm-upt-02	10.10.10.7	10.236.153.7

ESXi/ESXio Installation using a Unified Image

Before you start, make sure the following required items are available:

• ESXi/ESXio IPs and FQDNs registered in DNS/DHCP
• Management network netmask and gateway
• NTP servers
• Password for ESXi/ESXio root user

To install the unified ESXi image on the BlueField-2 DPU equipped Dell server:

1. Download the ESXi/ESXio server hypervisor ISO file from https://customerconnect.vmware.com/ website.

2. Access the iDRAC web interface and open the Virtual Console.

3. In the Virtual Console, select Virtual Media.

4. Click on Connect Virtual Media.

5. Click on Choose File.

6. Select the image file.

7. Click on Map Device and afterwards on Close.

8. In the Virtual Console window, select Boot.

9. From the Boot Controls popup window, select Virtual CD/DVD/ISO.

10. On Confirm Boot Action popup window, click Yes.

11. On the Virtual Console window select Power.

12. On Confirm Power Action popup window, click Yes.

13. From the Power Controls popup window, select Power On System or Power Cycle System (cold boot).**

14. Let the system boot up (take several minutes) and reach the VMware ESXi 8.0.0 Installation. Press Enter to start the installation.

15. Press F11 to accept the End User License Agreement (EULA).

16. At the Installer

Configuration window, ensure both the Install/Upgrade ESXi and the detected Dell NVIDIA Bluefield-2 Dual Port 25 GbE SFP Crypto DPU 2 are selected, and press Enter.

1. Select the disk from the list where ESXi operating system will be installed and press Enter. In our case, we selected the DELLBOSS VD.

2. Keep the default for keyboard layout - US Default, press Enter to proceed.

3. Provide a root password. After password match confirmation press Enter.

   Warning: ESXio password on newer builds will be same as password set for ESXi.

   

4. Ensure the installation on both the Dell BOSS card and the BlueField-2 DPU, and press F11.

   

5. Press Enter to reboot once the installation in completed on both on ESXi and BlueField-2 DPU. The installation is performed first on the BlueField-2 DPU and then the host.

6. Upon successful ESXi installation, press Enter to reboot.

7. Upon successful ESXi installation on x86 and ESXi on BlueField-2 DPU (ESXio), the Server then reboots into ESXi booted on x86 and ESXio is booted on BlueField-2 DPU. Now your system is ready for use, both the ESXi on x86 and the ESXio on BlueField-2 DPU. Below is the DCUI screen, which will be configured in next steps.

8. Press F2 to configure the ESXi. Authentication is required here, Enter an authorized login name root and password (you set during install) for the ESXi host.

9. Select Configure Management Network and ensure/apply the appropriate host network configuration.

10. Check and Enable ESXi Shell and SSH on ESXi (x86).

11. Connect to the ESXi console and run sshdpu to access to the ESXio on the BlueField-2 DPU DCUI screen.

    Warning: You can enable ssh on ESXio by running the following command on ESXi (x86) console. [root@clx-host-154:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0 [root@clx-host-154:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0

    [root@clx-host-154:~] sshdpu
    ssh: connect to host 169.254.100.2 port 22: Connection refused
    [root@clx-host-154:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0
    [root@clx-host-154:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0
    [root@clx-host-154:~] sshdpu
    hostfile_replace_entries: link /.ssh/known_hosts to /.ssh/known_hosts.old: Function not implemented
    update_known_hosts: hostfile_replace_entries failed for /.ssh/known_hosts: Function not implemented
    The time and date of this login have been sent to the system logs.
    
    WARNING:
       All commands run on the ESXi shell are logged and may be included in
       support bundles. Do not provide passwords directly on the command line.
       Most tools can prompt for secrets or accept them from standard input.
    
    VMware offers supported, powerful system administration tools.  Please
    see www.vmware.com/go/sysadmintools for details.
    
    The ESXi Shell can be disabled by an administrative user. See the
    vSphere Security documentation for more information.
    [root@clx-host-154-oob:~] dcui
    

12. Press F2 to configure the ESXi. Authentication is required here.

    Warning: You can access and login to the ESXio shell on BlueField-2 DPU console DCUI using following keys Ctrl+G Ctrl+B 2 Enter You can switch back to DCUI window using following keys Ctrl+G Ctrl+B 3 Enter

13. Enter an authorized login name root and password (you set during install) for the ESXi host.

1. 选择 Configure Management Network 并确保/应用适当的主机网络配置。

   

   

   

   

   

   

2. 检查并 启用 ESXi Shell 和 SSH（在 ESXi (x86) 上）。

   

   

vCenter 部署

开始之前，请确保以下必需项可用：

• vCenter IP 和 FQDN 已在 DNS/DHCP 中注册

• 管理网络的子网掩码和网关

• NTP 服务器

• 确保 vCenter 主机名和 IP 地址的 nslookup 正常工作

  

• vCenter root 用户的密码

• administrator@vsphere.local 用户的密码

要在配备 BlueField-2 DPU 的 Dell 服务器上安装统一的 ESXi 映像：

1. 从 https://customerconnect.vmware.com/ 下载 VCSA 8.0 ISO，并将其挂载到 Windows 系统上。

2. 从 ISO 文件夹位置 CD\DVD Drive <VMware VCSA>:\vcsa-ui-installer\win32 执行 Installer.exe 应用程序。

   

3. 在介绍屏幕上选择“Install”，然后单击 Next。

   

   

4. 接受 最终用户许可协议，然后单击 Next。

   

5. 指定部署 vCenter Server 设备的 目标。目标可以是 ESXi 主机或现有的 vCenter Server。 在此部署中，我们将使用管理集群 vCenter Server。 填写表单后，单击 Next 继续。

   

6. 可能会出现 证书警告。单击 Yes 继续。

   

7. 在 选择文件夹 中，选择一个数据中心或 VM 文件夹来创建 vCenter Server VM，然后单击 Next。

   

8. 在 选择计算资源 中，选择将部署 vCenter Server 设备的 ESXi 服务器。

   

9. 指定 vCenter Server 的 VM 设置。单击 Next 继续。

   • VM Name - 虚拟机名称
   • Set root password - 提供密码
   • Confirm root password - 确认 root 密码

   

10. 为您的环境选择 VCSA 部署大小。在此部署中，我们将使用“Small”作为 部署大小，使用“Default”作为 存储大小。

11. 单击 Next 继续。

    

    警告：请参阅 vCenter Server Appliance 的硬件要求。

RDG for Redis on vSphere Distributed Services Engine (Project Monterey) deployment over NVIDIA BlueField-2 DPU

vSphere Configuration

Deploying vCenter Server Appliance

1. Select the datastore to identify the storage location for this vCenter server in Select datastore. Click Next to continue.

   Warning: For this environment, we selected "Enable Thin Disk Mode". If you enable thin disk mode, the datastore size should have a minimum of 25GB space.

   

2. Configure the network settings of the VCSA. Please provide following information:

   • Network - Select the port group to be used for the VCSA and ping the ESXi management network
   • IP Version - Select either IPv4 or IPv6
   • IP assignment - Select either Static or dhcp
   • FQDN - Provide the fully qualified domain name (FQDN) or IP address of the VCSA (vcentername.domain.com or x.x.x.x)
   • IP Address - Provide the IP address of the VCSA (x.x.x.x)
   • Subnet Mask or Prefix Length - Provide the Subnet mask or Prefix Length of the VCSA network (x.x.x.x or xx)
   • Default Gateway - Provide the Default Gateway (x.x.x.x)
   • DNS Servers - Provide the DNS server IP address for the VCSA (x.x.x.x,x.x.x.x)
   • Common Ports - Leave Defaults unless you need to customize them for your environment

3. Click Next to continue.

   

4. Review your configuration and click Finish to initiate stage 1 of deploying the VCSA.

   

5. Wait until the vCenter server appliance deployment is completed. Note, this might take awhile.

   

6. Click Continue to proceed with stage 2 of the deployment process once stage 1 is completed successfully.

   Warning: If you exit the setup installer window, you can continue with the vCenter Server setup at any time by logging in to the vCenter Server Management Interface (VAMI) https://vCenter_FQDN_or_IP:5480/

   

7. Click Next to proceed with Stage 2: setting up the vCenter Server.

   

8. Set the NTP and SSH configuration for the VCSA and click Next to continue.

   

9. Choose your SSO Configuration and click Next to continue. In this deployment we will be creating a new SSO domain.

   

10. Configure CEIP and click Next to continue.

    

11. Review Ready to complete screen and verify configuration. Click Finish to initiate stage 2.

    

12. After completing successfully stage 2, you will be presented with a URL. Click the URL to launch the vSphere Client. You can now close the vCenter Server Installer.

    

13. At the vCenter server page, accept the SSL certificate.

14. Click Launch vSphere Client.

    

15. Log into vSphere client using administrator@vsphere.local account.

    

16. You have deployed the vCenter Server Appliance successfully.

    

vSphere Configuration

Create vSphere Datacenter, Cluster and add the ESXi Host to the Cluster

1. Create a new Datacenter, navigate to Home and then Hosts and Clusters.
2. Right-click the mouse on the vCenter name to show the menu.

and select New Datacenter.

1. Name the new Datacenter. For example: SL-Datacenter.

   

2. Right-click on the Datacenter name to show the menu and to create a New Cluster.

   

3. In the Basic menu in the New Cluster window:

   1. Name the new Cluster (e.g, SL-WL01-Cluster01).
   2. Make sure the image the settings are set as shown in the figure below.

4. Click Next.

   

5. In the Image menu in the New Cluster window, make sure the right ESXi Version is set, then click Next.

   

6. In the Review menu in the New Cluster window, make sure the settings are correct, then click Finish.

   

7. Add the ESXi host to the Cluster.

   

8. Right-click on the Cluster name to show the menu, and select Add and Manage Hosts.

   

9. In the Select task menu, add the ESXi host that was created (name, user root, and password). If the user and password are the same for all ESXi hosts, you can enable the option to use the same credentials for all ESXi hosts (no need to add to each one).

10. Click Next.

    

11. In the Select Hosts menu, Select all ESXi hosts and import all certificates from the ESXi host to Center.

12. Click OK.

    

    1. In the Host Summary menu check for warnings. In our case, we have an warning that explains that one VM already exists in one of the ESXi hosts and is powered on. Such notification is harmless and just informed us about the ESXi host we are adding to the Center.

    2. In the Import Image menu, select Don't import an image. Click Next.

       

13. In the Manage physical adapters menu, check the ESXi host information. If all is ok, click Finish and add the ESXi hosts to your Cluster.

    

    If there is no problem with the IP/FQDN, or the network of your ESXi hosts, vCenter will start to import them to the Cluster.

    

14. Select the ESXi host and in Summary tab check the BlueField-2 DPU Information.

    

Add Additional MFT Depot to the Cluster Image

To load the MFT depot to the VMware Lifecycle Manager (vLCM).

1. Download MFT v4.22.1 LTS (required for vSphere 8.0b) from the web.

   

2. Open a browser, connect to vSphere web interface at https://<vcenter_fqdn>, and login with the administrator@vsphere.local account.

   ![

在 NVIDIA BlueField-2 DPU 上为 vSphere Distributed Services Engine (Project Monterey) 部署 RDG for Redis

创建于 2023 年 12 月 18 日

目录

• 范围
• 缩写和缩略语
• 简介
• 参考资料
• 解决方案架构
• 关键组件和技术
• 逻辑设计

---

范围

本文档描述了在 NVIDIA BlueField-2 DPU 上为 vSphere Distributed Services Engine (Project Monterey) 部署 RDG for Redis 的解决方案。

缩写和缩略语

缩写	全称
DPU	Data Processing Unit
RDG	Redis on vSphere Distributed Services Engine
SR-IOV	Single Root I/O Virtualization
MFT	Mellanox Firmware Tools
vCLS	vSphere Cluster Services

简介

本文档提供了在 NVIDIA BlueField-2 DPU 上为 vSphere Distributed Services Engine (Project Monterey) 部署 RDG for Redis 的详细步骤。

参考资料

• NVIDIA BlueField-2 DPU 用户指南
• VMware vSphere 文档

解决方案架构

关键组件和技术

• NVIDIA BlueField-2 DPU：提供硬件加速和隔离功能。
• VMware vSphere：虚拟化平台。
• Redis：内存数据库。

逻辑设计

---

配置步骤

在 ESXi 主机上启用 SR-IOV

要设置 SR-IOV 环境，需要满足以下要求：

1. 确保在特定服务器的 BIOS 中启用了 SR-IOV。

2. 通过 Web 浏览器访问 iDRAC。

   

3. 从 Configuration → BIOS Settings → Integrated Devices，将 SR-IOV Global Enable 设置为 Enabled。

   

4. 打开浏览器，连接到 vSphere Web 界面 https://<vcenter_fqdn>，并使用 administrator@vsphere.local 账户登录。

   

5. 在顶部左侧菜单中，点击树形图标并选择 Lifecycle Manager。

   

6. 打开 Action 下拉菜单并选择 Import Updates。

   

7. 点击 Browse。

   

8. 在弹出的窗口中，选择下载的 MFT NMST 文件，然后点击 Open。

   

9. 重复步骤 4 到 6 以导入第二个 depot 包。

   

   

10. 在 Inventory 选项卡中，选择集群，然后选择 Updates 选项卡。

11. 选择镜像，然后点击 Edit。

    

12. 点击 Show details。

    

13. 点击 ADD COMPONENTS。

    

14. 选择 MFT 文件，然后点击 SELECT。

    

15. 点击 SAVE。

    

16. 合规性检查将自动开始。

    

17. 点击 REMEDIATE ALL 以在主机上开始 MFT 安装。

    

18. 点击 START REMEDIATION。

    

    

    

19. 如果需要，手动关闭主机上的 vCLS VM 以将主机置于 维护 模式。

    

    

    

所有主机现在都已安装 MFT 工具。

在 ESXi 主机上启用 SR-IOV

要设置 SR-IOV 环境，需要满足以下要求：

1. 确保在特定服务器的 BIOS 中启用了 SR-IOV。

2. 通过 Web 浏览器访问 iDRAC。

   

3. 从 Configuration → BIOS Settings → Integrated Devices，将 SR-IOV Global Enable 设置为 Enabled。

   

4. 打开浏览器，连接到 vSphere Web 界面 https://<vcenter_fqdn>，并使用 administrator@vsphere.local 账户登录。

   

5. At the Inventory tab, click on a ESXi host, and select Configure → Hardware → PCI Devices, and click on ALL PCI DEVICES.

   

6. Filter by Vendor Name.

   

7. Select first port (in our deployment) and click CONFIGURE SR-IOV.

   

8. At the Configure SR-IOV menu, Enable SR-IOV and select number of Virtual functions(VF).

9. Click on OK.

   

10. Now you can see 8 Virtual functions.

    

11. To see the VF in host console. Log into the ESXi console with root permissions and run the following commands.

    

    

NTP Configuration

Verify that in your environment the NTP is configured and works properly on both the host and vCenter.

Create and Configure a Regular (non-offloaded) Distributed Virtual Switch for vMotion and vSAN

A regular (non-offloaded) Distributed Virtual Switch with portgroups for the vMotion, and vSAN traffic need to be created to ensure consistent network settings between the hosts.

1. Launch the vSphere Web Client and connect to a vCenter Server instance.

   

2. On the vSphere Web Client home screen, choose the Inventory Networking tab.

3. Right click on the datacenter (SL-Datacenter), select the Distributed Switch, then click New Distributed Switch (see image below) to launch the New VDS creation wizard.

   

4. In the Name and location menu, set the name (SL-WL01-DS01) of the new distributed switch and select the location within the vCenter inventory where you would like to store the new VDS (a data center object or a folder).

5. Click NEXT.

   

6. In the Select version menu, select the 8.0.0 – ESXi 8.0.0 and later switch version.

7. Click NEXT.

   

8. In the Configure settings menu:

   1. set the Network Offloads compatibility as None
   2. set the Number of uplinks as 2
   3. uncheck the Create a default port group box

9. Click NEXT.

   

10. In the Ready to complete menu, click Finish.

    

Set the MTU and Discovery protocol for the newly created distributed switch.

Right-click the new distributed switch in the list of objects and select Settings → Edit Settings... from the Actions menu.

In the SL_W01-DS01 Distributed Switch - Edit Settings dialog box, in the Advanced tab, set:

1. MTU to 9000
2. Discovery protocol Type to Link Layer Discovery Protocol
3. Operation to Both

Click OK.

Adding Hosts to VDS

To add an ESXi host to an existing VDS:

1. Launch the vSphere Web Client, and connect to the vCenter Server instance.

2. Navigate to the list of Hosts in the SL-WL01-Cluster01 and select ESXi host.

3. Select Configure → Networking → Physical adapters.

4. Check the network ports that you are going to use. In this case, vmnic1 will be used.

   

5. Navigate to the list of distributed switches.

6. Right-click the new distributed switch in the list of objects and select Add and Manage Hosts from the Actions menu.

7. In the Select task menu, check the Add hosts option and click NEXT.

   

8. In the Select Hosts menu, select all the ESXi hosts you would like to add to the VDS.

9. Click NEXT.

   

10. In the Manage physical adapters menu click on the 网卡 on all hosts tab and configure vmnic1 in an ESXi host as Uplink 1 for the VDS.

    Warning: In case you added the ConnectX-6DX please select the card ports as Uplink1 and 2.

    

11. In the Manage VMkernel adapters and Migrate VM networking menus click NEXT to continue.

    

12. In the Ready to complete menu click FINISH.

    

Creating Distributed Port Groups for vMotion vSAN Traffic

1. Right-click the Distributed switch, and select Distributed Port Group → New Distributed Port Group.

   

2. In the Name and location menu, set the Name as and click NEXT.

   

3. In the Configure settings menu:

   1. set the VLAN type as VLAN
   2. set the VLAN ID as

1620

• uncheck the Customize default policies configuration checkbox

1. Click NEXT

   

2. In the Ready to complete menu, make sure all the changes are OK and then click FINISH.

   

3. Repeat steps 1 to 4 for the vSAN.

   

Adding a VMkernel Network for vMotion and vSAN Port Group

To add VMkernel adapters for distributed port groups, follow the steps below.

1. Right click the distributed port group and select Add VMkernel 网卡.

   

2. In the Select hosts menu, select the hosts and click NEXT.

   

3. In the Configure VMkernel adapter menu, select vMotion and Provisioning in Available services, and click NEXT.

   

4. In the IPv4 settings menu, set the Network Settings and Gateway details, and click NEXT.

   

5. In the Ready to complete menu, click FINISH.

   

6. Repeat steps 1 to 5 for the vSAN.

   

   

   

   

Create and Configure a VMware vSAN ESA

To create and configure a VMware vSAN ESA cluster please use the following document RDG for VMware vSAN ESA over NVIDIA RoCE on VMware vSphere 8.0 - 解决方案 - NVIDIA Networking Docs

Warning RDMA is not supported in VMware vSphere 8.0b Distributed Services Engine. To run RDMA vSAN you can use additional ConnectX-6 Dx network card and use the card for vSAN RDMA traffic.

Samples of vSAN configuration screens.

Enable DRS and HA on Cluster

To enable DRS and HA on the SL-WL01-Cluster01 vSphere Cluster.

1. Open a browser, connect to vSphere web interface at https://<vcenter_fqdn>, and login with the administrator@vsphere.local account.

2. On the Inventory tab, select the cluster (SL-WL01-Cluster01), select Configure, select vSphere DRS, then select Edit….

3. On the Edit Cluster Settings page, enable vSphere DRS, then click Ok.

   

   

4. Check and apply LCM compliance.

   

To enable HA on the SL-WL01-Cluster01 vSphere Cluster.

1. Open a browser, connect to vSphere web interface at https://<vcenter_fqdn>, and login with the administrator@vsphere.local account.

2. On the Inventory tab, select the cluster (SL-WL01-Cluster01), select Configure, select vSphere DRS, then select Edit….

3. Enable vSphere HA, and click OK.

   

4. Check and apply LCM compliance.

   

Create and Configure an Offloaded Distributed Virtual Switch

A regular distributed switch with portgroups for the vMotion, and vSAN traffic should be created to ensure consistent network settings between the hosts.

1. Launch the vSphere Web Client and connect to a vCenter Server instance.

   

2. On the vSphere Web Client home screen, click on Inventory Networking tab.

3. Right click on the datacenter (SL-Datacenter), select the Distributed Switch, then click New Distributed Switch (see image below) to launch the New VDS creation wizard.

   

4. In the Name and location menu, set the name (SL-WL01-DS02) of the new distributed switch and select the location within the vCenter inventory where you would like to store the new VDS (a data center object or a folder).

5. Click NEXT.

   

6. In the Switch version menu, select 8.0.0 – ESXi 8.0.0 and later.

7. Click NEXT.

   

8. In the Configure settings menu:

   1. set the Network Offloads compatibility as NNVDIA BlueField
   2. set the Number of uplinks as 2
   3. uncheck the Create a default port group box

9. Click NEXT.

   

10. In the Ready to complete menu, click Finish.

• Set the MTU and Discovery protocol for the newly created distributed switch.

• Right-click the new distributed switch in the list of objects and select Settings → Edit Settings... from the Actions menu.

  

• In the SL_W01-DS02 Distributed Switch - Edit Settings dialog box, in the Advanced tab, set:

  1. MTU to 9000
  2. Discovery protocol Type to Link Layer Discovery Protocol
  3. Operation to Both

• Click OK.

  

Adding Hosts to VDS

To add an ESXi host to an existing VDS:

1. Launch the vSphere Web Client, and connect to the vCenter Server instance.

2. Navigate to the list of Hosts in the SL-WL01-Cluster01 and select ESXi host.

3. Select Configure → Networking → Physical adapters.

4. Check the network ports that you are going to use. In this case, vmnic0 will be used.

5. Navigate to the list of distributed switches.

6. Right-click the new distributed switch in the list of objects and select Add and Manage Hosts from the Actions menu.

   

7. In the Select task menu, check the Add hosts option and click NEXT.

   

8. In the Select hosts menu, select all the ESXi hosts you would like to add to the VDS.

9. Click NEXT.

   

10. In the Manage physical adapters menu, select the 网卡 on all hosts tab and configure vmnic0 in an ESXi host as Uplink 1 for the VDS.

    Warning: In case you added the ConnectX-6DX please select the card first port (vmnic0) as Uplink1 and second port (vmnic1) as Uplink2.

    

11. In the Manage VMkernel adapters and Migrate VM networking menus click NEXT to continue.

    

12. In the Ready to complete menu, click FINISH.

    

NSX Manager Deployment and Configuration

NSX Manager Deployment

Prerequisites

Below are the prerequisites for deploying NSX Manager.

• Necessary resources exist on the cluster to deploy NSX-Manager appliance

• Port group for Management network (it is preferable to keep on the same network as vCenter)

• 4 free IPs from management network for NSX Managers (in this guide only one appliance is deployed)

• NSX License

• Create DNS A records for NSX-Manager Hostnames

• Management network netmask and gateway

• NTP servers

• Make sure nslookup is working for vCenter hostname and IP Address

  

• Password for the NSX root user

• Password for the NSX admin

user

• Password for the NSX audit user

Network Latency Requirements

• The maximum network latency between NSX Managers in a NSX Manager cluster is 10ms
• The maximum network latency between NSX Managers and Transport Nodes is 150ms

Storage Requirements

• The maximum disk access latency is under 10ms

• It is recommended that NSX Managers are placed on a shared storage

• Storage should be highly available to avoid a storage outage causing all NSX Manager file systems to be placed into read-only mode upon a storage failure event

  Please see the documentation for your storage technology on how to optimally design a highly available storage solution.

Deployment

1. Download the NSX manager OVA from VMware Customer Connect portal.

   

2. Login to the Management Cluster vCenter.

   

3. Click Host and Clusters, right click on a ESXi where you would like the NSX Manager to be deployed at and click Deploy OVF Template.

   

4. Select Local file → browse the OVA file for NSX Unified appliance.

   

5. Click NEXT.

6. Assign a Virtual machine name (sl-wl01-nsx01) and select the location were you want to place the NSX manager and click NEXT.

   

7. Select the compute resource where you want to place your NSX manager (i.e., the cluster, resource pool, or the host), check its compatibility, and click NEXT.

   

8. Review the details and click NEXT.

   

9. Select deployment size (Medium), click NEXT.

   

   Warning: When selecting "Small" as the deployment configuration, some of the services do not work. Always use "Medium" or "Large".

10. Select the Storage/datastore where NSX manager should be placed and click NEXT.

    

11. Select the Management network and click NEXT to move to the "Customize Template" screen.

    

12. Specify the System GRUB root, System root account’s, admin and audit user passwords.

13. Scroll down.

    Warning: Strong password is required.

    

14. Provide the Hostname, Rolename (NSX Manager has 3 roles, as seen below), and Management networking details.

15. Scroll down.

• Assign the DNS and NTP details and choose whether SSH needs to be enabled on the NSX Manager.

• Click NEXT.

• Review the details and click FINISH. Wait for the NSX manager appliance to be successfully deployed.

• The NSX Manager will be deployed after about 5-10 minutes. Once the appliance is successfully deployed, Power on and wait for at least 15 minutes for all the services to come up.

• Once the services are up, a login prompt is shown. At this point, wait 10 more minutes as backend services need to start the web-portal-related services.

Post-Deployment Health Checks

Once done, do the following post-deployment steps:

1. Open the console and try to log in with admin credentials using putty. You will see the version number and the role.

   

2. Verify the network configuration by using the command below.

   get interface eth0
   

   

3. Check the services status by running the following command.

   get services
   

   

   Warning: Liagent, migration-coordinator and SNMP services are not started by default.

NSX Manager Configuration

1. Log in to NSX manager UI login page by using the URL "https://".

   

2. Accept the EULA on the first login to the NSX manager.

   

3. Decide whether or not to join the Customer Experience Improvement Program.

   

4. Take a quick tour, in case you need.

   

5. Finally, the NSX manager is presented.

   

6. Navigate to System → Licenses and click on +ADD LICENSE.

   

7. Add a license and click ADD.

   

Adding a Compute Manager

1. To add a compute manager, navigate to System → Fabric → Compute Managers, click ADD COMPUTE MANAGER to add vCenter as compute manager.

   

2. Fill in the vCenter details → click ADD.

   

3. Click ADD, when presented with the vCenter server thumbprint window.

   

4. Wait for registration to complete. The status will be changed to registered and connection status to "Up".

   

Create an Uplink Profile

An uplink profile defines the policies for links from a hypervisor hosts to the NSX logical switches or from NSX Edge nodes to top-of-rack switches. The settings defined by these profiles may include teaming policies, active/standby links, transport VLAN ID, and MTU settings. Uplink profiles enables consistent configuration of identical capabilities for network adapters across multiple hosts and nodes. By default, there are two uplink profiles that are provided with NSX and cannot be edited, that is why new profiles for the Edge uplink should be created (for hosts' uplinks as well).

To create a Host uplink profile in NSX Manager, do the following:

1. Navigate to System → Fabric → Profiles → Uplink Profiles → +ADD PROFILE.

   

2. Assign a name to the profile and fill the description.

3. Under Teamings, set the Teaming Policy to Failover Order.

4. Set the Active Uplinks to uplink-1.

5. The Transport VLAN will be an Overlay VLAN ID (in the example provided, 1640) since these uplinks are connected directly to the hosts and need to be tagged accordingly. MTU not fill.

6. Click ADD.

   

   

Create a Transport Zone

Transport Zones dictate which hosts (and consequently which VMs) can participate in the use of a particular network. There are two types of transport zones: an overlay and a VLAN.

• The overlay transport zone is used by both host transport nodes and NSX Edges and is responsible for communication over the overlay network.
• The VLAN transport zone is used by the NSX Edge for its VLAN uplinks.

Both types create an N-VDS on the host or Edge to allow for virtual-to-physical packet flow by binding logical router uplinks and downlinks to physical NICs.

To create Transport Zone with Overlay Traffic Type

1. To create a Transport Zone in the NSX Manager, navigate to System → Fabric → Transport Zones → +ADD ZONE.

   

2. Provide a Name and select the Traffic Type as Overlay (Geneve) and then click ADD.

   

   

Create a NSX TEPs IP Pool

Each transport node (i.e., hypervisors) is assigned with an IP address for the TEP interface. DHCP, Static IP List, and IP address Pool can all be used for assigning IP addresses for the TEP (Tunnel Endpoint) interfaces. To configure the ESXi hosts as Transport nodes, NSX IP Pool will be created to assign IP addresses for the TEP interfaces.

To create an IP Pool, in the NSX Manager, do the following:

1. Navigate to Networking → IP Address Pools → ADD IP ADDRESS POOL.

   

2. Specify the name and description of the IP address pool. Click Set.

   

3. Click ADD SUBNET and select IP Ranges.

4. 

5. Specify the IP Address Ranges, along with CIDR and Gateway IP, address for the specified IP ranges. Click ADD.

   

Click Apply.

Click SAVE to create the IP Pool.

Once the IP Pool is created, the status will be changed to Success.

Installing NSX Components on a Host Transport Node(ESXi/ESXio).

1. Navigate to System → Fabric → Hosts → Clusters. Select SL-WL01-Cluster01. Click Configure NSX.

   

2. In the Transport Node Profile, click the three dots menu to Create a New Transport Nodes Profile.

   

   1. Specify the Name → SL-WL01-TNP-EDP-STD and description, and click the Set.

      

   2. Click ADD HOST SWITCH.

   3. Select Name of vCenter → sl-wl01-vc01 and VDS → SL-WL01-DS02.

   4. Select Transport Zones were created before → SL-WL01-Overlay-TZ.

   5. Select the Uplink Profile → Uplink-profile-01.

   6. Select the IP Assignment (TEP) → Use IP Pool.

   7. Select the IP Pool → SL-WL01-TEP-IPPool.

   8. Select Mode → Enhanced Datapath - Standard.

      VDS Modes:

      • Standard applies to all the supported hosts.
      • Enhanced Datapath - Standard is an interrupt driven variant of the Enhanced Data Path mode.
      • Enhanced Datapath - Performance is the Enhanced Data Path switch mode. This mode provides accelerated networking performance, but also introduces additional prerequisites. In order to benefit from this mode, workloads must be compiled with DPDK and use VMNET3 for their vNIC.

   9. In Team Policy Uplink Mapping Select uplink-1 → Uplink 1.

      In case you added the ConnectX-6DX please select uplink-1 → Uplink 1 and uplink-2 → Uplink 2.

   10. Click ADD.

       

3. Click APPLY.

   

4. Click SAVE.

   

5. Select the Transport Nodes Profile we created (SL-WL01-TNP-EDP-STD) and click SAVE.

   

   You can see the installation is progressing.

   

6. In the vSphere client.

   

7. Wait until the hosts are successfully installed, the NSX Configuration status is "Success" and the Status indicators are "Up".

   

8. In the Inventory tab, select the cluster (SL-WL01-Cluster01), select Updates, select Image. Image compliance should show compliant prior to moving on to next steps.

   

Create an NSX Tier-1 Gateway

1. Navigate to Networking -> Tier-1 Gateway and click on Add Tier-1 Gateway.

   

2. Provide a name for the Tier-1 Gateway (T1-GW) and click on Save to complete creation of the Tier-1 Gateway.

   

3. Select NO in response to Do you want to continue configuring this Tier-1 Gateway?

   

4. The T1-GW Tier-1 Gateway is successfully created**.**

   

Create an Overlay Segment with Offload 交换机

1. Navigate to Networking -> Segments and click on Add Segments.

   

2. Provide a name for the Segment (SL-WL01-Seg01), Connected Gateway (T1-GW), Transport Zone (SL-WL01-Overlay-TZ), Subnets: 10.10.10.1/24.

3. Click on Save to complete creation of the Segment.

   

4. Select NO in response to Do you want to continue configuring this Segment?.

   

5. The SL-WL01-Seg01 segment is successfully created.

   

6. Ensure that you see the segment in vSphere client.

   

Validation

1. Log into the ESXi console.

2. Check the FPO model on ESXi.

3. Log into the ESXi console with root permissions, and run the following commands in the ESXi CLI to view information about the VDS switch and VMK interface(s).

   [root@clx-host-153:~] esxcfg-vswitch -l
   

   

   [root@clx-host-153:~] esxcfg-vmknic -l
   

   

4. Check the FPO model on the VDS (SL-WL01-DS02) switch.

   [root@clx-host-153:~] nsxdp-cli ens fpo get -dvs SL-WL01-DS02
   

   

5. On ESXi check the running FPO Model on the vmnic0 (in our environment).

   [root@clx-host-153:~] nsxdp-cli ens fpo status get -n vmnic0
   

   

6. Check the FPO model on ESXio.

7. Log into the ESXio console with sshdpu command.

   

   Warning: In case you see the "Connection refused" message. Run the following commands on the ESXi host to open SSH to ESXio.

   ssh: connect to host 169.254.100.2 port 22: Connection refused

   [root@clx-host-153:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0

   [root@clx-host-153:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0

8. Run the following commands in the ESXio CLI to view information about the VDS switch and VMK interface(s).

   [root@clx-host-153-oob:~] esxcfg-vswitch -l
   

   

   [root@clx-host-153-oob:~] esxcfg-vmknic -l
   

   

9. Check the FPO model on the VDS (SL-WL01-DS02) switch.

   [root@clx-host-153-oob:~] nsxdp-cli ens fpo get -dvs SL-WL01-DS02
   

   

10. On ESXio check the running FPO Model on the vmnic0 (in our environment).

    [root@clx-host-153-oob:~] nsxdp-cli ens fpo status get -n vmnic0
    

    

Create a Linux VM.

Create a Linux VM.

To create a Linux VM.

1. Right-click Host in the VMware Host Client inventory and select New Virtual Machine... The New Virtual Machine wizard opens.

2. Select Create a new virtual machine and click NEXT.

3. Select a name and a location for the VM and click NEXT.

4. On the Select a compute resource, select an ESXi host and click NEXT.

5. On the Select storage page, select the storage type for the virtual machine and a datastore where to store the virtual machine files.

   1. To save all the virtual machine disks and configuration files on a standard datastore, click the Standard button.
   2. To save the virtual machine hard disks on the host-local PMem datastore, click the Persistent Memory button.
   3. Select a datastore from the list and click Next. Ensure that the VM Storage is mapped to vSANDatastore or another shared datastore, if the intent is to be able to vMotion.

6. Click Next.

7. On the Select compatibility page, select Compatible with: ESXI 8.0 and later. Click NEXT.

8. On the Select a guest OS page, configure the guest operating system.

   1. From the Guest OS family drop-down menu, select the guest operating system.

   2. From the Guest OS version drop-down menu, select the guest operating system version.

      Warning: For UPTv2 VM make sure what OS from the supported list - https://kb.vmware.com/s/article/90548?lang=en_US. In our case we are using Ubuntu 22.04 with kernel 6.4.0.

9. Click Next.

10. On the Customize hardware page, configure the virtual machine hardware and options and click Next.

    1. On the Customize settings page, click Virtual Hardware and add a new virtual hardware device.

       1. Add the first 网卡 "New Network" to the Management Port Group(VM network).
       2. Click the ADD NEW DEVICE → 网卡 icon to add a second NIC to the virtual machine. And add the second 网卡 "**New Network ***" to the SL-WL01-Seg01 NSX Segment in the lab network. Select Adapter Type: VMXNET3. For UPTv2 VM select Use UPT Support: Enabled.

    2. Connect ISO File with an Linux OS.

    3. (Optional) To customize virtual machine options, click the VM Options button.

11. On the Ready to complete page, review the details and click Finish.

12. Power ON the VM and open the VM console to start to install OS from mapped ISO image.

13. After the installation is completed, check the IP for the second interface.

UPTv2 Interface Validation

• To validate if programming the interface in UPTv2 mode was successful, verify that the state of the Port is set to PT_UPT.

On the host run the following commands:

[root@clx-host-153:~] net-stats -l

Make sure the PortNum and the SwitchName corresponding to the interface on the UPTv2 VM attached to the NSX Segment (highlighted in the output above):

[root@clx-host-153:~] vsish -e get /net/portsets/DvsPortset-1/ports/134217763/vmxnet3/ptSummary

• To verify network connectivity and vMotion.

Log into the ESXi console.

Log into the ESXio console with sshdpu command.

Warning: In case you see the "Connection refused" message. Run the following commands on the ESXi host to open SSH to ESXio. ssh: connect to host 169.254.100.2 port 22: Connection refused [root@clx-host-153:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0 [root@clx-host-153:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0

Run the following command in the ESXio CLI to view information about the VDS switch using ports.

[root@clx-host-153-oob:~] esxcfg-vswitch -l

Check or set IP on UPT VMs second interface (ens35).

Run the ping command from the first UPT VM (clx-vm-upt-01) to the second UPT VM (clx-vm-upt-02) and migrate the first UPT VM (clx-vm-upt-01) to another host(not when the second UPT VM (clx-vm-upt-02) is runing).

• To verify Traffic Offloadinging.

Run the iperf commands from the first UPT VM (clx-vm-upt-01) to second UPT VM (clx-vm-upt-02).

Log into the ESXi console.

Run the following command in the ESXi CLI to view the flow table dump from x86.

[root@clx-host-153:~] nsxdp-cli ens flow-table dump

As expected, the table is empty.

Now, log into the ESXio console with sshdpu command.

Warning: In case you see the "Connection refused" message. Run the following commands on the ESXi host to open SSH to ESXio.

ssh: connect to host 169.254.100.2 port 22: Connection refused

[root@clx-host-153:~] vim-cmd combinersvc/dpu_services/set_policy TSM-SSH on vmdpu0

[root@clx-host-153:~] vim-cmd combinersvc/dpu_services/start TSM-SSH vmdpu0

Run the following commands in in the ESXio CLI to view information about the VDS switch using ports.

[root@clx-host-153-oob:~] nsxdp-cli ens flow-table dump

The tables shows that the iperf operation for UPT VMs is offloaded.

Stop the iperf command from the first UPT VM (clx-vm-upt-01) to the second UPT VM (clx-vm-upt-02).

Redis Install

For the purpose of installing Redis on Ubuntu 22.04, follow the given instructions.

Prerequisites

• A server (VM) with Ubuntu 22.04 as OS and kernel 6.4.0.
• User privileges: root or non-root user with sudo privileges

1. Update the packages to their latest versions available after a fresh install of Ubuntu 22.04.

   sudo apt-get update -y && sudo apt-get upgrade -y
   

2. Install the Redis server.

   sudo apt-get install redis-server -y
   

3. Once installed, start and enable the redis service.

   sudo systemctl enable redis-server.service && sudo systemctl start redis-server.service
   

4. Check Redis server version:

   redis-server -v
   

   

5. Check if the service is up and running.

   sudo systemctl status redis-server.service
   

   You should receive the following output:

   

6. Test Redis server and ping service.

   redis-cli
   127.0.0.1:6379> ping
   PONG
   

7. Check Redis information.

   127.0.0.1:6379> INFO
   # Server
   redis_version:7.2.3
   redis_git_sha1:00000000
   redis_git_dirty:0
   redis_build_id:7f52fd1717e1b756
   redis_mode:standalone
   os:Linux 6.4.0-060400-generic x86_64
   arch_bits:64
   monotonic_clock:POSIX clock_gettime
   multiplexing_api:epoll
   atomicvar_api:c11-builtin
   gcc_version:11.4.0
   process_id:46090
   process_supervised:systemd
   run_id:ce8123f949fe1367a31affe218d17f38da81eb3b
   tcp_port:6379
   server_time_usec:1704885230333632
   uptime_in_seconds:1465
   uptime_in_days:0
   hz:10
   configured_hz:10
   lru_clock:10386414
   executable:/usr/bin/redis-server
   config_file:/etc/redis/redis.conf
   io_threads_active:0
   listener0:name=tcp,bind=127.0.0.1,bind=-::1,port=6379
   
   # Clients
   connected_clients:1
   cluster_connections:0
   maxclients:10000
   client_recent_max_input_buffer:8
   client_recent_max_output_buffer:0
   blocked_clients:0
   tracking_clients:0
   clients_in_timeout_table:0
   total_blocking_keys:0
   total_blocking_keys_on_nokey:0
   
   # Memory
   used_memory:940664
   used_memory_human:918.62K
   used_memory_rss:11534336
   used_memory_rss_human:11.00M
   used_memory_peak:1134032
   used_memory_peak_human:1.08M
   used_memory_peak_perc:82.95%
   used_memory_overhead:868032
   used_memory_startup:865920
   used_memory_dataset:72632
   used_memory_dataset_perc:97.17%
   allocator_allocated:1816960
   allocator_active:1970176
   allocator_resident:5959680
   total_system_memory:33645490176
   total_system_memory_human:31.33G
   used_memory_lua:31744
   used_memory_vm_eval:31744
   used_memory_lua_human:31.00K
   used_memory_scripts_eval:0
   number_of_cached_scripts:0
   number_of_functions:0
   number_of_libraries:0
   used_memory_vm_functions:32768
   used_memory_vm_total:64512
   used_memory_vm_total_human:63.00K
   used_memory_functions:184
   used_memory_scripts:184
   used_memory_scripts_human:184B
   maxmemory:0
   maxmemory_human:0B
   maxmemory_policy:noeviction
   allocator_frag_ratio:1.08
   allocator_frag_bytes:153216
   allocator_rss_ratio:3.02
   allocator_rss_bytes:3989504
   rss_overhead_ratio:1.94
   rss_overhead_bytes:5574656
   mem_fragmentation_ratio:12.57
   mem_fragmentation_bytes:10616560
   mem_not_counted_for_evict:0
   mem_replication_backlog:0
   mem_total_replication_buffers:0
   mem_clients_slaves:0
   

mem_clients_normal:1928 mem_cluster_links:0 mem_aof_buffer:0 mem_allocator:jemalloc-5.3.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0

Persistence

loading:0 async_loading:0 current_cow_peak:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1704883765 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_saves:0 rdb_last_cow_size:0 rdb_last_load_keys_expired:0 rdb_last_load_keys_loaded:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_rewrites:0 aof_rewrites_consecutive_failures:0 aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0

Stats

total_connections_received:1 total_commands_processed:2 instantaneous_ops_per_sec:0 total_net_input_bytes:55 total_net_output_bytes:205212 total_net_repl_input_bytes:0 total_net_repl_output_bytes:0 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.00 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:3 evicted_keys:0 evicted_clients:0 total_eviction_exceeded_time:0 current_eviction_exceeded_time:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 pubsubshard_channels:0 latest_fork_usec:0 total_forks:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 total_active_defrag_time:0 current_active_defrag_time:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:0 dump_payload_sanitizations:0 total_reads_processed:3 total_writes_processed:4 io_threaded_reads_processed:0 io_threaded_writes_processed:0 reply_buffer_shrinks:1 reply_buffer_expands:0 eventloop_cycles:14633 eventloop_duration_sum:463427 eventloop_duration_cmd_sum:740 instantaneous_eventloop_cycles_per_sec:9 instantaneous_eventloop_duration_usec:29 acl_access_denied_auth:0 acl_access_denied_cmd:0 acl_access_denied_key:0 acl_access_denied_channel:0

Replication

role:master connected_slaves:0 master_failover_state:no-failover master_replid:80635a1ad25fe22b9b37498932d6537b467f14e7 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:0.224864 used_cpu_user:0.417350 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000 used_cpu_sys_main_thread:0.224397 used_cpu_user_main_thread:0.417483

Modules

Errorstats

Cluster

cluster_enabled:0

Keyspace

127.0.0.1:6379>

1. Exit Redis CLI.

   127.0.0.1:6379> exit
   user@base-redis1:~$
   

memtier Install

For the purpose of installing Redis on Ubuntu 22.04, follow the given instructions.

Appendix

Test the Environment

Redis includes a benchmark as part of its distribution, but more precise results are obtained by a separate benchmarking tool called Memtier, which is used in our tests. Memtier supports many parameters to simulate different scenarios, with different numbers of simulated customers accessing the Redis database on different threads, with different operation mixes, data sizes, and key sets. Memtier also supports different protocols (e.g., Redis, memcached binary, memcached text) for accessing a remote database. Here, several Redis clients (typically 50-200) in Memtier request remote transactions (like a Set or a Get using a key) from a remote Redis database, and each client awaits the result before requesting a new transaction. The benchmark results are the average number of transactions/second supported by the system, the average latency of those transactions, and the network bandwidth utilized (network throughput). By varying the Memtier parameters, we can move the workload from a CPU-bound scenario to one which is network-bound (i.e., we reach the limit of our network card), while maintaining average latency at or below our target.

Hardware and Software Components

Host under test:

• Server, Dell PowerEdge R750 vSAN Ready Node, with 2 x Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz 16 cores, 384GB of RAM.
• NVIDIA ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AN-CDA), with the driver 4.23.0.36-8vmw.800.1.0.20513097 and FW 22.34.10.02 versions.
• 4xDell Ent NVMe P5600 MU U.2 1.6TB ( vSAN ESA Default Plicy - RAID5 )
• VMware ESXi™ 8.0 GA

Network:

• NVIDIA Spectrum® SN3700 Open Ethernet Switch NVIDIA® Cumulus® Linux v5.1 Network OS
• NVIDIA MCP1650-H001E30 Passive Copper Cable InfiniBand HDR up to 200Gb/s QSFP56 LSZH 1m Black Pulltab 30AWG

Virtual Machine and Benchmark Configuration

We used memtier benchmark workloads to measure performance with the following parameter configurations:

memtier_benchmark install.

For the purpose of installingmemtier_benchmark on Ubuntu 22.04, follow the given instructions.

Prerequisites

• A server (VM) with Ubuntu 22.04 as OS and kernel 6.4.0.
• User privileges: root or non-root user with sudo privileges

1. Update the packages to their latest versions available after a fresh install of Ubuntu 22.04.

   sudo apt-get update -y && sudo apt-get upgrade -y
   

2. Install all prerequisites by running.

   sudo apt-get -y install build-essential autoconf automake libpcre3-dev libevent-dev pkg-config zlib1g-dev libssl-dev
   

3. Download the source tree and set 1.4.0 version.

   sudo git clone https://github.com/RedisLabs/memtier_benchmark.git
   cd memtier_benchmark/
   sudo git checkout 1.4.0 -b v1.4-branch
   

4. Use standard autoconf/automake commands:

   sudo autoreconf -ivf
   sudo ./configure
   sudo make
   sudo make install
   

Performance Results

Note: The performance results listed in this document are indicative, and should not be considered as formal performance targets for NVIDIA products.

Conclusion

Done!

Authors

Boris KovalevBoris Kovalev has worked for the past several years as a 解决方案 Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the NVIDIA Documents website.