RDG for DPF Host Trusted with HBN DPU Service
Created on December 31, 2025 Scope This Reference Deployment Guide (RDG) provides detailed instructions for deploying a Kubernetes (K8s) cluster using the DOCA
文档目录
Created on December 31, 2025
Scope
This Reference Deployment Guide (RDG) provides detailed instructions for deploying a Kubernetes (K8s) cluster using the DOCA Platform Framework (DPF). The guide focuses on setting up an accelerated Host-Based Networking (HBN) service on NVIDIA® BlueField®-3 DPU to deliver secure, isolated, and hardware-accelerated environments.
This guide is designed for experienced system administrators, system engineers, and solution architects who seek to deploy high-performance Kubernetes clusters with Host-Based Networking enabled on NVIDIA BlueField DPU.
Warning
- This reference implementation, as the name implies, is a specific, opiniated deployment example designed to address the use case described above.
- While other approaches may exist to implement similar solutions, this document provides a detailed guide for this particular method.
Abbreviations and Acronyms
| Term | Definition | Term | Definition |
|---|---|---|---|
| BFB | BlueField Bootstream (OS Image) | MAAS | Metal as a Service |
| BGP | Border Gateway Protocol | RDG | Reference Deployment Guide |
| CNI | Container Network Interface | RDMA | Remote Direct Memory Access |
| CSI | Container Storage Interface | SFC | Service Function Chaining |
| DOCA | Data Center Infrastructure-on-a-Chip Architecture | SR-IOV | Single Root Input/Output Virtualization |
| DPF | DOCA Platform Framework | TOR | Top of Rack |
| DPU | Data Processing Unit | VLAN | Virtual LAN (Local Area Network) |
| GENEVE | Generic Network Virtualization Encapsulation | VNI | Virtual Network Interface |
| HBN | Host Based Networking | VRF | Virtual Router/Forwarder |
| IPAM | IP Address Management | VRR | Virtual Router Redundancy |
| K8S | Kubernetes | VTEP | Virtual Tunnel End Point |
Introduction
The NVIDIA BlueField-3 Data Processing Unit (DPU) is a 400 Gb/s infrastructure compute platform designed for line-rate processing of software-defined networking, storage, and cybersecurity workloads. It combines powerful compute resources, high-speed networking, and advanced programmability to deliver hardware-accelerated, software-defined solutions for modern data centers.
NVIDIA DOCA unleashes the full potential of the BlueField platform by enabling rapid development of applications and services that offload, accelerate, and isolate data center workloads.
One such service is Host-Based Networking (HBN) - a DOCA-enabled solution that allows network architects to design networks based on Layer 3 (L3) protocols. HBN enables routing on the server side by using BlueField as a BGP router. It encapsulates key networking functions in a containerized service pod, deployed directly on the BlueField’s ARM cores.
However, deploying and managing DPU and their associated DOCA services, especially at scale, presents operational challenges. Without a robust provisioning and orchestration system, tasks such as lifecycle management, service deployment, and network configuration for service function chaining (SFC) can quickly become complex and error prone. This is where the DOCA Platform Framework (DPF) comes into play.
DPF automates the full DPU lifecycle, streamlines the deployment of DOCA services, and simplifies advanced network configurations. With DPF, services such as HBN can be deployed seamlessly, allowing for efficient offloading and intelligent routing of traffic through the DPU data plane.
By leveraging DPF, users can scale and automate DPU management across Kubernetes customer environments - optimizing performance while simplifying operations.
As part of the reference implementation, open-source components outside the scope of DPF (e.g., MAAS, pfSense, Kubespray) are used to simulate a realistic customer deployment environment.
The guide includes the full end-to-end deployment process, including:
- Infrastructure provisioning
- DPF deployment
- DPU provisioning
- Service configuration and deployment
- Service chaining
It also demonstrates some performance optimizations, with results validated through standard RDMA and TCP workload tests.
References
- NVIDIA BlueField DPU
- NVIDIA DOCA
- NVIDIA DOCA HBN Service
- NVIDIA DPF Release Notes
- NVIDIA DPF GitHub Repository
- NVIDIA DPF System Overview
- NVIDIA DPF with HBN User Guide
- NVIDIA Ethernet Switching
- NVIDIA Cumulus Linux
- NVIDIA Network Operator
- What is K8s?
- Kubespray
Solution Architecture
Key Components and Technologies
-
NVIDIA BlueField® Data Processing Unit (DPU) The NVIDIA® BlueField® data processing unit (DPU) ignites unprecedented innovation for modern data centers and supercomputing clusters. With its robust compute power and integrated software-defined hardware accelerators for networking, storage, and security, BlueField creates a secure and accelerated infrastructure for any workload in any environment, ushering in a new era of accelerated computing and AI.
-
NVIDIA DOCA Software Framework NVIDIA DOCA™ unlocks the potential of the NVIDIA® BlueField® networking platform. By harnessing the power of BlueField DPU and SuperNICs, DOCA enables the rapid creation of applications and services that offload, accelerate, and isolate data center workloads. It lets developers create software-defined, cloud-native, DPU- and SuperNIC-accelerated services with zero-trust protection, addressing the performance and security demands of modern data centers.
10/25/40/50/100/200和400G以太网网卡
业界领先的NVIDIA® ConnectX®系列智能网卡（SmartNICs）提供先进的硬件卸载和加速功能。
NVIDIA以太网网卡为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供最高的ROI和最低的总拥有成本。
NVIDIA® LinkX®线缆和收发器产品系列提供业界最完整的10、25、40、50、100、200和400GbE以太网以及100、200和400Gb/s InfiniBand产品，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。
灵活的外形规格，支持16到128个物理端口，速度从1GbE到400GbE。
基于突破性的硅技术，针对性能和可扩展性进行了优化，NVIDIA Spectrum交换机非常适合构建高性能、高性价比、高效的云数据中心网络、以太网存储结构和深度学习互连。
NVIDIA将基于业界领先的专用集成电路（ASIC）技术的NVIDIA Spectrum™交换机的优势与多种现代网络操作系统选择相结合，包括NVIDIA Cumulus® Linux、SONiC和NVIDIA Onyx®。
NVIDIA® Cumulus® Linux是业界最具创新性的开放网络操作系统，可让您像其他操作系统一样自动化、定制和扩展数据中心网络。
NVIDIA Network Operator简化了Kubernetes集群中NVIDIA网络资源的配置和管理。该运算符自动安装所需的主机网络软件，汇集所有必要组件以提供高速网络连接。这些组件包括NVIDIA网络驱动程序、Kubernetes设备插件、CNI插件、IP地址管理（IPAM）插件等。NVIDIA Network Operator与NVIDIA GPU Operator协同工作，为可扩展的GPU计算集群提供高吞吐量、低延迟的网络。
Kubernetes是一个开源容器编排平台，用于容器化应用程序的部署自动化、扩展和管理。
Kubespray由Ansible剧本、清单、配置工具和通用OS/Kubernetes集群配置管理任务的领域知识组成，并提供：
- 高可用集群
- 可组合属性
- 支持大多数流行的Linux发行版
RDMA是一种允许网络中的计算机在不涉及任一计算机的处理器、缓存或操作系统的情况下交换数据的技术。
与本地DMA类似，RDMA提高了吞吐量和性能，并释放了计算资源。
Solution Design
Solution Logical Design
逻辑设计包括以下组件：
- 1个Hypervisor节点（基于KVM），配备ConnectX-7
- 1个防火墙VM
- 1个跳板VM
- 1个MaaS VM
- 3个K8s Master VM，运行所有K8s管理组件
- 2个工作节点（PCI Gen5），每个配备1个BlueField-3网卡
- 单个高速（HS）交换机
- 1 Gb主机管理网络
HBN service Logical Design
在本文档中，我们将在每个工作节点上创建两个隔离网络；一个基于PF0的虚拟功能VF10，另一个基于PF1的虚拟功能VF10。
每个网络通过HBN服务连接到单独的VLAN/VNI，位于单独的VRF - RED和BLUE上。
我们将使用hostdev插件将这些虚拟功能作为辅助网络分配给工作负载Pod。然后，我们将演示同一网络内（例如RED网络）不同工作节点上Pod之间的加速RDMA和TCP流量，并验证连接到不同网络（RED vs BLUE）的Pod之间的网络隔离。
如果您对加速主Kubernetes网络感兴趣，请参阅此RDG，其中涵盖了DPF与HBN和OVN-Kubernetes服务以及额外DOCA服务的部署。
K8s Cluster Logical Design
以下K8s逻辑设计图突出了DPF系统的主要组件，包括：
- 3个K8s Master Node VM，运行所有K8s管理组件
- 2个K8s Worker Nodes（x86）
- 2个K8s DPU Workers，运行DOCA服务（HBN）
- 1个Kamaji（K8s控制平面管理器）
- 1个Tenant DPU Control Plane（租户集群）
- 连接到高速和1GbE网络
Firewall Design
本解决方案中的pfSense防火墙扮演两个关键角色：
- 防火墙 – 为DPF系统提供隔离环境，确保安全操作
- 路由器 – 为管理网络提供互联网访问
防火墙上配置了SSH和RDP的端口转发规则，将流量路由到跳板节点在主机管理网络上的IP地址。从跳板节点，管理员可以管理和访问设置中的各种设备，以及处理Kubernetes（K8s）集群和DPF组件的部署。
下图说明了本解决方案中使用的防火墙设计：
Software Stack Components
注意： 确保使用上述软件堆栈的完全相同版本。
Bill of Materials
物料清单
物料清单
部署与配置
节点与交换机定义
以下定义和参数用于部署演示的组网：
交换机端口使用
| Hostname | Rack ID | Ports |
|---|---|---|
hs-switch |
1 | swp1-4 |
mgmt-switch |
1 | swp1-3 |
主机
| Rack | Server Type | Server Name | Switch Port | IP and NICs | Default Gateway |
|---|---|---|---|---|---|
| Rack1 | Hypervisor Node | hypervisor |
mgmt-switch: swp1 |
mgmt-br (interface eno2): -lab-br (interface eno1): Trusted LAN IP | Trusted LAN GW |
| Rack1 | Worker Node | worker1 |
mgmt-switch: swp2hs-switch: swp1-swp2 |
ens15f0: 10.0.110.21/24ens5f0np0/ens5f1np1: 10.0.120.0/22 | 10.0.110.254 |
| Rack1 | Worker Node | worker2 |
mgmt-switch: swp3hs-switch: swp3-swp4 |
ens15f0: 10.0.110.22/24ens5f0np0/ens5f1np1: 10.0.120.0/22 | 10.0.110.254 |
| Rack1 | Firewall (Virtual) | fw |
- | LAN (mgmt-br): 10.0.110.254/24WAN (lab-br): Trusted LAN IP | Trusted LAN GW |
| Rack1 | Jump Node (Virtual) | jump |
- | enp1s0: 10.0.110.253/24 | 10.0.110.254 |
| Rack1 | MaaS (Virtual) | maas |
- | enp1s0: 10.0.110.252/24 | 10.0.110.254 |
| Rack1 | Master Node (Virtual) | master1 |
- | enp1s0: 10.0.110.1/24 | 10.0.110.254 |
| Rack1 | Master Node (Virtual) | master2 |
- | enp1s0: 10.0.110.2/24 | 10.0.110.254 |
| Rack1 | Master Node (Virtual) | master3 |
- | enp1s0: 10.0.110.3/24 | 10.0.110.254 |
布线
Hypervisor节点
K8s Worker节点
组网配置
更新Cumulus Linux
作为最佳实践，请确保使用最新发布的Cumulus Linux NOS版本。
有关如何升级Cumulus Linux的信息，请参阅Cumulus Linux用户指南。
配置Cumulus Linux交换机
按如下方式配置SN3700交换机（hs-switch）：
# 配置命令示例
nv set interface lo ip address 11.0.0.101/32
nv set interface lo type loopback
nv set interface swp1-4 link state up
nv set interface swp1-4 type swp
nv set router bgp autonomous-system 65001
nv set router bgp enable on
nv set router bgp graceful-restart mode full
nv set router bgp router-id 11.0.0.101
nv set vrf default router bgp address-family ipv4-unicast enable on
nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
nv set vrf default router bgp address-family ipv6-unicast enable on
nv set vrf default router bgp address-family ipv6-unicast redistribute connected enable on
nv set vrf default router bgp enable on
nv set evpn enable on
nv set vrf default router bgp neighbor swp1-4 peer-group hbn
nv set vrf default router bgp neighbor swp1-4 type unnumbered
nv set vrf default router bgp path-selection multipath aspath-ignore on
nv set vrf default router bgp peer-group hbn remote-as external
nv set vrf default router bgp address-family l2vpn-evpn enable on
nv set vrf default router bgp peer-group hbn address-family l2vpn-evpn enable on
nv config apply -y
按如下方式配置 SN2201 交换机（mgmt-switch）：
nv set bridge domain br_default untagged 1
nv set interface swp1-3 link state up
nv set interface swp1-3 type swp
nv set interface swp1-3 bridge domain br_default
nv config apply -y
主机配置
注意： 确保工作节点服务器的 BIOS 设置中已启用 SR-IOV，并且服务器已调优至最佳性能。
注意： 确保所有工作节点的 BlueField-3 网卡具有相同的 PCIe 位置，并显示相同的接口名称。
虚拟机管理程序安装与配置
本参考部署指南（RDG）使用的虚拟机管理程序基于 Ubuntu 24.04 和 KVM。
本文档不详细说明 KVM 安装过程，但请注意，部署防火墙、跳板机和 MaaS 虚拟机（VM）需要以下 ISO：
- Ubuntu 24.04
- pfSense-CE-2.7.2
为实现该解决方案，必须在虚拟机管理程序上创建两个 Linux 桥接：
警告： 确保在受信任的 LAN 中为
lab-br桥接接口配置 DHCP 记录，以分配 IP 地址。
lab-br– 将防火墙 VM 连接到受信任的 LAN。mgmt-br– 将各个 VM 连接到主机管理网络。
虚拟机管理程序 netplan 配置
network:
ethernets:
eno1:
dhcp4: false
eno2:
dhcp4: false
bridges:
lab-br:
interfaces: [eno1]
dhcp4: true
mgmt-br:
interfaces: [eno2]
dhcp4: false
version: 2
应用配置：
虚拟机管理程序控制台
$ sudo netplan apply
准备基础设施服务器
防火墙 VM - pfSense 安装与接口配置
将 pfSense CE（社区版）ISO 下载到虚拟机管理程序，然后进行软件安装。
建议规格：
- vCPU：2
- 内存：2GB
- 存储：10GB
- 网络接口
- 连接到
lab-br的桥接设备 - 连接到
mgmt-br的桥接设备
- 连接到
防火墙 VM 必须连接到虚拟机管理程序上的两个 Linux 桥接。在开始安装之前，请确保配置了三个类型为 “桥接设备” 的虚拟网络接口。每个接口应连接到不同的桥接（lab-br 和 mgmt-br），如下图所示。
安装完成后，设置向导会显示一个包含多个选项的菜单，例如“分配接口”和“重启系统”。在此阶段，配置防火墙 VM 的网络接口：
- 选择 选项 2：“设置接口 IP 地址”，并按如下方式配置接口：
- WAN – 受信任的 LAN IP（静态/DHCP）
- LAN – 静态 IP
10.0.110.254/24
- 接口配置完成后，使用主机管理网络内的 Web 浏览器访问防火墙 Web 界面，完成配置。
接下来，安装 跳板机 VM。该 VM 将作为运行浏览器的平台，用于访问防火墙的 Web 界面进行安装后配置。
跳板机 VM
建议规格：
- vCPU：4
- 内存：8GB
- 存储：25GB
- 网络接口：桥接设备，连接到
mgmt-br
步骤：
-
进行标准 Ubuntu 24.04 安装。在此设置的所有主机上使用以下登录凭据：
用户名 密码 depuser user -
通过创建以下 Netplan 配置，启用互联网连接和 DNS 解析：
警告： 在 MaaS VM 安装并配置完成之前，使用
10.0.110.254作为临时 DNS 名称服务器。完成 MaaS 安装后，更新 Netplan 文件，将该地址替换为 MaaS IP：10.0.110.252。跳板节点 netplan
network: ethernets: enp1s0: dhcp4: false addresses: [10.0.110.253/24] nameservers: search: [dpf.rdg.local.domain] addresses: [10.0.110.254] routes: - to: default via: 10.0.110.254 version: 2 -
应用配置：
跳板节点控制台
depuser@jump:~$ sudo netplan apply -
更新并升级系统：
跳板节点控制台
depuser@jump:~$ sudo apt update -y depuser@jump:~$ sudo apt upgrade -y -
安装并配置 Xfce 桌面环境 和 XRDP（RDP 的补充包）：
跳板节点控制台
depuser@jump:~$ sudo apt install -y xfce4 xfce4-goodies depuser@jump:~$ sudo apt install -y lightdm-gtk-greeter depuser@jump:~$ sudo apt install -y xrdp depuser@jump:~$ echo "xfce4-session" | tee .xsession depuser@jump:~$ sudo systemctl restart xrdp -
安装 Firefox 以访问防火墙 Web 界面：
跳板节点控制台
$ sudo apt install -y firefox -
安装并配置 NFS 服务器，使用
/mnt/dpf_share目录：跳板节点控制台
$ sudo apt install -y nfs-server $ sudo mkdir -m 777 /mnt/dpf_share $ sudo vi /etc/exports -
将以下行添加到
/etc/exports：跳板节点控制台
/mnt/dpf_share
10.0.110.0/24(rw,sync,no_subtree_check)
1. Restart the NFS server:
**Jump Node Console**
```bash
$ sudo systemctl restart nfs-server
```
1. Create the directory `bfb` under `/mnt/dpf_share` with the same permissions as the parent directory:
**Jump Node Console**
```bash
$ sudo mkdir -m 777 /mnt/dpf_share/bfb
```
1. Generate an SSH key pair for `depuser` in the jump node (later on will be imported to the admin user in MaaS to enable password-less login to the provisioned servers):
**Jump Node Console**
```bash
depuser@jump:~$ ssh-keygen -t rsa
```
1. Finally, reboot the VM to load the graphical interface:
**Jump Node Console**
```bash
depuser@jump:~$ sudo reboot
```
> **Warning**
> After setting up the port-forwarding rules on the firewall (see next steps), you will be able to remotely log into the graphical interface of the Jump node via RDP and SSH.
> Please note that you can not be logged in to both the local graphical console and the RDP client at the same time. Be sure to log out before switching to an RDP connection.
##### Firewall VM – Web Configuration
From the Jump node graphical interface, open a Firefox web browser and go to the pfSense web UI (`http://10.0.110.254`; default credentials are `admin/pfsense`). You should see a page similar to the following:
> **Warning**
> The IP addresses from the trusted LAN network under "DNS servers" and "Interfaces - WAN" are blurred.
Proceed with the following configurations:
> **Warning**
> The following screenshots display only part of the configuration view. Be sure to follow all of the steps mentioned below!
- Interfaces
- WAN (lab-br) – mark "Enable interface", unmark "Block private networks and loopback addresses"
- LAN (mgmt-br) – mark "Enable interface", "IPv4 configuration type": **Static IPv4** ("IPv4 Address": **10.0.110.254/24**, "IPv4 Upstream Gateway": **None**)
- Firewall:
- NAT -> Port Forward -> Add rule -> "Interface": **WAN**, "Address Family": **IPv4**, "Protocol": **TCP**, "Destination": **WAN address**, "Destination port range": ("From port": **SSH**, "To port": **SSH**), "Redirect target IP": ("Type": **Address or Alias**, "Address": **10.0.110.253**), "Redirect target port": **SSH**, "Description": **NAT SSH**
- NAT -> Port Forward -> Add rule -> "Interface": **WAN**, "Address Family": **IPv4**, "Protocol": **TCP**, "Destination": **WAN address**, "Destination port range": ("From port": **MS RDP**, "To port": **MS RDP**), "Redirect target IP": ("Type": **Address or Alias**, "Address": **10.0.110.253**), "Redirect target port": **MS RDP**, "Description": **NAT RDP**
##### MaaS VM
Suggested specifications:
- vCPU: 4
- RAM: 4GB
- Storage: 50GB
- Network interface: Bridge device, connected to `mgmt-br`
Procedure:
1. Perform a regular Ubuntu installation on the MaaS VM.
1. Create the following Netplan configuration to enable internet connectivity and DNS resolution:
> **Warning**
> Use `10.0.110.254` as a temporary DNS nameserver. After the MaaS installation, replace this with the MaaS IP address (`10.0.110.252`) in both the Jump and MaaS VM Netplan files.
**MaaS netplan**
```yaml
network:
ethernets:
enp1s0:
dhcp4: false
addresses: [10.0.110.252/24]
nameservers:
search: [dpf.rdg.local.domain]
addresses: [10.0.110.254]
routes:
- to: default
via: 10.0.110.254
version: 2
```
1. Apply the netplan configuration:
**MaaS Console**
```bash
depuser@maas:~$ sudo netplan apply
```
1. Update and upgrade the system:
**MaaS Console**
```bash
depuser@maas:~$ sudo apt update -y
depuser@maas:~$ sudo apt upgrade -y
```
1. Install PostgreSQL and configure the database for MaaS:
**MaaS Console**
```bash
$ sudo -i
# apt install -y postgresql
# systemctl disable --now systemd-timesyncd
# export MAAS_DBUSER=maasuser
# export MAAS_DBPASS=maaspass
# export MAAS_DBNAME=maas
# sudo -i -u postgres psql -c "CREATE USER \"$MAAS_DBUSER\" WITH ENCRYPTED PASSWORD '$MAAS_DBPASS'"
# sudo -i -u postgres createdb -O "$MAAS_DBUSER" "$MAAS_DBNAME"
```
1. Install MaaS:
**MaaS Console**
```bash
# snap install maas
```
1. Initialize MaaS:
**MaaS Console**
```bash
# maas init region+rack --maas-url http://10.0.110.252:5240/MAAS --database-uri "postgres://$MAAS_DBUSER:$MAAS_DBPASS@localhost/$MAAS_DBNAME"
```
1. Create an admin account:
**MaaS Console**
```bash
# maas createadmin --username admin --password admin --email admin@example.com
```
1. Save the admin API key:
**MaaS Console**
```bash
# maas apikey --username admin > admin-apikey
```
1. Log in to the MaaS server:
**MaaS Console**
```bash
# maas login admin http://localhost:5240/MAAS "$(cat admin-apikey)"
```
1. Configure MaaS (Substitute <Trusted_LAN_NTP_IP> and <Trusted_LAN_DNS_IP> with the IP addresses in your environment):
**MaaS Console**
```bash
# maas admin domain update maas name="dpf.rdg.local.domain"
# maas admin maas set-config name=ntp_servers value="<Trusted_LAN_NTP_IP>"
# maas admin maas set-config name=network_discovery value="disabled"
# maas admin maas set-config name=upstream_dns value="<Trusted_LAN_DNS_IP>"
# maas admin maas set-config name=dnssec_validation value="no"
# maas admin maas set-config name=default_osystem value="ubuntu"
```
1. Define and configure IP ranges and subnets:
**MaaS Console**
```bash
# maas admin ipranges create type=dynamic start_ip="10.0.110.51" end_ip="10.0.110.120"
# maas admin ipranges create type=dynamic start_ip="10.0.110.21" end_ip="10.0.110.30"
# maas admin ipranges create type=reserved start_ip="10.0.110.10" end_ip="10.0.110.10" comment="c-plane VIP"
# maas admin ipranges create type=reserved start_ip="10.0.110.200" end_ip="10.0.110.200" comment="kamaji VIP"
# maas admin ipranges create type=reserved start_ip="10.0.110.251" end_ip="10.0.110.254" comment="dpfmgmt"
# maas admin vlan
update 0 untagged dhcp_on=True primary_rack=maas
# maas admin dnsresources create fqdn=kube-vip.dpf.rdg.local.domain ip_addresses=10.0.110.10
# maas admin dnsresources create fqdn=jump.dpf.rdg.local.domain ip_addresses=10.0.110.253
# maas admin dnsresources create fqdn=fw.dpf.rdg.local.domain ip_addresses=10.0.110.254
- Configure static DHCP leases for the worker nodes (replace MAC address as appropriate with your workers MGMT interface MAC):
**MaaS Console**
```bash
# maas admin reserved-ips create ip="10.0.110.21" mac_address="04:32:01:60:0d:da" comment="worker1"
# maas admin reserved-ips create ip="10.0.110.22" mac_address="04:32:01:5f:cb:e0" comment="worker2"
-
Complete MaaS setup:
- Connect to the Jump node GUI and access the MaaS UI at
http://10.0.110.252:5240/MAAS. - On the first page, verify the "Region Name" and "DNS Forwarder," then continue.
- Make sure Ubuntu 24.04 LTS (amd64) and sync the image is selected.
- Import the previously generated SSH key (
id_rsa.pub) for thedepuserinto the MaaS admin user profile and finalize the setup.
- Connect to the Jump node GUI and access the MaaS UI at
-
Update the DNS nameserver IP address in both Jump and MaaS VM Netplan files from
10.0.110.254to10.0.110.252and reapply the configuration.
K8s Master VMs
Suggested specifications:
- vCPU: 8
- RAM: 16GB
- Storage: 100GB
- Network interface: Bridge device, connected to
mgmt-br
-
Before provisioning the Kubernetes (K8s) Master VMs with MaaS, create the required virtual disks with empty storage. Use the following one-liner to create three 100 GB QCOW2 virtual disks:
Hypervisor Console
$ for i in $(seq 1 3); do qemu-img create -f qcow2 /var/lib/libvirt/images/master$i.qcow2 100G; doneThis command generates the following disks in the
/var/lib/libvirt/images/directory:master1.qcow2master2.qcow2master3.qcow2
-
Configure VMs in virt-manager:
- Open virt-manager and create three virtual machines:
- Assign the corresponding virtual disk (
master1.qcow2,master2.qcow2, ormaster3.qcow2) to each VM. - Configure each VM with the suggested specifications (vCPU, RAM, storage, and network interface).
- Assign the corresponding virtual disk (
- During the VM setup, ensure the NIC is selected under the Boot Options tab. This ensures the VMs can PXE boot for MaaS provisioning.
- Once the configuration is complete, shut down all the VMs.
- Open virt-manager and create three virtual machines:
-
After the VMs are created and configured, proceed to provision them via the MaaS interface. MaaS will handle the OS installation and further setup as part of the deployment process.
Provision Master VMs and Worker Nodes Using MaaS
Master VMs
Install virsh and Set Up SSH Access
-
SSH to the MaaS VM from the Jump node:
MaaS Console
depuser@jump:~$ ssh maas depuser@maas:~$ sudo -i -
Install the
virshclient to communicate with the hypervisor:MaaS Console
# apt install -y libvirt-clients -
Generate an SSH key for the
rootuser and copy it to the hypervisor user in thelibvirtdgroup:MaaS Console
# ssh-keygen -t rsa # ssh-copy-id ubuntu@<hypervisor_MGMT_IP> -
Verify SSH access and
virshcommunication with the hypervisor:MaaS Console
# virsh -c qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system list --allExpected output:
MaaS Console
Id Name State ------------------------------ 1 fw running 2 jump running 3 maas running - master1 shut off - master2 shut off - master3 shut off -
Copy the SSH key to the required MaaS directory (for snap-based installations):
MaaS Console
# mkdir -p /var/snap/maas/current/root/.ssh # cp .ssh/id_rsa* /var/snap/maas/current/root/.ssh/
Get MAC Addresses of the Master VMs
Retrieve the MAC addresses of the Master VMs:
MaaS Console
# for i in $(seq 1 3); do virsh -c qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system dumpxml master$i | grep 'mac address'; done
Example output:
MaaS Console
<mac address='52:54:00:a9:9c:ef'/>
<mac address='52:54:00:19:6b:4d'/>
<mac address='52:54:00:68:39:7f'/>
Add Master VMs to MaaS
-
Add the Master VMs to MaaS:
Once added, MaaS will automatically start the newly added VMs commissioning (discovery and introspection).
MaaS Console
# maas admin machines create hostname=master1 architecture=amd64/generic mac_addresses='52:54:00:a9:9c:ef' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master1 skip_bmc_config=1 testing_scripts=none Success. Machine-readable output follows: { "description": "", "status_name": "Commissioning", ... "status": 1, ... "system_id": "c3seyq", ... "fqdn": "master1.dpf.rdg.local.domain", "power_type": "virsh", ... "status_message": "Commissioning", "resource_uri": "/MAAS/api/2.0/machines/c3seyq/" } # maas admin machines create hostname=master2 architecture=amd64/generic mac_addresses='52:54:00:19:6b:4d' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master2 skip_bmc_config=1 testing_scripts=none # maas admin machines create hostname=master3 architecture=amd64/generic mac_addresses='52:54:00:68:39:7f' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master3 skip_bmc_config=1 testing_scripts=none -
Repeat the command for
master2andmaster3with their respective MAC addresses. -
Verify commissioning by waiting for the status to change to "Ready" in MaaS.
After commissioning, the next phase is deployment (OS provisioning).
Configure Master VMs Network
To ensure persistence across reboots, assign a static IP address to the management interface of the master nodes.
For each Master VM:
- Navigate to Network and click "actions" near the management interface (a small arrowhead pointing down). Then select "Edit Physical".
- Configure as follows:
- Subnet: 10.0.110.0/24
- Configure as follows:
- IP Mode: Static Assign
- Address: Assign
10.0.110.1formaster1,10.0.110.2formaster2, and10.0.110.3formaster3.
- Save the interface settings for each VM.
Deploy Master VMs Using Cloud-Init
-
Use the following cloud-init script to configure the necessary software and ensure persistency:
Master nodes cloud-init
#cloud-config system_info: default_user: name: depuser passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/" lock_passwd: false groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video] sudo: ["ALL=(ALL) NOPASSWD:ALL"] shell: /bin/bash ssh_pwauth: True package_upgrade: true runcmd: - apt-get update - apt-get -y install nfs-common -
Deploy the master VMs:
- Select all three Master VMs → Actions → Deploy.
- Toggle Cloud-init user-data and paste the cloud-init script.
- Start the deployment and wait until the status to changes to "Ubuntu 24.04 LTS".
Verify Deployment
-
SSH into the Master VMs from the Jump node:
Jump Node Console
depuser@jump:~$ ssh master1 depuser@master1:~$ -
Run
sudowithout a password:Master1 Console
depuser@master1:~$ sudo -i root@master1:~# -
Verify the installed packages:
Master1 Console
root@master1:~# apt list --installed | egrep 'nfs-common' nfs-common/noble,now 1:2.6.4-3ubuntu5 amd64 [installed]
Finalize Setup
Reboot the Master VMs to complete the provisioning.
Master1 Console
root@master1:~# reboot
Worker Nodes
Create Worker Machines in MaaS
-
Add the worker nodes to MaaS using
ipmias the power type. Replace placeholders with your specific IPMI credentials and IP addresses:MAAS Console
# maas admin machines create hostname=worker1 architecture=amd64 power_type=ipmi power_parameters_power_driver=LAN_2_0 power_parameters_power_user=<IPMI_username_worker1> power_parameters_power_pass=<IPMI_password_worker1> power_parameters_power_address=<IPMI_address_worker1>Output example:
MaaS Console
... Success. Machine-readable output follows: { "description": "", "status_name": "Commissioning", ... "status": 1, ... "system_id": "pbskd3", ... "fqdn": "worker1.dpf.rdg.local.domain", ... "power_type": "ipmi", ... "resource_uri": "/MAAS/api/2.0/machines/pbskd3/" } -
Repeat the command for
worker2with its respective credentials:MAAS Console
# maas admin machines create hostname=worker2 architecture=amd64 power_type=ipmi power_parameters_power_driver=LAN_2_0 power_parameters_power_user=<IPMI_username_worker2> power_parameters_power_pass=<IPMI_password_worker2> power_parameters_power_address=<IPMI_address_worker2>
Once added, MaaS automatically starts commissioning the worker nodes (discovery and introspection).
Warning: Please ensure the worker nodes are properly configured to allow PXE booting on their management interface. Specific BIOS settings might be required.
Create a Tag for Kernel Parameters
Create an entity called "Tag" to configure kernel parameters for the worker nodes.
-
In the MaaS UI sidebar, go to Organization → Tags → Create New Tag and define:
- "Tag name":
compute_performance - "Kernel options":
- "Tag name":
-
Substitute the values for
isolcpus,nohz_full, andrcu_nocbswith the CPU cores from the NUMA node which the BlueField-3 is connected to:Warning: If you are not sure in which NUMA node BlueField is connected to, you can later perform this step after the worker node is deployed (although redeployment would be necessary).
MAAS Console
intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111 -
Apply the tag:
- Go to Machines → Select a worker node → Configuration → Edit Tag → Select
compute_performance→ Save. - Repeat for the other worker node.
- Go to Machines → Select a worker node → Configuration → Edit Tag → Select
Adjust Network Settings
For each worker node, configure the network interfaces:
- Management Adapter:
- Go to Network → Select the host management adapter (e.g.,
ens15f0) → Create Bridge - Name:
br-dpu - Bridge Type: Standard
- Subnet:
10.0.110.0/24 - IP Mode: Dynamic
- Save the interface
- Go to Network → Select the host management adapter (e.g.,
Repeat the previous steps for the second worker node.
Deploy Worker Nodes Using Cloud-Init
-
Use the following cloud-init script for deployment:
Worker node cloud-init
#cloud-config system_info: default_user: name: depuser passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/" lock_passwd: false groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video] sudo: ["ALL=(ALL) NOPASSWD:ALL"] shell: /bin/bash ssh_pwauth: True package_upgrade: true runcmd: - apt-get update - apt-get -y install nfs-common -
Deploy the worker nodes by selecting the worker nodes in MaaS → Actions → Deploy → Customize options → Enable Cloud-init user-data → Paste the cloud-init script → Deploy.
Verify Deployment
After the deployment is complete, verify that the worker nodes have been deployed successfully using the following commands:
-
SSH without password from the jump node:
Jump Node Console
depuser@jump:~$ ssh worker1 depuser@worker1:~$ -
Run
sudowithout a password:Worker1 Console
depuser@worker1:~$ sudo -i root@worker1:~# -
Validate that the
nfs-commonpackage was installed:Worker1 Console
root@worker1:~# apt list --installed | grep 'nfs-common' nfs-common/noble,now 1:2.6.4-3ubuntu5 amd64 [installed] -
Validate that
/proc/cmdlineis configured with the correct parameters and that IOMMU is indeed inpassthroughmode:Worker1 Console
root@worker1:~# cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-6.8.0-90-generic root=UUID=5b74560e-130e-42db-a939-58a8d3003cbd ro intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111 root@worker1:~# dmesg | grep 'type: Passthrough' [ 5.068360] iommu: Default domain type: Passthrough (set via kernel command line)
Finalize Deployment
Reboot the worker nodes:
Jump Node Console
root@worker1:~# reboot
The infrastructure is now ready for the K8s deployment.
K8s Cluster Deployment and Configuration
Kubespray Deployment and Configuration
In this solution, the Kubernetes (K8s) cluster is deployed using a modified Kubespray (based on release v2.28.1) from the Jump Node, utilizing a non-root depuser account. These Kubespray modifications align with the DPF prerequisites as described in the User Manual and facilitate both cluster deployment and scaling.
Our modified Kubespray installs Flannel CNI as the primary Kubernetes network plugin.
-
Download the modified Kubespray archive: modified_kubespray_v2.28.1.tar.gz.
-
Extract the contents and navigate to the extracted directory:
Jump Node Console
$ tar -xzf /home/depuser/modified_kubespray_v2.28.1.tar.gz $ cd kubespray/ depuser@jump:~/kubespray$ -
Set the K8s API VIP address and DNS record. Replace the values with your own IP address and DNS record if they differ:
Jump Node Console
depuser@jump:~/kubespray$ sed -i '/# kube_vip_address:/s/.*/kube_vip_address: 10.0.110.10/' inventory/mycluster/group_vars/k8s_cluster/addons.yml depuser@jump:~/kubespray$ sed -i '/apiserver_loadbalancer_domain_name:/s/.*/apiserver_loadbalancer_domain_name: "kube-vip.dpf.rdg.local.domain"/' roles/kubespray_defaults/defaults/main/main.yml -
Install the necessary dependencies and set up the Python virtual environment:
Jump Node Console
depuser@jump:~/kubespray$ sudo apt -y install python3-pip jq python3.12-venv depuser@jump:~/kubespray$ python3 -m venv .venv depuser@jump:~/kubespray$ source .venv/bin/activate (.venv) depuser@jump:~/kubespray$ python3 -m pip install --upgrade pip (.venv) depuser@jump:~/kubespray$ pip install -U -r requirements.txt -
Review and edit the
inventory/mycluster/hosts.yamlfile to define the cluster nodes. The following is the configuration for this deployment:Warning
- All of the nodes are already labeled and annotated by Kubespray as required by DPF prerequisites.
- The worker nodes include additional kubelet configuration which will be applied during their deployment to achieve best performance, allowing:
- Container in Guaranteed pods, requesting an integer number of CPUs, will have dedicated CPU cores on the node.
- The NIC in our example is wired to NUMA node 1. To achieve maximum performance, we would need to prevent the pods from getting cores from NUMA 0, so we reserve these cores for the system using the
reservedSystemCPUsoption.
- The workers under the
kube_nodegroup are marked with#to only deploy the cluster with control plane nodes at the beginning (worker nodes will be added later on after the various components that are necessary for the DPF system are installed).
inventory/mycluster/hosts.yaml
all: hosts: master1: ansible_host: 10.0.110.1 ip: 10.0.110.1 access_ip: 10.0.110.1 master2: ansible_host: 10.0.110.2 ip: 10.0.110.2 access_ip: 10.0.110.2 master3: ansible_host: 10.0.110.3 ip: 10.0.110.3 access_ip: 10.0.110.3 worker1: ansible_host: 10.0.110.21 ip: 10.0.110.21 access_ip: 10.0.110.21 node_labels: node-role.kubernetes.io/worker: "" kubelet_cpu_manager_policy: static kubelet_reservedSystemCPUs: 0-27,56-83 worker2: ansible_host: 10.0.110.22 ip: 10.0.110.22 access_ip: 10.0.110.22 node_labels: node-role.kubernetes.io/worker: "" kubelet_cpu_manager_policy: static kubelet_reservedSystemCPUs: 0-27,56-83 children: kube_control_plane: hosts: master1: master2: master3: kube_node: hosts: # worker1: # worker2: etcd: hosts: master1: master2: master3: k8s_cluster: children: kube_control_plane: kube_node:
Deploying Cluster Using Kubespray Ansible Playbook
-
Run the following command from the Jump Node to initiate the deployment process:
Warning Ensure you are in the Python virtual environment (
.venv) when running the command.Jump Node Console
(.venv) depuser@jump:~/kubespray$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml -
It takes a while for this deployment to complete. Make sure there are no errors. Successful result example:
Success It is recommended to keep the shell where Kubespray has been running open, as it will be useful later for scaling out the cluster and adding worker nodes.
K8s Deployment Verification
To simplify K8s cluster management from the Jump Host, set up kubectl with bash auto-completion.
-
Copy
kubectland the kubeconfig file frommaster1to the Jump Host:Jump Node Console
## Connect to master1 depuser@jump:~$ ssh master1 depuser@master1:~$ cp /usr/local/bin/kubectl /tmp/ depuser@master1:~$ sudo cp /root/.kube/config /tmp/kube-config depuser@master1:~$ sudo chmod 644 /tmp/kube-config -
In another terminal tab, copy the files to the Jump Host:
Jump Node Console
depuser@jump:~$ scp master1:/tmp/kubectl /tmp/ depuser@jump:~$ sudo chown root:root /tmp/kubectl depuser@jump:~$ sudo mv /tmp/kubectl /usr/local/bin/ depuser@jump:~$ mkdir -p ~/.kube depuser@jump:~$ scp master1:/tmp/kube-config ~/.kube/config depuser@jump:~$ chmod 600 ~/.kube/config -
Enable bash auto-completion for
kubectl:-
Verify if bash-completion is installed:
Jump Node
-
Console
depuser@jump:~$ type _init_completion
如果已安装，输出将包含：
Jump Node Console
_init_completion is a function
-
如果未安装，请安装：
Jump Node Console
depuser@jump:~$ sudo apt install -y bash-completion -
设置
kubectl补全脚本：Jump Node Console
depuser@jump:~$ kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null depuser@jump:~$ bash -
检查集群中节点的状态：
Jump Node Console
depuser@jump:~$ kubectl get nodes预期输出：
Jump Node Console
NAME STATUS ROLES AGE VERSION master1 Ready control-plane 42m v1.31.12 master2 Ready control-plane 41m v1.31.12 master3 Ready control-plane 41m v1.31.12 -
检查所有命名空间中的Pod：
Jump Node Console
depuser@jump:~$ kubectl get pods -A预期输出：
Jump Node Console
NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-776bb9db5d-cr56m 1/1 Running 0 12m kube-system coredns-776bb9db5d-dnhct 1/1 Running 0 12m kube-system dns-autoscaler-6ffb84bd6-5kvfc 1/1 Running 0 12m kube-system kube-apiserver-master1 1/1 Running 0 14m kube-system kube-apiserver-master2 1/1 Running 0 14m kube-system kube-apiserver-master3 1/1 Running 0 13m kube-system kube-controller-manager-master1 1/1 Running 1 14m kube-system kube-controller-manager-master2 1/1 Running 1 14m kube-system kube-controller-manager-master3 1/1 Running 1 13m kube-system kube-flannel-fm7fr 1/1 Running 0 13m kube-system kube-flannel-gtv6l 1/1 Running 0 13m kube-system kube-flannel-nqvxs 1/1 Running 0 13m kube-system kube-proxy-dspz6 1/1 Running 0 14m kube-system kube-proxy-tntld 1/1 Running 0 13m kube-system kube-proxy-ttfct 1/1 Running 0 14m kube-system kube-scheduler-master1 1/1 Running 1 14m kube-system kube-scheduler-master2 1/1 Running 1 13m kube-system kube-scheduler-master3 1/1 Running 1 13m kube-system kube-vip-master1 1/1 Running 0 14m kube-system kube-vip-master2 1/1 Running 0 13m kube-system kube-vip-master3 1/1 Running 0 13m
DPF Installation
Software Prerequisites and Required Variables
首先安装剩余的软件先决条件。
Jump Node Console
## 连接到master1以复制kubespray部署期间安装的helm客户端工具
$ depuser@jump:~$ ssh master1
depuser@master1:~$ cp /usr/local/bin/helm /tmp/
## 在另一个标签页中
depuser@jump:~$ scp master1:/tmp/helm /tmp/
depuser@jump:~$ sudo chown root:root /tmp/helm
depuser@jump:~$ sudo mv /tmp/helm /usr/local/bin/
## 验证envsubst工具是否已安装
depuser@jump:~$ which envsubst
/usr/bin/envsubst
继续克隆doca-platform Git仓库（确保使用标签v25.10.0）：
Jump Node Console
$ git clone https://github.com/NVIDIA/doca-platform.git
$ cd doca-platform
$ git checkout v25.10.0
切换到包含hbn-only readme.md的目录，所有命令都将从此位置运行：
Jump Node Console
$ cd docs/public/user-guides/host-trusted/use-cases/hbn
使用以下文件定义安装所需的变量：
警告：将以下文件中的变量值替换为适合您设置的值。特别注意
DPU_P0和DPUCLUSTER_INTERFACE。
manifests/00-env-vars/envvars.env
## DPU集群负载均衡器使用的虚拟IP。必须是管理子网中的保留IP，且不由DHCP分配。
export DPUCLUSTER_VIP=10.0.110.200
## DPU_P0是DPU的第一个端口名称。此名称在所有工作节点上必须相同。
export DPU_P0=ens5f0np0
## DPUCluster负载均衡器监听的接口。应为控制平面节点的管理接口。
export DPUCLUSTER_INTERFACE=ens160
## 用作BFB存储的NFS服务器的IP地址。
export NFS_SERVER_IP=10.0.110.253
## NVIDIA Helm图表仓库的仓库URL。
## 通常是NVIDIA Helm NGC仓库。出于开发目的，可以设置为其他仓库。
export HELM_REGISTRY_REPO_URL=https://helm.ngc.nvidia.com/nvidia/doca
## HBN容器镜像的仓库URL。
## 通常是NVIDIA NGC仓库。出于开发目的，可以设置为其他仓库。
export HBN_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_hbn
## DPF REGISTRY是DPF Operator Chart所在的Helm仓库URL。
## 通常是NVIDIA Helm NGC仓库。出于开发目的，可以设置为其他仓库。
export REGISTRY=https://helm.ngc.nvidia.com/nvidia/doca
## DPF TAG是本指南中部署的DPF组件的版本。
export TAG=v25.10.0
## 在`bfb.yaml`中使用并由DPUSet链接的BFB的URL。
export BFB_URL="https://content.mellanox.com/BlueField/BFBs/Ubuntu24.04/bf-bundle-3.2.1-34_25.11_ubuntu-24.04_64k_prod.bfb"
导出安装所需的环境变量：
Jump Node Console
$ source manifests/00-env-vars/envvars.env
DPF Operator Installation
Create Storage Required by the DPF Operator
以下YAML文件定义了DPF Operator所需的存储（用于BFB镜像）。
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: bfb-pv
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
nfs:
path: /mnt/dpf_share/bfb
server: $NFS_SERVER_IP
persistentVolumeReclaimPolicy: Delete
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: bfb-pvc
namespace: dpf-operator-system
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
volumeMode: Filesystem
storageClassName: ""
运行以下命令创建DPF Operator的命名空间。使用 envsubst 替换环境变量并应用YAML文件：
Jump Node Console
$ kubectl create namespace dpf-operator-system
$ cat manifests/01-dpf-operator-installation/*.yaml | envsubst | kubectl apply -f -
Additional Dependencies
- DPF Operator需要在Kubernetes环境中正常运行多个先决组件。从DPF v25.7开始，所有Helm依赖项已从DPF chart中移除。这意味着所有
dependencies must be installed manually before installing the DPF chart itself. The following commands describe an opiniated approach to install those dependencies (for more information, check: Helm Prerequisites - NVIDIA Docs).
-
Install
helmfilebinary:Jump Node Console
$ wget https://github.com/helmfile/helmfile/releases/download/v1.1.2/helmfile_1.1.2_linux_amd64.tar.gz $ tar -xvf helmfile_1.1.2_linux_amd64.tar.gz $ sudo mv ./helmfile /usr/local/bin/ -
Change directory to doca-platform:
Use another shell from the one where you run all the other installation commands for DPF.
Jump Node Console
$ cd doca-platform/ -
Install Helm dependencies using the following command:
Jump Node Console
$ make HELMFILE_FILE=deploy/helmfiles/prereqs.yaml test-deploy-helmfile
DPF Operator Deployment
Run the following commands to install the DPF Operator:
Jump Node Console
$ helm repo add --force-update dpf-repository ${REGISTRY}
$ helm repo update
$ helm upgrade --install -n dpf-operator-system dpf-operator dpf-repository/dpf-operator --version=$TAG
Verify the DPF Operator installation by ensuring the deployment is available and all pods are ready:
You may need to run the following verification commands multiple times to confirm that all conditions are met.
Jump Node Console
$ kubectl rollout status deployment --namespace dpf-operator-system dpf-operator-controller-manager
deployment "dpf-operator-controller-manager" successfully rolled out
$ kubectl wait --for=condition=ready --namespace dpf-operator-system pods --all
pod/argo-cd-argocd-application-controller-0 condition met
pod/argo-cd-argocd-redis-6c6b84f6fb-dhfqx condition met
pod/argo-cd-argocd-repo-server-65cfb96746-fjqhn condition met
pod/argo-cd-argocd-server-5bbdb4b6b9-hbbsw condition met
pod/dpf-operator-controller-manager-5dd7555c6d-gpbss condition met
pod/kamaji-95587fbc7-xzfmn condition met
pod/kamaji-etcd-0 condition met
pod/kamaji-etcd-1 condition met
pod/kamaji-etcd-2 condition met
pod/maintenance-operator-74bd5774b7-v4kct condition met
pod/node-feature-discovery-gc-6b48f49cc4-l4p2w condition met
pod/node-feature-discovery-master-747d789485-x5hcb condition met
DPF System Installation
This section involves creating the DPF system components and some basic infrastructure required for a functioning DPF-enabled cluster.
The following YAML files define the DPFOperatorConfig to install the DPF System components and the DPUCluster to serve as Kubernetes control plane for DPU nodes.
Note that to achieve high performance results you need to adjust the
operatorconfig.yamlto support MTU 9000.
---
apiVersion: operator.dpu.nvidia.com/v1alpha1
kind: DPFOperatorConfig
metadata:
name: dpfoperatorconfig
namespace: dpf-operator-system
spec:
provisioningController:
bfbPVCName: "bfb-pvc"
dmsTimeout: 900
kamajiClusterManager:
disable: false
networking:
highSpeedMTU: 9000
---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUCluster
metadata:
name: dpu-cplane-tenant1
namespace: dpu-cplane-tenant1
spec:
type: kamaji
maxNodes: 10
clusterEndpoint:
# deploy keepalived instances on the nodes that match the given nodeSelector.
keepalived:
# interface on which keepalived will listen. Should be the oob interface of the control plane node.
interface: $DPUCLUSTER_INTERFACE
# Virtual IP reserved for the DPU Cluster load balancer. Must not be allocatable by DHCP.
vip: $DPUCLUSTER_VIP
# virtualRouterID must be in range [1,255], make sure the given virtualRouterID does not duplicate with any existing keepalived process running on the host
virtualRouterID: 126
nodeSelector:
node-role.kubernetes.io/control-plane: ""
Create a namespace for the Kubernetes control plane of the DPU nodes:
Jump Node Console
$ kubectl create ns dpu-cplane-tenant1
Apply the previous YAML files:
Jump Node Console
$ cat manifests/02-dpf-system-installation/*.yaml | envsubst | kubectl apply -f -
Verify the DPF system by ensuring that the provisioning and DPUService controller manager deployments are available, all other deployments in the DPF Operator system are available, and the DPUCluster is ready for nodes to join.
Jump Node Console
$ kubectl rollout status deployment --namespace dpf-operator-system dpf-provisioning-controller-manager dpuservice-controller-manager
deployment "dpf-provisioning-controller-manager" successfully rolled out
deployment "dpuservice-controller-manager" successfully rolled out
$ kubectl rollout status deployment --namespace dpf-operator-system
deployment "argo-cd-argocd-applicationset-controller" successfully rolled out
deployment "argo-cd-argocd-redis" successfully rolled out
deployment "argo-cd-argocd-repo-server" successfully rolled out
deployment "argo-cd-argocd-server" successfully rolled out
deployment "dpf-operator-controller-manager" successfully rolled out
deployment "dpf-provisioning-controller-manager" successfully rolled out
deployment "dpuservice-controller-manager" successfully rolled out
deployment "kamaji" successfully rolled out
deployment "kamaji-cm-controller-manager" successfully rolled out
deployment "maintenance-operator" successfully rolled out
deployment "node-feature-discovery-gc" successfully rolled out
deployment "node-feature-discovery-master" successfully rolled out
$ kubectl wait --for=condition=ready --namespace dpu-cplane-tenant1 dpucluster --all
dpucluster.provisioning.dpu.nvidia.com/dpu-cplane-tenant1 condition met
Install components to enable Accelerated Interfaces
The HBN service can accelerate pod traffic by attaching a VF to each pod, which offloads flows to the DPU. This section details the components needed to connect pods to HBN.
Install Multus and SRIOV Network Operator using NVIDIA Network Operator.
Start by adding the NVIDIA Network Operator Helm repository:
Jump Node Console
$ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia --force-update
The following network-operator.yaml values file are applied:
nfd:
enabled: false
deployNodeFeatureRules: false
sriovNetworkOperator:
enabled: true
sriov-network-operator:
operator:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
- matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: Exists crds: enabled: true sriovOperatorConfig: deploy: true configDaemonNodeSelector: null operator: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/master operator: Exists - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: Exists
Deploy the operator:
Jump Node Console
$ helm upgrade --no-hooks --install --create-namespace --namespace nvidia-network-operator network-operator nvidia/network-operator --version 25.7.0 -f ./manifests/03-enable-accelerated-interfaces/helm-values/network-operator.yml
Ensure that all the pods in the nvidia-network-operator namespace are ready:
Jump Node Console
$ kubectl wait --for=condition=ready --namespace nvidia-network-operator pods --all
pod/network-operator-66b5cdbc79-8bv9c condition met
pod/network-operator-sriov-network-operator-6b87b5cf96-j6xx5 condition met
The following NicClusterPolicy and SriovNetworkNodePolicy configuration files should be applied.
---
apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
name: nic-cluster-policy
spec:
secondaryNetwork:
multus:
image: multus-cni
imagePullSecrets: []
repository: ghcr.io/k8snetworkplumbingwg
version: v3.9.3
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
name: bf3-p0-vfs
namespace: nvidia-network-operator
spec:
nicSelector:
deviceID: "a2dc"
vendor: "15b3"
pfNames:
- $DPU_P0#2-45
nodeSelector:
node-role.kubernetes.io/worker: ""
numVfs: 46
resourceName: bf3-p0-vfs
isRdma: true
externallyManaged: true
deviceType: netdevice
linkType: eth
Apply the following configuration files:
Jump Node Console
$ cat manifests/03-enable-accelerated-interfaces/*.yaml | envsubst | kubectl apply -f -
Verify the DPF system by ensuring that all pods in nvidia-network-operator namespace are ready, and that the following DaemonSets have been successfully rolled out:
Jump Node Console
$ kubectl wait --for=condition=ready --namespace nvidia-network-operator pods --all
pod/network-operator-7bc7b45d67-jftqg condition met
pod/network-operator-sriov-network-operator-86c9cd4899-5blhf condition met
$ kubectl rollout status daemonset --namespace nvidia-network-operator kube-multus-ds sriov-network-config-daemon sriov-device-plugin
daemon set "kube-multus-ds" successfully rolled out
daemon set "sriov-network-config-daemon" successfully rolled out
daemon set "sriov-device-plugin" successfully rolled out
DPU Deployment Installation
Before deploying the objects under manifests/04-dpudeployment-installation/ directory, a few adjustments are needed to achieve better performance results.
Edit the DPUFlavor using the following YAML:
Warning: The parameter
NUM_VF_MSIXis set to 48 in the provided example, which is suitable for the servers used in this RDG. Set this value to match the physical number of cores in the NUMA node where the NIC is located.
---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUFlavor
metadata:
name: hbn-$TAG
namespace: dpf-operator-system
spec:
bfcfgParameters:
- UPDATE_ATF_UEFI=yes
- UPDATE_DPU_OS=yes
- WITH_NIC_FW_UPDATE=yes
configFiles:
- operation: override
path: /etc/mellanox/mlnx-bf.conf
permissions: "0644"
raw: |
ALLOW_SHARED_RQ="no"
IPSEC_FULL_OFFLOAD="no"
ENABLE_ESWITCH_MULTIPORT="yes"
- operation: override
path: /etc/mellanox/mlnx-ovs.conf
permissions: "0644"
raw: |
CREATE_OVS_BRIDGES="no"
OVS_DOCA="yes"
- operation: override
path: /etc/mellanox/mlnx-sf.conf
permissions: "0644"
raw: ""
grub:
kernelParameters:
- console=hvc0
- console=ttyAMA0
- earlycon=pl011,0x13010000
- fixrttc
- net.ifnames=0
- biosdevname=0
- iommu.passthrough=1
- cgroup_no_v1=net_prio,net_cls
- hugepagesz=2048kB
- hugepages=8072
nvconfig:
- device: '*'
parameters:
- PF_BAR2_ENABLE=0
- PER_PF_NUM_SF=1
- PF_TOTAL_SF=20
- PF_SF_BAR_SIZE=10
- NUM_PF_MSIX_VALID=0
- PF_NUM_PF_MSIX_VALID=1
- PF_NUM_PF_MSIX=228
- INTERNAL_CPU_MODEL=1
- INTERNAL_CPU_OFFLOAD_ENGINE=0
- SRIOV_EN=1
- NUM_OF_VFS=46
- LAG_RESOURCE_ALLOCATION=1
- LINK_TYPE_P1=ETH
- LINK_TYPE_P2=ETH
- NUM_VF_MSIX=48
ovs:
rawConfigScript: |
_ovs-vsctl() {
ovs-vsctl --no-wait --timeout 15 "$@"
}
_ovs-vsctl set Open_vSwitch . other_config:doca-init=true
_ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000
_ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
_ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
_ovs-vsctl set Open_vSwitch . other_config:max-idle=20000
_ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000
_ovs-vsctl --if-exists del-br ovsbr1
_ovs-vsctl --if-exists del-br ovsbr2
_ovs-vsctl --may-exist add-br br-sfc
_ovs-vsctl set bridge br-sfc datapath_type=netdev
_ovs-vsctl set bridge br-sfc fail_mode=secure
_ovs-vsctl --may-exist add-port br-sfc p0
_ovs-vsctl set Interface p0 type=dpdk
_ovs-vsctl set Interface p0 mtu_request=9216
_ovs-vsctl set Port p0 external_ids:dpf-type=physical
_ovs-vsctl --may-exist add-port br-sfc p1
_ovs-vsctl set Interface p1 type=dpdk
_ovs-vsctl set Interface p1 mtu_request=9216
_ovs-vsctl set Port p1 external_ids:dpf-type=physical
_ovs-vsctl --may-exist add-br br-hbn
_ovs-vsctl set bridge br-hbn datapath_type=netdev
_ovs-vsctl set bridge br-hbn fail_mode=secure
The rest of the configuration files remain the same.
As explained in the introduction, they create service chains to connect two virtual functions (VF10 on PF0 and VF10 on PF1) to the outer fabric through HBN, providing EVPN VXLAN overlay, VNI based isolation, and ECMP redundancy through both DPU uplinks (p0 and p1).
These are the configuration files:
-
BFB to download BlueField BFB to a shared volume.
--- apiVersion: provisioning.dpu.nvidia.com/v1alpha1 kind: BFB metadata: name: bf-bundle-$TAG namespace: dpf-operator-system spec: url: $BFB_URL -
HBN DPUServiceConfig and DPUServiceTemplate to deploy HBN workloads to the DPU.
--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceConfiguration metadata: name: doca-hbn namespace: dpf-operator-system spec: deploymentServiceName: "doca-hbn" serviceConfiguration: serviceDaemonSet: annotations: k8s.v1.cni.cncf.io/networks: |- [ {"name": "iprequest", "interface": "ip_lo", "cni-args": {"poolNames": ["loopback"], "poolType": "cidrpool"}}, {"name": "iprequest", "interface": "ip_pf0vf10", "cni-args": {"poolNames": ["pool1"], "poolType": "cidrpool", "allocateDefaultGateway": true}}, {"name": "iprequest", "interface": "ip_pf1vf10", "cni-args": {"poolNames": ["pool2"], "poolType": "cidrpool", "allocateDefaultGateway": true}} ] helmChart: values: configuration: perDPUValuesYAML: | - hostnamePattern: "*" values: bgp_peer_group: hbn vrf1: RED vrf2: BLUE l3vni1: 100001 l3vni2: 100002 startupYAMLJ2: | - header: model: bluefield nvue-api-version: nvue_v1 rev-id: 1.0 version: HBN 2.4.0 - set: evpn: enable: on route-advertise: {} interface: lo: ip: address: {{ ipaddresses.ip_lo.ip }}/32: {} type: loopback p0_if,p1_if,pf0vf10_if,pf1vf10_if: type: swp link: mtu: 9000 pf0vf10_if: ip: address: {{ ipaddresses.ip_pf0vf10.cidr }}: {} vrf: {{ config.vrf1 }} pf1vf10_if: ip: address: {{ ipaddresses.ip_pf1vf10.cidr }}: {} vrf: {{ config.vrf2 }} nve: vxlan: arp-nd-suppress: on enable: on source: address:
{{ ipaddresses.ip_lo.ip }} router: bgp: enable: on graceful-restart: mode: full vrf: default: router: bgp: address-family: ipv4-unicast: enable: on redistribute: connected: enable: on multipaths: ebgp: 16 l2vpn-evpn: enable: on autonomous-system: {{ ( ipaddresses.ip_lo.ip.split(".")[3] | int ) + 65101 }} enable: on neighbor: p0_if: peer-group: {{ config.bgp_peer_group }} type: unnumbered address-family: l2vpn-evpn: enable: on add-path-tx: off p1_if: peer-group: {{ config.bgp_peer_group }} type: unnumbered address-family: l2vpn-evpn: enable: on add-path-tx: off path-selection: multipath: aspath-ignore: on peer-group: {{ config.bgp_peer_group }}: address-family: ipv4-unicast: enable: on l2vpn-evpn: enable: on remote-as: external router-id: {{ ipaddresses.ip_lo.ip }} {{ config.vrf1 }}: evpn: enable: on vni: {{ config.l3vni1 }}: {} router: bgp: address-family: ipv4-unicast: enable: on redistribute: connected: enable: on route-export: to-evpn: enable: on autonomous-system: {{ ( ipaddresses.ip_lo.ip.split(".")[3] | int ) + 65101 }} enable: on {{ config.vrf2 }}: evpn: enable: on vni: {{ config.l3vni2 }}: {} router: bgp: address-family: ipv4-unicast: enable: on redistribute: connected: enable: on route-export: to-evpn: enable: on autonomous-system: {{ ( ipaddresses.ip_lo.ip.split(".")[3] | int ) + 65101 }} enable: on
interfaces:
- name: p0_if network: mybrhbn
- name: p1_if network: mybrhbn
- name: pf0vf10_if network: mybrhbn
- name: pf1vf10_if network: mybrhbn
```yaml
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
name: doca-hbn
namespace: dpf-operator-system
spec:
deploymentServiceName: "doca-hbn"
helmChart:
source:
repoURL: $HELM_REGISTRY_REPO_URL
version: 1.0.5
chart: doca-hbn
values:
image:
repository: $HBN_NGC_IMAGE_URL
tag: 3.2.1-doca3.2.1
resources:
memory: 6Gi
nvidia.com/bf_sf: 4
- Physical Interfaces for physical ports on the DPU.
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: p0
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
uplink: "p0"
spec:
interfaceType: physical
physical:
interfaceName: p0
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: p1
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
uplink: "p1"
spec:
interfaceType: physical
physical:
interfaceName: p1
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: pf0vf10-rep
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
vf: "pf0vf10"
spec:
interfaceType: vf
vf:
parentInterfaceRef: p0
pfID: 0
vfID: 10
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: pf1vf10-rep
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
vf: "pf1vf10"
spec:
interfaceType: vf
vf:
parentInterfaceRef: p1
pfID: 1
vfID: 10
- DPU Service IPAM objects to set up IP Address Management on the DPUCluster.
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
name: pool1
namespace: dpf-operator-system
spec:
ipv4Network:
network: "10.0.121.0/24"
gatewayIndex: 2
prefixSize: 29
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
name: pool2
namespace: dpf-operator-system
spec:
ipv4Network:
network: "10.0.122.0/24"
gatewayIndex: 2
prefixSize: 29
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
name: loopback
namespace: dpf-operator-system
spec:
ipv4Network:
network: "11.0.0.0/24"
prefixSize: 32
Apply all of the YAML files mentioned above using the following command:
Jump Node Console
$ cat manifests/04-dpudeployment-installation/*.yaml | envsubst | kubectl apply -f -
Verify the DPUService installation by ensuring that:
- HBN DPUService is created and reconciled
- DPUServiceIPAMs are reconciled
- DPUServiceInterfaces are reconciled, and
- DPUServiceChains are reconciled.
Notes These verification commands may need to be run multiple times to ensure the conditions are met.
Jump Node Console
## Successful output will require some few minutes of waiting##
$ kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_hbn-only
dpuservice.svc.dpu.nvidia.com/doca-hbn-g5md8 condition met
$ kubectl wait --for=condition=DPUIPAMObjectReconciled --namespace dpf-operator-system dpuserviceipam --all
dpuserviceipam.svc.dpu.nvidia.com/loopback condition met
dpuserviceipam.svc.dpu.nvidia.com/pool1 condition met
dpuserviceipam.svc.dpu.nvidia.com/pool2 condition met
$ kubectl wait --for=condition=ServiceInterfaceSetReconciled --namespace dpf-operator-system dpuserviceinterface --all
dpuserviceinterface.svc.dpu.nvidia.com/hbn-only-doca-hbn-p0-if condition met
dpuserviceinterface.svc.dpu.nvidia.com/hbn-only-doca-hbn-p1-if condition met
dpuserviceinterface.svc.dpu.nvidia.com/hbn-only-doca-hbn-pf0vf10-if condition met
dpuserviceinterface.svc.dpu.nvidia.com/hbn-only-doca-hbn-pf1vf10-if condition met
dpuserviceinterface.svc.dpu.nvidia.com/p0 condition met
dpuserviceinterface.svc.dpu.nvidia.com/p1 condition met
dpuserviceinterface.svc.dpu.nvidia.com/pf0vf10-rep condition met
dpuserviceinterface.svc.dpu.nvidia.com/pf1vf10-rep condition met
$ kubectl wait --for=condition=ServiceChainSetReconciled --namespace dpf-operator-system dpuservicechain --all
dpuservicechain.svc.dpu.nvidia.com/hbn-only condition met
K8s Cluster Scale-out
At this point, workers should be added to the cluster. As they are added, DPU will be provisioned and DPUServices will begin to be spun up.
For the scale-out procedure, return to the shell where Kubespray was previously used to deploy the cluster. Uncomment the workers under the kube_node group in the hosts.yaml file, then add the worker nodes to the cluster:
Ensure you are in the Python virtual environment (
.venv) when running the command.
Jump Node Console
(.venv) depuser@jump:~/kubespray$ cat inventory/mycluster/hosts.yaml
...
kube_node:
hosts:
worker1:
worker2:
...
(.venv) depuser@jump:~/kubespray$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root scale.yml
The scale-out process should not take long. A successful run should produce output similar to the following:
要跟踪DPU配置的进度，运行以下命令检查其当前阶段：
Jump Node Console
$ watch -n10 "kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase'"
Dpu Node Name: worker1
Type: InternalIP
Type: Hostname
Last Transition Time: 2025-12-25T20:29:38Z
Type: Ready
Last Transition Time: 2025-12-25T19:59:18Z
Type: BFBPrepared
Last Transition Time: 2025-12-25T19:58:41Z
Type: BFBReady
Last Transition Time: 2025-12-25T20:25:57Z
Type: DPUClusterReady
Last Transition Time: 2025-12-25T19:58:41Z
Type: Initialized
Last Transition Time: 2025-12-25T19:59:16Z
Type: NodeEffectReady
Last Transition Time: 2025-12-25T20:29:38Z
Type: NodeEffectRemoved
Last Transition Time: 2025-12-25T20:15:31Z
Type: CheckedHostRebootNeed
Last Transition Time: 2025-12-25T19:59:18Z
Type: FWConfigured
Last Transition Time: 2025-12-25T20:25:45Z
Type: HostNetworkReady
Last Transition Time: 2025-12-25T19:59:17Z
Type: InterfaceInitialized
Last Transition Time: 2025-12-25T20:15:29Z
Type: OSInstalled
Last Transition Time: 2025-12-25T20:24:25Z
Type: Rebooted
Phase: Ready
Dpu Node Name: worker2
Type: InternalIP
Type: Hostname
Last Transition Time: 2025-12-25T20:28:37Z
Type: Ready
Last Transition Time: 2025-12-25T19:58:38Z
Type: BFBPrepared
Last Transition Time: 2025-12-25T19:58:11Z
Type: BFBReady
Last Transition Time: 2025-12-25T20:24:31Z
Type: DPUClusterReady
Last Transition Time: 2025-12-25T19:58:11Z
Type: Initialized
Last Transition Time: 2025-12-25T19:58:36Z
Type: NodeEffectReady
Last Transition Time: 2025-12-25T20:28:37Z
Type: NodeEffectRemoved
Last Transition Time: 2025-12-25T20:15:01Z
Type: CheckedHostRebootNeed
Last Transition Time: 2025-12-25T19:58:38Z
Type: FWConfigured
Last Transition Time: 2025-12-25T20:24:12Z
Type: HostNetworkReady
Last Transition Time: 2025-12-25T19:58:36Z
Type: InterfaceInitialized
Last Transition Time: 2025-12-25T20:14:58Z
Type: OSInstalled
Last Transition Time: 2025-12-25T20:22:51Z
Type: Rebooted
Phase: Ready
一旦两个DPU都进入"Ready"阶段，验证DPU已成功配置：
Jump Node Console
$ kubectl wait --for=condition=ready --namespace dpf-operator-system dpu --all
dpu.provisioning.dpu.nvidia.com/worker1-mt2414600gq5 condition met
dpu.provisioning.dpu.nvidia.com/worker2-mt2412600eq8 condition met
确保以下DaemonSet有2个就绪副本：
Jump Node Console
$ kubectl wait ds --for=jsonpath='{.status.numberReady}'=2 --namespace nvidia-network-operator kube-multus-ds sriov-network-config-daemon sriov-device-plugin
daemonset.apps/kube-multus-ds condition met
daemonset.apps/sriov-network-config-daemon condition met
daemonset.apps/sriov-device-plugin condition met
最后，验证所有DPUServices、DPUServiceIPAMs、DPUServiceInterfaces和DPUServiceChains对象现在都处于Ready状态：
Jump Node Console
$ kubectl wait --for=condition=ApplicationsReady --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_hbn-only
dpuservice.svc.dpu.nvidia.com/doca-hbn-g5md8 condition met
$ kubectl wait --for=condition=DPUIPAMObjectReady --namespace dpf-operator-system dpuserviceipam --all
dpuserviceipam.svc.dpu.nvidia.com/loopback condition met
dpuserviceipam.svc.dpu.nvidia.com/pool1 condition met
dpuserviceipam.svc.dpu.nvidia.com/pool2 condition met
$ kubectl wait --for=condition=ServiceInterfaceSetReady --namespace dpf-operator-system dpuserviceinterface --all
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-p0-if-xgrcl condition met
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-p1-if-hpcbz condition met
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-pf0vf10-if-52jnw condition met
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-pf1vf10-if-4djh6 condition met
dpuserviceinterface.svc.dpu.nvidia.com/p0 condition met
dpuserviceinterface.svc.dpu.nvidia.com/p1 condition met
dpuserviceinterface.svc.dpu.nvidia.com/pf0vf10-rep condition met
dpuserviceinterface.svc.dpu.nvidia.com/pf1vf10-rep condition met
$ kubectl wait --for=condition=ServiceChainSetReady --namespace dpf-operator-system dpuservicechain --all
dpuservicechain.svc.dpu.nvidia.com/hbn-only-ljcqz condition met
$ kubectl -n dpf-operator-system exec deployment/dpf-operator-controller-manager -- /dpfctl describe all --show-resources=dpu --show-conditions=dpu
NAME NAMESPACE STATUS REASON SINCE MESSAGE
DPFOperatorConfig/dpfoperatorconfig dpf-operator-system Ready: True Success 1h
└─DPU
└─2 DPU... dpf-operator-system Ready: True DPUReady 1h See worker1-mt2414600gq5, worker2-mt2412600eq8
恭喜，DPF系统已成功安装！
基础设施延迟与带宽验证
通过使用各种测试验证部署，并确保在DPF系统上达到链路速度性能和良好的延迟结果：
- RDMA - 用于延迟测量
- Iperf TCP - 用于带宽测量
每个测试都有详细描述。在每个测试结束时，您将看到所达到的性能。
注意 确保服务器已调优以获得最大性能（本文档未涵盖）。
性能测试
RoCE延迟测试
创建网络附加定义（NAD）和测试部署，使用以下YAML文件创建4个跨两个不同工作节点运行的Pod。
每个Pod将使用hostdev插件将主机虚拟功能编号10（VF10）设备插入其中——一个来自PF0，一个来自PF1。
此部署演示了在每个节点上，Pod被隔离并连接到不同VNI上的不同网络。
我们将展示：
- 我们可以在同一网络上不同工作节点上运行的Pod之间通信并实现加速流量（性能测试）
- 我们无法在不同网络上的Pod之间通信（隔离测试）
以下是用于创建NAD的yaml文件。将<PF0_VF_DEVICE>和<PF1_VF_DEVICE>替换为工作节点上的正确设备名称（例如ens5f0v10和ens5f1v10）：
nad-hostdev.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: hostdev-pf0vf10-worker1
spec:
config: '{
"cniVersion": "0.3.1",
"name": "hostpf0vf10",
"type": "host-device",
"device": "<PF0_VF_DEVICE>",
"ipam": {
"type": "static",
"addresses": [
{
"address": "10.0.121.1/29"
}
],
"routes": [
{
"dst": "10.0.121.8/29",
"gw": "10.0.121.2"
}
]
}
}'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: hostdev-pf1vf10-worker1
spec:
config: '{
"cniVersion": "0.3.1",
"name": "hostpf1vf10",
"type": "host-device",
"device": "<PF1_VF_DEVICE>",
"ipam": {
"type": "static",
"addresses": [
{
"address": "10.0.122.1/29"
}
],
"routes": [
{
"dst": "10.0.122.8/29",
"gw": "10.0.122.2"
}
]
}
}'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: hostdev-pf0vf10-worker2
spec:
config: '{
"cniVersion": "0.3.1",
"name": "hostpf0vf10",
"type": "host-device",
"device": "<PF0_VF_DEVICE>",
"ipam": {
"type": "static",
"addresses": [
{
"address": "10.0.121.9/29"
}
],
"routes": [
{
"dst": "10.0.121.0/29",
"gw": "10.0.121.10"
}
]
}
}'
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: hostdev-pf1vf10-worker2
spec:
config: '{
"cniVersion": "0.3.1",
"name": "hostpf1vf10",
"type": "host-device",
"device": "<PF1_VF_DEVICE>",
"ipam": {
"type": "static",
"addresses": [
{
"address": "10.0.122.9/29"
}
],
"routes": [
{
"dst": "10.0.122.0/29",
"gw": "10.0.122.10"
}
]
}
}'
"10.0.122.0/29",
"gw": "10.0.122.10"
}
]
}
}'
This is the YAML file used to create the deployments for PF0 (RED network). Replace <IMAGE_URL> with a relevant container image URL (The container image must include NVIDIA user space drivers and the perftest and iperf3 packages):
testapp-performance-test-deployment-pf0.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sriov-hostdev-pf0vf10-test-worker1
labels:
app: sriov-hostdev-pf0vf10-test-worker1
spec:
replicas: 1
selector:
matchLabels:
app: sriov-hostdev-pf0vf10-test-worker1
template:
metadata:
labels:
app: sriov-hostdev-pf0vf10-test-worker1
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-pf0vf10-worker1
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: sriov-test-worker
nodeSelector:
feature.node.kubernetes.io/dpu-enabled: "true"
kubernetes.io/hostname: "worker1"
containers:
- name: testapp
securityContext:
privileged: true
capabilities:
add:
- NET_ADMIN
- IPC_LOCK
image: <IMAGE_URL>
command: ["sleep", "infinity"]
ports:
- containerPort: 5000
name: tcp-server
resources:
requests:
cpu: 24
memory: 6Gi
limits:
cpu: 24
memory: 6Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sriov-hostdev-pf0vf10-test-worker2
labels:
app: sriov-hostdev-pf0vf10-test-worker2
spec:
replicas: 1
selector:
matchLabels:
app: sriov-hostdev-pf0vf10-test-worker2
template:
metadata:
labels:
app: sriov-hostdev-pf0vf10-test-worker2
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-pf0vf10-worker2
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: sriov-test-worker
nodeSelector:
feature.node.kubernetes.io/dpu-enabled: "true"
kubernetes.io/hostname: "worker2"
containers:
- name: testapp
securityContext:
privileged: true
capabilities:
add:
- NET_ADMIN
- IPC_LOCK
image: <IMAGE_URL>
command: ["sleep", "infinity"]
ports:
- containerPort: 5000
name: tcp-server
resources:
requests:
cpu: 24
memory: 6Gi
limits:
cpu: 24
memory: 6Gi
Apply the following resources:
Jump Node Console
$ kubectl apply -f nad-hostdev.yaml
$ kubectl apply -f testapp-performance-test-deployment-pf0.yaml
Validate that the deployment is running successfully:
Jump Node Console
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb 1/1 Running 0 3d1h 10.233.69.102 worker1 <none> <none>
sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw 1/1 Running 0 3d1h 10.233.70.177 worker2 <none> <none>
Now that the test deployment is running, perform a latency performance test between two pods.
Connect to one of the pods in the deployment, specifically the pod running on the PF0 network on the first worker node:
Jump Node Console
$ kubectl exec -it sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb -- bash
From within the container, set an MTU value of 9000 and check the IP address on the net1 interface. Also identify the relevant RDMA device:
First Pod Console
root@sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb:/# ip link set net1 mtu 9000
root@sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0@if185: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether 8a:d6:45:6a:b4:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.233.67.39/24 brd 10.233.67.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::88d6:45ff:fe6a:b43d/64 scope link
valid_lft forever preferred_lft forever
59: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
link/ether 0e:a4:2e:4e:1f:9f brd ff:ff:ff:ff:ff:ff
altname enp134s0f0v10
inet 10.0.121.1/29 brd 10.0.121.7 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::ca4:2eff:fe4e:1f9f/64 scope link
valid_lft forever preferred_lft forever
root@testapp-performance-6c69b69d9b-24gpj:/# rdma link | grep net1
link mlx5_12/1 state ACTIVE physical_state LINK_UP netdev net1
Start the ib_read_lat server side:
First Pod Console
root@sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb:/# ib_read_lat -F -n 20000 -d mlx5_12
************************************
* Waiting for client to connect... *
************************************
Using another console window, reconnect to the jump node and connect to a second pod running on the same network (PF0), on the second worker node.
Jump Node Console
$ kubectl exec -it sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw -- bash
From within the container, set an MTU value of 9000 on the net1 interface, then identify the relevant RDMA Device and start the ib_read_lat client using the IP address from the server-side container, finally let's check the latency results:
First Pod Console
root@sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw:/# ip link set net1 mtu 9000
root@sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw:/# rdma link | grep net1
link mlx5_12/1 state ACTIVE physical_state LINK_UP netdev net1
root@testapp-performance-6c69b69d9b-mrg4g:/# ib_read_lat -F -n 20000 -d mlx5_12 10.0.121.1
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx5_12
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : ON
TX depth : 1
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x04b9 PSN 0x4f96f3 OUT 0x10 RKey 0x078507 VAddr 0x006344ee3b0000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:121:09
remote address: LID 0000 QPN 0x04b9 PSN 0x7fc74c OUT 0x10 RKey 0x07b607 VAddr 0x005a042c76c000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:121:01
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 20000 3.99 166.68 4.12 8.26 9.08 44.07 58.11
---------------------------------------------------------------------------------------
iPerf TCP Bandwidth Test
Connect to the first pods:
Jump Node Console
$ kubectl exec -it sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb -- bash
$ kubectl exec -it sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb -- bash
在启动 iperf3 服务器监听器之前，并在另一个标签页中检查 Pod 当前运行的核，以获得良好结果：
Jump Node Console
$ ssh worker1
depuser@worker1:~$ sudo -i
root@worker1:~# crictl ps | grep testapp
68fe4bc7f6854 25a2fe7e1cbce 3 hours ago Running testapp 0 4fa7100bee05e sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb
root@worker2:~# crictl inspect 68fe4bc7f6854 | jq '.status.resources.linux.cpusetCpus'
"28-51"
使用以下脚本在不同端口上启动多个 iperf3 服务器（每个核一个）：
iperf_server.sh
#!/bin/bash
# Cores to bind the iperf3 server processes to
CORES=$1
# Calculate the first_core and last_core to provide the CPU range
first_core=$(echo $CORES | cut -d "-" -f1)
last_core=$(echo $CORES | cut -d "-" -f2)
ports_num=$(($last_core - $first_core + 1))
# Loop over the ports (5201 + i*2) for i in the given CPU range and run iperf3 servers
for i in $(seq 1 $ports_num); do
echo "Running iperf3 server $i"
taskset -c $(($i + $first_core - 1)) iperf3 -s -p $((5201 + i * 2)) > /dev/null 2>&1 &
done
使用之前的 CPU 范围启动脚本（留 1 个核作为缓冲）：
First Pod Console
root@sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb:/# chmod +x iperf_server.sh
root@sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb:/# ./iperf_server.sh 28-51
Running iperf3 server 1
Running iperf3 server 2
...
...
Running iperf3 server 23
Running iperf3 server 24
root@sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb:/# ps -ef | grep iperf3
root 2136 1 0 15:54 pts/2 00:00:00 iperf3 -s -p 5257
root 2137 1 0 15:54 pts/2 00:00:00 iperf3 -s -p 5259
...
...
root 2157 1 0 15:54 pts/2 00:00:00 iperf3 -s -p 5303
root 2158 1 0 15:54 pts/2 00:00:00 iperf3 -s -p 5305
连接到第二个 Pod：
Jump Node Console
$ kubectl exec -it sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw -- bash
按照之前显示的方法识别第二个 Pod 运行的 CPU 核。
使用以下脚本启动多个 iperf3 客户端，连接到第一个 Pod 中的每个 iperf3 服务器：
注意：
- 脚本接收 3 个参数：要连接的服务器 IP、生成
iperf3进程的 CPU 核以及iperf3测试的持续时间。启动脚本时请确保提供所有 3 个参数，并将 CPU 核指定为范围（如 28-51）。- 确保 Pod 上安装了
jq和bc，以便脚本正常运行。
iperf_client.sh
#!/bin/bash
# IP address of the server where iperf3 servers are running
SERVER_IP=$1 # Change to your server's IP
# Cores to bind the iperf3 client processes to
CORES=$2
# Duration to run the iperf3 test
DUR=$3
# Variable to accumulate the total bandwidth in Gbit/sec
total_bandwidth_Gbit=0
# Calculate the first_core and last_core to provide the CPU range
first_core=$(echo $CORES | cut -d "-" -f1)
last_core=$(echo $CORES | cut -d "-" -f2)
ports_num=$(($last_core - $first_core + 1))
# Array to store the PIDs of background tasks
pids=()
# Loop over the ports (5201 + i*2) for i in the given CPU range
for i in $(seq 1 $ports_num); do
port=$((5201 + i * 2))
cpu_core=$(($i + $first_core - 1)) # Assign CPU core based on the value of i
output_file="iperf3_client_results_$port.log"
# Run the iperf3 client in the background with CPU core binding
timeout $(( DUR +5 )) taskset -c $cpu_core iperf3 -c $SERVER_IP -p $port -t $DUR -J > $output_file &
pid=$!
pids+=("$pid")
done
# Wait for all background tasks to complete and check their status
for pid in "${pids[@]}"; do
wait $pid
if [[ $? -ne 0 ]]; then
echo "Process with PID $pid failed or timed out."
fi
done
# Summarize the results from each log file
echo "Summary of iperf3 client results:"
for i in $(seq 1 $ports_num); do
port=$((5201 + i * 2))
output_file="iperf3_client_results_$port.log"
if [[ -f $output_file ]]; then
echo "Results for port $port:"
# Parse the results and print a summary
bandwidth_bps=$(jq '.end.sum_received.bits_per_second' $output_file)
if [[ -n $bandwidth_bps ]]; then
# Convert bandwidth from bps to Gbit/sec
bandwidth_Gbit=$(echo "scale=3; $bandwidth_bps / 1000000000" | bc)
echo " Bandwidth: $bandwidth_Gbit Gbit/sec"
# Accumulate the bandwidth for the total summary
total_bandwidth_Gbit=$(echo "scale=3; $total_bandwidth_Gbit + $bandwidth_Gbit" | bc)
# Delete current log file
rm $output_file
else
echo "No bandwidth data found in $output_file"
fi
else
echo "No results found for port $port"
fi
done
# Print the total bandwidth summary
echo "Total Bandwidth across all streams: $total_bandwidth_Gbit Gbit/sec"
运行脚本并检查性能结果：
Second Pod Console
root@sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw:/# chmod +x iperf_client.sh
root@sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw:/# ./iperf_client.sh 10.0.121.1 28-51 30
Summary of iperf3 client results:
Results for port 5203:
Bandwidth: 14.207 Gbit/sec
Results for port 5205:
Bandwidth: 22.445 Gbit/sec
Results for port 5207:
Bandwidth: 8.868 Gbit/sec
Results for port 5209:
Bandwidth: 11.115 Gbit/sec
Results for port 5211:
Bandwidth: 14.104 Gbit/sec
Results for port 5213:
Bandwidth: 13.387 Gbit/sec
Results for port 5215:
Bandwidth: 22.743 Gbit/sec
Results for port 5217:
Bandwidth: 12.132 Gbit/sec
Results for port 5219:
Bandwidth: 13.927 Gbit/sec
Results for port 5221:
Bandwidth: 13.470 Gbit/sec
Results for port 5223:
Bandwidth: 22.720 Gbit/sec
Results for port 5225:
Bandwidth: 14.771 Gbit/sec
Results for port 5227:
Bandwidth: 12.752 Gbit/sec
Results for port 5229:
Bandwidth: 9.174 Gbit/sec
Results for port 5231:
Bandwidth: 14.265 Gbit/sec
Results for port 5233:
Bandwidth: 24.338 Gbit/sec
Results for port 5235:
Bandwidth: 14.087 Gbit/sec
Results for port 5237:
Bandwidth: 13.353 Gbit/sec
Results for port 5239:
Bandwidth: 14.555 Gbit/sec
Results for port 5241:
Bandwidth: 20.808 Gbit/sec
Results for port 5243:
Bandwidth: 13.056 Gbit/sec
Results for port 5245:
Bandwidth: 16.648 Gbit/sec
Results for port 5247:
Bandwidth: 17.545 Gbit/sec
Results for port 5249:
Bandwidth: 20.905 Gbit/sec
Total Bandwidth across all streams: 375.375 Gbit/sec
Network Isolation Test
最后，验证运行在不同网络（使用 PF0 和 PF1 上的虚拟功能）上的两个 Pod 无法相互通信。
应用以下 YAML 创建 PF1（BLUE 网络）的部署 - 确保将 <IMAGE_URL> 替换为相关的容器镜像 URL：
testapp-performance-test-deployment-pf1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sriov-hostdev-pf1vf10-test-worker1
labels:
app: sriov-hostdev-pf1vf10-test-worker1
spec:
replicas: 1
selector:
matchLabels:
app: sriov-hostdev-pf1vf10-test-worker1
template:
metadata:
labels:
app: sriov-hostdev-pf1vf10-test-worker1
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-pf1vf10-worker1
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: sriov-test-worker
nodeSelector:
feature.node.kubernetes.io/dpu-enabled: "true"
kubernetes.io/hostname: "worker1"
containers:
- name: testapp
securityContext:
privileged: true
capabilities:
add:
- NET_ADMIN
- IPC_LOCK
image: <IMAGE_URL>
command: ["sleep", "infinity"]
ports:
- containerPort: 5000
name: tcp-server
resources:
requests:
cpu: 24
memory: 6Gi
limits:
cpu: 24
memory: 6Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sriov-hostdev-pf1vf10-test-worker2
labels:
app: sriov-hostdev-pf1vf10-test-worker2
spec:
replicas: 1
selector:
matchLabels:
app: sriov-hostdev-pf1vf10-test-worker2
template:
metadata:
labels:
app: sriov-hostdev-pf1vf10-test-worker2
annotations:
k8s.v1.cni.cncf.io/networks: hostdev-pf1vf10-worker2
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: sriov-test-worker
nodeSelector:
feature.node.kubernetes.io/dpu-enabled: "true"
kubernetes.io/hostname: "worker2"
containers:
- name: testapp
securityContext:
privileged: true
capabilities:
add:
- NET_ADMIN
- IPC_LOCK
image: <IMAGE_URL>
command: ["sleep", "infinity"]
ports:
- containerPort: 5000
name: tcp-server
resources:
requests:
cpu: 24
memory: 6Gi
limits:
cpu: 24
memory: 6Gi
add: - NET_ADMIN - IPC_LOCK image: <IMAGE_URL> command: ["sleep", "infinity"] ports: - containerPort: 5000 name: tcp-server resources: requests: cpu: 24 memory: 6Gi limits: cpu: 24 memory: 6Gi
Jump Node Console
$ kubectl apply -f testapp-performance-test-deployment-pf1.yaml
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
sriov-hostdev-pf0-test-worker1-76bbff86d-4gvsb 1/1 Running 0 3d1h 10.233.69.102 worker1 <none> <none>
sriov-hostdev-pf0-test-worker2-649f58574c-5xrgw 1/1 Running 0 3d1h 10.233.70.177 worker2 <none> <none>
sriov-hostdev-pf1-test-worker1-f9bc5f88f-msl4r 1/1 Running 0 3d1h 10.233.69.11 worker1 <none> <none>
sriov-hostdev-pf1-test-worker2-f9bc5f88f-pqsb6 1/1 Running 0 3d1h 10.233.70.73 worker2 <none> <none>
Connect to the pod running on the second worker node, with the PF1 network, add the required routing entry and try to ping the first pod running on the first worker node, with the PF0 network:
Jump Node Console
$ kubectl exec -it sriov-hostdev-pf1-test-worker2-f9bc5f88f-pqsb6 -- bash
root@sriov-hostdev-pf1-test-worker2-f9bc5f88f-pqsb6:/# ip route add 10.0.121.0/29 via 10.0.122.2
root@sriov-hostdev-pf1-test-worker2-f9bc5f88f-pqsb6:/# ping 10.0.121.1
This ping operation should fail due to the network isolation implemented in HBN using different VLANs, VNIs and VRFs.
Authors
 |
Guy ZilbermanGuy Zilberman is a solution architect at NVIDIA's 网络解决方案 Labs, bringing extensive experience from several leadership roles in cloud computing. He specializes in designing and implementing solutions for cloud and containerized workloads, leveraging NVIDIA's advanced networking technologies. His work primarily focuses on open-source cloud infrastructure, with expertise in platforms such as Kubernetes (K8s) and OpenStack. |
 |
Shachar DorShachar Dor joined the 解决方案 Lab team after working more than ten years as a software architect at NVIDIA Networking (previously Mellanox Technologies), where he was responsible for the architecture of network management products and solutions. Shachar's focus is on networking technologies, especially around fabric bring-up, configuration, monitoring, and life-cycle management.Shachar has a strong background in software architecture, design, and programming through his work on multiple projects and technologies also prior to joining the company. |
NVIDIA and the NVIDIA logo, are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.™
2025 NVIDIA Corporation. All rights reserved.©