RDG for DPF Zero-Trust Multi-DPU: DPU1 with HBN and DPU2 with DTS/Blueman services
Created on Dec 08, 2025 Updated on Jan 06, 2026 (DPF 25.10.0 GA) Scope This Reference Deployment Guide (RDG) provides comprehensive instructions for deploying
文档目录
Created on Dec 08, 2025
Updated on Jan 06, 2026 (DPF 25.10.0 GA)
Scope
This Reference Deployment Guide (RDG) provides comprehensive instructions for deploying the NVIDIA DOCA Platform Framework (DPF) on high-performance, bare-metal infrastructure in Zero-Trust mode. The guide focuses on setting up an accelerated Host-Based Networking (HBN) service on NVIDIA® BlueField®-3 DPU to deliver secure, isolated, and hardware-accelerated environments. The guide also covers deploying the DOCA Telemetry Service (DTS) and BlueMan Service on additional workload NVIDIA® BlueField®-3 DPU, enabling a unified interface to accessing essential DPU information, health status, and telemetry metrics.
The guide is intended for experienced system administrators, systems engineers, and solution architects who build highly secure bare-metal environments with Host-Based Networking enabled using NVIDIA BlueField DPU for acceleration, isolation, and infrastructure offload.
This document is an extension of the RDG for DPF Zero Trust (DPF-ZT) - NVIDIA Docs (referred to as the Baseline RDG). It details the additional steps and modifications required to deploy the HBN, DTS, and BlueMan Services into the Baseline RDG environment.
Warning
- This reference implementation, as the name implies, is a specific, opinionated deployment example designed to address the use case described above.
- Although other approaches may exist for implementing similar solutions, this document provides a detailed guide for this specific method.
Abbreviations and Acronyms
| Term | Definition | Term | Definition |
|---|---|---|---|
| BFB | BlueField Bootstream | NFS | Network File System |
| BGP | Border Gateway Protocol | OOB | Out-of-Band |
| DOCA | Data Center Infrastructure-on-a-Chip Architecture | PF | Physical Function |
| DPF | DOCA Platform Framework | RDG | Reference Deployment Guide |
| DPU | Data Processing Unit | RDMA | Remote Direct Memory Access |
| DTS | DOCA Telemetry Service | RoCE | RDMA over Converged Ethernet |
| HBN | Host Based Networking | SFC | Service Function Chaining |
| IPAM | IP Address Management | SR-IOV | Single Root Input/Output Virtualization |
| K8S | Kubernetes | VLAN | Virtual LAN (Local Area Network) |
| KVM | Kernel-based Virtual Machine | VNI | Virtual Network Interface |
| MAAS | Metal as a Service | VRF | Virtual Router/Forwarder |
| MTU | Maximum Transmission Unit | ZT | Zero Trust |
| NGC | NVIDIA GPU Cloud |
Introduction
The NVIDIA BlueField-3 Data Processing Unit (DPU) is a 400 Gb/s infrastructure compute platform designed for line-rate processing of software-defined networking, storage, and cybersecurity workloads. It combines powerful compute resources, high-speed networking, and advanced programmability to deliver hardware-accelerated, software-defined solutions for modern data centers.
NVIDIA DOCA unleashes the full potential of the BlueField platform by enabling rapid development of applications and services that offload, accelerate, and isolate data center workloads.
One such service is Host-Based Networking (HBN) - a DOCA-enabled solution that allows network architects to design networks based on Layer 3 (L3) protocols. HBN enables routing on the server side by using BlueField as a BGP router. It encapsulates key networking functions in a containerized service pod, deployed directly on the BlueField's Arm cores.
DOCA Telemetry Service (DTS) collects data from built-in providers (data providers such as sysfs, ethtool and tc, and aggregation providers such as fluent_aggr and prometheus_aggr), and from external telemetry applications.
DOCA BlueMan runs in the DPU as a standalone web dashboard and consolidates all the basic information, health, and telemetry counters into a single interface. All the information that BlueMan provides is gathered from the DOCA Telemetry Service (DTS).
However, deploying and managing DPU and their associated DOCA services, especially at scale, presents operational challenges. Without a robust provisioning and orchestration system, tasks such as lifecycle management, service deployment, and network configuration for service function chaining (SFC) can quickly become complex and error prone. This is where the DOCA Platform Framework (DPF) comes into play.
DPF automates the full DPU lifecycle, streamlines the deployment of DOCA services, and simplifies advanced network configurations. With DPF, services such as HBN can be deployed seamlessly, allowing for efficient offloading and intelligent routing of traffic through the DPU data plane.
By leveraging DPF, users can scale and automate DPU management across Bare Metal, Virtual, and Kubernetes customer environments - optimizing performance while simplifying operations.
DPF supports multiple deployment models. This guide focuses on the Zero Trust bare-metal deployment model. In this scenario:
- The DPU is managed through its Baseboard Management Controller (BMC)
- All management traffic occurs over the DPU's out-of-band (OOB) network
- The host is considered as an untrusted entity towards the data center network. The DPU acts as a barrier between the host and the network.
- The host sees the DPU as a standard NIC, with no access to the internal DPU management plane (Zero Trust Mode)
This Reference Deployment Guide (RDG) provides a step-by-step example for installing DPF in Zero-Trust mode and HBN. It also includes practical demonstrations of performance optimization, validated using standard RDMA and TCP workloads.
As part
of the reference implementation, open-source components outside the scope of DPF (e.g., MAAS, pfSense, Kubespray) are used to simulate a realistic customer deployment environment. The guide includes the full end-to-end deployment process, including:
- Infrastructure provisioning
- DPF deployment
- DPU provisioning (redfish)
- Service configuration and deployment
- Service chaining.
This document extends the capabilities of the DPF-managed Kubernetes cluster described in the RDG for DPF Zero Trust (DPF-ZT) - NVIDIA Docs (referred to as the Baseline RDG) by deploying the NVIDIA DOCA HBN, DTS and BlueMan Services within the existing DPF deployment to achieve a comprehensive, accelerated infrastructure.
References
- NVIDIA BlueField DPU
- NVIDIA DOCA
- NVIDIA DPF Release Notes
- NVIDIA DPF GitHub Repository
- NVIDIA DPF System Overview
- NVIDIA Ethernet Switching
- NVIDIA Cumulus Linux
- What is K8s?
- Kubespray
Solution Architecture
Key Components and Technologies
-
NVIDIA BlueField® Data Processing Unit (DPU) The NVIDIA® BlueField® data processing unit (DPU) ignites unprecedented innovation for modern data centers and supercomputing clusters. With its robust compute power and integrated software-defined hardware accelerators for networking, storage, and security, BlueField creates a secure and accelerated infrastructure for any workload in any environment, ushering in a new era of accelerated computing and AI.
-
NVIDIA DOCA Software Framework NVIDIA DOCA™ unlocks the potential of the NVIDIA® BlueField® networking platform. By harnessing the power of BlueField DPU and SuperNICs, DOCA enables the rapid creation of applications and services that offload, accelerate, and isolate data center workloads. It lets developers create software-defined, cloud-native, DPU- and SuperNIC-accelerated services with zero-trust protection, addressing the performance and security demands of modern data centers.
-
NVIDIA ConnectX SmartNICs 10/25/40/50/100/200 and 400G Ethernet Network 网卡 The industry-leading NVIDIA® ConnectX® family of smart network interface cards (SmartNICs) offer advanced hardware offloads and accelerations. NVIDIA Ethernet adapters enable the highest ROI and lowest Total Cost of Ownership for hyperscale, public and private clouds, storage, machine learning, AI, big data, and telco platforms.
-
NVIDIA LinkX Cables The NVIDIA® LinkX® product family of cables and transceivers provides the industry’s most complete line of 10, 25, 40, 50, 100, 200, and 400GbE in Ethernet and 100, 200 and 400Gb/s InfiniBand products for Cloud, HPC, hyperscale, Enterprise, telco, storage and artificial intelligence, data center applications.
-
NVIDIA Spectrum 以太网交换机 Flexible form-factors with 16 to 128 physical ports, supporting 1GbE through 400GbE speeds. Based on a ground-breaking silicon technology optimized for performance and scalability, NVIDIA Spectrum switches are ideal for building high-performance, cost-effective, and efficient Cloud Data Center Networks, Ethernet Storage Fabric, and Deep Learning Interconnects. NVIDIA combines the benefits of NVIDIA Spectrum™ switches, based on an industry-leading application-specific integrated circuit (ASIC) technology, with a wide variety of modern network operating system choices, including NVIDIA Cumulus® Linux, SONiC and NVIDIA Onyx®.
-
NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.
-
Kubernetes Kubernetes is an open-source container orchestration platform for deployment automation, scaling, and management of containerized applications.
-
Kubespray Kubespray is a composition of Ansible playbooks, inventory, provisioning tools, and domain knowledge for generic OS/Kubernetes clusters configuration management tasks and provides:
- A highly available cluster
- Composable attributes
- Support for most popular Linux distributions
Solution Design
Solution Logical Design
The logical design includes the following components:
- 1 x Hypervisor node (KVM-based) with ConnectX-7:
- 1 x Firewall VM
- 1 x Jump Node VM
- 1 x MaaS VM
- 3 x K8s Master VMs running all K8s management components
- 4 x Worker nodes (PCI Gen5), each with 2 x BlueField-3 NIC
- Single High-Speed (HS) switch
- 1 Gb Host Management network
HBN service Logical Design
As part of this RDG, we will:
- Create two fully isolated logical networks per bare-metal workload server using a single physical function (PF0).
- Connect each network through the HBN service to a dedicated VLAN/VNI, mapped to separate VRFs (RED or BLUE).
- Route all workload traffic through the HBN service, with routing and isolation enforced inside the DPU.
- Assign PF0 as the sole network interface for each bare-metal workload server, with no networking configuration on the host.
- Demonstrate accelerated RDMA and TCP traffic between workload servers running on different bare-metal hosts within the same network (for example, RED ↔ RED).
- Validate strict network isolation by confirming that traffic between workloads in different networks (RED vs BLUE) is blocked.
RDG for DPF Zero-Trust Multi-DPU: DPU1 with HBN and DPU2 with DTS/Blueman services
Note: This is a translated and converted version of the original HTML content.
Firewall Design
The pfSense firewall in this solution serves a dual purpose:
- Firewall — provides an isolated environment for the DPF system, ensuring secure operations
- Router — enables Internet access for the management network
Port-forwarding rules for SSH and RDP are configured on the firewall to route traffic to the jump node's IP address in the host management network. From the jump node, administrators can manage and access various devices in the setup, as well as handle the deployment of the Kubernetes (K8s) cluster and DPF components.
The following diagram illustrates the firewall design used in this solution:
Software Stack Components
Important: Make sure to use the exact same versions for the software stack as described above.
Bill of Materials
Deployment and Configuration
Node and Switch Definitions
These are the definitions and parameters used for deploying the demonstrated fabric:
交换机 Ports Usage
| Hostname | Rack ID | Ports |
|---|---|---|
mgmt-switch |
1 | swp1-3 |
hs-switch |
1 | swp1-17 |
Hosts
| Rack | Server Type | Server Name | Switch Port | IP and NICs | Default Gateway |
|---|---|---|---|---|---|
| Rack1 | Hypervisor Node | hypervisor |
mgmt-switch: swp1hs-switch: swp1 |
lab-br (interface eno1): Trusted LAN IPmgmt-br (interface eno2): -hs-br (interface enp1s0): - | Trusted LAN GW |
| Rack1 | Firewall (Virtual) | fw |
- | WAN (lab-br): Trusted LAN IPLAN (mgmt-br): 10.0.110.254/24OPT1 (hs-br): 10.0.123.254/22 | Trusted LAN GW |
| Rack1 | Jump Node (Virtual) | jump |
- | enp1s0: 10.0.110.253/24 | 10.0.110.254 |
| Rack1 | MaaS (Virtual) | maas |
- | enp1s0: 10.0.110.252/24 | 10.0.110.254 |
| Rack1 | Master Node (Virtual) | master1 |
- | enp1s0: 10.0.110.1/24 | 10.0.110.254 |
| Rack1 | Master Node (Virtual) | master2 |
- | enp1s0: 10.0.110.2/24 | 10.0.110.254 |
| Rack1 | Master Node (Virtual) | master3 |
- | enp1s0: 10.0.110.3/24 | 10.0.110.254 |
| Rack1 | Worker Node | worker1 |
mgmt-switch: swp2 (DPU OOB)hs-switch: swp2-swp3-swp4-swp5 |
dpubmc: 10.0.110.21/24ens1f0np0/ens1f1np1: 10.0.120.0/22 | 10.0.110.254 |
| Rack | Node Type | Hostname | Switch Ports | IP Addresses | Gateway |
|---|---|---|---|---|---|
| Rack1 | Worker Node | worker2 |
mgmt-switch: swp3(DPU OOB)hs-switch: swp6-swp7-swp8-swp9 |
dpubmc: 10.0.110.22/24ens1f0np0/ens1f1np1: 10.0.120.0/22 | 10.0.110.254 |
| Rack1 | Worker Node | worker3 |
mgmt-switch: swp2(DPU OOB)hs-switch: swp10-swp11-swp12-swp13 |
dpubmc: 10.0.110.23/24ens1f0np0/ens1f1np1: 10.0.120.0/22 | 10.0.110.254 |
| Rack1 | Worker Node | worker4 |
mgmt-switch: swp3(DPU OOB)hs-switch: swp14-swp15-swp16-swp17 |
dpubmc: 10.0.110.24/24ens1f0np0/ens1f1np1: 10.0.120.0/22 | 10.0.110.254 |
Wiring
Hypervisor Node
Bare Metal Worker Node
Fabric Configuration
Updating Cumulus Linux
作为最佳实践，请确保使用最新发布的 Cumulus Linux NOS 版本。
有关如何升级 Cumulus Linux 的信息，请参阅 Cumulus Linux 用户指南。
Configuring the Cumulus Linux Switch
SN3700 交换机（hs-switch）配置如下：
nv set evpn state enable
nv set interface eth0 ip address dhcp
nv set interface eth0 ip vrf mgmt
nv set interface eth0 type eth
nv set interface lo ipv4 address 11.0.0.101/32
nv set interface lo type loopback
nv set interface swp1-17 link state up
nv set interface swp1-17 type swp
nv set interface swp1 ipv4 address 10.0.123.253/22
nv set router bgp autonomous-system 65001
nv set router bgp state enabled
nv set router bgp graceful-restart mode full
nv set router bgp router-id 11.0.0.101
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute static state enabled
nv set vrf default router bgp address-family ipv6-unicast state enabled
nv set vrf default router bgp address-family ipv6-unicast redistribute connected state enabled
nv set vrf default router bgp address-family l2vpn-evpn state enabled
nv set vrf default router bgp state enabled
nv set vrf default router bgp neighbor swp2 peer-group hbn
nv set vrf default router bgp neighbor swp2 type unnumbered
nv set vrf default router bgp neighbor swp3 peer-group hbn
nv set vrf default router bgp neighbor swp3 type unnumbered
nv set vrf default router bgp neighbor swp4 peer-group hbn
nv set vrf default router bgp neighbor swp4 type unnumbered
nv set vrf default router bgp neighbor swp5 peer-group hbn
nv set vrf default router bgp neighbor swp5 type unnumbered
nv set vrf default router bgp neighbor swp6 peer-group hbn
nv set vrf default router bgp neighbor swp6 type unnumbered
nv set vrf default router bgp neighbor swp7 peer-group hbn
nv set vrf default router bgp neighbor swp7 type unnumbered
nv set vrf default router bgp neighbor swp8 peer-group hbn
nv set vrf default router bgp neighbor swp8 type unnumbered
nv set vrf default router bgp neighbor swp9 peer-group hbn
nv set vrf default router bgp neighbor swp9 type unnumbered
nv set vrf default router bgp neighbor swp10 peer-group hbn
nv set vrf default router bgp neighbor swp10 type unnumbered
nv set vrf default router bgp neighbor swp11 peer-group hbn
nv set vrf default router bgp neighbor swp11 type unnumbered
nv set vrf default router bgp neighbor swp12 peer-group hbn
nv set vrf default router bgp neighbor swp12 type unnumbered
nv set vrf default router bgp neighbor swp13 peer-group hbn
nv set vrf default router bgp neighbor swp13 type unnumbered
nv set vrf default router bgp neighbor swp14 peer-group hbn
nv set vrf default router bgp neighbor swp14 type unnumbered
nv set vrf default router bgp neighbor swp15 peer-group hbn
nv set vrf default router bgp neighbor swp15 type unnumbered
nv set vrf default router bgp neighbor swp16 peer-group hbn
nv set vrf default router bgp neighbor swp16 type unnumbered
nv set vrf default router bgp neighbor swp17 peer-group hbn
nv set vrf default router bgp neighbor swp17 type unnumbered
nv set vrf default router bgp path-selection multipath aspath-ignore enabled
nv set vrf default router bgp peer-group hbn address-family l2vpn-evpn state enabled
nv set vrf default router bgp peer-group hbn remote-as external
nv set vrf default router static 0.0.0.0/0 address-family ipv4-unicast
nv set vrf default router static 0.0.0.0/0 via 10.0.123.254 type ipv4-address
nv config apply -y
nv config save
SN2201 交换机（mgmt-switch）配置如下：
nv set interface swp1-3 link state up
nv set interface swp1-3 type swp
nv set interface swp1-3 bridge domain br_default
nv set bridge domain br_default untagged 1
nv config apply
nv config save -y
Host Configuration
注意： 确保工作节点服务器的 BIOS 设置中启用了 SR-IOV，并且服务器已调整为最大性能。
所有工作节点必须具有相同的 BlueField-3 网卡 PCIe 位置，并且必须显示相同的接口名称。
确保您拥有 DPU BMC 和 OOB MAC 地址。
与参考部署指南（基线 RDG）（“部署和配置”部分，“主机配置”小节）相比无变化。
Hypervisor Installation and Configuration
与基线 RDG（“部署和配置”部分，“虚拟机监控程序安装和配置”小节）相比无变化。
Prepare Infrastructure Servers
关于防火墙虚拟机、跳板虚拟机、MaaS 虚拟机，与基线 RDG（“部署和配置”部分，“准备基础设施服务器”小节）相比无变化。
（可选）防火墙虚拟机 – 裸机服务器外部连接
为了通过高速网络提供裸机主机的外部连接，请打开 Firefox 浏览器并访问 pfSense Web UI（http://10.0.110.254/）。
- 系统：
- 路由 → 网关 → 添加 → “接口”： OPT1，“地址族”：
IPv4, "Name": 交换机, "Gateway": 10.0.123.253 → 点击 "Save" → 在 "Default Gateway" - "Default gateway IPv4" 下选择 WAN_DHCP → 点击 "Save"
注意： Trusted LAN 网络下 "Gateway" 和 "Monitor IP" 的 IP 地址已模糊处理。
使用 MaaS 配置主虚拟机
与基线 RDG（第 "部署与配置" 节，子节 "使用 MaaS 配置主虚拟机"）相比无变化。
K8s 集群部署与配置
使用 Kubespray 进行初始 Kubernetes 集群部署（主节点）及后续验证的步骤与基线 RDG（第 "K8s 集群部署与配置" 节，子节："Kubespray 部署与配置"、"使用 Kubespray Ansible Playbook 部署集群"、"K8s 部署验证"）相比保持不变。
DPF 安装
DPF 安装过程（Operator、系统组件）基本遵循基线 RDG。
软件前提条件与所需变量
-
首先安装剩余的软件前提条件。
跳板机控制台
## 连接到 master1 以复制 kubespray 部署期间安装的 helm 客户端工具 $ depuser@jump:~$ ssh master1 depuser@master1:~$ cp /usr/local/bin/helm /tmp/ ## 在另一个标签页中 depuser@jump:~$ scp master1:/tmp/helm /tmp/ depuser@jump:~$ sudo chown root:root /tmp/helm depuser@jump:~$ sudo mv /tmp/helm /usr/local/bin/ ## 验证 envsubst 工具是否已安装 depuser@jump:~$ which envsubst /usr/bin/envsubst -
继续克隆 doca-platform Git 仓库：
跳板机控制台
$ git clone https://github.com/NVIDIA/doca-platform.git -
切换到 doca-platform 目录并检出 标签 v25.10.0：
跳板机控制台
$ cd doca-platform/ $ git checkout v25.10.0 -
切换到 readme.md 目录，所有命令将在此运行：
跳板机控制台
$ cd doca-platform/docs/public/user-guides/zero-trust/use-cases/hbn -
更改 BMC root 密码。 在零信任模式下，配置 DPU 需要通过 Redfish 进行身份验证。 为此，您必须为 DPF 将要管理的所有 DPU 设置相同的 BMC root 密码。有关如何设置 BMC root 密码的更多信息，请参阅 BlueField DPU 管理员快速入门指南。
通过 SSH 连接到 DPU BMC，更改所有 DPU 上的 BMC root 密码。
跳板机控制台
$ ssh root@10.0.110.201 root@10.0.110.201's password: <BMC Root Password. 默认 root/0penBmc，首次需更改为 manifests/00-env-vars/envvars.env 文件中的 $BMC_ROOT_PASSWORD> -
修改
manifests/00-env-vars/envvars.env中的变量以匹配您的环境，然后 source 该文件：错误： 将以下文件中的变量值替换为适合您设置的值。特别注意
DPUCLUSTER_INTERFACE、BMC_ROOT_PASSWORD和DPU 的序列号。 要获取DPU 的序列号，可以使用以下命令。示例：$ curl -k -u root:'BMC root password' https://10.0.110.201/redfish/v1/Systems/Bluefield | jq -r '.SerialNumber | ascii_downcase'% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed100 4970 100 4970 0 0 4211 0 0:00:01 0:00:01 --:--:-- 4211mt2402xz0f7xmanifests/00-env-vars/envvars.env
## 目标集群（DPF 安装于其上）的 Kubernetes API 服务器的 IP 地址。 ## 不应包含协议或端口。 ## 例如 10.10.10.10 export TARGETCLUSTER_API_SERVER_HOST=10.0.110.10 ## DPU 集群负载均衡器使用的虚拟 IP。必须是管理子网中的保留 IP，且不由 DHCP 分配。 export DPUCLUSTER_VIP=10.0.110.200 ## DPUCluster 负载均衡器将监听的接口。应为控制平面节点的管理接口。 export DPUCLUSTER_INTERFACE=ens160 ## 用作 BFB 存储的 NFS 服务器的 IP 地址。 export NFS_SERVER_IP=10.0.110.253 ## NVIDIA Helm 图表仓库的 URL。 ## 通常为 NVIDIA Helm NGC 仓库。出于开发目的，可设置为其他仓库。 export HELM_REGISTRY_REPO_URL=https://helm.ngc.nvidia.com/nvidia/doca ## HBN 容器镜像的仓库 URL。 ## 通常为 NVIDIA NGC 仓库。出于开发目的，可设置为其他仓库。 export HBN_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_hbn ## DPF REGISTRY 是 DPF Operator Chart 所在的 Helm 仓库 URL。 ## 通常为 NVIDIA Helm NGC 仓库。出于开发目的，可设置为其他仓库。 export REGISTRY=https://helm.ngc.nvidia.com/nvidia/doca ## DPF TAG 是本指南中将部署的 DPF 组件的版本。 export TAG=v25.10.0 ## `bfb.yaml` 中使用的 BFB 的 URL，由 DPUSet 链接。 export BFB_URL="https://content.mellanox.com/BlueField/BFBs/Ubuntu24.04/bf-bundle-3.2.1-34_25.11_ubuntu-24.04_64k_prod.bfb" ## IP_RANGE_START 和 IP_RANGE_END ## 定义通过 Redfish/BMC 接口进行 DPU 发现的 IP 范围。 ## 示例：如果您的 DPU 的 BMC IP 在 10.0.110.201-224 范围内 ## export IP_RANGE_START=10.0.110.201 ## export IP_RANGE_END=10.0.110.224 ## DPUDiscovery IpRange 起始 export IP_RANGE_START=10.0.110.201 ## DPUDiscovery IpRange 结束 export IP_RANGE_END=10.0.110.208 # 所有 DPU 的 BMC root 登录密码，必须相同 # 有关如何设置 BMC root 密码的更多信息，请参阅 BlueField DPU 管理员快速入门指南。 export BMC_ROOT_PASSWORD=<设置您的 BMC_ROOT_PASSWORD> ## DPU 的序列号。如果您有超过 2 个 DPU，则需要相应地参数化系统并暴露 ## 额外的变量。 ## 所有序列号必须为小写。 ## DPU1 的序列号 export DPU1_SERIAL=mt2402xz0f7x ## DPU2 的序列号 export DPU2_SERIAL=mt2402xz0f80 ## DPU3 的序列号 export DPU2_SERIAL=mt2402xz0f9n ## DPU4 的序列号 export DPU2_SERIAL=mt2402xz0f8g -
导出安装所需的环境变量：
跳板机控制台
$ source manifests/00-env-vars/envvars.env
DPF Operator 安装
与基线 RDG（第 "DPF 安装" 节，子节 "DPF Operator 安装"）相比无变化。
DPF 系统安装
与基线 RDG（第 "DPF 安装" 节，子节 "DPF 系统安装"）相比无变化。
DPU 服务安装
HBN DPU 服务安装
HBN DPU Service Installation
This section focuses on provisioning NVIDIA® BlueField®-3 DPU using DPF, installing the HBN DPU Service on those DPU and enabling workload traffic to pass through HBN before leaving the DPU.
-
Export environment variables for the installation:
Jump Node Console
$ source manifests/00-env-vars/envvars.env -
Use the following YAML to define a
BFBresource that downloads the Bluefield Bitstream to a shared volume:--- apiVersion: provisioning.dpu.nvidia.com/v1alpha1 kind: BFB metadata: name: bf-bundle-$TAG namespace: dpf-operator-system spec: url: $BFB_URL -
Change the
DPUFlavorusing the following YAML.--- apiVersion: provisioning.dpu.nvidia.com/v1alpha1 kind: DPUFlavor metadata: name: hbn-$TAG namespace: dpf-operator-system spec: dpuMode: zero-trust bfcfgParameters: - UPDATE_ATF_UEFI=yes - UPDATE_DPU_OS=yes - WITH_NIC_FW_UPDATE=yes configFiles: - operation: override path: /etc/mellanox/mlnx-bf.conf permissions: "0644" raw: | ALLOW_SHARED_RQ="no" IPSEC_FULL_OFFLOAD="no" ENABLE_ESWITCH_MULTIPORT="yes" - operation: override path: /etc/mellanox/mlnx-ovs.conf permissions: "0644" raw: | CREATE_OVS_BRIDGES="no" OVS_DOCA="yes" - operation: override path: /etc/mellanox/mlnx-sf.conf permissions: "0644" raw: "" grub: kernelParameters: - console=hvc0 - console=ttyAMA0 - earlycon=pl011,0x13010000 - fixrttc - net.ifnames=0 - biosdevname=0 - iommu.passthrough=1 - cgroup_no_v1=net_prio,net_cls - hugepagesz=2048kB - hugepages=3072 nvconfig: - device: '*' parameters: - PF_BAR2_ENABLE=0 - PER_PF_NUM_SF=1 - PF_TOTAL_SF=20 - PF_SF_BAR_SIZE=10 - NUM_PF_MSIX_VALID=0 - PF_NUM_PF_MSIX_VALID=1 - PF_NUM_PF_MSIX=228 - INTERNAL_CPU_MODEL=1 - INTERNAL_CPU_OFFLOAD_ENGINE=0 - SRIOV_EN=1 - NUM_OF_VFS=46 - LAG_RESOURCE_ALLOCATION=1 - LINK_TYPE_P1=ETH - LINK_TYPE_P2=ETH - EXP_ROM_UEFI_x86_ENABLE=1 ovs: rawConfigScript: | _ovs-vsctl() { ovs-vsctl --no-wait --timeout 15 "$@" } _ovs-vsctl set Open_vSwitch . other_config:doca-init=true _ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000 _ovs-vsctl set Open_vSwitch . other_config:hw-offload=true _ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true _ovs-vsctl set Open_vSwitch . other_config:max-idle=20000 _ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000 _ovs-vsctl --if-exists del-br ovsbr1 _ovs-vsctl --if-exists del-br ovsbr2 _ovs-vsctl --may-exist add-br br-sfc _ovs-vsctl set bridge br-sfc datapath_type=netdev _ovs-vsctl set bridge br-sfc fail_mode=secure _ovs-vsctl --may-exist add-port br-sfc p0 _ovs-vsctl set Interface p0 type=dpdk _ovs-vsctl set Interface p0 mtu_request=9216 _ovs-vsctl set Port p0 external_ids:dpf-type=physical _ovs-vsctl --may-exist add-port br-sfc p1 _ovs-vsctl set Interface p1 type=dpdk _ovs-vsctl set Interface p1 mtu_request=9216 _ovs-vsctl set Port p1 external_ids:dpf-type=physical _ovs-vsctl --may-exist add-br br-hbn _ovs-vsctl set bridge br-hbn datapath_type=netdev _ovs-vsctl set bridge br-hbn fail_mode=secure -
In multi-DPU configurations—where a single host worker node includes two or more NVIDIA® BlueField® DPU—using a standard
nodeSelectortargets the host node rather than individual DPU. As a result, all DPU-scoped services (HBN, DTS, BlueMan) are deployed onto every DPU on that node, which may lead to service conflicts and prevents proper role separation across DPU.The
dpuSelectormechanism provides fine-grained control over service placement by enabling operators to target specific DPU directly. This approach improves resource allocation, enforces service isolation, and enables clean scalability in multi-DPU deployments.Using
dpuSelector, you can:- Run the HBN service exclusively on the first DPU.
- Deploy the DTS and BlueMan services on the second DPU.
To target a specific DPU, apply labels to the corresponding
DPUDeviceobject. The labeled device can then be referenced bydpuSelector.Below is an example (replace the serial number with the one from your environment):
Jump Node Console
$ kubectl label dpudevice -n dpf-operator-system mt2402xz0f7x mt2402xz0f80 mt2402xz0f9n mt2402xz0f8g provisioning.dpu.nvidia.com/dpudevice-service-name=hbn $ kubectl label dpudevice -n dpf-operator-system mt2511600rc3 mt2511600ruh mt2511600r8p mt2511600rp1 provisioning.dpu.nvidia.com/dpudevice-service-name=dts-blueman -
Change the
dpudeployment.yamlfile to reference theDPUFlavor.--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUDeployment metadata: name: hbn-only namespace: dpf-operator-system spec: dpus: bfb: bf-bundle-$TAG flavor: hbn-$TAG nodeEffect: hold: true dpuSets: - nameSuffix: "dpuset1" nodeSelector: matchLabels: feature.node.kubernetes.io/dpu-enabled: "true" dpuSelector: provisioning.dpu.nvidia.com/dpudevice-service-name: hbn services: doca-hbn: serviceTemplate: doca-hbn serviceConfiguration: doca-hbn serviceChains: switches: - ports: - serviceInterface: matchLabels: uplink: p0 - service: name: doca-hbn interface: p0_if - ports: - serviceInterface: matchLabels: uplink: p1 - service: name: doca-hbn interface: p1_if - ports: - serviceInterface: matchLabels: interface: pf0hpf - service: interface: pf0hpf_if name: doca-hbnWarning: Please notice that with default nodeEffect above, DPU provisioning workflow will be paused and wait for an external signal (annotation) in order to proceed, as demonstrated in upcoming steps. To implement a fully automated process that won’t require user intervention, see customAction option.
-
Change the rest of the configuration files.
As explained in the introduction, these files create service chains that connect two physical functions PF0(RED) or PF0(BLUE) to the outer fabric through HBN, providing EVPN VXLAN overlay, VNI based isolation, and ECMP redundancy across both DPU uplinks (p0 and p1).
These are the configuration files.
-
HBN DPUServiceConfig and DPUServiceTemplate to deploy HBN workloads to the DPU.
--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceConfiguration metadata: name: doca-hbn namespace: dpf-operator-system spec: deploymentServiceName: "doca-hbn" serviceConfiguration: serviceDaemonSet: annotations: k8s.v1.cni.cncf.io/networks: |- [ {"name": "iprequest", "interface": "ip_lo", "cni-args": {"poolNames": ["loopback"], "poolType": "cidrpool"}}, {"name": "iprequest", "interface": "ip_pf0hpf_red", "cni-args": {"poolNames": ["pool1"], "poolType": "cidrpool", "allocateDefaultGateway": true}}, {"name": "iprequest", "interface": "ip_pf0hpf_blue", "cni-args": {"poolNames": ["pool2"], "poolType": "cidrpool", "allocateDefaultGateway": true}} ] helmChart: values: configuration: perDPUValuesYAML: | - hostnamePattern: "*" values: bgp_peer_group: hbn # ---- DPU1, DPU2 => RED only ---- - hostnamePattern: "dpu-node-${DPU1_SERIAL}-${DPU1_SERIAL}" values: role: RED vrf: RED vlan: 11 l2vni: 10010 l3vni: 100001 bgp_autonomous_system: 65101 - hostnamePattern: "dpu-node-${DPU2_SERIAL}-${DPU2_SERIAL}" values: role: RED vrf: RED vlan: 11 l2vni: 10010 l3vni: 100001 bgp_autonomous_system: 65201 # ---- DPU3, DPU4 => BLUE only ---- - hostnamePattern: "dpu-node-${DPU3_SERIAL}-${DPU3_SERIAL}" values: role: BLUE vrf: BLUE vlan: 21 l2vni: 10020 l3vni: 100002 bgp_autonomous_system: 65301 - hostnamePattern: "dpu-node-${DPU4_SERIAL}-${DPU4_SERIAL}" values: role: BLUE vrf: BLUE vlan: 21 l2vni: 10020 l3vni: 100002 bgp_autonomous_system: 65401 startupYAMLJ2: | - header: model: bluefield nvue-api-version: nvue_v1 rev-id: 1.0 version: HBN 2.4.0 - set: bridge: domain: br_default: vlan: {{ config.vlan }}: vni: {{ config.l2vni }}: {} evpn: enable: on route-advertise: {} interface: lo: ip: address: {{ ipaddresses.ip_lo.ip }}/32: {} type: loopback p0_if,p1_if,pf0hpf_if: type: swp link: mtu: 9000
-
pf0hpf_if:
bridge:
domain:
br_default:
access: {{ config.vlan }}
vlan{{ config.vlan }}:
type: svi
vlan: {{ config.vlan }}
ip:
address:
{% if config.role == "RED" %}
{{ ipaddresses.ip_pf0hpf_red.cidr }}: {}
{% else %}
{{ ipaddresses.ip_pf0hpf_blue.cidr }}: {}
{% endif %}
vrf: {{ config.vrf }}
nve:
vxlan:
arp-nd-suppress: on
enable: on
source:
address: {{ ipaddresses.ip_lo.ip }}
router:
bgp:
enable: on
graceful-restart:
mode: full
vrf:
default:
router:
bgp:
address-family:
ipv4-unicast:
enable: on
redistribute:
connected:
enable: on
l2vpn-evpn:
enable: on
autonomous-system: {{ config.bgp_autonomous_system }}
enable: on
neighbor:
p0_if:
peer-group: {{ config.bgp_peer_group }}
type: unnumbered
p1_if:
peer-group: {{ config.bgp_peer_group }}
type: unnumbered
path-selection:
multipath:
aspath-ignore: on
peer-group:
{{ config.bgp_peer_group }}:
address-family:
ipv4-unicast:
enable: on
l2vpn-evpn:
enable: on
remote-as: external
router-id: {{ ipaddresses.ip_lo.ip }}
{{ config.vrf }}:
evpn:
enable: on
vni:
{{ config.l3vni }}: {}
loopback:
ip:
address:
{{ ipaddresses.ip_lo.ip }}/32: {}
router:
bgp:
address-family:
ipv4-unicast:
enable: on
redistribute:
connected:
enable: on
route-export:
to-evpn:
enable: on
autonomous-system: {{ config.bgp_autonomous_system }}
enable: on
router-id: {{ ipaddresses.ip_lo.ip }}
interfaces:
- name: p0_if
network: mybrhbn
- name: p1_if
network: mybrhbn
- name: pf0hpf_if
network: mybrhbn
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
name: doca-hbn
namespace: dpf-operator-system
spec:
deploymentServiceName: "doca-hbn"
helmChart:
source:
repoURL: $HELM_REGISTRY_REPO_URL
version: 1.0.5
chart: doca-hbn
values:
image:
repository: $HBN_NGC_IMAGE_URL
tag: 3.2.1-doca3.2.1
resources:
memory: 6Gi
nvidia.com/bf_sf: 4
Physical Interfaces for physical ports on the DPU.
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: p0
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
uplink: "p0"
spec:
interfaceType: physical
physical:
interfaceName: p0
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: p1
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
uplink: "p1"
spec:
interfaceType: physical
physical:
interfaceName: p1
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
name: pf0hpf
namespace: dpf-operator-system
spec:
template:
spec:
template:
metadata:
labels:
interface: "pf0hpf"
spec:
interfaceType: pf
pf:
pfID: 0
DPU Service IPAM objects to set up IP Address Management on the DPUCluster.
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
name: pool1
namespace: dpf-operator-system
spec:
ipv4Network:
network: "10.0.121.0/24"
gatewayIndex: 2
prefixSize: 29
# These preallocations are not necessary. We specify them so that the validation commands are straightforward.
allocations:
dpu-node-${DPU1_SERIAL}-${DPU1_SERIAL}: 10.0.121.0/29
dpu-node-${DPU2_SERIAL}-${DPU2_SERIAL}: 10.0.121.8/29
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
name: pool2
namespace: dpf-operator-system
spec:
ipv4Network:
network: "10.0.122.0/24"
gatewayIndex: 2
prefixSize: 29
allocations:
dpu-node-${DPU3_SERIAL}-${DPU3_SERIAL}: 10.0.122.0/29
dpu-node-${DPU4_SERIAL}-${DPU4_SERIAL}: 10.0.122.8/29
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
name: loopback
namespace: dpf-operator-system
spec:
ipv4Network:
network: "11.0.0.0/24"
prefixSize: 32
Warning: It is necessary to set several environment variables before running this command.
$ source manifests/00-env-vars/envvars.env
Apply all of the YAML files mentioned above using the following command:
Jump Node Console
$ cat manifests/03.1-dpudeployment-installation-pf/*.yaml | envsubst | kubectl apply -f -
Jump Node Console
$ kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices --all
dpuservice.svc.dpu.nvidia.com/doca-hbn-wb5pg condition met
dpuservice.svc.dpu.nvidia.com/flannel condition met
dpuservice.svc.dpu.nvidia.com/multus condition met
dpuservice.svc.dpu.nvidia.com/nvidia-k8s-ipam condition met
dpuservice.svc.dpu.nvidia.com/ovs-cni condition met
dpuservice.svc.dpu.nvidia.com/servicechainset-controller condition met
dpuservice.svc.dpu.nvidia.com/servicechainset-rbac-and-crds condition met
dpuservice.svc.dpu.nvidia.com/sfc-controller condition met
dpuservice.svc.dpu.nvidia.com/sriov-device-plugin condition met
$ kubectl wait --for=condition=DPUIPAMObjectReconciled --namespace dpf-operator-system dpuserviceipam --all
dpuserviceipam.svc.dpu.nvidia.com/loopback condition met
dpuserviceipam.svc.dpu.nvidia.com/pool1 condition met
dpuserviceipam.svc.dpu.nvidia.com/pool2 condition met
$ kubectl wait --for=condition=ServiceInterfaceSetReconciled --namespace dpf-operator-system dpuserviceinterface --all
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-p0-if-vjqn5 condition met
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-p1-if-nl8rj condition met
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-pf0hpf-if-kbfj4 condition met
dpuserviceinterface.svc.dpu.nvidia.com/doca-hbn-pf1hpf-if-79zsq condition met
dpuserviceinterface.svc.dpu.nvidia.com/p0 condition met
dpuserviceinterface.svc.dpu.nvidia.com/p1 condition met
dpuserviceinterface.svc.dpu.nvidia.com/pf0hpf condition met
dpuserviceinterface.svc.dpu.nvidia.com/pf1hpf condition met
$ kubectl wait --for=condition=ServiceChainSetReconciled --namespace dpf-operator-system dpuservicechain --all
dpuservicechain.svc.dpu.nvidia.com/hbn-only-8xrrx condition met
To follow the progress of DPU provisioning, run the following command to check its current phase:
Jump Node Console
$ watch -n10 "kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase'"
Wait for the NodeEffect stage (at this point the provisioning is paused, waiting for external signal). Run following command on all/specific DPU nodemaintenance object/s to proceed with provisioning:
Jump Node Console
$ kubectl annotate dpunodemaintenances -n dpf-operator-system --all provisioning.dpu.nvidia.com/wait-for-external-nodeeffect=false --overwrite
To follow the progress of DPU provisioning, run the following command to check its current phase:
Jump Node Console
$ watch -n10 "kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase'"
Every 10.0s: kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase' setup5-jump: Wed Jan 7 10:47:25 2026
Dpu Node Name: dpu-node-mt2402xz0f7x
Last Transition Time: 2026-01-07T08:31:53Z
Type: BFBPrepared
Last Transition Time: 2026-01-07T08:31:49Z
Type: BFBReady
等待 Rebooted 阶段，然后手动对裸金属主机进行 Power Cycle。
DPU 启动后，对每个 DPU worker 运行以下命令：
跳转节点控制台
$ kubectl -n dpf-operator-system annotate dpunode dpu-node-mt2402xz0f7x dpu-node-mt2402xz0f80 dpu-node-mt2402xz0f9n dpu-node-mt2402xz0f8g provisioning.dpu.nvidia.com/dpunode-external-reboot-required-
此时，DPU worker 应已添加到集群。随着它们被添加到集群，DPU 被配置。
跳转节点控制台
$ watch -n10 "kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase'"
Every 10.0s: kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase' setup5-jump: Wed Jan 7 11:10:49 2026
Dpu Node Name: dpu-node-mt2402xz0f7x
Type: InternalIP
Type: Hostname
Last Transition Time: 2026-01-07T09:09:57Z
Type: Ready
Last Transition Time: 2026-01-07T08:31:53Z
Type: BFBPrepared
Last Transition Time: 2026-01-07T08:31:49Z
Type: BFBReady
Last Transition Time: 2026-01-07T08:36:38Z
Type: BFBTransferred
Last Transition Time: 2026-01-07T09:09:57Z
Type: DPUClusterReady
Last Transition Time: 2026-01-07T08:31:52Z
Type: FWConfigured
Last Transition Time: 2026-01-07T08:31:49Z
Type: Initialized
Last Transition Time: 2026-01-07T08:31:50Z
Type: InterfaceInitialized
Last Transition Time: 2026-01-07T08:31:49Z
Type: NodeEffectReady
Last Transition Time: 2026-01-07T09:09:57Z
Type: NodeEffectRemoved
Last Transition Time: 2026-01-07T08:43:33Z
Reason: OemLastState
Type: OSInstalled
Last Transition Time: 2026-01-07T09:09:57Z
Type: Rebooted
Phase: Ready
Dpu Node Name: dpu-node-mt2402xz0f80
Type: InternalIP
Type: Hostname
Last Transition Time: 2026-01-07T09:10:24Z
Type: Ready
Last Transition Time: 2026-01-07T08:31:52Z
Type: BFBPrepared
Last Transition Time: 2026-01-07T08:31:49Z
Type: BFBReady
Last Transition Time: 2026-01-07T08:36:33Z
Type: BFBTransferred
Last Transition Time: 2026-01-07T09:10:24Z
Type: DPUClusterReady
Last Transition Time: 2026-01-07T08:31:51Z
Type: FWConfigured
Last Transition Time: 2026-01-07T08:31:49Z
Type: Initialized
Last Transition Time: 2026-01-07T08:31:49Z
Type: InterfaceInitialized
Last Transition Time: 2026-01-07T08:31:49Z
Type: NodeEffectReady
Last Transition Time: 2026-01-07T09:10:24Z
Type: NodeEffectRemoved
Last Transition Time: 2026-01-07T08:43:19Z
Reason: OemLastState
Type: OSInstalled
Last Transition Time: 2026-01-07T09:10:24Z
Type: Rebooted
Phase: Ready
...
最后，验证所有不同的 DPU 相关对象现在都处于 Ready 状态：
跳转节点控制台
$ kubectl get secrets -n dpu-cplane-tenant1 dpu-cplane-tenant1-admin-kubeconfig -o json | jq -r '.data["admin.conf"]' | base64 --decode > /home/depuser/dpu-cluster.config
$ echo "alias ki='KUBECONFIG=/home/depuser/dpu-cluster.config kubectl'" >> ~/.bashrc
$ echo 'alias dpfctl="kubectl -n dpf-operator-system exec deploy/dpf-operator-controller-manager -- /dpfctl "' >> ~/.bashrc
$ dpfctl describe dpudeployments
NAME NAMESPACE STATUS REASON SINCE MESSAGE
DPFOperatorConfig/dpfoperatorconfig dpf-operator-system Ready: True Success 3m3s
└─DPUDeployments
└─DPUDeployment/hbn dpf-operator-system Ready: True Success 22s
├─DPUServiceChains
│ └─DPUServiceChain/hbn-wd7fs dpf-operator-system Ready: True Success 65s
├─DPUServiceInterfaces
│ └─3 DPUServiceInterfaces... dpf-operator-system Ready: True Success 70s See doca-hbn-p0-if-749n9, doca-hbn-p1-if-fn8w5, doca-hbn-pf0hpf-if-9s8c6
├─DPUSets
│ └─DPUSet/hbn-dpuset1 dpf-operator-system Ready: True Success 71s
│ ├─BFB/bf-bundle-v25.10.0 dpf-operator-system Ready: True Ready 39m File: bf-bundle-3.2.1-34_25.11_ubuntu-24.04_64k_prod.bfb, DOCA: 3.2.1
│ ├─DPUNodes
│ │ └─4 DPUNodes... dpf-operator-system Ready: True Ready 98s See dpu-node-mt2402xz0f7x, dpu-node-mt2402xz0f80, dpu-node-mt2402xz0f8g, dpu-node-mt2402xz0f9n
│ └─DPU
│ └─4 DPU... dpf-operator-system Ready: True DPUReady 98s See dpu-node-mt2402xz0f7x-mt2402xz0f7x, dpu-node-mt2402xz0f80-mt2402xz0f80,
│ dpu-node-mt2402xz0f8g-mt2402xz0f8g, dpu-node-mt2402xz0f9n-mt2402xz0f9n
└─Services
├─DPUServiceTemplates
│ └─DPUServiceTemplate/doca-hbn dpf-operator-system Ready: True Success 39m
└─DPUServices
└─1 DPUServices... dpf-operator-system Ready: True Success 50s See doca-hbn-jxkxw
$ ki get node -A
NAME STATUS ROLES AGE VERSION
dpu-node-mt2402xz0f7x-mt2402xz0f7x Ready <none> 5m18s v1.34.3
dpu-node-mt2402xz0f80-mt2402xz0f80 Ready <none> 6m12s v1.34.3
dpu-node-mt2402xz0f8g-mt2402xz0f8g Ready <none> 6m14s v1.34.3
dpu-node-mt2402xz0f9n-mt2402xz0f9n Ready <none> 6m22s v1.34.3
$ kubectl get dpu -A
NAMESPACE NAME READY PHASE AGE
dpf-operator-system dpu-node-mt2402xz0f7x-mt2402xz0f7x True Ready 36m
dpf-operator-system dpu-node-mt2402xz0f80-mt2402xz0f80 True Ready 36m
dpf-operator-system dpu-node-mt2402xz0f8g-mt2402xz0f8g True Ready 36m
dpf-operator-system dpu-node-mt2402xz0f9n-mt2402xz0f9n True Ready 36m
$ kubectl wait --for=condition=ready --namespace dpf-operator-system dpu --all
dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f7x-mt2402xz0f7x condition met
dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f80-mt2402xz0f80 condition met
dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f8g-mt2402xz0f8g condition met
dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f9n-mt2402xz0f9n condition met
$ ki get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dpf-operator-system dpu-cplane-tenant1-cni-installer-89kn4 1/1 Running 0 6m50s 10.244.2.3 dpu-node-mt2402xz0f80-mt2402xz0f80 <none> <none>
dpf-operator-system dpu-cplane-tenant1-cni-installer-s8h4z 1/1 Running 0 7m1s 10.244.0.5 dpu-node-mt2402xz0f9n-mt2402xz0f9n <none> <none>
dpf-operator-system dpu-cplane-tenant1-cni-installer-wb29j 1/1 Running 0 5m57s 10.244.3.2 dpu-node-mt2402xz0f7x-mt2402xz0f7x <none> <none>
dpf-operator-system dpu-cplane-tenant1-cni-installer-zhzqh 1/1 Running 0 6m53s 10.244.1.4 dpu-node-mt2402xz0f8g-mt2402xz0f8g <none> <none>
dpf-operator-system dpu-cplane-tenant1-doca-hbn-jxkxw-ds-5sbzs 2/2 Running 0 2m54s 10.244.0.6 dpu-node-mt2402xz0f9n-mt2402xz0f9n <none> <none>
dpf-operator-system dpu-cplane-tenant1-doca-hbn-jxkxw-ds-ftnpn 2/2 Running 0 2m54s 10.244.1.5 dpu-node-mt2402xz0f8g-mt2402xz0f8g <none> <none>
dpf-operator-system dpu-cplane-tenant1-doca-hbn-jxkxw-ds-gjsqq 2/2 Running 0 3m21s 10.244.3.4 dpu-node-mt2402xz0f7x-mt2402xz0f7x <none> <none>
dpf-operator-system dpu-cplane-tenant1-doca-hbn-jxkxw-ds-k78vb 2/2 Running 0 2m54s 10.244.2.4 dpu-node-mt2402xz0f80-mt2402xz0f80 <none> <none>
dpf-operator-system dpu-cplane-tenant1-nvidia-k8s-ipam-controller-5c77854fcc-grchr 1/1 Running 0 127m 10.244.0.3 dpu-node-mt2402xz0f9n-mt2402xz0f9n <none> <none>
dpf-operator-system dpu-cplane-tenant1-nvidia-k8s-ipam-node-ds-krgzw 1/1 Running 0 6m53s 10.244.1.2 dpu-node-mt2402xz0f8g-mt2402xz0f8g <none> <none>
dpf-operator-system dpu-cplane-tenant1-nvidia-k8s-ipam-node-ds-pr85m 1/1 Running 0 5m57s 10.244.3.3 dpu-node-mt2402xz0f7x-mt2402xz0f7x <none> <none>
dpf-operator-system dpu-cplane-tenant1-nvidia-k8s-ipam-node-ds-x4lfs 1/1 Running 0 7m1s 10.244.0.2 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpf-operator-system dpu-cplane-tenant1-nvidia-k8s-ipam-node-ds-zlzvf 1/1 Running 0 6m50s 10.244.2.2 dpu-node-mt2402xz0f80-mt2402xz0f80 dpf-operator-system dpu-cplane-tenant1-ovs-cni-arm64-bpljq 1/1 Running 0 7m1s 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpf-operator-system dpu-cplane-tenant1-ovs-cni-arm64-gls6h 1/1 Running 0 6m50s 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 dpf-operator-system dpu-cplane-tenant1-ovs-cni-arm64-j8wr4 1/1 Running 0 5m57s 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x dpf-operator-system dpu-cplane-tenant1-ovs-cni-arm64-kbrrn 1/1 Running 0 6m53s 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g dpf-operator-system dpu-cplane-tenant1-sfc-controller-node-ds-vmfq4 1/1 Running 0 5m57s 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x dpf-operator-system dpu-cplane-tenant1-sfc-controller-node-ds-x45nl 1/1 Running 0 6m53s 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g dpf-operator-system dpu-cplane-tenant1-sfc-controller-node-ds-xskh9 1/1 Running 0 7m1s 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpf-operator-system dpu-cplane-tenant1-sfc-controller-node-ds-zfmt5 1/1 Running 1 (5m46s ago) 6m50s 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 dpf-operator-system kube-flannel-ds-2shh7 1/1 Running 0 7m2s 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpf-operator-system kube-flannel-ds-42mlq 1/1 Running 0 6m54s 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g dpf-operator-system kube-flannel-ds-m7xgt 1/1 Running 0 5m58s 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x dpf-operator-system kube-flannel-ds-vd574 1/1 Running 0 6m52s 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 dpf-operator-system kube-multus-ds-d5kb4 1/1 Running 0 6m53s 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g dpf-operator-system kube-multus-ds-gnv88 1/1 Running 0 6m50s 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 dpf-operator-system kube-multus-ds-l66tm 1/1 Running 0 7m1s 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpf-operator-system kube-multus-ds-mh4cj 1/1 Running 0 5m57s 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x dpf-operator-system kube-sriov-device-plugin-64c29 1/1 Running 0 7m1s 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpf-operator-system kube-sriov-device-plugin-6js9j 1/1 Running 0 6m50s 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 dpf-operator-system kube-sriov-device-plugin-g5gkx 1/1 Running 0 6m53s 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g dpf-operator-system kube-sriov-device-plugin-lk4z7 1/1 Running 0 5m57s 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x kube-system coredns-66bc5c9577-gqn8d 1/1 Running 0 127m 10.244.0.4 dpu-node-mt2402xz0f9n-mt2402xz0f9n kube-system coredns-66bc5c9577-p2xnm 1/1 Running 0 127m 10.244.1.3 dpu-node-mt2402xz0f8g-mt2402xz0f8g kube-system kube-proxy-64865 1/1 Running 0 5m58s 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x kube-system kube-proxy-hvjjp 1/1 Running 0 6m52s 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 kube-system kube-proxy-qfbwh 1/1 Running 0 6m54s 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g kube-system kube-proxy-w9gg4 1/1 Running 0 7m2s 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n
Congratulations! The DPF system with the HBN service has been successfully installed.
DTS and BlueMan DPU Services Installation
This section focuses on provisioning NVIDIA® BlueField®-3 DPU using DPF, installing the DTS and BlueMan DPU Services on the second DPU in the first bare-metal host, and enabling a unified interface for accessing essential DPU information, health status, and telemetry metrics.
Before deploying the objects under doca-platform/dpuservices/dts-blueman/ directory, a few adjustments are required.
-
Export environment variables for the installation:
Jump Node Console
$ source manifests/00-env-vars/envvars.env -
Create a directory from where all the commands will be run:
Jump Node Console
$ mkdir /home/depuser/doca-platform/dpuservices/dts-blueman/ $ cd /home/depuser/doca-platform/dpuservices/dts-blueman/ -
Create the
DPUFlavorusing the following YAML:--- apiVersion: provisioning.dpu.nvidia.com/v1alpha1 kind: DPUFlavor metadata: name: dpf-provisioning-dts-blueman namespace: dpf-operator-system spec: bfcfgParameters: - UPDATE_ATF_UEFI=yes - UPDATE_DPU_OS=yes - WITH_NIC_FW_UPDATE=yes configFiles: - operation: override path: /etc/mellanox/mlnx-bf.conf permissions: "0644" raw: | ALLOW_SHARED_RQ="no" IPSEC_FULL_OFFLOAD="no" ENABLE_ESWITCH_MULTIPORT="yes" - operation: override path: /etc/mellanox/mlnx-ovs.conf permissions: "0644" raw: | CREATE_OVS_BRIDGES="no" OVS_DOCA="yes" - operation: override path: /etc/mellanox/mlnx-sf.conf permissions: "0644" raw: "" grub: kernelParameters: - console=hvc0 - console=ttyAMA0 - earlycon=pl011,0x13010000 - fixrttc - net.ifnames=0 - biosdevname=0 - iommu.passthrough=1 - cgroup_no_v1=net_prio,net_cls - hugepagesz=2048kB - hugepages=3072 nvconfig: - device: '*' parameters: - PF_BAR2_ENABLE=0 - PER_PF_NUM_SF=1 - PF_TOTAL_SF=20 - PF_SF_BAR_SIZE=10 - NUM_PF_MSIX_VALID=0 - PF_NUM_PF_MSIX_VALID=1 - PF_NUM_PF_MSIX=228 - INTERNAL_CPU_MODEL=1 - INTERNAL_CPU_OFFLOAD_ENGINE=0 - SRIOV_EN=1 - NUM_OF_VFS=46 - LAG_RESOURCE_ALLOCATION=1 - LINK_TYPE_P1=ETH - LINK_TYPE_P2=ETH - EXP_ROM_UEFI_x86_ENABLE=1 ovs: rawConfigScript: | _ovs-vsctl() { ovs-vsctl --no-wait --timeout 15 "$@" } _ovs-vsctl set Open_vSwitch . other_config:doca-init=true _ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000 _ovs-vsctl set Open_vSwitch . other_config:hw-offload=true _ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true _ovs-vsctl set Open_vSwitch . other_config:max-idle=20000 _ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000 _ovs-vsctl set Open_vSwitch . other_config:ctl-pipe-size=1024 _ovs-vsctl --if-exists del-br ovsbr1 _ovs-vsctl --if-exists del-br ovsbr2 _ovs-vsctl --may-exist add-br br-sfc _ovs-vsctl set bridge br-sfc datapath_type=netdev _ovs-vsctl set bridge br-sfc fail_mode=secure _ovs-vsctl --may-exist add-port br-sfc p0 _ovs-vsctl set Interface p0 type=dpdk _ovs-vsctl set Port p0 external_ids:dpf-type=physical _ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev _ovs-vsctl --may-exist add-br br-ovn _ovs-vsctl set bridge br-ovn datapath_type=netdev _ovs-vsctl br-set-external-id br-ovn bridge-id br-ovn _ovs-vsctl br-set-external-id br-ovn bridge-uplink puplinkbrovntobrsfc _ovs-vsctl --may-exist add-port br-ovn pf0hpf _ovs-vsctl set Interface pf0hpf type=dpdk -
Create the
DPUDeployment.yamlfile:--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUDeployment metadata: name: dts-blueman namespace: dpf-operator-system spec: dpus: bfb: bf-bundle-$TAG dpuSets: - nameSuffix: dpuset-dts-blueman nodeSelector: matchLabels: feature.node.kubernetes.io/dpu-enabled: "true" dpuSelector: provisioning.dpu.nvidia.com/dpudevice-service-name: dts-blueman flavor: dpf-provisioning-dts-blueman nodeEffect: noEffect: true services: dts: serviceTemplate: dts serviceConfiguration: dts blueman: serviceTemplate: blueman serviceConfiguration: blueman -
Create the
DPUServiceconfig_dts.yamlfile:
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
name: dts
namespace: dpf-operator-system
spec:
deploymentServiceName: "dts"
-
Create the
DPUServicetemplate_dts.yamlfile:--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceTemplate metadata: name: dts namespace: dpf-operator-system spec: deploymentServiceName: "dts" helmChart: source: repoURL: $HELM_REGISTRY_REPO_URL version: 1.0.8 chart: doca-telemetry -
Create the
DPUServiceconfig_blueman.yamlfile:--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceConfiguration metadata: name: blueman namespace: dpf-operator-system spec: deploymentServiceName: "blueman" -
Create the
DPUServicetemplate_blueman.yamlfile:--- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceTemplate metadata: name: blueman namespace: dpf-operator-system spec: deploymentServiceName: "blueman" helmChart: source: repoURL: $HELM_REGISTRY_REPO_URL version: 1.0.8 chart: doca-blueman -
Apply all of the YAML files mentioned above using the following command:
Jump Node Console
$ cat *.yaml | envsubst | kubectl apply -f - -
To follow the progress of DPU provisioning, run the following command several time (take 20-30 minutes) to check its current phase:
Jump Node Console
$ dpfctl describe dpudeployments ... │ │ └─DPU │ │ ├─DPU/dpu-node-mt2511600r8p-mt2511600r8p dpf-operator-system │ │ │ ├─Rebooted False WaitingForManualPowerCycleOrReboot 13m │ │ │ └─Ready False Rebooting 13m │ │ ├─DPU/dpu-node-mt2511600rc3-mt2511600rc3 dpf-operator-system │ │ │ ├─Rebooted False WaitingForManualPowerCycleOrReboot 11m │ │ │ └─Ready False Rebooting 11m │ │ ├─DPU/dpu-node-mt2511600rp1-mt2511600rp1 dpf-operator-system │ │ │ ├─Rebooted False WaitingForManualPowerCycleOrReboot 12m │ │ │ └─Ready False Rebooting 12m │ │ └─DPU/dpu-node-mt2511600ruh-mt2511600ruh dpf-operator-system │ │ ├─Rebooted False WaitingForManualPowerCycleOrReboot 13m │ │ └─Ready False Rebooting 13m ... -
Wait for the Rebooted stage and then Power Cycle the bare-metal host manual. After the DPU is up, run following command for each DPU worker:
Jump Node Console
$ kubectl -n dpf-operator-system annotate dpunode dpu-node-mt2511600rc3 dpu-node-mt2511600ruh dpu-node-mt2511600r8p dpu-node-mt2511600rp1 provisioning.dpu.nvidia.com/dpunode-external-reboot-required- -
At this point, the DPU workers should be added to the cluster. As they being added to the cluster, the DPU are provisioned.
Jump Node Console
$ dpfctl describe dpudeployments NAME NAMESPACE STATUS REASON SINCE MESSAGE DPFOperatorConfig/dpfoperatorconfig dpf-operator-system Ready: True Success 118s └─DPUDeployments └─2 DPUDeployments... dpf-operator-system Ready: True Success 3m49s See dts-blueman, hbn -
Finally, validate that all the different DPU-related objects are now in the Ready state:
Jump Node Console
$ echo "alias ki='KUBECONFIG=/home/depuser/dpu-cluster.config kubectl'" >> ~/.bashrc $ kubectl get secrets -n dpu-cplane-tenant1 dpu-cplane-tenant1-admin-kubeconfig -o json | jq -r '.data["admin.conf"]' | base64 --decode > /home/depuser/dpu-cluster.config $ ki get node -A NAME STATUS ROLES AGE VERSION dpu-node-mt2402xz0f7x-mt2402xz0f7x Ready <none> 113m v1.34.3 dpu-node-mt2402xz0f80-mt2402xz0f80 Ready <none> 114m v1.34.3 dpu-node-mt2402xz0f8g-mt2402xz0f8g Ready <none> 114m v1.34.3 dpu-node-mt2402xz0f9n-mt2402xz0f9n Ready <none> 114m v1.34.3 dpu-node-mt2511600r8p-mt2511600r8p Ready <none> 5m41s v1.34.3 dpu-node-mt2511600rc3-mt2511600rc3 Ready <none> 5m20s v1.34.3 dpu-node-mt2511600rp1-mt2511600rp1 Ready <none> 5m34s v1.34.3 dpu-node-mt2511600ruh-mt2511600ruh Ready <none> 5m56s v1.34.3 $ kubectl get dpu -A NAMESPACE NAME READY PHASE AGE dpf-operator-system dpu-node-mt2402xz0f7x-mt2402xz0f7x True Ready 118m dpf-operator-system dpu-node-mt2402xz0f80-mt2402xz0f80 True Ready 118m dpf-operator-system dpu-node-mt2402xz0f8g-mt2402xz0f8g True Ready 118m dpf-operator-system dpu-node-mt2402xz0f9n-mt2402xz0f9n True Ready 118m dpf-operator-system dpu-node-mt2511600r8p-mt2511600r8p True Ready 39m dpf-operator-system dpu-node-mt2511600rc3-mt2511600rc3 True Ready 39m dpf-operator-system dpu-node-mt2511600rp1-mt2511600rp1 True Ready 39m dpf-operator-system dpu-node-mt2511600ruh-mt2511600ruh True Ready 39m $ kubectl wait --for=condition=ready --namespace dpf-operator-system dpu --all dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f7x-mt2402xz0f7x condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f80-mt2402xz0f80 condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f8g-mt2402xz0f8g condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2402xz0f9n-mt2402xz0f9n condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2511600r8p-mt2511600r8p condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2511600rc3-mt2511600rc3 condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2511600rp1-mt2511600rp1 condition met dpu.provisioning.dpu.nvidia.com/dpu-node-mt2511600ruh-mt2511600ruh condition metCongratulations! The DTS and BlueMan services have been successfully deployed on the second DPU in the first bare-metal host.
Zero-Trust Mode Checking
Here's a step-by-step procedure to check the Zero-Trust Mode on your NVIDIA BlueField DPU from the host server, including the installation of the Mellanox Firmware Tools (MFT).
Warning Ubuntu 24.04 was installed on the servers.
-
Navigate to the NVIDIA 下载 Site: Open your web browser and go to the official NVIDIA Mellanox software downloads page.
-
Select the Latest Version for your OS:
-
Transfer and Extract MFT Tools on the Worker 1 BareMetal Host.
First Pod Console
root@worker1:~# tar -xvzf /tmp/mft-4.33.0-169-x86_64-deb.tgz -
Navigate into the Extracted Directory.
First Pod Console
root@worker1:~# cd mft-4.33.0-169-x86_64-deb/ -
Run
following commands.
First Pod Console
root@worker1:~# apt-get install gcc make dkms
root@worker1:~# ./install.sh
-
Start MST (Mellanox Software Tools) Service and Identify DPU Device Name.
First Pod Console
root@worker1:~# mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices Unloading MST PCI module (unused) - Success root@worker1:~# mst status MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt41692_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:2b:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 01 -
Perform Zero-Trust Checking.
First Pod Console
root@worker1:~# mlxprivhost -d 2b:00.0 q Host configurations ------------------- level : RESTRICTED Port functions status: ----------------------- disable_rshim : TRUE disable_tracer : TRUE disable_port_owner : TRUE disable_counter_rd : TRUE #Expected Zero-Trust Output.This is the most definitive confirmation.
level : RESTRICTEDmeans the host is in Zero-Trust Mode, and theTRUEflags confirm individual security restrictions are active. -
Check Firmware Access with
mlxfwmanager:First Pod Console
root@worker1:~# mlxfwmanager -d 2b:00.0 --query Querying Mellanox devices firmware ... Device #1: ---------- Device Type: BlueField3 Part Number: -- Description: PSID: PCI Device Name: 2b:00.0 Base MAC: N/A Versions: Current Available FW -- Status: Failed to open device"Failed to open device" indicates the host is blocked from accessing the DPU for firmware operations, a key aspect of Zero-Trust.
-
Check Device Configuration with
mlxconfig:First Pod Console
root@worker1:~# mlxconfig -d 2b:00.0 q Device #1: ---------- Device type: BlueField3 Name: 900-9D3B6-00CV-A_Ax Description: NVIDIA BlueField-3 B3220 P-Series FHHL DPU; 200GbE (default mode) / NDR200 IB; Dual-port QSFP112; PCIe Gen5.0 x16 with x16 PCIe extension option; 16 Arm cores; 32GB on-board DDR; integrated BMC; Crypto Enabled Device: 2b:00.0 Configurations: Next Boot ... ALLOW_RD_COUNTERS True(1) # No RO, but restricted by mlxprivhost ... PORT_OWNER True(1) # No RO, but restricted by mlxprivhost ... TRACER_ENABLE True(1) # No RO, but restricted by mlxprivhostMost configuration parameters will be prefixed with
RO(Read-Only). Parameters related to direct host control, likePORT_OWNER,ALLOW_RD_COUNTERS,TRACER_ENABLE, even if shown asTrue(1)for the DPU's internal capability, will be unenforcible by the host due to themlxprivhostrestrictions. The widespreadROstatus shows that the host cannot modify these configurations, reinforcing the DPU's autonomous and secure state. The few parameters withoutROare still overridden by themlxprivhostsecurity policy. -
Check Low-Level Hardware Access with
ethtool:First Pod Console
root@worker1:~# ethtool -d ens1f0np0 Cannot get register dump: Operation not supportedThis confirms the DPU is preventing deep, low-level hardware access from the host, aligning with Zero-Trust's isolation goals.
Conclusion
The command outputs of mlxprivhost, mlxfwmanager, mlxconfig (showing RO flags), and ethtool (showing "Operation not supported"), then your NVIDIA BlueField DPU is indeed operating in Zero-Trust Mode.
This means the host has significantly restricted privileges and cannot perform sensitive operations on the DPU, ensuring its security and isolation.
Infrastructure Bandwidth & Latency Validation
Verify the deployment and confirm that the DPU system achieves link-speed performance and low latency by running various tests:
- Iperf TCP—for bandwidth measurements
- RDMA—for bandwidth and latency measurements
- Network isolation
Each test is described in detail. At the end of each test, the achieved performance is displayed.
Notes Make sure that the servers are tuned for maximum performance (not covered in this document).
Performance and Isolation Tests
Now that the test deployment is running, perform bandwidth and latency performance tests between two bare-metal workload servers.
Ubuntu 24.04 was installed on the servers.
-
Before running the tests, check the Gateway address on each HBN pod:
Jump Node Console
$ ki -n dpf-operator-system get pod -o wide | grep doca-hbn dpu-cplane-tenant1-doca-hbn-jxkxw-ds-5sbzs 2/2 Running 0 15m 10.244.0.6 dpu-node-mt2402xz0f9n-mt2402xz0f9n <none> <none> dpu-cplane-tenant1-doca-hbn-jxkxw-ds-ftnpn 2/2 Running 0 15m 10.244.1.5 dpu-node-mt2402xz0f8g-mt2402xz0f8g <none> <none> dpu-cplane-tenant1-doca-hbn-jxkxw-ds-gjsqq 2/2 Running 0 16m 10.244.3.4 dpu-node-mt2402xz0f7x-mt2402xz0f7x <none> <none> dpu-cplane-tenant1-doca-hbn-jxkxw-ds-k78vb 2/2 Running 0 15m 10.244.2.4 dpu-node-mt2402xz0f80-mt2402xz0f80 <none> <none> $ ki exec -it -n dpf-operator-system dpu-cplane-tenant1-doca-hbn-jxkxw-ds-gjsqq -- bash Defaulted container "doca-hbn" out of: doca-hbn, hbn-sidecar, hbn-init (init) root@dpu-cplane-tenant1-doca-hbn-jxkxw-ds-gjsqq:/tmp# ip a s ... 9: vlan11@br_default: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master RED state UP group default qlen 1000 link/ether 0a:ff:4e:3e:99:24 brd ff:ff:ff:ff:ff:ff inet 10.0.121.2/29 scope global vlan11 valid_lft forever preferred_lft forever inet6 fe80::8ff:4eff:fe3e:9924/64 scope link valid_lft forever preferred_lft forever ... $ exit $ ki exec -it -n dpf-operator-system dpu-cplane-tenant1-doca-hbn-jxkxw-ds-k78vb -- bash Defaulted container "doca-hbn" out of: doca-hbn, hbn-sidecar, hbn-init (init) root@dpu-cplane-tenant1-doca-hbn-jxkxw-ds-k78vb:/tmp# ip a s ... 9: vlan11@br_default: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master RED state UP group default qlen 1000 link/ether 0e:7d:99:41:2e:11 brd ff:ff:ff:ff:ff:ff inet 10.0.121.10/29 scope global vlan11 valid_lft forever preferred_lft forever inet6 fe80::c7d:99ff:fe41:2e11/64 scope link valid_lft forever preferred_lft forever ... $ exit -
Connect to a first Workload Server console, install iperf, perftest, check DPU High Speed Interfaces, set route to ethernet and identify the relevant RDMA device:
First Pod Console
root@worker1:~# apt install iperf3 root@worker1:~# apt install perftest root@worker1:~# ip a s ... 6: ens1f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 58:a2:e1:73:69:e6 brd ff:ff:ff:ff:ff:ff altname enp43s0f0np0 ... root@worker1:~# ip route add 10.0.123.0/22 via 10.0.121.2 depuser@worker2:~$ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=5.35 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=5.10 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=5.15 ms root@worker1:~# rdma link | grep ens1f0np0 link mlx5_0/1 state DOWN physical_state DISABLED netdev ens1f0np0 -
Configure the
ens1f0np0
interface on Ubuntu 24.04 using iproute2.
Configuration Overview
| Interface | IP Address | Default Gateway |
|---|---|---|
| ens1f0np0 | 10.0.121.1/29 | 10.0.121.2/29 |
First Pod Console
# Bring up physical interfaces
root@worker1:~# ip link set dev ens1f0np0 up
# Assign IP addresses
root@worker1:~# ip addr add 10.0.121.1/29 dev ens1f0np0
# Set default route
root@worker1:~# ip route add default via 10.0.121.2 dev ens1f0np0
-
Using another console window, reconnect to the jump node and connect to a second Workload Server. From within the servers, install iperf, perftest, check DPU High Speed Interfaces, set route to ethernet and identify the relevant RDMA device:
First Pod Console
root@worker2:~# apt install iperf3 root@worker2:~# apt install perftest root@worker2:~# ip a s ... 6: ens1f0np0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000 link/ether 58:a2:e1:73:6a:58 brd ff:ff:ff:ff:ff:ff altname enp43s0f0np0 ... root@worker2:~# ip route add 10.0.123.0/22 via 10.0.121.10 depuser@worker2:~$ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=5.35 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=5.10 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=5.15 ms root@worker2:~# rdma link | grep ens1f0np0 link mlx5_0/1 state DOWN physical_state DISABLED netdev ens1f0np0 -
Configure the
ens1f0np0interface on Ubuntu 24.04 usingiproute2.Configuration Overview
Interface IP Address Default Gateway ens1f0np0 10.0.121.9/29 10.0.121.10/29 First Pod Console
# Bring up physical interfaces root@worker2:~# ip link set dev ens1f0np0 up # Assign IP addresses root@worker2:~# ip addr add 10.0.121.9/29 dev ens1f0np0 # Set default route root@worker2:~# ip route add default via 10.0.121.10 dev ens1f0np0
iPerf TCP Bandwidth Test
Move back to the first server console.
-
Start the
iperf3server side:First BM Server Console
root@worker1:~# iperf3 -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ -
Move to the second server console. Start the
iperfclient side:Second BM Server Console
root@worker2:~# iperf3 -c 10.0.121.1 -P 16 ------------------------------------------------------------ Client connecting to 10.0.121.1, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 9] local 10.0.121.9 port 48620 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/827) [ 10] local 10.0.121.9 port 48610 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/881) [ 1] local 10.0.121.9 port 48712 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/608) [ 14] local 10.0.121.9 port 48728 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/722) [ 11] local 10.0.121.9 port 48710 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/870) [ 4] local 10.0.121.9 port 48622 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/945) [ 7] local 10.0.121.9 port 48690 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/906) [ 15] local 10.0.121.9 port 48736 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/689) [ 2] local 10.0.121.9 port 48616 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/796) [ 3] local 10.0.121.9 port 48618 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/940) [ 12] local 10.0.121.9 port 48706 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/892) [ 16] local 10.0.121.9 port 48696 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/810) [ 8] local 10.0.121.9 port 48626 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/801) [ 6] local 10.0.121.9 port 48692 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/891) [ 5] local 10.0.121.9 port 48624 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/931) [ 13] local 10.0.121.9 port 48686 connected with 10.0.121.1 port 5001 (icwnd/mss/irtt=14/1448/903) [ ID] Interval Transfer Bandwidth [ 3] 0.0000-10.0058 sec 14.1 GBytes 12.1 Gbits/sec [ 13] 0.0000-10.0057 sec 14.2 GBytes 12.2 Gbits/sec [ 7] 0.0000-10.0056 sec 13.4 GBytes 11.5 Gbits/sec [ 12] 0.0000-10.0057 sec 15.2 GBytes 13.1 Gbits/sec [ 4] 0.0000-10.0058 sec 14.1 GBytes 12.1 Gbits/sec [ 11] 0.0000-10.0058 sec 15.8 GBytes 13.6 Gbits/sec [ 8] 0.0000-10.0057 sec 13.9 GBytes 11.9 Gbits/sec [ 9] 0.0000-10.0058 sec 13.8 GBytes 11.9 Gbits/sec [ 15] 0.0000-10.0057 sec 14.3 GBytes 12.3 Gbits/sec [ 16] 0.0000-10.0058 sec 14.6 GBytes 12.5 Gbits/sec [ 1] 0.0000-10.0057 sec 14.6 GBytes 12.6 Gbits/sec [ 6] 0.0000-10.0058 sec 13.1 GBytes 11.3 Gbits/sec [ 14] 0.0000-10.0059 sec 13.6 GBytes 11.6 Gbits/sec [ 10] 0.0000-10.0055 sec 13.5 GBytes 11.6 Gbits/sec [ 2] 0.0000-10.0057 sec 14.0 GBytes 12.0 Gbits/sec [ 5] 0.0000-10.0058 sec 14.6 GBytes 12.6 Gbits/sec [SUM] 0.0000-10.0010 sec 227 GBytes 195 Gbits/sec
RoCE Latency Test
Return to the first server console.
-
Start the
ib_read_latserver side:First BM Server Console
root@worker1:~# ib_read_lat -F -n 20000 -d mlx5_0 ************************************ * Waiting for client to connect... * ************************************ -
Move to the second server console. Start the
ib_read_latclient side:Second BM Server Console
root@worker2:~# ib_read_lat -F -n 20000 -d mlx5_0 10.0.121.1 --------------------------------------------------------------------------------------- RDMA_Read Latency Test Dual-port : OFF Device : mlx5_0 Number of qps : 1 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 1 Mtu : 1024[B] Link type : Ethernet GID index : 3 Outstand reads : 16 rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- local address: LID 0000 QPN 0x0048 PSN 0x77ae88 OUT 0x10 RKey 0x186ded VAddr 0x005fe0b3e3a000 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:00:121:09 remote address: LID 0000 QPN 0x0048 PSN 0x51948d OUT 0x10 RKey 0x186ded VAddr 0x00577584a67000 GID:
00:00:00:00:00:00:00:00:00:00:255:255:10:00:121:01
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 20000 3.98 65.30 4.08 7.89 7.17 31.51 36.33
---------------------------------------------------------------------------------------
RoCE Bandwidth Test
返回第一个服务器控制台。
-
启动
ib_write_bw服务器端：第一个BM服务器控制台
root@worker1:~# ib_write_bw -s 1048576 -F -D 30 -q 64 -d mlx5_0 ************************************ * Waiting for client to connect... * ************************************ -
切换到第二个服务器控制台。 启动
ib_write_bw客户端：第二个BM服务器控制台
root@worker2:~# ib_write_bw -s 1048576 -F -D 30 -q 64 -d mlx5_0 10.0.121.1 --report_gbit --------------------------------------------------------------------------------------- RDMA_Write BW Test Dual-port : OFF Device : mlx5_0 Number of qps : 64 Transport type : IB Connection type : RC Using SRQ : OFF PCIe relax order: ON ibv_wr* API : ON TX depth : 128 CQ Moderation : 1 Mtu : 1024[B] Link type : Ethernet GID index : 3 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet --------------------------------------------------------------------------------------- … --------------------------------------------------------------------------------------- #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 1048576 420000 0.00 220.72 0.026312 ---------------------------------------------------------------------------------------
Network Isolation Test
最后，验证运行在不同网络上的两台服务器——使用 PF0 和 PF0 的虚拟功能——无法相互通信。
连接到第一个工作负载服务器（使用 PF0 网络），并尝试 ping 第二个节点上的 PF0。
-
从 PF0 到 PF0 运行
ping命令：第一个BM服务器控制台
root@worker1:~# ping -c 3 10.0.121.9 PING 10.0.121.9 (10.0.121.9) 56(84) bytes of data. 64 bytes from 10.0.121.9: icmp_seq=1 ttl=62 time=0.896 ms 64 bytes from 10.0.121.9: icmp_seq=2 ttl=62 time=0.241 ms 64 bytes from 10.0.121.9: icmp_seq=3 ttl=62 time=0.258 ms -
尝试 ping 节点3和4 上的 PF0。从 PF0 到 PF0 运行
ping命令：第一个BM服务器控制台
root@worker1:~# ping -c 3 10.0.122.1 PING 10.0.122.1 (10.0.122.1) 56(84) bytes of data. From 10.0.121.2 icmp_seq=1 Destination Host Unreachable From 10.0.121.2 icmp_seq=2 Destination Host Unreachable From 10.0.121.2 icmp_seq=3 Destination Host Unreachable --- 10.0.122.1 ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2045ms root@worker1:~# ping -c 3 10.0.122.9 PING 10.0.122.9 (10.0.122.9) 56(84) bytes of data. From 10.0.121.2 icmp_seq=1 Destination Host Unreachable From 10.0.121.2 icmp_seq=2 Destination Host Unreachable From 10.0.121.2 icmp_seq=3 Destination Host Unreachable --- 10.0.122.9 ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2067ms
此ping操作应失败，因为HBN通过不同的VLAN、VNI和VRF实现了网络隔离。
DTS and BlueMan Services Verification
以下是检查 DTS 和 Blueman DPUServices 是否已部署在NVIDIA BlueField DPU上的逐步过程。
为了能够登录BlueMan并以便捷方式查看 本地DTS实例 数据，应将DPU的管理IP地址输入到与DPU位于同一网络的Web浏览器中。在本RDG中，将通过使用 RDP 连接到 Jump 节点并在其中打开Web浏览器（与MaaS、Firewall相同）来演示。
-
要找出
10.0.110.0/24子网中的DPU管理IP地址，请获取DPU名称。Jump节点控制台
$ kubectl get dpus -n dpf-operator-system NAME READY PHASE AGE dpu-node-mt2402xz0f7x-mt2402xz0f7x True Ready 150m dpu-node-mt2402xz0f80-mt2402xz0f80 True Ready 150m dpu-node-mt2402xz0f8g-mt2402xz0f8g True Ready 150m dpu-node-mt2402xz0f9n-mt2402xz0f9n True Ready 150m dpu-node-mt2511600r8p-mt2511600r8p True Ready 70m dpu-node-mt2511600rc3-mt2511600rc3 True Ready 70m dpu-node-mt2511600rp1-mt2511600rp1 True Ready 70m dpu-node-mt2511600ruh-mt2511600ruh True Ready 70m -
获取DPU管理IP：
Jump节点控制台
$ $ kubectl get dpus -n dpf-operator-system -o json \ | jq -r ' .items[] | "\(.metadata.name)\t\(.status.addresses[].address)" ' dpu-node-mt2402xz0f7x-mt2402xz0f7x 10.0.110.211 dpu-node-mt2402xz0f7x-mt2402xz0f7x dpu-node-mt2402xz0f7x-mt2402xz0f7x dpu-node-mt2402xz0f80-mt2402xz0f80 10.0.110.212 dpu-node-mt2402xz0f80-mt2402xz0f80 dpu-node-mt2402xz0f80-mt2402xz0f80 dpu-node-mt2402xz0f8g-mt2402xz0f8g 10.0.110.214 dpu-node-mt2402xz0f8g-mt2402xz0f8g dpu-node-mt2402xz0f8g-mt2402xz0f8g dpu-node-mt2402xz0f9n-mt2402xz0f9n 10.0.110.213 dpu-node-mt2402xz0f9n-mt2402xz0f9n dpu-node-mt2402xz0f9n-mt2402xz0f9n dpu-node-mt2511600r8p-mt2511600r8p 10.0.110.217 dpu-node-mt2511600r8p-mt2511600r8p dpu-node-mt2511600r8p-mt2511600r8p dpu-node-mt2511600rc3-mt2511600rc3 10.0.110.215 dpu-node-mt2511600rc3-mt2511600rc3 dpu-node-mt2511600rc3-mt2511600rc3 dpu-node-mt2511600rp1-mt2511600rp1 10.0.110.218 dpu-node-mt2511600rp1-mt2511600rp1 dpu-node-mt2511600rp1-mt2511600rp1 dpu-node-mt2511600ruh-mt2511600ruh 10.0.110.216 dpu-node-mt2511600ruh-mt2511600ruh dpu-node-mt2511600ruh-mt2511600ruh -
在RDP会话中，打开Web浏览器并输入 https://<DPU_INTERNAL_IP>。应出现自签名证书警告；点击接受风险并继续。 之后将打开登录页面：
登录凭据与用于SSH连接到DPU的凭据相同（
ubuntu/ubuntu）。但是，直接登录将不起作用，需要在浏览器中额外添加证书例外。 -
在浏览器中打开另一个标签页并输入 https://<DPU_INTERNAL_IP>:10000。它将再次提示自签名证书警告；点击接受风险以将其添加到浏览器中。
exception list. An error message similar to the following will be displayed, but it doesn't matter since it's an internal address to fetch resources from–in other words, the error message can be ignored.
-
Return to the BlueMan login page, enter the credentials, and you should be able to login.
Done.
Authors
 |
Boris KovalevBoris Kovalev has worked for the past several years as a 解决方案 Architect, focusing on NVIDIA Networking/Mellanox technology, and is responsible for complex machine learning, Big Data and advanced VMware-based cloud research and design. Boris previously spent more than 20 years as a senior consultant and solutions architect at multiple companies, most recently at VMware. He has written multiple reference designs covering VMware, machine learning, Kubernetes, and container solutions which are available at the NVIDIA Documents website. |
NVIDIA, the NVIDIA logo, and BlueField are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.™
2025 NVIDIA Corporation. All rights reserved.©