超擎数智超擎数智

[2026/01/05][AZ] RDG for DPF Host Trusted with OVN-Kubernetes v25.10

Created on September 17, 2025. This Reference Deployment Guide (RDG) provides detailed instructions for deploying a Kubernetes (K8s) cluster using the DOCA Platform Framework (DPF), focusing on setting up an accelerated OVN-Kubernetes service on NVIDIA BlueField-3 DPU to deliver secure, isolated, and hardware-accelerated environments.

文档目录

Created on September 17, 2025

Scope

This Reference Deployment Guide (RDG) provides detailed instructions for deploying a Kubernetes (K8s) cluster using the DOCA Platform Framework (DPF). The guide focuses on setting up an accelerated OVN-Kubernetes service on NVIDIA® BlueField®-3 DPU to deliver secure, isolated, and hardware-accelerated environments.

This guide is designed for experienced system administrators, system engineers, and solution architects who seek to deploy high-performance Kubernetes clusters with Host-Based Networking enabled on NVIDIA BlueField DPU.

Warning

  • This reference implementation, as the name implies, is a specific, opinionated deployment example designed to address the use case described above.
  • While other approaches may exist to implement similar solutions, this document provides a detailed guide for this particular method.

Abbreviations and Acronyms

Term Definition Term Definition
BFB BlueField Bootstream MAAS Metal as a Service
BGP Border Gateway Protocol OVN Open Virtual Network
CNI Container Network Interface RDG Reference Deployment Guide
CSI Container Storage Interface RDMA Remote Direct Memory Access
DOCA Data Center Infrastructure-on-a-Chip Architecture SFC Service Function Chaining
DPF DOCA Platform Framework SR-IOV Single Root Input/Output Virtualization
DPU Data Processing Unit TOR Top of Rack
DTS DOCA Telemetry Service VLAN Virtual LAN (Local Area Network)
GENEVE Generic Network Virtualization Encapsulation VRR Virtual Router Redundancy
IPAM IP Address Management VTEP Virtual Tunnel End Point
K8S Kubernetes

Introduction

The NVIDIA BlueField-3 Data Processing Unit (DPU) is a 400 Gb/s infrastructure compute platform designed for line-rate processing of software-defined networking, storage, and cybersecurity workloads. It combines powerful compute resources, high-speed networking, and advanced programmability to deliver hardware-accelerated, software-defined solutions for modern data centers.

NVIDIA DOCA unleashes the full potential of the BlueField platform by enabling rapid development of applications and services that offload, accelerate, and isolate data center workloads.

OVN-Kubernetes is a Kubernetes CNI network plugin that provides robust networking for Kubernetes clusters. Built on Open Virtual Network (OVN) and Open vSwitch (OVS), it supports hardware acceleration to offload OVS packet processing to NIC/DPU hardware. With OVS-DOCA, an extension of traditional OVS-DPDK and OVS-Kernel, accelerated OVN-Kubernetes delivers industry-leading performance, functionality, and efficiency. Running OVN-Kubernetes on the DPU reserves host CPUs exclusively for workloads, maximizing system resources.

Deploying and managing DPU and their associated DOCA services, especially at scale, presents operational challenges. Without a robust provisioning and orchestration system, tasks such as lifecycle management, service deployment, and network configuration for service function chaining (SFC) can quickly become complex and error prone. This is where the DOCA Platform Framework (DPF) comes into play.

DPF automates the full DPU lifecycle, streamlines the deployment of DOCA services, and simplifies advanced network configurations. With DPF, services such as HBN can be deployed seamlessly, allowing for efficient offloading and intelligent routing of traffic through the DPU data plane.

By leveraging DPF, users can scale and automate DPU management across Kubernetes customer environments - optimizing performance while simplifying operations.

As part of the reference implementation, open-source components outside the scope of DPF (e.g., MAAS, pfSense, Kubespray) are used to simulate a realistic customer deployment environment.

The guide includes the full end-to-end deployment process, including:

  • Infrastructure provisioning
  • DPF deployment
  • DPU provisioning
  • Service configuration and deployment
  • Service chaining

It also demonstrates some performance optimizations, with results validated through standard RDMA and TCP workload tests.

Warning We will deploy OVN-K8s over a simple bridged network, using a single highspeed uplink on each worker node

image-2025-5-27_10-32-5.png

If you are interested in a DPF deployment that incorporates DOCA's Host-Based Networking Service (HBN), utilizing ECMP-based dual uplinks and a large-scale BGP/EVPN fabric, please refer to this RDG that covers DPF with both the HBN and OVN-Kubernetes services and the deployment of additional DOCA Services.

References

解决方案架构

关键组件与技术

  • NVIDIA BlueField® 数据处理单元 (DPU) NVIDIA® BlueField® 数据处理单元 (DPU) 为现代数据中心和超级计算集群带来了前所未有的创新。凭借其强大的计算能力和集成软件定义的硬件加速器（用于网络、存储和安全），BlueField 为任何环境中的任何工作负载创建了安全且加速的基础设施，开启了加速计算和人工智能的新时代。

  • NVIDIA DOCA 软件框架 NVIDIA DOCA™ 释放了 NVIDIA® BlueField® 网络平台的潜力。通过利用 BlueField DPU 和 SuperNIC 的强大功能，DOCA 能够快速创建卸载、加速和隔离数据中心工作负载的应用程序和服务。它使开发人员能够创建软件定义的、云原生的、DPU 和 SuperNIC 加速的服务，并具有零信任保护，满足现代数据中心的性能和安全需求。

  • NVIDIA ConnectX 智能网卡 10/25/40/50/100/200 和 400G 以太网网卡 业界领先的 NVIDIA® ConnectX® 系列智能网卡 (SmartNIC) 提供先进的硬件卸载和加速功能。 NVIDIA 以太网网卡为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供最高的 ROI 和最低的总拥有成本。

  • NVIDIA LinkX 线缆 NVIDIA® LinkX® 产品系列包括线缆和收发器，提供业界最完整的 10、25、40、50、100、200 和 400GbE 以太网以及 100、200 和 400Gb/s InfiniBand 产品，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。

  • NVIDIA Spectrum 以太网交换机 灵活的外形，支持 16 到 128 个物理端口，速度从 1GbE 到 400GbE。 基于突破性的硅技术，针对性能和可扩展性进行了优化，NVIDIA Spectrum 交换机非常适合构建高性能、经济高效且高效的云数据中心网络、以太网存储结构和深度学习互连。 NVIDIA 将基于业界领先的专用集成电路 (ASIC) 技术的 NVIDIA Spectrum™ 交换机的优势与多种现代网络操作系统选择相结合，包括 NVIDIA Cumulus® Linux、SONiC 和 NVIDIA Onyx®。

  • NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux 是业界最具创新性的开放网络操作系统，可让您像其他系统一样自动化、定制和扩展数据中心网络。

  • NVIDIA Network Operator NVIDIA Network Operator 简化了 Kubernetes 集群中 NVIDIA 网络资源的配置和管理。该操作员自动安装所需的主机网络软件，汇集所有必要组件以提供高速网络连接。这些组件包括 NVIDIA 网络驱动程序、Kubernetes 设备插件、CNI 插件、IP 地址管理 (IPAM) 插件等。NVIDIA Network Operator 与 NVIDIA GPU Operator 协同工作，为可扩展的 GPU 计算集群提供高吞吐量、低延迟的网络。

  • Kubernetes Kubernetes 是一个开源容器编排平台，用于容器化应用程序的部署自动化、扩展和管理。

  • Kubespray Kubespray 由 Ansible playbooks、清单、配置工具和领域知识组成，用于通用 OS/Kubernetes 集群配置管理任务，并提供：

    • 高可用集群
    • 可组合属性
    • 支持大多数流行的 Linux 发行版

  • OVN-Kubernetes OVN-Kubernetes（开放虚拟网络 - Kubernetes）是一个开源项目，以 OVN（开放虚拟网络）和 Open vSwitch（开放虚拟交换机）为核心，为 Kubernetes 集群提供强大的网络解决方案。它是一个根据 CNI（容器网络接口）规范编写的 Kubernetes 网络合规插件。

  • RDMA RDMA 是一种允许网络中的计算机在不涉及任一计算机的处理器、缓存或操作系统的情况下交换数据的技术。 与本地 DMA 类似，RDMA 提高了吞吐量和性能，并释放了计算资源。

解决方案设计

解决方案逻辑设计

逻辑设计包括以下组件：

  • 1 个 Hypervisor 节点（基于 KVM），配备 ConnectX-7：
    • 1 个防火墙 VM
    • 1 个跳板 VM
    • 1 个 MaaS VM
    • 3 个 K8s Master VM，运行所有 K8s 管理组件
  • 2 个 Worker 节点（PCI Gen5），每个节点配备 1 个 BlueField-3 网卡
  • 单个高速 (HS) 交换机，1 个 L3 HS 底层网络
  • 1 Gb 主机管理网络

image-2025-6-4_9-54-18.png

K8s 集群逻辑设计

以下 K8s 逻辑设计示意图展示了 DPF 系统的主要组件，其中包括：

  • 3 个 K8s Master VM，运行所有 K8s 管理组件
  • 2 个 K8s Worker 节点 (x86)
  • 2 个 K8s DPU Worker，运行 OVN-K8s DOCA 服务
  • 1 个 Kamaji（K8s 控制平面管理器）
  • 1 个 DPU 控制平面（租户集群）
  • 连接到高速/1Gb 网络

image-2025-5-18_16-24-32-1.png

防火墙设计

本解决方案中的 pfSense 防火墙具有双重用途：

  • 防火墙 – 为 DPF 系统提供隔离环境，确保安全操作
  • 路由器 – 实现互联网访问以及主机管理网络与高速网络之间的连接
  • 高速网络的 DHCP 服务器

防火墙上配置了 SSH 和 RDP 的端口转发规则，用于路由

流量将被转发至跳转节点在主机管理网络中的 IP 地址。管理员可通过跳转节点管理和访问设置中的各种设备，并处理 Kubernetes (K8s) 集群及 DPF 组件的部署。

下图展示了本解决方案中使用的防火墙设计：

FW_Design_Updated_2.png

软件栈组件

SW_stack_OVN_only.png

注意： 请务必使用上述完全相同的软件栈版本。

物料清单

image-2025-6-5_10-58-39-1.png

部署与配置

节点与交换机定义

以下是部署所演示网络架构时使用的定义和参数：

交换机端口使用
主机名 机架 ID 端口
hs-switch 1 swp1-3
mgmt-switch 1 swp1-3
主机
机架 服务器类型 服务器名称 交换机端口 IP 和网卡 默认网关
Rack1 虚拟机管理节点 hypervisor mgmt-switch: swp1hs-switch: swp1 mgmt-br (接口 eno2): -hs-br (接口 ens2f0np0): -lab-br (接口 eno1): 可信 LAN IP 可信 LAN GW
Rack1 工作节点 worker1 mgmt-switch: swp2hs-switch: swp2 ens15f0: 10.0.110.21/24ens5f0np0: 10.0.120.0/22 10.0.110.254
Rack1 工作节点 worker2 mgmt-switch: swp3hs-switch: swp4 ens15f0: 10.0.110.22/24ens5f0np0: 10.0.120.0/22 10.0.110.254
Rack1 防火墙（虚拟） fw - LAN (mgmt-br): 10.0.110.254/24OPT1 (hs-br): 10.0.123.254/22WAN (lab-br): 可信 LAN IP 可信 LAN GW
Rack1 跳转节点（虚拟） jump - enp1s0: 10.0.110.253/24 10.0.110.254
Rack1 MaaS（虚拟） maas - enp1s0: 10.0.110.252/24 10.0.110.254
Rack1 主节点（虚拟） master1 - enp1s0: 10.0.110.1/24 10.0.110.254
Rack1 主节点（虚拟） master2 - enp1s0: 10.0.110.2/24 10.0.110.254
Rack1 主节点（虚拟） master3 - enp1s0: 10.0.110.3/24 10.0.110.254

布线

虚拟机管理节点

image-2025-5-18_16-17-13-1.png

K8s 工作节点

image-2025-6-5_12-14-8-1.png

网络架构配置

更新 Cumulus Linux

最佳实践是使用最新发布的 Cumulus Linux NOS 版本。

有关如何升级 Cumulus Linux 的信息，请参阅

参考Cumulus Linux用户指南。

配置Cumulus Linux交换机

SN3700交换机（hs-switch）配置如下：

以下命令在hs-switch上配置一个普通网桥。

nv set bridge domain br_default untagged 1
nv set interface swp1-3 link state up
nv set interface swp1-3 type swp
nv set interface swp1-3 bridge domain br_default
nv config apply -y

SN2201交换机（mgmt-switch）配置如下：

nv set bridge domain br_default untagged 1
nv set interface swp1-3 link state up
nv set interface swp1-3 type swp
nv set interface swp1-3 bridge domain br_default
nv config apply -y

主机配置

确保工作节点服务器的BIOS设置中启用了SR-IOV，并且服务器已调整为最大性能。

所有工作节点的BlueField-3网卡必须具有相同的PCIe位置，并且必须显示相同的接口名称。

虚拟机管理程序安装和配置

本参考部署指南（RDG）中使用的虚拟机管理程序基于Ubuntu 24.04和KVM。

虽然本文档不详细说明KVM安装过程，但需要注意的是，部署防火墙、Jump和MaaS虚拟机（VM）需要以下ISO：

  • Ubuntu 24.04
  • pfSense-CE-2.7.2

要实现该解决方案，必须在虚拟机管理程序上创建三个Linux网桥：

确保在受信任的LAN中为lab-br网桥接口配置DHCP记录，以分配IP地址。

  • lab-br – 将防火墙VM连接到受信任的LAN。
  • mgmt-br – 将各个VM连接到主机管理网络。
  • hs-br – 将防火墙VM连接到高速网络。

此外，必须在管理和高速网桥（mgmt-br和hs-br）及其上行接口上配置MTU为9000，以确保最佳性能。

network:
    ethernets:
        eno1:
            dhcp4: false
        eno2:
            dhcp4: false
            mtu: 9000
        ens2f0np0:
            dhcp4: false
            mtu: 9000
    bridges:
      lab-br:
         interfaces: [eno1]
         dhcp4: true
      mgmt-br:
         interfaces: [eno2]
         dhcp4: false
         mtu: 9000
      hs-br:
         interfaces: [ens2f0np0]
         dhcp4: false
         mtu: 9000
    version: 2

应用配置：

$ sudo netplan apply

准备基础设施服务器

防火墙VM - pfSense安装和接口配置

将pfSense CE（社区版）ISO下载到虚拟机管理程序，然后进行软件安装。

建议规格：

  • vCPU：2
  • RAM：2GB
  • 存储：10GB
  • 网络接口
    • 连接到lab-br的网桥设备
    • 连接到mgmt-br的网桥设备
    • 连接到hs-br的网桥设备

防火墙VM必须连接到虚拟机管理程序上的所有三个Linux网桥。在开始安装之前，请确保配置了三个类型为**“网桥设备”**的虚拟网络接口。每个接口应连接到不同的网桥（lab-br、mgmt-br和hs-br），如下图所示。

FW_VM_NIC.png

安装完成后，设置向导会显示一个菜单，其中包含多个选项，例如“分配接口”和“重启系统”。在此阶段，必须配置防火墙VM的网络接口。

  1. 选择选项2：“设置接口IP地址”，并按如下方式配置接口：
    • WAN – 受信任的LAN IP（静态/DHCP）
    • LAN – 静态IP 10.0.110.254/24
    • OPT1 – 静态IP 10.0.123.254/22
  2. 接口配置完成后，使用主机管理网络内的Web浏览器访问防火墙Web界面，完成配置。

接下来，继续安装Jump VM。该VM将作为运行浏览器的平台，用于访问防火墙的Web界面以进行安装后配置。

Jump VM

建议规格：

  • vCPU：4
  • RAM：8GB
  • 存储：25GB
  • 网络接口：网桥设备，连接到mgmt-br

步骤：

  1. 进行标准的Ubuntu 24.04安装。在此设置中的所有主机上使用以下登录凭据：

    • 用户名：depuser
    • 密码：user

  2. 通过创建以下Netplan配置来启用互联网连接和DNS解析：

    在MaaS VM安装和配置之前，使用10.0.110.254作为临时DNS名称服务器。完成MaaS安装后，更新Netplan文件，将此地址替换为MaaS IP：10.0.110.252。

    network:
    ethernets:
        enp1s0:
            dhcp4: false
            addresses: [10.0.110.253/24]
            nameservers:
              search: [dpf.rdg.local.domain]
              addresses: [10.0.110.254]
            routes:
              - to: default
                via: 10.0.110.254
    version: 2

  3. 应用配置：

    depuser@jump:~$ sudo netplan apply

  4. 更新和升级系统：

    depuser@jump:~$ sudo apt update -y
depuser@jump:~$ sudo apt upgrade -y

  5. 安装和配置Xfce桌面环境和XRDP（RDP的补充包）：

    depuser@jump:~$ sudo apt install -y xfce4 xfce4-goodies
depuser@jump:~$ sudo apt install -y lightdm-gtk-greeter
depuser@jump:~$ sudo apt install -y xrdp
depuser@jump:~$ echo "xfce4-session" | tee .xsession
depuser@jump:~$ sudo systemctl restart xrdp

  6. 安装Firefox以访问防火墙Web界面：

    depuser@jump:~$ sudo apt install -y firefox

跳板节点配置

  1. 安装Firefox浏览器：

    $ sudo apt install -y firefox

  2. 安装并配置NFS服务器，共享目录为/mnt/dpf_share：

    $ sudo apt install -y nfs-server
$ sudo mkdir -m 777 /mnt/dpf_share
$ sudo vi /etc/exports

  3. 在/etc/exports中添加以下行：

    /mnt/dpf_share 10.0.110.0/24(rw,sync,no_subtree_check)

  4. 重启NFS服务器：

    $ sudo systemctl restart nfs-server

  5. 在/mnt/dpf_share下创建bfb目录，权限与父目录相同：

    $ sudo mkdir -m 777 /mnt/dpf_share/bfb

  6. 为跳板节点上的depuser生成SSH密钥对（稍后将导入MaaS的管理用户，以实现对已配置服务器的无密码登录）：

    depuser@jump:~$ ssh-keygen -t rsa

  7. 重启跳板节点以显示图形用户界面：

    depuser@jump:~$ sudo reboot

    注意： 在防火墙上设置端口转发规则（后续步骤）后，即可远程登录跳板节点的图形界面。 本地图形控制台与RDP不能同时登录，切换到RDP连接时请先退出本地控制台。

防火墙VM – Web配置

从跳板节点打开Firefox浏览器，访问pfSense Web UI（http://10.0.110.254，默认凭据为admin/pfsense）。您将看到类似以下页面：

警告： 受信任LAN网络下的“DNS服务器”和“接口 - WAN”中的IP地址已模糊处理。

firewall_main_page_blur.png

请继续执行以下配置：

警告： 以下截图仅显示配置视图的一部分。请确保不要遗漏任何步骤！

  • 接口：

    • WAN (lab-br) – 勾选“启用接口”，取消勾选“阻止私有网络和环回地址”

    • LAN (mgmt-br) – 勾选“启用接口”，“IPv4配置类型”：静态IPv4（“IPv4地址”：10.0.110.254/24，“IPv4上游网关”：无），“MTU”：9000

    • OPT1 (hs-br) – 勾选“启用接口”，“IPv4配置类型”：静态IPv4（“IPv4地址”：10.0.123.254/22，“IPv4上游网关”：无），“MTU”：9000

      Firewall_LAN_Interface.png

  • 防火墙：

    • NAT -> 端口转发 -> 添加规则 -> “接口”：WAN，“地址族”：IPv4，“协议”：TCP，“目标”：WAN地址，“目标端口范围”：（“从端口”：SSH，“到端口”：SSH），“重定向目标IP”：（“类型”：地址或别名，“地址”：10.0.110.253），“重定向目标端口”：SSH，“描述”：NAT SSH

    • NAT -> 端口转发 -> 添加规则 -> “接口”：WAN，“地址族”：IPv4，“协议”：TCP，“目标”：WAN地址，“目标端口范围”：（“从端口”：MS RDP，“到端口”：MS RDP），“重定向目标IP”：（“类型”：地址或别名，“地址”：10.0.110.253），“重定向目标端口”：MS RDP，“描述”：NAT RDP

      pfsense_nat_forward_ssh.png

      Firewall_NAT_rules.png

  • 规则 -> OPT1 -> 添加规则 -> “动作”：通过，“接口”：OPT1，“地址族”：IPv4+IPv6，“协议”：任意，“源”：任意，“目标”：任意

    Firewall_OPT1_Rules.png

  • 服务

    • DHCP服务器 -> OPT1：启用DHCP服务器，设置地址池范围：10.0.120.1 - 10.0.123.253

      image-2025-5-26_17-14-6.png

      向下滚动到“其他DHCP选项”：

      • 网关："none"（我们不发送默认网关地址）
      • 域名："dpf.rdg.local.domain"

      image-2025-5-26_17-22-21.png

      向下滚动到“自定义DHCP选项”：

      • 添加选项编号 121，字符串值为 "20:a9:fe:63:64:0a:00:7b:fe"。该值编码了路由条目 "to 169.254.99.100/32 via 10.0.123.254"，DPF将使用该路由通过高速网络为OVN-K8s内部分配网关。
      • 添加选项编号 26，无符号16位整数值为 "9000"。这将把主机接口的MTU设置为9000。

      image-2025-5-27_13-46-54.png

MaaS VM

建议规格：

  • vCPU：4
  • 内存：4GB
  • 存储：50GB
  • 网络接口：桥接设备，连接到mgmt-br

步骤：

  1. 执行常规Ubuntu安装。

在 MaaS VM 上安装。

This command generates the following disks in the /var/lib/libvirt/images/ directory:

  • master1.qcow2
  • master2.qcow2
  • master3.qcow2
  1. Configure VMs in virt-manager:
    1. Open virt-manager and create three virtual machines:
      • Assign the corresponding virtual disk (master1.qcow2, master2.qcow2, or master3.qcow2) to each VM.
      • Configure each VM with the suggested specifications (vCPU, RAM, storage, and network interface).
    2. During the VM setup, ensure the NIC is selected under the Boot Options tab. This ensures the VMs can PXE boot for MaaS provisioning.
    3. Once the configuration is complete, shut down all the VMs.
  2. After the VMs are created and configured, proceed to provision them via the MaaS interface. MaaS will handle the OS installation and further setup as part of the deployment process.

Provision Master VMs and Worker Nodes Using MaaS

Master VMs
Install virsh and Set Up SSH Access

  1. SSH to the MaaS VM from the Jump node:

    depuser@jump:~$ ssh maas
depuser@maas:~$ sudo -i

  2. Install the virsh client to communicate with the hypervisor:

    # apt install -y libvirt-clients

  3. Generate an SSH key for the root user and copy it to the hypervisor user in the libvirtd group:

    # ssh-keygen -t rsa
# ssh-copy-id ubuntu@<hypervisor_MGMT_IP>

  4. Verify SSH access and virsh communication with the hypervisor:

    # virsh -c qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system list --all

    Expected output:

     Id   Name          State
------------------------------
 1    fw     running
 2    jump   running
 3    maas   running
 -    master1       shut off
 -    master2       shut off
 -    master3       shut off

  5. Copy the SSH key to the required MaaS directory (for snap-based installations):

    # mkdir -p /var/snap/maas/current/root/.ssh
# cp .ssh/id_rsa* /var/snap/maas/current/root/.ssh/
Get MAC Addresses of the Master VMs

Retrieve the MAC addresses of the Master VMs:

# for i in $(seq 1 3); do virsh -c qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system dumpxml master$i | grep 'mac address'; done

Example output:

<mac address='52:54:00:a9:9c:ef'/>
<mac address='52:54:00:19:6b:4d'/>
<mac address='52:54:00:68:39:7f'/>
Add Master VMs to MaaS

  1. Add the Master VMs to MaaS:

    Once added, MaaS will automatically start the newly added VMs commissioning (discovery and introspection).

    # maas admin machines create hostname=master1 architecture=amd64/generic mac_addresses='52:54:00:a9:9c:ef' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master1 skip_bmc_config=1 testing_scripts=none
Success.
Machine-readable output follows:
{
    "description": "",
    "status_name": "Commissioning",
...
    "status": 1,
...
    "system_id": "c3seyq",
...
    "fqdn": "master1.dpf.rdg.local.domain",
    "power_type": "virsh",
...
    "status_message": "Commissioning",
    "resource_uri": "/MAAS/api/2.0/machines/c3seyq/"
}

# maas admin machines create hostname=master2 architecture=amd64/generic mac_addresses='52:54:00:19:6b:4d' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master2 skip_bmc_config=1 testing_scripts=none

# maas admin machines create hostname=master3 architecture=amd64/generic mac_addresses='52:54:00:68:39:7f' power_type=virsh power_parameters_power_address=qemu+ssh://ubuntu@<hypervisor_MGMT_IP>/system power_parameters_power_id=master3 skip_bmc_config=1 testing_scripts=none

    Repeat the command for master2 and master3 with their respective MAC addresses.

  2. Verify commissioning by waiting for the status to change to "Ready" in MaaS.

    maas_masters_commission_virsh_updated.png

    After commissioning, the next phase is the deployment (OS provisioning).

Configure OVS Bridges on Master VMs

To be able to have persistency across reboots, create an OVS-bridge from each management interface of the master nodes and assign it a static IP address.

For each Master VM:

  1. Create an OVS bridge in the MaaS Network tab:

    1. Navigate to Network → Management Interface → Create Bridge.
    2. Configure as follows:

      1. Name: brenp1s0 (prefix br added to the interface name)

      2. Bridge Type: Open vSwitch (ovs)

      3. Subnet: 10.0.110.0/24

      4. IP Mode: Static Assign

      5. Address: Assign 10.0.110.1 for master1, 10.0.110.2 for master2, and 10.0.110.3 for master3.

        maas_master1_ovs_bridge_updated.png

  2. Save the interface settings for each VM.

Deploy Master VMs Using Cloud-Init

  1. Use the following cloud-init script to configure the necessary software and ensure OVS bridge persistency:

    Replace enp1s0 and brenp1s0 in the following cloud-init with your interface names as

显示在MaaS网络选项卡中。

Master节点cloud-init

#cloud-config
system_info:
  default_user:
    name: depuser
    passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/"
    lock_passwd: false
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
ssh_pwauth: True
package_upgrade: true
runcmd:
    - apt-get update
    - apt-get -y install openvswitch-switch nfs-common
    - |
      UPLINK_MAC=$(cat /sys/class/net/enp1s0/address)
      ovs-vsctl set Bridge brenp1s0 other-config:hwaddr=$UPLINK_MAC
      ovs-vsctl br-set-external-id brenp1s0 bridge-id brenp1s0 -- br-set-external-id brenp1s0 bridge-uplink enp1s0
  1. 部署Master虚拟机：

    1. 选中所有三个Master虚拟机 → Actions → Deploy。

    2. 切换 Cloud-init user-data 并粘贴cloud-init脚本。

    3. 开始部署，等待状态变为 "Ubuntu 24.04 LTS"。

      maas_master_vms_deployment_before.png

      maas_master_vms_deployment_complete_updated.png

验证部署

  • 从Jump节点SSH登录Master虚拟机：

    depuser@jump:~$ ssh master1
depuser@master1:~$

  • 运行sudo无需密码：

    depuser@master1:~$ sudo -i
root@master1:~#

  • 验证已安装的软件包：

    root@master1:~# apt list --installed | egrep 'openvswitch-switch|nfs-common'
nfs-common/noble,now 1:2.6.4-3ubuntu5.1 amd64 [installed]
openvswitch-switch/noble-updates,now 3.3.0-1ubuntu3.1 amd64 [installed]

  • 检查OVS网桥属性：

    root@master1:~# ovs-vsctl list bridge brenp1s0

    输出示例：

    ...
external_ids        : {bridge-id=brenp1s0, bridge-uplink=enp1s0, netplan="true", "netplan/global/set-fail-mode"=standalone, "netplan/mcast_snooping_enable"="false", "netplan/rstp_enable"="false"}
...
other_config        : {hwaddr="52:54:00:a9:9c:ef"}
...

  • 验证enp1s0和brenp1s0配置了9000 MTU（将enp1s0和brenp1s0替换为您的接口名称）：

    root@master1:~# ip a show enp1s0; ip a show brenp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
    link/ether 52:54:00:a9:9c:ef brd ff:ff:ff:ff:ff:ff
4: brenp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 52:54:00:a9:9c:ef brd ff:ff:ff:ff:ff:ff
    inet 10.0.110.1/24 brd 10.0.110.255 scope global brenp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fea9:9cef/64 scope link
       valid_lft forever preferred_lft forever

完成设置

重启Master虚拟机以完成配置。

root@master1:~# reboot

Worker节点

在MaaS中创建Worker机器

  1. 使用ipmi作为电源类型将Worker节点添加到MaaS。将占位符替换为您的具体IPMI凭据和IP地址：

    # maas admin machines create hostname=worker1 architecture=amd64 power_type=ipmi power_parameters_power_driver=LAN_2_0 power_parameters_power_user=<IPMI_username_worker1> power_parameters_power_pass=<IPMI_password_worker1> power_parameters_power_address=<IPMI_address_worker1>

    输出示例：

    ...
Success.
Machine-readable output follows:
{
    "description": "",
    "status_name": "Commissioning",
...
    "status": 1,
...
    "system_id": "pbskd3",
...
    "fqdn": "worker1.dpf.rdg.local.domain",
...
    "power_type": "ipmi",
...
    "resource_uri": "/MAAS/api/2.0/machines/pbskd3/"
}

  2. 使用相应的凭据重复worker2的命令：

    # maas admin machines create hostname=worker2 architecture=amd64 power_type=ipmi power_parameters_power_driver=LAN_2_0 power_parameters_power_user=<IPMI_username_worker2> power_parameters_power_pass=<IPMI_password_worker2> power_parameters_power_address=<IPMI_address_worker2>

添加后，MaaS将自动开始对Worker节点进行上线（发现和自检）。

创建内核参数标签

创建一个名为"Tag"的实体，用于配置Worker节点的内核参数。

  1. 在MaaS UI侧边栏中，转到 Organization → Tags → Create New Tag 并定义：

    • "Tag name": compute_performance
    • "Kernel options":

  2. 将isolcpus、nohz_full和rcu_nocbs的值替换为BlueField-3所连接的NUMA节点中的CPU核心：

    注意： 如果不确定BlueField连接在哪个NUMA节点，可以在Worker节点部署后执行此步骤（尽管需要重新部署）。

    Kernel options for worker nodes

[2026/01/05][AZ] RDG for DPF Host Trusted with OVN-Kubernetes v25.10

Adjust Network Settings

For each worker node, configure the network interfaces:

  • Management Adapter:
    • Go to Network → Select the host management adapter (e.g., ens15f0) → Create Bridge
    • Name: br-dpu
    • Bridge Type: Standard
    • Subnet: 10.0.110.0/24
    • IP Mode: DHCP
    • Save the interface
  • BlueField Adapter:
    • Select P0 on the BlueField adapter (e.g., ens5f0np0) → Actions → Edit Physical
    • Fabric: Fabric-1
    • Subnet: 20.20.20.0/24 (fake-dpf)
    • IP Mode: DHCP
    • Save the interface

Repeat these steps for the second worker node.

maas_edit_physical_interface.png

Deploy Worker Nodes Using Cloud-Init

  1. Use the following cloud-init script for deployment. Replace ens5f0np0 with your actual interface name:

    Worker node cloud-init

    #cloud-config
system_info:
  default_user:
    name: depuser
    passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/"
    lock_passwd: false
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
ssh_pwauth: True
package_upgrade: true
runcmd:
  - apt-get update
  - apt-get -y install nfs-common
  - sysctl --system
  - sed -i '/^\s*ens5f0np0:/,/^\s*mtu:/ { /^\s*mtu:/d }' /etc/netplan/*.yaml
  - netplan apply

  2. Deploy the worker nodes by selecting the worker nodes in MaaS → Actions → Deploy → Customize options → Enable Cloud-init user-data → Paste the cloud-init script → Deploy.

Verify Deployment

After the deployment is complete, verify that the worker nodes have been deployed successfully with the following commands:

  • SSH without password from the jump node:

    Jump Node Console

    depuser@jump:~$ ssh worker1
depuser@worker1:~$

  • Run sudo without password:

    Worker1 Console

    depuser@worker1:~$ sudo -i
root@worker1:~#

  • Validate that nfs-common package was installed:

    Worker1 Console

    root@worker1:~# apt list --installed | grep 'nfs-common'
nfs-common/noble,now 1:2.6.4-3ubuntu5.1 amd64 [installed]

  • /proc/cmdline is configured with the correct parameters and that IOMMU is indeed in passthrough mode:

    Worker1 Console

    root@worker1:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-90-generic root=UUID=60af5180-8c82-45cb-ba04-84a587d14317 ro intel_iommu=on iommu=pt numa_balancing=disable processor.max_cstate=0 isolcpus=28-55,84-111 nohz_full=28-55,84-111 rcu_nocbs=28-55,84-111

root@worker1:~# dmesg | grep 'type: Passthrough'
[    5.068360] iommu: Default domain type: Passthrough (set via kernel command line)

  • P0 interface has dhcp4 set to true and does not have mtu line in netplan configuration file:

    Worker1 Console

    root@worker1:~# cat /etc/netplan/50-cloud-init.yaml
network:
...
		ens5f0np0:
            dhcp4: true
            match:
                macaddress: a0:88:c2:46:78:c4
            set-name: ens5f0np0
...

  • ens15f0 and br-dpu are with 9000 MTU (replace ens15f0 with your interface name):

    Worker1 Console

    root@worker1:~# ip a show ens15f0; ip a show br-dpu
2: ens15f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master br-dpu state UP group default qlen 1000
    link/ether 04:32:01:60:0d:da brd ff:ff:ff:ff:ff:ff
    altname enp53s0f0
8: br-dpu: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 04:32:01:60:0d:da brd ff:ff:ff:ff:ff:ff
    inet 10.0.110.21/24 metric 100 brd 10.0.110.255 scope global dynamic br-dpu
       valid_lft 403sec preferred_lft 403sec
    inet6 fe80::632:1ff:fe60:dda/64 scope link
       valid_lft forever preferred_lft forever

Finalize Deployment

Reboot the worker nodes:

root@worker1:~# reboot

The infrastructure is now ready for the K8s deployment.

maas_worker_nodes_after_deployment_updated_2.png

K8s Cluster Deployment and Configuration

Kubespray Deployment and Configuration

In this solution, the Kubernetes (K8s) cluster is deployed using a modified Kubespray (based on tag v2.28.1) with a non-root depuser account from the Jump Node. The modifications in Kubespray are designed to meet the DPF prerequisites as described in the User Manual and facilitate cluster deployment and scaling.

  1. Download the modified Kubespray archive: modified_kubespray_v2.28.1.tar.gz.

  2. Extract the contents and navigate to the extracted directory:

    Jump Node Console

    $ tar -xzf /home/depuser/modified_kubespray_v2.28.1.tar.gz
$ cd kubespray/
depuser@jump:~/kubespray$

  3. Set the K8s API VIP address and DNS record. Replace it with your own.

IP地址和DNS记录（如果不同）：

Jump Node Console

depuser@jump:~/kubespray$ sed -i '/# kube_vip_address:/s/.*/kube_vip_address: 10.0.110.10/' inventory/mycluster/group_vars/k8s_cluster/addons.yml
depuser@jump:~/kubespray$ sed -i '/apiserver_loadbalancer_domain_name:/s/.*/apiserver_loadbalancer_domain_name: "kube-vip.dpf.rdg.local.domain"/' roles/kubespray_defaults/defaults/main/main.yml

  1. 安装必要的依赖并设置Python虚拟环境：

    Jump Node Console

    depuser@jump:~/kubespray$ sudo apt -y install python3-pip jq python3.12-venv
depuser@jump:~/kubespray$ python3 -m venv .venv
depuser@jump:~/kubespray$ source .venv/bin/activate
(.venv) depuser@jump:~/kubespray$ python3 -m pip install --upgrade pip
(.venv) depuser@jump:~/kubespray$ pip install -U -r requirements.txt
(.venv) depuser@jump:~/kubespray$ pip install ruamel-yaml

  2. 查看并编辑 inventory/mycluster/hosts.yaml 文件以定义集群节点。以下是本次部署的配置：

    注意：

    • 所有节点已按照DPF用户指南的先决条件进行了标记和注释。
    • 工作节点包含额外的kubelet配置，将在部署期间应用以实现最佳性能，允许：
      • 具有整数CPU requests 的 Guaranteed pod中的容器访问节点上的独占CPU。
      • 使用 reservedSystemCPUs 选项为系统预留一些核心（启用静态策略时，kubelet要求CPU预留大于零），并确保它们属于NUMA 0（因为示例中的网卡连接到NUMA节点1，如果网卡连接到NUMA节点0，则使用NUMA 1的核心）。
      • 将拓扑定义为 single-numa-node，以便仅当所有请求的CPU和设备都可以从恰好一个NUMA节点分配时才允许pod被接纳。
    • kube_node 组下的工作节点用 # 标记，以便在开始时仅部署控制平面节点（工作节点将在安装DPF系统所需的各种组件后添加）。

    inventory/mycluster/hosts.yaml

    all:
  hosts:
    master1:
      ansible_host: 10.0.110.1
      ip: 10.0.110.1
      access_ip: 10.0.110.1
      node_labels:
        "k8s.ovn.org/zone-name": "master1"
    master2:
      ansible_host: 10.0.110.2
      ip: 10.0.110.2
      access_ip: 10.0.110.2
      node_labels:
        "k8s.ovn.org/zone-name": "master2"
    master3:
      ansible_host: 10.0.110.3
      ip: 10.0.110.3
      access_ip: 10.0.110.3
      node_labels:
        "k8s.ovn.org/zone-name": "master3"
    worker1:
      ansible_host: 10.0.110.21
      ip: 10.0.110.21
      access_ip: 10.0.110.21
      node_labels:
        "node-role.kubernetes.io/worker": ""
        "k8s.ovn.org/dpu-host": ""
        "k8s.ovn.org/zone-name": "worker1"
      node_annotations:
        "k8s.ovn.org/remote-zone-migrated": "worker1"
      kubelet_cpu_manager_policy: static
      kubelet_topology_manager_policy: single-numa-node
      kubelet_reservedSystemCPUs: 0-7
    worker2:
      ansible_host: 10.0.110.22
      ip: 10.0.110.22
      access_ip: 10.0.110.22
      node_labels:
        "node-role.kubernetes.io/worker": ""
        "k8s.ovn.org/dpu-host": ""
        "k8s.ovn.org/zone-name": "worker2"
      node_annotations:
        "k8s.ovn.org/remote-zone-migrated": "worker2"
      kubelet_cpu_manager_policy: static
      kubelet_topology_manager_policy: single-numa-node
      kubelet_reservedSystemCPUs: 0-7
  children:
    kube_control_plane:
      hosts:
        master1:
        master2:
        master3:
    kube_node:
      hosts:
#       worker1:
#       worker2:
    etcd:
      hosts:
        master1:
        master2:
        master3:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:

使用Kubespray Ansible Playbook部署集群

  1. 从Jump Node运行以下命令以启动部署过程：

    注意： 确保在Python虚拟环境（.venv）中运行该命令。

    Jump Node Console

    (.venv) depuser@jump:~/kubespray$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

  2. 此部署需要一段时间才能完成。确保没有错误。成功结果示例：

    image-2025-9-15_16-24-5.png

    提示： 建议保持运行Kubespray的shell打开，稍后在执行集群扩展以添加工作节点时会用到。

K8s部署验证

为简化从Jump Host管理K8s集群，设置带有bash自动补全的 kubectl。

  1. 将 kubectl 和kubeconfig文件从 master1 复制到Jump Host：

    Jump Node Console

    ## 连接到master1
depuser@jump:~$ ssh master1
depuser@master1:~$ cp /usr/local/bin/kubectl /tmp/
depuser@master1:~$ sudo cp /root/.kube/config /tmp/kube-config
depuser@master1:~$ sudo chmod 644 /tmp/kube-config

  2. 在另一个终端标签页中，将文件复制到Jump Host：

    Jump Node Console

    depuser@jump:~$ scp master1:/tmp/kubectl /tmp/
depuser@jump:~$ sudo chown root:root /tmp/kubectl
depuser@jump:~$ sudo mv /tmp/kubectl /usr/local/bin/
depuser@jump:~$ mkdir -p ~/.kube
depuser@jump:~$ scp master1:/tmp/kube-config ~/.kube/config
depuser@jump:~$ chmod 600 ~/.kube/config

  3. 启用 kubectl 的bash自动补全：

    1. 验证bash-completion是否已安装：

      Jump Node Console

      depuser@jump:~$ type _init_completion

      如果已安装，输出将包含：

      Jump Node Console

      _init_completion is a function

    2. 如果未安装，请安装：

      Jump Node Console

      depuser@jump:~$ sudo apt install -y bash-completion

  4. 设置 kubectl 补全脚本：

    跳板机控制台

    depuser@jump:~$ kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl > /dev/null
depuser@jump:~$ bash

  5. 检查集群中节点的状态：

    跳板机控制台

    depuser@jump:~$ kubectl get nodes

    注意： 节点将处于 NotReady 状态，因为部署未包含 CNI 组件。

    跳板机控制台

    NAME      STATUS     ROLES           AGE   VERSION
master1   NotReady   control-plane   42m   v1.31.12
master2   NotReady   control-plane   41m   v1.31.12
master3   NotReady   control-plane   41m   v1.31.12

  6. 检查所有命名空间中的 Pod：

    跳板机控制台

    depuser@jump:~$ kubectl get pods -A

    注意： coredns 和 dns-autoscaler Pod 将处于 Pending 状态，因为缺少 CNI 组件。

    跳板机控制台

    NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-776bb9db5d-ndr7j          0/1     Pending   0          41m
kube-system   dns-autoscaler-6ffb84bd6-xj9bv    0/1     Pending   0          41m
kube-system   kube-apiserver-master1            1/1     Running   0          43m
kube-system   kube-apiserver-master2            1/1     Running   0          42m
kube-system   kube-apiserver-master3            1/1     Running   0          42m
kube-system   kube-controller-manager-master1   1/1     Running   1          43m
kube-system   kube-controller-manager-master2   1/1     Running   1          42m
kube-system   kube-controller-manager-master3   1/1     Running   1          42m
kube-system   kube-scheduler-master1            1/1     Running   1          43m
kube-system   kube-scheduler-master2            1/1     Running   1          42m
kube-system   kube-scheduler-master3            1/1     Running   1          42m
kube-system   kube-vip-master1                  1/1     Running   0          43m
kube-system   kube-vip-master2                  1/1     Running   0          42m
kube-system   kube-vip-master3                  1/1     Running   0          42m

DPF 安装

软件先决条件和必需变量

  1. 首先安装剩余的软件先决条件。

    跳板机控制台

    ## 连接到 master1 以复制 kubespray 部署期间安装的 helm 客户端工具
$ depuser@jump:~$ ssh master1
depuser@master1:~$ cp /usr/local/bin/helm /tmp/

## 在另一个标签页中
depuser@jump:~$ scp master1:/tmp/helm /tmp/
depuser@jump:~$ sudo chown root:root /tmp/helm
depuser@jump:~$ sudo mv /tmp/helm /usr/local/bin/

## 验证 envsubst 工具是否已安装
depuser@jump:~$ which envsubst
/usr/bin/envsubst

  2. 继续克隆 doca-platform Git 仓库：

    跳板机控制台

    $ git clone https://github.com/NVIDIA/doca-platform.git

  3. 切换到 doca-platform 目录并检出到 标签 v25.10.0：

    $ cd doca-platform/
$ git checkout v25.10.0

  4. 切换到 readme.md 所在的目录，所有命令将在此运行：

    跳板机控制台

    $ cd docs/public/user-guides/host-trusted/use-cases/ovnk/

  5. 使用以下文件定义安装所需的变量：

    警告： 将以下文件中的变量值替换为适合您设置的值。特别注意 DPU_P0、DPU_P0_VF1 和 DPUCLUSTER_INTERFACE。

    manifests/00-env-vars/envvars.env

    ## 目标集群（安装 DPF）的 Kubernetes API 服务器的 IP 地址。
## 不应包含协议或端口。
## 例如 10.10.10.10
export TARGETCLUSTER_API_SERVER_HOST=10.0.110.10

## 目标集群（安装 DPF）的 Kubernetes API 服务器的端口。
export TARGETCLUSTER_API_SERVER_PORT=6443

## 目标集群（安装 DPF）中主机的 IP 地址范围。
## 格式为 CIDR，例如 10.10.10.0/24
export TARGETCLUSTER_NODE_CIDR=10.0.110.0/24

## OVN Kubernetes 使用的 VTEP 的 IP 地址范围。应与为高速网络提供服务的 DHCP 服务器中使用的 VTEP CIDR 对齐。
## 在不同机架使用不同范围的配置中，该值应设置为包含所有这些范围的超集 CIDR。
## 格式为 CIDR，例如 10.0.120.0/22
export VTEP_CIDR=10.0.120.0/22

## DPU 集群负载均衡器使用的虚拟 IP。必须是管理子网中的保留 IP，且不由 DHCP 分配。
export DPUCLUSTER_VIP=10.0.110.200

## DPU_P0 是 DPU 的第一个端口的名称。此名称在所有工作节点上必须相同。
export DPU_P0=ens5f0np0

## DPU_P0_VF1 是 DPU 第一个端口的第二个虚拟功能（VF）的名称。此名称在所有工作节点上必须相同。
## 注意：VF 将在 DPU 配置完成且“主机网络配置”阶段完成后创建。
export DPU_P0_VF1=ens5f0v1

## DPUCluster 负载均衡器将监听的接口/网桥。应为控制平面节点的管理接口/网桥。
export DPUCLUSTER_INTERFACE=brenp1s0

## 用作 BFB 存储的 NFS 服务器的 IP 地址。
export NFS_SERVER_IP=10.0.110.253

## NVIDIA Helm 图表仓库的仓库 URL。
## 通常为 NVIDIA Helm NGC 仓库。出于开发目的，可设置为其他仓库。
export HELM_REGISTRY_REPO_URL=https://helm.ngc.nvidia.com/nvidia/doca

## OVN-Kubernetes Helm 图表的仓库 URL。
## 通常为 NVIDIA GHCR 仓库。出于开发目的，可设置为其他仓库。
export OVN_KUBERNETES_REPO_URL=oci://ghcr.io/nvidia

## POD_CIDR 是目标 Kubernetes 集群中 Pod 使用的 CIDR。
export POD_CIDR=10.233.64.0/18

## SERVICE_CIDR 是目标 Kubernetes 集群中服务使用的 CIDR。
## 格式为 CIDR，例如 10.10.10.0/24
export SERVICE_CIDR=10.233.0.0/18

## DPF REGISTRY 是 DPF Operator Chart 所在的 Helm 仓库 URL。
## 通常为 NVIDIA Helm NGC 仓库。出于开发目的，可设置为其他仓库。
export REGISTRY=https://helm.ngc.nvidia.com/nvidia/doca

## DPF TAG 是本指南中将部署的 DPF 组件的版本。
export TAG=v25.10.0

## 在 `bfb.yaml` 中使用并由 DPUSet 链接的 BFB 的 URL。
export

BFB_URL="https://content.mellanox.com/BlueField/BFBs/Ubuntu24.04/bf-bundle-3.2.1-34_25.11_ubuntu-24.04_64k_prod.bfb"

  1. Export environment variables for the installation:

    Jump Node Console

    $ source manifests/00-env-vars/envvars.env

CNI Installation

OVN Kubernetes is used as the primary CNI for the cluster. On worker nodes, the primary CNI will be accelerated by offloading work to the DPU. On control plane nodes, OVN Kubernetes will run without offloading.

  1. Create the NS for the CNI:

    Jump Node Console

    $ kubectl create ns ovn-kubernetes

  2. Install the OVN Kubernetes CNI components from the helm chart substituting the environment variables with the ones we defined before.

    Note: Note that MTU field with value of 8940 has been added to the yaml to override the default value and to be able to achieve better performance results.

    manifests/01-cni-installation/helm-values/ovn-kubernetes.yml

    commonManifests:
  enabled: true
nodeWithoutDPUManifests:
  enabled: true
controlPlaneManifests:
  enabled: true
nodeWithDPUManifests:
  enabled: true
  nodeMgmtPortNetdev: $DPU_P0_VF1
  dpuServiceAccountNamespace: dpf-operator-system
gatewayOpts: --gateway-interface=$DPU_P0
k8sAPIServer: https://$TARGETCLUSTER_API_SERVER_HOST:$TARGETCLUSTER_API_SERVER_PORT
## Note this CIDR is followed by a trailing /24 which informs OVN Kubernetes on how to split the CIDR per node.
podNetwork: $POD_CIDR/24
serviceNetwork: $SERVICE_CIDR
mtu: 8940

  3. Run the following command:

    Jump Node Console

    $ envsubst < manifests/01-cni-installation/helm-values/ovn-kubernetes.yml | helm upgrade --install -n ovn-kubernetes ovn-kubernetes

  4. Verify the CNI installation:

    Note: The following verification commands may need to be run multiple times to ensure the condition is met.

    Jump Node Console

    $ kubectl wait --for=condition=ready --namespace ovn-kubernetes pods --all --timeout=300s
pod/ovn-kubernetes-cluster-manager-54b48f96d4-9b29q condition met
pod/ovn-kubernetes-node-7bg2p condition met
pod/ovn-kubernetes-node-jfmbh condition met
pod/ovn-kubernetes-node-pt75h condition met

$ kubectl wait --for=condition=ready nodes --all
node/master1 condition met
node/master2 condition met
node/master3 condition met

DPF Operator Installation

Create Storage Required by the DPF Operator

  • YAML

    ---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: bfb-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  nfs:
    path: /mnt/dpf_share/bfb
    server: $NFS_SERVER_IP
  persistentVolumeReclaimPolicy: Delete
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: bfb-pvc
  namespace: dpf-operator-system
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeMode: Filesystem
  storageClassName: ""

  • Run the following command to substitute the environment variables using envsubst and apply the yaml files:

    Jump Node Console

    $ kubectl create ns dpf-operator-system
$ cat manifests/02-dpf-operator-installation/*.yaml | envsubst | kubectl apply -f -

Additional Dependencies

  1. The DPF Operator requires several prerequisite components to function properly in a Kubernetes environment. Starting with DPF v25.7, all Helm dependencies have been removed from the DPF chart. This means that all dependencies must be installed manually before installing the DPF chart itself. The following commands describe an opiniated approach to install those dependencies (for more information, check: Helm Prerequisites - NVIDIA Docs).

    1. Install helmfile binary:

      $ wget https://github.com/helmfile/helmfile/releases/download/v1.1.2/helmfile_1.1.2_linux_amd64.tar.gz
$ tar  -xvf helmfile_1.1.2_linux_amd64.tar.gz
$ sudo mv ./helmfile /usr/local/bin/

    2. Change directory to doca-platform:

      Note: Use another shell from the one where you run all the other installation commands for DPF.

      $ cd doca-platform/

    3. Install Helm dependencies using the following command:

      $ make HELMFILE_FILE=deploy/helmfiles/prereqs.yaml test-deploy-helmfile

  2. Ensure that the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables are set in the node-feature-discovery-worker DaemonSet:

    Warning: Run this command from the previous shell where the environment variables were exported.

    $ kubectl -n dpf-operator-system set env daemonset/node-feature-discovery-worker \

KUBERNETES_SERVICE_HOST=$TARGETCLUSTER_API_SERVER_HOST
KUBERNETES_SERVICE_PORT=$TARGETCLUSTER_API_SERVER_PORT

DPF Operator 部署

  1. 运行以下命令替换环境变量并安装DPF Operator：

    跳板机控制台

    $ helm repo add --force-update dpf-repository ${REGISTRY}
$ helm repo update
$ helm upgrade --install -n dpf-operator-system dpf-operator dpf-repository/dpf-operator --version=$TAG

  2. 验证DPF Operator安装，确保部署可用且所有Pod就绪：

    注意： 以下验证命令可能需要多次运行以确保条件满足。

    跳板机控制台

    $ kubectl rollout status deployment --namespace dpf-operator-system dpf-operator-controller-manager
deployment "dpf-operator-controller-manager" successfully rolled out

$ kubectl wait --for=condition=ready --namespace dpf-operator-system pods --all
pod/argo-cd-argocd-application-controller-0 condition met
pod/argo-cd-argocd-redis-6c6b84f6fb-xj5jg condition met
pod/argo-cd-argocd-repo-server-65cfb96746-r2rmr condition met
pod/argo-cd-argocd-server-5bbdb4b6b9-4dwhm condition met
pod/dpf-operator-controller-manager-5dd7555c6d-dqmdt condition met
pod/kamaji-95587fbc7-sn45q condition met
pod/kamaji-etcd-0 condition met
pod/kamaji-etcd-1 condition met
pod/kamaji-etcd-2 condition met
pod/maintenance-operator-74bd5774b7-lssgq condition met
pod/node-feature-discovery-gc-6b48f49cc4-6mmsd condition met
pod/node-feature-discovery-master-747d789485-d5x2s condition met

DPF 系统安装

本节介绍创建DPF系统组件以及运行DPF集群所需的基本基础设施。

  1. 以下YAML文件定义了用于安装DPF系统组件的 DPFOperatorConfig 和作为DPU节点Kubernetes控制平面的 DPUCluster。

    注意： 为获得高性能结果，需要调整 operatorconfig.yaml 以支持MTU 9000。

    manifests/03-dpf-system-installation/operatorconfig.yaml

    ---
apiVersion: operator.dpu.nvidia.com/v1alpha1
kind: DPFOperatorConfig
metadata:
  name: dpfoperatorconfig
  namespace: dpf-operator-system
spec:
  overrides:
    kubernetesAPIServerVIP: $TARGETCLUSTER_API_SERVER_HOST
    kubernetesAPIServerPort: $TARGETCLUSTER_API_SERVER_PORT
  provisioningController:
    bfbPVCName: "bfb-pvc"
    dmsTimeout: 900
  kamajiClusterManager:
    disable: false
  networking:
    controlPlaneMTU: 9000
    highSpeedMTU: 9000

    manifests/03-dpf-system-installation/dpucluster.yaml

    ---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUCluster
metadata:
  name: dpu-cplane-tenant1
  namespace: dpu-cplane-tenant1
spec:
  type: kamaji
  maxNodes: 10
  clusterEndpoint:
    # deploy keepalived instances on the nodes that match the given nodeSelector.
    keepalived:
      # interface on which keepalived will listen. Should be the oob interface of the control plane node.
      interface: $DPUCLUSTER_INTERFACE
      # Virtual IP reserved for the DPU Cluster load balancer. Must not be allocatable by DHCP.
      vip: $DPUCLUSTER_VIP
      # virtualRouterID must be in range [1,255], make sure the given virtualRouterID does not duplicate with any existing keepalived process running on the host
      virtualRouterID: 126
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""

  2. 为DPU节点的Kubernetes控制平面创建命名空间：

    跳板机控制台

    $ kubectl create ns dpu-cplane-tenant1

  3. 应用上述YAML文件：

    跳板机控制台

    $ cat manifests/03-dpf-system-installation/*.yaml | envsubst | kubectl apply -f -

  4. 验证DPF系统，确保provisioning和DPUService controller manager部署可用，DPF Operator系统中的所有其他部署可用，并且DPUCluster已准备好让节点加入。

    跳板机控制台

    $ kubectl rollout status deployment --namespace dpf-operator-system dpf-provisioning-controller-manager dpuservice-controller-manager
deployment "dpf-provisioning-controller-manager" successfully rolled out
deployment "dpuservice-controller-manager" successfully rolled out

$ kubectl rollout status deployment --namespace dpf-operator-system
deployment "argo-cd-argocd-applicationset-controller" successfully rolled out
deployment "argo-cd-argocd-redis" successfully rolled out
deployment "argo-cd-argocd-repo-server" successfully rolled out
deployment "argo-cd-argocd-server" successfully rolled out
deployment "dpf-operator-controller-manager" successfully rolled out
deployment "dpf-provisioning-controller-manager" successfully rolled out
deployment "dpuservice-controller-manager" successfully rolled out
deployment "kamaji" successfully rolled out
deployment "kamaji-cm-controller-manager" successfully rolled out
deployment "maintenance-operator" successfully rolled out
deployment "node-feature-discovery-gc" successfully rolled out
deployment "node-feature-discovery-master" successfully rolled out
deployment "servicechainset-controller-manager" successfully rolled out

$ kubectl wait --for=condition=ready --namespace dpu-cplane-tenant1 dpucluster --all
dpucluster.provisioning.dpu.nvidia.com/dpu-cplane-tenant1 condition met

安装组件以启用加速CNI节点

OVN Kubernetes通过使用主CNI为每个Pod附加一个VF来加速流量。该VF用于将流卸载到DPU。本节详细介绍将Pod连接到卸载的OVN Kubernetes CNI所需的组件。

使用NVIDIA Network Operator安装Multus和SRIOV Network Operator

  1. 添加NVIDIA Network Operator Helm仓库：

    跳板机控制台

    $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia --force-update

  2. 将应用以下 network-operator.yaml values文件：

    manifests/04-enable-accelerated-cni/helm-values/network-operator.yml

    # 内容省略，请参考原始文件

class="expand-control-text">展开源代码 nfd: enabled: false deployNodeFeatureRules: false sriovNetworkOperator: enabled: true sriov-network-operator: operator: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/master operator: Exists - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: Exists crds: enabled: true sriovOperatorConfig: deploy: true configDaemonNodeSelector: null operator: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-role.kubernetes.io/master operator: Exists - matchExpressions: - key: node-role.kubernetes.io/control-plane operator: Exists

</code-snippet>

cgroup_no_v1=net_prio,net_cls

  • hugepagesz=2048kB
  • hugepages=8072 nvconfig:
  • device: "*" parameters:
  • PF_BAR2_ENABLE=0
  • PER_PF_NUM_SF=1
  • PF_TOTAL_SF=20
  • PF_SF_BAR_SIZE=10
  • NUM_PF_MSIX_VALID=0
  • PF_NUM_PF_MSIX_VALID=1
  • PF_NUM_PF_MSIX=228
  • INTERNAL_CPU_MODEL=1
  • INTERNAL_CPU_OFFLOAD_ENGINE=0
  • SRIOV_EN=1
  • NUM_OF_VFS=46
  • LAG_RESOURCE_ALLOCATION=1
  • LINK_TYPE_P1=ETH
  • LINK_TYPE_P2=ETH
  • NUM_VF_MSIX=48 ovs: rawConfigScript: | _ovs-vsctl() { ovs-vsctl --no-wait --timeout 15 "$@" }

_ovs-vsctl set Open_vSwitch . other_config:doca-init=true _ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000 _ovs-vsctl set Open_vSwitch . other_config:hw-offload=true _ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true _ovs-vsctl set Open_vSwitch . other_config:max-idle=20000 _ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000 _ovs-vsctl set Open_vSwitch . other_config:ctl-pipe-size=1024 _ovs-vsctl --if-exists del-br ovsbr1 _ovs-vsctl --if-exists del-br ovsbr2 _ovs-vsctl --may-exist add-br br-sfc _ovs-vsctl set bridge br-sfc datapath_type=netdev _ovs-vsctl set bridge br-sfc fail_mode=secure _ovs-vsctl --may-exist add-port br-sfc p0 _ovs-vsctl set Interface p0 type=dpdk _ovs-vsctl set Interface p0 mtu_request=9216 _ovs-vsctl set Port p0 external_ids:dpf-type=physical

_ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev _ovs-vsctl --may-exist add-br br-ovn _ovs-vsctl set bridge br-ovn datapath_type=netdev _ovs-vsctl br-set-external-id br-ovn bridge-id br-ovn _ovs-vsctl br-set-external-id br-ovn bridge-uplink puplinkbrovntobrsfc _ovs-vsctl set Interface br-ovn mtu_request=9216 _ovs-vsctl --may-exist add-port br-ovn pf0hpf _ovs-vsctl set Interface pf0hpf type=dpdk _ovs-vsctl set Interface pf0hpf mtu_request=9216

bfcfgParameters:

  • UPDATE_ATF_UEFI=yes
  • UPDATE_DPU_OS=yes
  • WITH_NIC_FW_UPDATE=yes

hostNetworkInterfaceConfigs:

  • portNumber: 0 dhcp: true mtu: 9000

configFiles:

  • path: /etc/mellanox/mlnx-bf.conf operation: override raw: | ALLOW_SHARED_RQ="no" IPSEC_FULL_OFFLOAD="no" ENABLE_ESWITCH_MULTIPORT="yes" permissions: "0644"
  • path: /etc/mellanox/mlnx-ovs.conf operation: override raw: | CREATE_OVS_BRIDGES="no" OVS_DOCA="yes" permissions: "0644"
  • path: /etc/mellanox/mlnx-sf.conf operation: override raw: "" permissions: "0644"

  1. Set the mtu to 8940 for the OVN DPUServiceConfig (to deploy the OVN Kubernetes workloads on the DPU with the same MTU as in the host):

    manifests/05-dpudeployment-installation/dpuserviceconfig_ovn.yaml

    ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "ovn"
  serviceConfiguration:
    helmChart:
      values:
        k8sAPIServer: https://$TARGETCLUSTER_API_SERVER_HOST:$TARGETCLUSTER_API_SERVER_PORT
        podNetwork: $POD_CIDR/24
        serviceNetwork: $SERVICE_CIDR
        mtu: 8940
        dpuManifests:
          kubernetesSecretName: "ovn-dpu" # user needs to populate based on DPUServiceCredentialRequest
          vtepCIDR: $VTEP_CIDR
          hostCIDR: $TARGETCLUSTER_NODE_CIDR
          externalDHCP: true
          gatewayDiscoveryNetwork: "169.254.99.100/32" # This is a "dummy" subnet used to get the default gateway address from DHCP server (via option 121)

  2. The rest of the configuration files remain the same, including:

    • OVN DPUServiceCredentialRequest to allow cross cluster communication.

      manifests/05-dpudeployment-installation/ovn-credentials.yaml

      ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceCredentialRequest
metadata:
  name: ovn-dpu
  namespace: dpf-operator-system
spec:
  serviceAccount:
    name: ovn-dpu
    namespace: dpf-operator-system
  duration: 24h
  type: tokenFile
  secret:
    name: ovn-dpu
    namespace: dpf-operator-system
  metadata:
    labels:
      dpu.nvidia.com/image-pull-secret: ""

    • DPUServiceInterfaces for physical ports on the DPU.

      manifests/05-dpudeployment-installation/physical-ifaces.yaml

      ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: p0
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            uplink: "p0"
        spec:
          interfaceType: physical
          physical:
            interfaceName: p0

    • OVN DPUServiceInterface to define the ports attached to OVN workloads on the DPU.

      manifests/05-dpudeployment-installation/ovn-iface.yaml

      ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            port: ovn
        spec:
          interfaceType: ovn

    • BFB to download BlueField Bitstream to a shared volume.

      manifests/05-dpudeployment-installation/bfb.yaml

      ---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: BFB
metadata:
  name: bf-bundle-$TAG
  namespace: dpf-operator-system
spec:
  url: $BFB_URL

    • OVN DPUServiceTemplate to deploy OVN Kubernetes workloads to the DPU.

      manifests/05-dpudeployment-installation/dpuservicetemplate_ovn.yaml

      ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "ovn"
  helmChart:
    source:
      repoURL: $OVN_KUBERNETES_REPO_URL
      chart: ovn-kubernetes-chart
      version: $TAG
    values:
      commonManifests:
        enabled: true
      dpuManifests:
        enabled: true
      leaseNamespace: "ovn-kubernetes"
      gatewayOpts: "--gateway-interface=br-ovn"

  3. Apply all of the YAML files mentioned above using the following command:

    Jump Node Console

    $ cat manifests/05-dpudeployment-installation/*.yaml | envsubst | kubectl apply -f -

  4. Verify the DPUService installation by ensuring the DPUServices are created and have been reconciled, that the DPUServiceInterfaces have been reconciled, and that the DPUServiceChains have been reconciled:

    Notes

    • These verification commands may need to be run multiple times to ensure the conditions are met.
    • When using DPUDeployment, the DPUService name will have the DPUDeployment name added as prefix. For example, ovn-vwxyz.

跳板机控制台

$ kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_ovn
a.com/owned-by-dpudeployment=dpf-operator-system_ovn
dpuservice.svc.dpu.nvidia.com/blueman-5bdx6 condition met
dpuservice.svc.dpu.nvidia.com/dts-s7xsm condition met
dpuservice.svc.dpu.nvidia.com/ovn-cpfjf condition met

$ kubectl wait --for=condition=ServiceInterfaceSetReady --namespace dpf-operator-system dpuserviceinterface --all
dpuserviceinterface.svc.dpu.nvidia.com/ovn condition met
dpuserviceinterface.svc.dpu.nvidia.com/p0 condition met

$ kubectl wait --for=condition=ServiceChainSetReady --namespace dpf-operator-system dpuservicechain --all
dpuservicechain.svc.dpu.nvidia.com/ovn-wqq8h condition met

K8s 集群扩展

向集群添加工作节点

此时，应将工作节点添加到集群中。随着工作节点的添加，DPU 将被配置，DPUServices 将开始启动。

  1. 返回之前运行 Kubespray 部署集群的 shell，取消注释 hosts.yaml 文件中 kube_node 组下的工作节点，并将工作节点添加到集群：

    注意： 确保在运行命令时处于 Python 虚拟环境（.venv）中。

    (.venv) depuser@jump:~/kubespray$ cat inventory/mycluster/hosts.yaml
...
    kube_node:
      hosts:
        worker1:
        worker2:
...

(.venv) depuser@jump:~/kubespray$ ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root scale.yml

  2. 扩展不应花费很长时间，成功运行后的输出应类似于以下内容：

    image-2025-9-16_10-45-46.png

验证

  1. 要跟踪 DPU 配置的进度，运行以下命令检查当前所处的阶段：

    $ watch -n10 "kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase'"
Every 10.0s: kubectl describe dpu -n dpf-operator-system | grep 'Node Name\|Type\|Last\|Phase'

Type:       InternalIP
    Type:       Hostname
    Last Transition Time:  2026-01-04T12:14:24Z
    Type:                  Ready
    Last Transition Time:  2026-01-04T11:39:31Z
    Type:                  BFBPrepared
    Last Transition Time:  2026-01-04T11:39:05Z
    Type:                  BFBReady
    Last Transition Time:  2026-01-04T12:02:55Z
    Type:                  DPUClusterReady
    Last Transition Time:  2026-01-04T11:39:05Z
    Type:                  Initialized
    Last Transition Time:  2026-01-04T11:39:30Z
    Type:                  NodeEffectReady
    Last Transition Time:  2026-01-04T12:14:24Z
    Type:                  NodeEffectRemoved
    Last Transition Time:  2026-01-04T11:55:43Z
    Type:                  CheckedHostRebootNeed
    Last Transition Time:  2026-01-04T11:39:31Z
    Type:                  FWConfigured
    Last Transition Time:  2026-01-04T12:02:51Z
    Type:                  HostNetworkReady
    Last Transition Time:  2026-01-04T11:39:30Z
    Type:                  InterfaceInitialized
    Last Transition Time:  2026-01-04T11:55:42Z
    Type:                  OSInstalled
    Last Transition Time:  2026-01-04T12:01:23Z
    Type:                  Rebooted
  Phase:                Ready
  Dpu Node Name:                                      worker2
    Type:       InternalIP
    Type:       Hostname
    Last Transition Time:  2026-01-04T12:13:24Z
    Type:                  Ready
    Last Transition Time:  2026-01-04T11:39:26Z
    Type:                  BFBPrepared
    Last Transition Time:  2026-01-04T11:39:00Z
    Type:                  BFBReady
    Last Transition Time:  2026-01-04T12:01:45Z
    Type:                  DPUClusterReady
    Last Transition Time:  2026-01-04T11:38:59Z
    Type:                  Initialized
    Last Transition Time:  2026-01-04T11:39:24Z
    Type:                  NodeEffectReady
    Last Transition Time:  2026-01-04T12:13:23Z
    Type:                  NodeEffectRemoved
    Last Transition Time:  2026-01-04T11:54:58Z
    Type:                  CheckedHostRebootNeed
    Last Transition Time:  2026-01-04T11:39:26Z
    Type:                  FWConfigured
    Last Transition Time:  2026-01-04T12:01:39Z
    Type:                  HostNetworkReady
    Last Transition Time:  2026-01-04T11:39:25Z
    Type:                  InterfaceInitialized
    Last Transition Time:  2026-01-04T11:54:56Z
    Type:                  OSInstalled
    Last Transition Time:  2026-01-04T12:00:09Z
    Type:                  Rebooted
  Phase:                Ready

  2. 通过确保 DPU 处于就绪状态，验证 DPU 是否已成功配置：

    $ kubectl wait --for=condition=ready --namespace dpf-operator-system dpu --all
dpu.provisioning.dpu.nvidia.com/worker1-mt2404xz0c98 condition met
dpu.provisioning.dpu.nvidia.com/worker2-mt2333xz0xq3 condition met

  3. 确保以下 DaemonSet 有 2 个就绪副本：

    $ kubectl wait ds --for=jsonpath='{.status.numberReady}'=2 --namespace nvidia-network-operator kube-multus-ds sriov-network-config-daemon sriov-device-plugin
daemonset.apps/kube-multus-ds condition met
daemonset.apps/sriov-network-config-daemon condition met
daemonset.apps/sriov-device-plugin condition met

$ kubectl wait ds --for=jsonpath='{.status.numberReady}'=2 --namespace ovn-kubernetes ovn-kubernetes-node-dpu-host
daemonset.apps/ovn-kubernetes-node-dpu-host condition met

  4. 验证所有不同的 DPUServices、DPUServiceInterfaces 和 DPUServiceChains 对象现在都处于就绪状态：

    $ kubectl wait --for=condition=ServiceInterfaceSetReady --namespace dpf-operator-system dpuserviceinterface --all
dpuserviceinterface.svc.dpu.nvidia.com/ovn condition met
dpuserviceinterface.svc.dpu.nvidia.com/p0 condition met

$ kubectl wait --for=condition=ServiceChainSetReady --namespace dpf-operator-system dpuservicechain --all
dpuservicechain.svc.dpu.nvidia.com/ovn-wqq8h condition met

$ kubectl -n dpf-operator-system exec deployment/dpf-operator-controller-manager -- /dpfctl describe all --show-resources=dpu --show-conditions=dpu
NAME NAMESPACE STATUS REASON SINCE MESSAGE
DPFOperatorConfig/dpfoperatorconfig dpf-operator-system Ready: True Success 17m
└─DPU
└─2 DPU... dpf-operator-system Ready: True DPUReady 12m See worker1-mt2404xz0c98, worker2-mt2333xz0xq3

恭喜，DPF 系统已成功安装！

基础设施带宽验证

通过使用各种测试验证部署，并确保可以在 DPF 系统上达到链路速度性能结果：

  1. RDMA 带宽测量
  2. Iperf TCP 带宽测量

每个测试都有详细说明。在每个测试结束时，您将看到所达到的性能。

警告： 确保服务器已针对最大性能进行调整（本文档未涵盖）。

性能测试

RoCE 带宽测试

  1. 应用以下 NetworkPolicy 以启用无状态流量：

stateless_netpolicy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: multi-port-egress
  namespace: default
  annotations:
    k8s.ovn.org/acl-stateless: "true"
spec:
  podSelector: {}
  policyTypes:
  - Egress
  - Ingress
  egress:
   - {}
  ingress:
   - {}

Jump Node Console

$ kubectl apply -f stateless_netpolicy.yaml

  1. 创建一个测试Deployment，使用以下YAML在2个不同的worker节点上创建2个副本：

    注意： 下面指定的容器镜像必须包含 NVIDIA 用户空间驱动 和 perftest。

    testapp-performance-test-deployment.yaml

    ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testapp-performance
  labels:
    app: testapp-performance
spec:
  replicas: 2
  selector:
    matchLabels:
      app: testapp-performance
  template:
    metadata:
      labels:
        app: testapp-performance
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: testapp-performance
      containers:
      - name: testapp-pod
        image: <container_image>
        imagePullPolicy: Always
        command: ['sh', '-c', 'trap : TERM INT; sleep infinity & wait']
        securityContext:
          capabilities:
            add: [ "IPC_LOCK" ]
        resources:
          requests:
            cpu: '24'
            memory: '8Gi'
          limits:
            cpu: '24'
            memory: '8Gi'

  2. 应用资源：

    Jump Node Console

    $ kubectl apply -f testapp-performance-test-deployment.yaml

  3. 验证Deployment是否成功运行：

    Jump Node Console

    $ kubectl get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP            NODE      NOMINATED NODE   READINESS GATES
testapp-performance-fd6954bd-9hq99   1/1     Running   0          2m39s   10.233.68.4   worker2   <none>           <none>
testapp-performance-fd6954bd-s56m7   1/1     Running   0          53s     10.233.67.4   worker1   <none>           <none>

  4. 连接到Deployment中的一个Pod：

    $ kubectl exec -it testapp-performance-fd6954bd-9hq99 -- bash

  5. 在容器内，检查其接口上的 IP地址，并确认它可被识别为RDMA设备：

    root@testapp-performance-fd6954bd-9hq99:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
130: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8940 qdisc mq state UP group default qlen 1000
    link/ether 0a:58:0a:e9:44:04 brd ff:ff:ff:ff:ff:ff permaddr c2:8e:07:eb:52:e5
    altname enp137s0f0v36
    inet 10.233.68.4/24 brd 10.233.68.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c08e:7ff:feeb:52e5/64 scope link
       valid_lft forever preferred_lft forever

root@testapp-performance-fd6954bd-9hq99:/# rdma link | grep eth0
link mlx5_38/1 state ACTIVE physical_state LINK_UP netdev eth0

  6. 启动 ib_write_bw 服务器端：

    root@testapp-performance-fd6954bd-9hq99:/# ib_write_bw -a

************************************
* Waiting for client to connect... *
************************************

  7. 使用另一个控制台窗口，重新连接到跳板机并连接到Deployment中的 第二个Pod。

    $ kubectl exec -it testapp-performance-fd6954bd-s56m7 -- bash

  8. 在容器内，启动 ib_read_lat 客户端（使用服务器端容器的IP地址）并检查带宽结果：

    root@testapp-performance-fd6954bd-s56m7:/# ib_write_bw -a --report_gbits -F 10.233.68.4
---------------------------------------------------------------------------------------
                    RDMA_Write BW Test
 Dual-port       : OFF          Device         : mlx5_10
 Number of qps   : 1            Transport type : IB
 Connection type : RC           Using SRQ      : OFF
 PCIe relax order: ON
 ibv_wr* API     : ON
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : Ethernet
 GID index       : 5
 Max inline data : 0[B]
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0000 QPN 0x01a8 PSN 0x9719ae RKey 0x058505 VAddr 0x007861d47cf000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:233:67:05
 remote address: LID 0000 QPN 0x0968 PSN 0x6a9c2e RKey 0x04a505 VAddr 0x0079c10c637000
 GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:233:68:04
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
 2          5000           0.012810            0.009982            0.623879
 4          5000             0.20               0.19               5.990129
 8          5000             0.39               0.39               6.037097
 16         5000             0.80               0.78               6.060446
 32         5000             1.55               1.54               5.998464
 64         5000             3.07               3.06               5.976294
 128        5000             6.32               6.14               6.000758
 256        5000             12.72              12.34              6.027505
 512        5000             24.60              24.50              5.982084
 1024       5000             49.20              49.01              5.983012
 2048       5000             98.40              97.84              5.971414
 4096       5000             169.78             168.60             5.145195
 8192       5000             192.47             192.30             2.934319
 16384      5000             192.90             192.86             1.471428
 32768      5000             193.18             193.13             0.736721
 65536      5000             193.29             193.26             0.368611
 131072     5000             193.32             193.32             0.184363
 262144     5000             193.34             193.34             0.092190
 524288     5000             193.37             193.37             0.046103
 1048576    5000             193.38             193.38             0.023053
 2097152    5000             193.38             193.38             0.011526
 4194304    5000             193.39             193.39             0.005763
 8388608    5000             193.39             193.39             0.002882
---------------------------------------------------------------------------------------

iPerf TCP Bandwidth Test

  1. 使用前一个示例中的YAML创建一个测试Deployment，在每个worker上创建一个Pod，用于测试TCP连接和性能。

    警告： 测试中指定的容器镜像必须

包括 iperf。

  1. 连接到部署中的一个 Pod：

    跳板机控制台

    $ kubectl exec -it testapp-performance-fd6954bd-9hq99 -- bash

  2. 在启动 iperf3 服务器监听器之前，为了获得良好结果，请检查 Pod 当前运行的 CPU 核心：

    警告： 要能够绑定到特定核心，请确保将 Pod 调度到 Guaranteed QoS 类。

    检查 Pod 运行在哪个工作节点上（本例中核心：28-51）：

    root@testapp-performance-fd6954bd-9hq99: taskset -pc 1
pid 1's current affinity list: 28-51

  3. 在 Pod 容器内，使用以下脚本在不同端口上启动多个 iperf3 服务器（每个核心一个）：

    iperf_server.sh

    #!/bin/bash

# Cores to bind the iperf3 server processes to
CORES=$1

# Calculate the first_core and last_core to provide the CPU range
first_core=$(echo $CORES | cut -d "-" -f1)
last_core=$(echo $CORES | cut -d "-" -f2)

# Loop over the ports (5201 + i*2) for i in the given CPU range and run iperf3 servers
for i in $(seq $first_core $last_core); do
   echo "Running iperf3 server on core $i"
   taskset -c $i iperf3 -s -p $((5201 + i * 2)) > /dev/null 2>&1 &
done

  4. 使用之前的 CPU 范围启动脚本（保留 1 个核心作为缓冲）：

    第一个 Pod 控制台

    root@testapp-performance-fd6954bd-9hq99:/# chmod +x iperf_server.sh
root@testapp-performance-fd6954bd-9hq99:/# ./iperf_server.sh 28-50
Running iperf3 server on core 28
Running iperf3 server on core 29
...
...
Running iperf3 server on core 49
Running iperf3 server on core 50

root@testapp-performance-fd6954bd-9hq99:/# ps -ef | grep iperf3
root          38       1  0 14:39 pts/0    00:00:00 iperf3 -s -p 5257
root          39       1  0 14:39 pts/0    00:00:00 iperf3 -s -p 5259
...
...
root          59       1  0 14:39 pts/0    00:00:00 iperf3 -s -p 5299
root          60       1  0 14:39 pts/0    00:00:00 iperf3 -s -p 5301

  5. 连接到第二个 Pod：

    跳板机控制台

    $ kubectl exec -it testapp-performance-fd6954bd-s56m7 -- bash

  6. 按照之前显示的方法识别第二个 Pod 运行的 CPU 核心。

  7. 使用以下脚本启动多个 iperf3 客户端，连接到第一个 Pod 中的每个 iperf3 服务器：

    注意：

    • 脚本接收 3 个参数：要连接的服务器 IP、生成 iperf3 进程的核心数以及 iperf3 测试的运行时长。启动脚本时请确保传递所有 3 个参数，并将 CPU 核心指定为范围（本例中为 28-50）。
    • Pod 上应安装 jq 和 bc 才能正常运行。

    iperf_client.sh

    #!/bin/bash

# IP address of the server where iperf3 servers are running
SERVER_IP=$1  # Change to your server's IP

# Cores to bind the iperf3 client processes to
CORES=$2

# Duration to run the iperf3 test
DUR=$3

# Variable to accumulate the total bandwidth in Gbit/sec
total_bandwidth_Gbit=0

# Calculate the first_core and last_core to provide the CPU range
first_core=$(echo $CORES | cut -d "-" -f1)
last_core=$(echo $CORES | cut -d "-" -f2)

# Array to store the PIDs of background tasks
pids=()

# Loop over the ports (5201 + i*2) for i in the given CPU range
for i in $(seq $first_core $last_core); do
    port=$((5201 + i * 2))
    cpu_core=$i  # Assign CPU core based on the value of i
    output_file="iperf3_client_results_$port.log"

    # Run the iperf3 client in the background with CPU core binding
    timeout $(( DUR +5 )) taskset -c $cpu_core iperf3 -c $SERVER_IP -p $port -t $DUR -J > $output_file &
    pid=$!
    pids+=("$pid")
done

# Wait for all background tasks to complete and check their status
for pid in "${pids[@]}"; do
    wait $pid
    if [[ $? -ne 0 ]]; then
        echo "Process with PID $pid failed or timed out."
    fi
done

# Summarize the results from each log file
echo "Summary of iperf3 client results:"
for i in $(seq $first_core $last_core); do
    port=$((5201 + i * 2))
    output_file="iperf3_client_results_$port.log"

    if [[ -f $output_file ]]; then
        echo "Results for port $port:"

        # Parse the results and print a summary
        bandwidth_bps=$(jq '.end.sum_received.bits_per_second' $output_file)

        if [[ -n $bandwidth_bps ]]; then
           # Convert bandwidth from bps to Gbit/sec
           bandwidth_Gbit=$(echo "scale=3; $bandwidth_bps / 1000000000" | bc)
           echo "  Bandwidth: $bandwidth_Gbit Gbit/sec"

           # Accumulate the bandwidth for the total summary
           total_bandwidth_Gbit=$(echo "scale=3; $total_bandwidth_Gbit + $bandwidth_Gbit" | bc)

           # Delete current log file
           rm $output_file
        else
           echo "No bandwidth data found in $output_file"
        fi

    else
        echo "No results found for port $port"
    fi
done

# Print the total bandwidth summary
echo "Total Bandwidth across all streams: $total_bandwidth_Gbit Gbit/sec"

  8. 运行脚本并检查性能结果：

    第二个 Pod 控制台

    root@testapp-performance-fd6954bd-s56m7:/# chmod +x iperf_client.sh
root@testapp-performance-fd6954bd-s56m7:/# ./iperf_client.sh 10.233.68.4 28-50 30
Summary of iperf3 client results:
Results for port 5257:
  Bandwidth: 3.843 Gbit/sec
Results for port 5259:
  Bandwidth: 11.506 Gbit/sec
Results for port 5261:
  Bandwidth: 11.492 Gbit/sec
Results for port 5263:
  Bandwidth: 11.492 Gbit/sec
Results for port 5265:
  Bandwidth: 5.734 Gbit/sec
Results for port 5267:
  Bandwidth: 5.700 Gbit/sec
Results for port 5269:
  Bandwidth: 5.769 Gbit/sec
Results for port 5271:
  Bandwidth: 3.873 Gbit/sec
Results for port 5273:
  Bandwidth: 5.772 Gbit/sec
Results for port 5275:
  Bandwidth: 11.556 Gbit/sec
Results for port 5277:
  Bandwidth: 11.513 Gbit/sec
Results for port 5279:
  Bandwidth: 5.820 Gbit/sec
Results for port 5281:
  Bandwidth: 5.816 Gbit/sec
Results for port 5283:
  Bandwidth: 11.501 Gbit/sec
Results for port 5285:
  Bandwidth: 3.820 Gbit/sec
Results for port 5287:
  Bandwidth: 11.505 Gbit/sec
Results for port 5289:
  Bandwidth: 5.815 Gbit/sec
Results for port 5291:
  Bandwidth: 11.507 Gbit/sec
Results for port 5293:
  Bandwidth: 11.559 Gbit/sec
Results for port 5295:
  Bandwidth: 11.550 Gbit/sec
Results for port 5297:
  Bandwidth: 11.541 Gbit/sec
Results for port 5299:
  Bandwidth: 5.737 Gbit/sec
Results for port 5301:
  Bandwidth: 5.822 Gbit/sec
Results for port 5303:
  Bandwidth: 5.742 Gbit/sec
Total Bandwidth across all streams: 195.985 Gbit/sec

作者
Unknown Attachment Guy ZilbermanGuy Zilberman 是 NVIDIA 网络解决方案实验室的解决方案架构师，在云计算领域拥有丰富的领导经验。他专注于利用 NVIDIA 先进的网络技术设计和实现云及容器化工作负载的解决方案。他的工作主要围绕开源云基础设施，精通 Kubernetes (K8s) 和 OpenStack 等平台。
Shachar Dor Shachar DorShachar Dor 在加入解决方案实验室团队之前，曾在 NVIDIA Networking（前 Mellanox Technologies）担任软件架构师超过十年，负责网络管理产品和解决方案的架构。Shachar 专注于网络技术，特别是网络部署、配置、监控和生命周期管理。在加入公司之前，Shachar 通过参与多个项目和技术，在软件架构、设计和编程方面拥有丰富的经验。