超擎数智超擎数智

RDG for DPF Host-Trusted with HBN and SNAP Virtio-FS

Created on January 6, 2026 Scope This Reference Deployment Guide (RDG) provides detailed instructions for deploying a Kubernetes (K8s) cluster using the DOCA Platform Framework (DPF) in Host-Trusted mode, and utilizing the SNAP DPU Service with Virtio-FS. The guide focuses on setting up an accelerated Host-Based Networking (HBN) service on NVIDIA BlueField-3 DPU to deliver secure, isolated, and hardware-accelerated environments, and utilizing the SNAP VirtIO-FS DPU service which provides a VirtIO-FS CSI to the cluster via the DPU using an external storage target (NFS).

文档目录

Created on January 6, 2026

Scope

This Reference Deployment Guide (RDG) provides detailed instructions for deploying a Kubernetes (K8s) cluster using the DOCA Platform Framework (DPF) in Host-Trusted mode, and utilizing the SNAP DPU Service with Virtio-FS. The guide focuses on setting up an accelerated Host-Based Networking (HBN) service on NVIDIA® BlueField®-3 DPU to deliver secure, isolated, and hardware-accelerated environments, and utilizing the SNAP VirtIO-FS DPU service which provides a VirtIO-FS CSI to the cluster via the DPU using an external storage target (NFS).

This guide is designed for experienced system administrators, system engineers, and solution architects who seek to deploy high-performance Kubernetes clusters with Host-Based Networking enabled on NVIDIA BlueField DPU and a VirtIO-FS CSI provided from an external storage target.

Warning

  • This reference implementation, as the name implies, is a specific, opiniated deployment example designed to address the use case described above.
  • While other approaches may exist to implement similar solutions, this document provides a detailed guide for this particular method.

Abbreviations and Acronyms

Term Definition Term Definition
BFB BlueField Bootstream (OS Image) RDG Reference Deployment Guide
BGP Border Gateway Protocol RDMA Remote Direct Memory Access
CNI Container Network Interface SFC Service Function Chaining
CSI Container Storage Interface SNAP Storage-Defined Network Accelerated Processing
DOCA Data Center Infrastructure-on-a-Chip Architecture SR-IOV Single Root Input/Output Virtualization
DPF DOCA Platform Framework TOR Top of Rack
DPU Data Processing Unit VLAN Virtual LAN (Local Area Network)
GENEVE Generic Network Virtualization Encapsulation VNI Virtual Network Interface
HBN Host Based Networking VRF Virtual Router/Forwarder
IPAM IP Address Management VRR Virtual Router Redundancy
K8S Kubernetes VTEP Virtual Tunnel End Point
MAAS Metal as a Service

Introduction

The NVIDIA BlueField-3 Data Processing Unit (DPU) is a 400 Gb/s infrastructure compute platform designed for line-rate processing of software-defined networking, storage, and cybersecurity workloads. It combines powerful compute resources, high-speed networking, and advanced programmability to deliver hardware-accelerated, software-defined solutions for modern data centers.

NVIDIA DOCA unleashes the full potential of the BlueField platform by enabling rapid development of applications and services that offload, accelerate, and isolate data center workloads.

One such service is Host-Based Networking (HBN) - a DOCA-enabled solution that allows network architects to design networks based on Layer 3 (L3) protocols. HBN enables routing on the server side by using BlueField as a BGP router. It encapsulates key networking functions in a containerized service pod, deployed directly on the BlueField's ARM cores.

Another such service is SNAP, which has both Block Device and File System modes. In this RDG, we will demonstrate its file system mode - Virtio-FS, that provides file system storage provided to the cluster from an external storage target (NFS).

In this solution, the SNAP Virtio-fs service deployed via NVIDIA DOCA Platform Framework (DPF) is composed of multiple functional components packaged into containers, which DPF orchestrates to run together with HBN. DPF simplifies DPU management by providing orchestration through a Kubernetes API. It handles the provisioning and lifecycle management of DPU, orchestrates specialized DPU services, and automates tasks such as service function chaining (SFC).

This RDG extends the capabilities of the DPF-managed Kubernetes cluster described in the RDG for DPF Host-Trusted with HBN DPU Service (referred to as the "Baseline RDG") by adding the SNAP DPU Service in Virtio-fs mode. It demonstrates performance optimizations, including Jumbo frame implementation, with results validated through an iperf3 TCP test and a standard FIO workload test.

References

Solution Architecture

关键组件与技术

  • NVIDIA BlueField® 数据处理单元 (DPU) NVIDIA® BlueField® 数据处理单元 (DPU) 为现代数据中心和超级计算集群带来了前所未有的创新。凭借其强大的计算能力和集成的软件定义硬件加速器（用于网络、存储和安全），BlueField 为任何环境中的任何工作负载创建了安全且加速的基础设施，开启了加速计算和人工智能的新时代。

  • NVIDIA DOCA 软件框架 NVIDIA DOCA™ 释放了 NVIDIA® BlueField® 网络平台的潜力。通过利用 BlueField DPU 和 SuperNIC 的强大功能，DOCA 能够快速创建卸载、加速和隔离数据中心工作负载的应用程序和服务。它使开发人员能够创建软件定义、云原生、DPU 和 SuperNIC 加速的服务，并具有零信任保护，满足现代数据中心对性能和安全性要求。

  • NVIDIA ConnectX 智能网卡 10/25/40/50/100/200 和 400G 以太网网卡 业界领先的 NVIDIA® ConnectX® 系列智能网卡提供先进的硬件卸载和加速功能。 NVIDIA 以太网网卡为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供了最高的投资回报率和最低的总拥有成本。

  • NVIDIA LinkX 线缆 NVIDIA® LinkX® 线缆和收发器产品系列提供了业界最完整的 10、25、40、50、100、200 和 400GbE 以太网以及 100、200 和 400Gb/s InfiniBand 产品线，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。

  • NVIDIA Spectrum 以太网交换机 灵活的外形规格，提供 16 到 128 个物理端口，支持 1GbE 到 400GbE 速度。 基于针对性能和可扩展性优化的突破性硅技术，NVIDIA Spectrum 交换机是构建高性能、高性价比、高效的云数据中心网络、以太网存储结构和深度学习互连的理想选择。 NVIDIA 结合了基于业界领先的专用集成电路 (ASIC) 技术的 NVIDIA Spectrum™ 交换机的优势，以及多种现代网络操作系统选择，包括 NVIDIA Cumulus® Linux、SONiC 和 NVIDIA Onyx®。

  • NVIDIA Cumulus Linux NVIDIA® Cumulus® Linux 是业界最具创新性的开放网络操作系统，可让您像其他系统一样自动化、定制和扩展数据中心网络。

  • NVIDIA Network Operator NVIDIA Network Operator 简化了 Kubernetes 集群中 NVIDIA 网络资源的配置和管理。该操作员自动安装所需的主机网络软件，汇集所有必要组件以提供高速网络连接。这些组件包括 NVIDIA 网络驱动程序、Kubernetes 设备插件、CNI 插件、IP 地址管理 (IPAM) 插件等。NVIDIA Network Operator 与 NVIDIA GPU Operator 协同工作，为可扩展的 GPU 计算集群提供高吞吐量、低延迟的网络。

  • Kubernetes Kubernetes 是一个开源容器编排平台，用于容器化应用程序的部署自动化、扩展和管理。

  • Kubespray Kubespray 是由 Ansible playbooks、清单、配置工具和领域知识组成的组合，用于通用操作系统/Kubernetes 集群配置管理任务，并提供：

    • 高可用集群
    • 可组合属性
    • 支持大多数流行的 Linux 发行版

  • RDMA RDMA 是一种允许网络中的计算机在不涉及任一计算机的处理器、缓存或操作系统的情况下交换数据的技术。 与本地 DMA 类似，RDMA 提高了吞吐量和性能，并释放了计算资源。

解决方案设计

解决方案逻辑设计

逻辑设计包括以下组件：

  • 1 个 Hypervisor 节点（基于 KVM），配备 ConnectX-7
    • 1 个防火墙虚拟机
    • 1 个跳板虚拟机
    • 1 个 MaaS 虚拟机
    • 3 个 K8s Master 虚拟机，运行所有 K8s 管理组件
    • 1 个存储目标虚拟机
  • 2 个工作节点（PCI Gen5），每个配备 1 个 BlueField-3 网卡
  • 1 个高速 (HS) 交换机
  • 1 Gb 主机管理网络

image-2025-12-15_18-35-29.png

HBN 服务逻辑设计

HBN+SNAP-VirtioFS 服务部署利用了 DPF 系统固有的服务功能链 (SFC) 能力，如 HBN DPU 服务的基线 RDG 中所述（请参阅“基础设施延迟和带宽验证”部分）。以下 SFC 逻辑图显示了所实现解决方案中涉及的所有服务的完整流程：

image-2026-1-1_15-8-37.png

卷仿真逻辑图

以下逻辑图展示了将卷挂载到工作负载 Pod 的过程中涉及的主要组件。

在 Host Trusted 模式下，主机运行 SNAP CSI 插件，该插件执行所有必要操作以使存储资源对主机可用。用户可以利用 Kubernetes 存储 API（StorageClass、PVC、PV、VolumeAttachment）来配置存储并将其挂载到主机。在主机集群中创建引用指定 SNAP CSI 插件作为其供应器的存储类的 PersistentVolumeClaim (PVC) 对象后，DPF 存储子系统组件通过 NFS 内核客户端将 NFS 卷带到所需的 DPU K8s 工作节点。然后，DOCA SNAP 服务将其仿真为 Virtio-fs 卷，并将网络存储作为本地文件系统设备呈现给主机，当 kubelet 请求时，SNAP CSI 插件将其挂载到 Pod 命名空间中。

注意： 有关仿真过程中涉及的不同组件及其协同工作的完整信息，请参阅：DPF 存储开发指南 - NVIDIA Docs。

image

VirtioFS Device Emulation Diagram

Firewall Design

The pfSense firewall in this solution serves two key roles:

  • Firewall – provides an isolated environment for the DPF system, ensuring secure operations
  • Router – enables Internet access for the management network

Port-forwarding rules for SSH and RDP are configured on the firewall to route traffic to the jump node's IP address on the host management network. From the jump node, administrators can manage and access various devices in the setup, as well as handle the deployment of both the Kubernetes (K8s) cluster and DPF components.

The following diagram illustrates the firewall design used in this solution:

image-2025-12-31_10-10-21-1.png

Software Stack Components

image-2026-1-1_15-0-29-1.png

Warning: Make sure to use the exact same versions for the software stack as described above.

Bill of Materials

image-2025-12-31_10-34-16-1.png

Deployment and Configuration

Node and Switch Definitions

The following definitions and parameters are used to deploy the demonstrated fabric:

交换机 Ports Usage
Hostname Rack ID Ports
hs-switch 1 swp1-5
mgmt-switch 1 swp1-3
Hosts
Rack Server Type Server Name Switch Port IP and NICs Default Gateway
Rack1 Hypervisor Node hypervisor mgmt-switch: swp1hs-switch: swp5 mgmt-br (interface eno2): -lab-br (interface eno1): Trusted LAN IP Trusted LAN GW
Rack1 Worker Node worker1 mgmt-switch: swp2hs-switch: swp1-swp2 ens15f0: 10.0.110.21/24 10.0.110.254
Rack1 Worker Node worker2 mgmt-switch: swp3hs-switch: swp3-swp4 ens15f0: 10.0.110.22/24 10.0.110.254
Rack1 Firewall (Virtual) fw - LAN (mgmt-br): 10.0.110.254/24WAN (lab-br): Trusted LAN IP Trusted LAN GW
Rack1 Jump Node (Virtual) jump - enp1s0: 10.0.110.253/24 10.0.110.254
Rack1 MaaS (Virtual) maas - enp1s0: 10.0.110.252/24 10.0.110.254
Rack1 Storage Target Node (Virtual) storage-target - enp1s0: 10.0.110.30/24enp5s0np1: 10.0.124.1/24 10.0.110.254
Rack1 Master Node (Virtual) master1 - enp1s0: 10.0.110.1/24 10.0.110.254
Rack1 Master Node (Virtual) master2 - enp1s0: 10.0.110.2/24 10.0.110.254
Rack Node Type Hostname OS Management Interface Gateway
Rack1 Master Node (Virtual) master3 - enp1s0: 10.0.110.3/24 10.0.110.254

Wiring

Hypervisor Node

image-2025-12-31_10-36-28-1.png

K8s Worker Node

image-2025-5-11_17-20-54-1.png

Fabric Configuration

Updating Cumulus Linux

As a best practice, make sure to use the latest released Cumulus Linux NOS version.

For information on how to upgrade Cumulus Linux, refer to the Cumulus Linux User Guide.

Configuring the Cumulus Linux Switch

Configure the SN3700 switch (hs-switch) as follows:

Note:

  • The following commands configure BGP unnumbered on hs-switch
  • Cumulus Linux enables the BGP equal-cost multipathing (ECMP) option by default
nv set bridge domain br_default vlan 10 vni 10
nv set evpn state enabled
nv set interface lo ipv4 address 11.0.0.101/32
nv set interface lo type loopback
nv set interface swp1-5 link state up
nv set interface swp1-5 type swp
nv set interface swp5 bridge domain br_default access 10
nv set nve vxlan state enabled
nv set nve vxlan source address 11.0.0.101
nv set router bgp autonomous-system 65001
nv set router bgp state enabled
nv set router bgp graceful-restart mode full
nv set router bgp router-id 11.0.0.101
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute static state enabled
nv set vrf default router bgp address-family ipv6-unicast state enabled
nv set vrf default router bgp address-family ipv6-unicast redistribute connected state enabled
nv set vrf default router bgp address-family l2vpn-evpn state enabled
nv set vrf default router bgp state enabled
nv set vrf default router bgp neighbor swp1-4 peer-group hbn
nv set vrf default router bgp neighbor swp1-4 type unnumbered
nv set vrf default router bgp path-selection multipath aspath-ignore enabled
nv set vrf default router bgp peer-group hbn remote-as external
nv set vrf default router bgp peer-group snap remote-as external
nv set vrf default router bgp peer-group snap address-family l2vpn-evpn state enabled
nv config apply -y

Configure the SN2201 switch (mgmt-switch) as follows:

nv set bridge domain br_default untagged 1
nv set interface swp1-3 link state up
nv set interface swp1-3 type swp
nv set interface swp1-3 bridge domain br_default
nv config apply -y

Host Configuration

Warning: Ensure that SR-IOV is enabled in the BIOS settings on the worker node servers, and that the servers are tuned for maximum performance.

Warning: Make sure all worker nodes have the same PCIe placement for the BlueField-3 NIC and that they show the same interface name.

Hypervisor Installation and Configuration

No change from the Baseline RDG (Section "Deployment and Configuration", Subsection "Prepare Infrastructure Servers") regarding Firewall VM, Jump VM, MaaS VM.

Provision Master VMs and Worker Nodes Using MaaS

Proceed with the instructions from the Baseline RDG until you reach the subsection "Deploy Master VMs using Cloud-Init".

Use the following cloud-init script instead of the one in the Baseline RDG to install the necessary software and also configure correct routing to the storage target node:

Master node cloud-init
#cloud-config
system_info:
  default_user:
    name: depuser
    passwd: "$6$jOKPZPHD9XbG72lJ$evCabLvy1GEZ5OR1Rrece3NhWpZ2CnS0E3fu5P1VcZgcRO37e4es9gmriyh14b8Jx8gmGwHAJxs3ZEjB0s0kn/"
    lock_passwd: false
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash
ssh_pwauth: True
package_upgrade: true
runcmd:
    - apt-get update
    - apt-get -y install nfs-common
    - |
      cat <<'EOF' | tee /etc/netplan/99-static-route.yaml
      network:
        version: 2
        ethernets:
          enp1s0:
            routes:
              - to: 10.0.124.1
                via: 10.0.110.30
      EOF
    - netplan apply

After that proceed exactly as instructed in the Baseline RDG, and in addition to the verification commands mentioned there, run the following command to verify that the static route has been configured correctly:

Master1 Console
root@master1:~# ip r
default via 10.0.110.254 dev enp1s0 proto static
10.0.110.0/24 dev enp1s0 proto kernel scope link src 10.0.110.1
10.0.124.1 via 10.0.110.30 dev enp1s0 proto static

No changes from the Baseline RDG to the worker nodes provisioning.

Storage Target Configuration

Warning:

  • The Storage target node is a separate, manually configured node in this RDG.
  • It will be a VM running on the hypervisor, with ConnectX-7 NIC and NVMe SSD disk attached to it as PCIe devices using PCI passthrough.

Suggested specifications:

  • vCPU: 8
  • RAM: 32GB
  • Storage:
    • VirtIO disk of 60GB size
    • NVMe SSD of 1.7TB size
  • Network interface:
    • Bridge device, connected to mgmt-br

Procedure:

  1. Perform a regular Ubuntu 24.04 installation on the Storage target VM.

  2. Create the following Netplan configuration to enable internet connectivity, DNS resolution and set an IP in the storage high-speed subnet:

    Warning: Replace enp1s0 and enp5s0np1 with your interface names.

    Storage Target netplan

    network:
  version: 2
  ethernets:
    enp1s0:
      addresses:
      - "10.0.110.30/24"
      mtu: 9000
      nameservers:

addresses:

  • 10.0.110.252 search:
  • dpf.rdg.local.domain routes:
  • to: "default" via: "10.0.110.254" enp5s0np1: addresses:
    • "10.0.124.1/24" mtu: 9000

  1. 应用netplan配置：

    存储目标控制台

    sudo netplan apply

  2. 更新和升级系统：

    存储目标控制台

    sudo apt update -y
sudo apt upgrade -y

  3. 在NVMe磁盘上创建XFS文件系统，并将其挂载到/srv/nfs目录：

    警告： 将/dev/nvme0n1替换为您的设备名称。

    存储目标控制台

    sudo mkfs.xfs /dev/nvme0n1
sudo mkdir -m 777 /srv/nfs/
sudo mount /dev/nvme0n1 /srv/nfs/

  4. 设置持久挂载：

    存储目标控制台

    $ sudo blkid
/dev/nvme0n1 /dev/nvme0n1: UUID="b37df0a9-d741-4222-82c9-7a3d66ffc0e1" BLOCK_SIZE="512" TYPE="xfs"

$ echo "/dev/disk/by-uuid/b37df0a9-d741-4222-82c9-7a3d66ffc0e1 /srv/nfs xfs defaults 0 1" | sudo tee -a /etc/fstab

  5. 安装并配置NFS服务器，共享/srv/nfs目录：

    存储目标控制台

    sudo apt install -y nfs-server
echo "/srv/nfs/ 10.0.110.0/24(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports
echo "/srv/nfs/ 10.0.124.0/24(rw,sync,no_subtree_check)" | sudo tee -a /etc/exports

  6. 重启NFS服务器：

    存储目标控制台

    sudo systemctl restart nfs-server

  7. 在/srv/nfs下创建share目录，权限与父目录相同：

    存储目标控制台

    sudo mkdir -m 777 /srv/nfs/share

K8s 集群部署与配置

使用Kubespray进行初始Kubernetes集群部署（主节点）以及后续验证的步骤与基线RDG（第"K8s集群部署与配置"节，子节："Kubespray部署与配置"、"使用Kubespray Ansible Playbook部署集群"、"K8s部署验证"）保持不变。

与基线RDG一样，工作节点在DPF和先决条件组件安装完成后添加。

DPF 安装

软件先决条件和所需变量

有关软件先决条件（如helm、envsubst），请参考基线RDG（第"DPF安装"节，子节"软件先决条件和所需变量"）。

继续克隆doca-platform Git仓库（确保使用标签v25.10.0）：

跳板机控制台

git clone https://github.com/NVIDIA/doca-platform.git
cd doca-platform
git checkout v25.10.0

切换到包含hbn-snap readme.md的目录，因为所有命令都将从此位置运行：

跳板机控制台

$ cd docs/public/user-guides/host-trusted/use-cases/hbn-snap

编辑以下文件以定义安装所需的变量：

警告：

  • 将以下文件中的变量值替换为适合您设置的值。 特别注意DPU_P0、DPUCLUSTER_INTERFACE以及DPU_P0_PF_NAME、DPU_P1_PF_NAME、DPU_P0_VF10_NAME、DPU_P1_VF10_NAME。

manifests/00-env-vars/envvars.env

## 用于DPU集群负载均衡器的虚拟IP。必须是管理子网中保留的IP，且不由DHCP分配。
export DPUCLUSTER_VIP=10.0.110.200

## DPUCluster负载均衡器将监听的接口。应为控制平面节点的管理接口。
export DPUCLUSTER_INTERFACE=enp1s0

## 用于存储BFB镜像的NFS服务器的IP地址。
## 注意：此环境变量不控制用作SNAP VirtioFS远程目标的NFS服务器的地址。
export NFS_SERVER_IP=10.0.110.253

## NVIDIA Helm Chart注册表的仓库URL。
## 通常为NVIDIA Helm NGC注册表。出于开发目的，可设置为其他仓库。
export HELM_REGISTRY_REPO_URL=https://helm.ngc.nvidia.com/nvidia/doca

## HBN容器镜像的仓库URL。
## 通常为NVIDIA NGC注册表。出于开发目的，可设置为其他仓库。
export HBN_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_hbn

## SNAP VFS容器镜像的仓库URL。
## 通常为NVIDIA NGC注册表。出于开发目的，可设置为其他仓库。
export SNAP_NGC_IMAGE_URL=nvcr.io/nvidia/doca/doca_vfs

## DPF REGISTRY是DPF Operator Chart所在的Helm仓库URL。
## 通常为NVIDIA Helm NGC注册表。出于开发目的，可设置为其他仓库。
export REGISTRY=https://helm.ngc.nvidia.com/nvidia/doca

## DPF TAG是本指南中部署的DPF组件的版本。
export TAG=v25.10.0

## 在`bfb.yaml`中使用并由DPUSet链接的BFB的URL。
export BFB_URL="https://content.mellanox.com/BlueField/BFBs/Ubuntu24.04/bf-bundle-3.2.1-34_25.11_ubuntu-24.04_64k_prod.bfb"

# 包含主机侧网络PF 0的名称，例如enp8s0f0np0
export DPU_P0_PF_NAME=ens4f0
# 包含主机侧网络PF 1的名称，例如enp8s0f1np1
export DPU_P1_PF_NAME=ens4f1
# 包含主机侧P0上网络VF 10的名称，例如enp8s0f0v10
export DPU_P0_VF10_NAME=ens4f0v10
# 包含主机侧P1上网络VF 10的名称，例如enp8s0f1v10
export DPU_P1_VF10_NAME=ens4f1v10

导出安装所需的环境变量：

跳板机控制台

source manifests/00-env-vars/envvars.env

DPF Operator 安装

与基线RDG（第"DPF安装"节，子节"DPF Operator安装"）无变化。

DPF 系统安装

与基线RDG（第"DPF安装"节，子节"DPF系统安装"）无变化。

安装组件以启用加速接口

请从基线RDG（第"DPF安装"节，子节"安装组件以启用加速接口"）执行此步骤。 注意：此时不应用sriov_network_operator_policy.yaml，将在稍后应用...

DPU 部署安装

DPU Deployment Installation

Before deploying the objects under manifests/04.2-dpudeployment-installation-virtiofs/ directory, a few adjustments are needed to achieve better performance results.

Edit the DPUFlavor YAML to add the NUM_VF_MSIX firmware parameter and increase the hugepages value in the grub:

Warning: The parameter NUM_VF_MSIX is set to 48 in the provided example, which is suitable for the servers used in this RDG. Set this value to match the physical number of cores in the NUMA node where the NIC is located.

---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: DPUFlavor
metadata:
  name: hbn-snap-virtiofs-$TAG
  namespace: dpf-operator-system
spec:
  bfcfgParameters:
  - UPDATE_ATF_UEFI=yes
  - UPDATE_DPU_OS=yes
  - WITH_NIC_FW_UPDATE=yes
  configFiles:
  - operation: override
    path: /etc/mellanox/mlnx-bf.conf
    permissions: "0644"
    raw: |
      ALLOW_SHARED_RQ="no"
      IPSEC_FULL_OFFLOAD="no"
      ENABLE_ESWITCH_MULTIPORT="yes"
      RDMA_SET_NETNS_EXCLUSIVE="no"
  - operation: override
    path: /etc/mellanox/mlnx-ovs.conf
    permissions: "0644"
    raw: |
      CREATE_OVS_BRIDGES="no"
      OVS_DOCA="yes"
  - operation: override
    path: /etc/mellanox/mlnx-sf.conf
    permissions: "0644"
    raw: ""
  grub:
    kernelParameters:
    - console=hvc0
    - console=ttyAMA0
    - earlycon=pl011,0x13010000
    - fixrttc
    - net.ifnames=0
    - biosdevname=0
    - iommu.passthrough=1
    - cgroup_no_v1=net_prio,net_cls
    - hugepagesz=2048kB
    - hugepages=8192
  nvconfig:
  - device: '*'
    parameters:
    - PF_BAR2_ENABLE=0
    - PER_PF_NUM_SF=1
    - PF_TOTAL_SF=20
    - PF_SF_BAR_SIZE=10
    - NUM_PF_MSIX_VALID=0
    - PF_NUM_PF_MSIX_VALID=1
    - PF_NUM_PF_MSIX=228
    - INTERNAL_CPU_MODEL=1
    - INTERNAL_CPU_OFFLOAD_ENGINE=0
    - SRIOV_EN=1
    - NUM_OF_VFS=46
    - LAG_RESOURCE_ALLOCATION=1
    - PCI_SWITCH_EMULATION_ENABLE=1
    - PCI_SWITCH_EMULATION_NUM_PORT=32
    - VIRTIO_FS_EMULATION_ENABLE=1
    - VIRTIO_FS_EMULATION_NUM_PF=0
    - LINK_TYPE_P1=ETH
    - LINK_TYPE_P2=ETH
    - NUM_VF_MSIX=48
  ovs:
    rawConfigScript: |
      _ovs-vsctl() {
        ovs-vsctl --no-wait --timeout 15 "$@"
      }

      _ovs-vsctl set Open_vSwitch . other_config:doca-init=true
      _ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000
      _ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
      _ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true
      _ovs-vsctl set Open_vSwitch . other_config:max-idle=20000
      _ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000
      _ovs-vsctl --if-exists del-br ovsbr1
      _ovs-vsctl --if-exists del-br ovsbr2
      _ovs-vsctl --may-exist add-br br-sfc
      _ovs-vsctl set bridge br-sfc datapath_type=netdev
      _ovs-vsctl set bridge br-sfc fail_mode=secure
      _ovs-vsctl --may-exist add-port br-sfc p0
      _ovs-vsctl set Interface p0 type=dpdk
      _ovs-vsctl set Interface p0 mtu_request=9216
      _ovs-vsctl set Port p0 external_ids:dpf-type=physical
      _ovs-vsctl --may-exist add-port br-sfc p1
      _ovs-vsctl set Interface p1 type=dpdk
      _ovs-vsctl set Interface p1 mtu_request=9216
      _ovs-vsctl set Port p1 external_ids:dpf-type=physical
      _ovs-vsctl --may-exist add-br br-hbn
      _ovs-vsctl set bridge br-hbn datapath_type=netdev
      _ovs-vsctl set bridge br-hbn fail_mode=secure

The rest of the configuration files remain the same, you would need to apply the following command:

Jump Node Console

cat manifests/04.2-dpudeployment-installation-virtiofs/*.yaml | envsubst | kubectl apply -f -

It will apply all the YAMLs required for the deployment - DPUDeployment, BFB, DPUFlavor, Service Templates and Configurations for the various DPU Services (7 separate service modules for SNAP and one for HBN), Physical Interfaces definitions and IPAM definitions.

Please proceed as described in the Baseline RDG until "Infrastructure Latency & Bandwidth Validation" section, including the cluster scale-out (adding the worker nodes).

Note that the first validation command after applying the above command should be (instead of the first command that appears in the Baseline RDG):

Jump Node Console

kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_hbn-snap

Testing Storage & Network Connectivity

In the next steps, we will configure and test the Virtio-FS storage and the accelerated network connection.

This will create the SriovNetworkNodePolicy and NetworkAttachmentDefinition objects:

Jump Node Console

cat manifests/05-network-configuration/*.yaml | envsubst | kubectl apply -f -

And this will create the test pods:

Warning: For achieving maximum TCP performance, please edit the pods in test-hostdev-pods.yaml to use 24 cores instead of 16

Jump Node Console

kubectl apply -f manifests/06-network-test/test-hostdev-pods.yaml

iPerf TCP Bandwidth Test

Connect to the first pod:

Jump Node Console

$ kubectl exec -it sriov-hostdev-pf0vf10-test-worker1-5bccdc4c75-97xms -- bash

Before starting the iperf3 server listeners, and to achieve good results, check which cores the pod is currently running on in another tab:

Jump Node Console

$ ssh worker1
depuser@worker1:~$ sudo -i
root@worker1:~# crictl ps | grep sriov-hostdev-pf0vf10
a4441f76405cf       0ac86781a84f1       14 minutes ago      Running             nginx                         0                   24f4c327d918f       sriov-hostdev-pf0vf10-test-worker1-5bccdc4c75-97xms   default

root@worker1:~# crictl inspect a4441f76405cf | jq '.status.resources.linux.cpusetCpus'
"28-51"

Back in the first pod - use vim to create the following script to start multiple iperf3 servers (1 for each core) on different ports:

iperf_server.sh

#!/bin/bash

# Cores to bind the iperf3 server processes to
CORES=$1

# Function to expand core ranges (e.g., "10-20,40-50" -> array of individual cores)
expand_core_ranges() {
    local ranges=$1
    local cores=()

    # Split by comma to handle multiple ranges
    IFS=',' read -ra RANGE_ARRAY <<< "$ranges"

    for range in "${RANGE_ARRAY[@]}"; do
        # Check if it's a range (contains '-') or a single core
        if [[ $range == *"-"* ]]; then
            first=$(echo $range | cut -d "-" -f1)
            last=$(echo $range | cut -d "-" -f2)
            for core in $(seq $first $last); do
                cores+=($core)
            done
        else
            cores+=($range)
        fi
    done

    echo "${cores[@]}"
}

# Expand the core ranges into an array
core_array=($(expand_core_ranges "$CORES"))
ports_num=${#core_array[@]}

echo "Starting $ports_num iperf3 server processes on cores: ${core_array[@]}"

# Loop over each core and run iperf3 servers with sequential port assignment
for i in $(seq 1 $ports_num); do
   core=${core_array[$((i-1))]}
   port=$((5201 + i * 2))
   echo "Running iperf3 server $i on core $core, port $port"
   taskset -c $core iperf3 -s -p $port > /dev/null 2>&1 &
done

For best performance please set 9K MTU on the net1 interface and then start the script using the previous CPU range (leave 1 core as a buffer):

First Pod Console

root@sriov-hostdev-pf0vf10-test-worker1-5bccdc4c75-97xms:/# ip link set net1 mtu 9000
root@sriov-hostdev-pf0vf10-test-worker1-5bccdc4c75-97xms:/# chmod +x iperf_server.sh
root@sriov-hostdev-pf0vf10-test-worker1-5bccdc4c75-97xms:/# ./iperf_server.sh 28-51
Starting 16 iperf3 server processes on cores: 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
Running iperf3 server 1
Running iperf3 server 2
...
...
Running iperf3 server 23
Running iperf3 server 24

root@sriov-hostdev-pf0vf10-test-worker1-5bccdc4c75-97xms:/# ps -ef | grep iperf3
   38 root      0:00 iperf3 -s -p 5203
   39 root      0:00 iperf3 -s -p 5205
...
...
   60 root      0:27 iperf3 -s -p 5247
   61 root

0:40 iperf3 -s -p 5249

Connect to the second pod:

Jump Node Console

$ kubectl exec -it sriov-hostdev-pf0vf10-test-worker2-85b7cb76fd-qmljl -- bash

Follow the previously displayed method to identify the CPU cores that the second pod is running on. In our case it was the same range (28-51).

Use vim to create the following script to start multiple iperf3 clients that will connect to each iperf3 server in the first pod:

Warning

  • The script receives 3 parameters: the server IP to connect to, the CPU cores on which to spawn the iperf3 processes, and the duration the iperf3 test. Make sure to provide all 3 when initiating the script and providing the CPU cores as a range (28-51).

iperf_client.sh

#!/bin/bash

# IP address of the server where iperf3 servers are running
SERVER_IP=$1  # Change to your server's IP

# Cores to bind the iperf3 client processes to
CORES=$2

# Duration to run the iperf3 test
DUR=$3

# Variable to accumulate the total bandwidth in Gbit/sec
total_bandwidth_Gbit=0

# Function to expand core ranges (e.g., "10-20,40-50" -> array of individual cores)
expand_core_ranges() {
    local ranges=$1
    local cores=()

    # Split by comma to handle multiple ranges
    IFS=',' read -ra RANGE_ARRAY <<< "$ranges"

    for range in "${RANGE_ARRAY[@]}"; do
        # Check if it's a range (contains '-') or a single core
        if [[ $range == *"-"* ]]; then
            first=$(echo $range | cut -d "-" -f1)
            last=$(echo $range | cut -d "-" -f2)
            for core in $(seq $first $last); do
                cores+=($core)
            done
        else
            cores+=($range)
        fi
    done

    echo "${cores[@]}"
}

# Expand the core ranges into an array
core_array=($(expand_core_ranges "$CORES"))
ports_num=${#core_array[@]}

echo "Starting $ports_num iperf3 client processes on cores: ${core_array[@]}"

# Array to store the PIDs of background tasks
pids=()

# Loop over each core and run iperf3 clients with sequential port assignment
for i in $(seq 1 $ports_num); do
    port=$((5201 + i * 2))
    cpu_core=${core_array[$((i-1))]}  # Assign CPU core from the expanded array
    output_file="iperf3_client_results_$port.log"

    echo "Running iperf3 client $i on core $cpu_core, connecting to port $port"
    # Run the iperf3 client in the background with CPU core binding
    timeout $(( DUR +5 )) taskset -c $cpu_core iperf3 -Z -c $SERVER_IP -p $port -t $DUR -J > $output_file &
    pid=$!
    pids+=("$pid")
done

# Wait for all background tasks to complete and check their status
for pid in "${pids[@]}"; do
    wait $pid
    if [[ $? -ne 0 ]]; then
        echo "Process with PID $pid failed or timed out."
    fi
done

# Summarize the results from each log file
echo "Summary of iperf3 client results:"
for i in $(seq 1 $ports_num); do
    port=$((5201 + i * 2))
    output_file="iperf3_client_results_$port.log"

    if [[ -f $output_file ]]; then
        echo "Results for port $port:"

        # Parse the results and print a summary
        bandwidth_bps=$(jq '.end.sum_received.bits_per_second' $output_file)

        if [[ -n $bandwidth_bps ]]; then
           # Convert bandwidth from bps to Gbit/sec
           bandwidth_Gbit=$(echo "scale=3; $bandwidth_bps / 1000000000" | bc)
           echo "  Bandwidth: $bandwidth_Gbit Gbit/sec"

           # Accumulate the bandwidth for the total summary
           total_bandwidth_Gbit=$(echo "scale=3; $total_bandwidth_Gbit + $bandwidth_Gbit" | bc)

           # Delete current log file
           rm $output_file
        else
           echo "No bandwidth data found in $output_file"
        fi

    else
        echo "No results found for port $port"
    fi
done

# Print the total bandwidth summary
echo "Total Bandwidth across all streams: $total_bandwidth_Gbit Gbit/sec"

Again, please set 9K MTU on net1 for maximum performance and run the script to check the performance results:

Second Pod Console

root@sriov-hostdev-pf0vf10-test-worker2-85b7cb76fd-qmljl:/# ip link set net1 mtu 9000
root@sriov-hostdev-pf0vf10-test-worker2-85b7cb76fd-qmljl:/# chmod +x iperf_client.sh
root@sriov-hostdev-pf0vf10-test-worker2-85b7cb76fd-qmljl:/# ./iperf_client.sh 10.0.121.1 28-51 30

Summary of iperf3 client results:
Results for port 5203:
  Bandwidth: 14.207 Gbit/sec
Results for port 5205:
  Bandwidth: 22.445 Gbit/sec
Results for port 5207:
  Bandwidth: 8.868 Gbit/sec
Results for port 5209:
  Bandwidth: 11.115 Gbit/sec
Results for port 5211:
  Bandwidth: 14.104 Gbit/sec
Results for port 5213:
  Bandwidth: 13.387 Gbit/sec
Results for port 5215:
  Bandwidth: 22.743 Gbit/sec
Results for port 5217:
  Bandwidth: 12.132 Gbit/sec
Results for port 5219:
  Bandwidth: 13.927 Gbit/sec
Results for port 5221:
  Bandwidth: 13.470 Gbit/sec
Results for port 5223:
  Bandwidth: 22.720 Gbit/sec
Results for port 5225:
  Bandwidth: 14.771 Gbit/sec
Results for port 5227:
  Bandwidth: 12.752 Gbit/sec
Results for port 5229:
  Bandwidth: 9.174 Gbit/sec
Results for port 5231:
  Bandwidth: 14.265 Gbit/sec
Results for port 5233:
  Bandwidth: 24.338 Gbit/sec
Results for port 5235:
  Bandwidth: 14.087 Gbit/sec
Results for port 5237:
  Bandwidth: 13.353 Gbit/sec
Results for port 5239:
  Bandwidth: 14.555 Gbit/sec
Results for port 5241:
  Bandwidth: 20.808 Gbit/sec
Results for port 5243:
  Bandwidth: 13.056 Gbit/sec
Results for port 5245:
  Bandwidth: 16.648 Gbit/sec
Results for port 5247:
  Bandwidth: 17.545 Gbit/sec
Results for port 5249:
  Bandwidth: 20.905 Gbit/sec
Total Bandwidth across all streams: 375.375 Gbit/sec

Storage Test

The following command will define the DPUStorageVendor for NFS CSI and the DPUStoragePolicy for filesystem policy:

Jump Node Console

cat manifests/07.2-storage-configuration-virtiofs/*.yaml | envsubst | kubectl apply -f -

Verify the DPUStorageVendor and DPUStoragePolicy objects are ready:

Jump Node Console

kubectl wait --for=condition=Ready --namespace dpf-operator-system dpustoragevendors --all
kubectl wait --for=condition=Ready --namespace dpf-operator-system dpustoragepolicies --all

Deploy storage test pods that mount a storage volume provided by SNAP VirtioFS:

Jump Node Console

kubectl apply -f manifests/08.2-storage-test-virtiofs

Check the virtiofs-tag name:

Jump Node Console

$ kubectl get dpuvolumeattachments.storage.dpu.nvidia.com -A -o json | jq '.items[0].status.dpu.virtioFSAttrs.filesystemTag'
"3e76e376579383d2tag"

Connect to the test pod, validate that the virtiofs filesystem is mounted with the previous tag name and install the fio software:

Jump Node Console

$ kubectl exec -it storage-test-pod-virtiofs-hotplug-pf-0 -- bash

root@storage-test-pod-virtiofs-hotplug-pf-0:/# df -Th
Filesystem          Type      Size  Used Avail Use% Mounted on
overlay             overlay   439G   17G  400G   4% /
tmpfs               tmpfs      64M     0   64M   0% /dev
3e76e376579383d2tag virtiofs  1.8T   45G  1.8T   3% /mnt/vol1
/dev/nvme0n1p2      ext4      439G   17G  400G   4% /etc/hosts
shm                 tmpfs      64M     0   64M   0% /dev/shm
tmpfs               tmpfs     251G   12K  251G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs               tmpfs     126G     0  126G   0% /proc/acpi
tmpfs               tmpfs     126G     0  126G   0% /proc/scsi
tmpfs               tmpfs     126G     0  126G   0% /sys/firmware
tmpfs               tmpfs     126G     0  126G   0% /sys/devices/virtual/powercap

root@storage-test-pod-virtiofs-hotplug-pf-0:/# apt update && apt install -y vim fio

Using vim, create the following file:

job-4k.fio

[global]
ioengine=libaio
direct=1
iodepth=32
rw=read
bs=4k
size=1G
numjobs=8
runtime=60
time_based
group_reporting

[job1]
filename=/mnt/vol1/test.fio

Finally, run the fio test:

Jump Node Console

root@storage-test-pod-virtiofs-hotplug-pf-0:/# fio job-4k.fio
job1:

(g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
...
fio-2.2.10
...
...
Starting 8 processes
job1: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 8 (f=8): [R(8)] [100.0% done] [826.1MB/0KB/0KB /s] [212K/0/0 iops] [eta 00m:00s]
job1: (groupid=0, jobs=8): err= 0: pid=1183: Mon Dec  1 10:31:32 2025
  read : io=47664MB, bw=813351KB/s, iops=203337, runt= 60008msec
    slat (usec): min=0, max=679, avg= 6.90, stdev= 4.13
    clat (usec): min=167, max=135036, avg=1250.42, stdev=4941.25
     lat (usec): min=170, max=135038, avg=1257.36, stdev=4940.79
    clat percentiles (usec):
     |  1.00th=[  258],  5.00th=[  278], 10.00th=[  286], 20.00th=[  298],
     | 30.00th=[  302], 40.00th=[  310], 50.00th=[  314], 60.00th=[  322],
     | 70.00th=[  326], 80.00th=[  338], 90.00th=[  358], 95.00th=[  470],
     | 99.00th=[27520], 99.50th=[32128], 99.90th=[46336], 99.95th=[52992],
     | 99.99th=[68096]
    bw (KB  /s): min=85832, max=121912, per=12.51%, avg=101789.00, stdev=5105.93
    lat (usec) : 250=0.39%, 500=95.22%, 750=0.55%, 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=1.05%, 50=2.70%
    lat (msec) : 100=0.07%, 250=0.01%
  cpu          : usr=2.78%, sys=24.20%, ctx=8652632, majf=0, minf=340
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=12201896/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=47664MB, aggrb=813351KB/s, minb=813351KB/s, maxb=813351KB/s, mint=60008msec, maxt=60008msec

Done!

## Authors

<br>

| | |
| --- | --- |
| ![GZ.jpg](https://networking-docs.nvidia.com/sol/__attachments/a_d030fb567c11478fd02f69a715dff79d1a797a9d5b0c1053e32501ae847376e7/GZ.jpg?cb=b27c02cdfcea319e0ef7919870379200) | **Guy Zilberman**<br>Guy Zilberman is a solution architect at NVIDIA's 网络解决方案 Labs, bringing extensive experience from several leadership roles in cloud computing. He specializes in designing and implementing solutions for cloud and containerized workloads, leveraging NVIDIA's advanced networking technologies. His work primarily focuses on open-source cloud infrastructure, with expertise in platforms such as Kubernetes (K8s) and OpenStack. |

<br>
<br>

| | |
| --- | --- |
| ![SD.jpg](https://networking-docs.nvidia.com/sol/__attachments/a_a9b5a707a2801018577a2cb51060e34c065d90841cbc2957ca6c708c2c263a36/SD.jpg?cb=3492c5f2a7749cdfd1781ad1e059b580) | **Shachar Dor**<br>Shachar Dor joined the 解决方案 Lab team after working more than ten years as a software architect at NVIDIA Networking (previously Mellanox Technologies), where he was responsible for the architecture of network management products and solutions. Shachar's focus is on networking technologies, especially around fabric bring-up, configuration, monitoring, and life-cycle management. <br>Shachar has a strong background in software architecture, design, and programming through his work on multiple projects and technologies also prior to joining the company. |

<br>
<br>

**NVIDIA and the NVIDIA logo, are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Other company and product names may be trademarks of the respective companies with which they are associated.**™

**2025 NVIDIA Corporation. All rights reserved.**©

<br>
<br>

<br>
<br>

<br>
<br>

<br>
<br>