超擎数智超擎数智

RDG for DPF with Firefly Time Synchronization, HBN and OVN Services - 25.04

Created on June 25, 2025 Scope This Reference Deployment Guide (RDG) provides detailed instructions on how to deploy, configure and validate the NVIDIA® DOCA™

文档目录

Created on June 25, 2025

Scope

This Reference Deployment Guide (RDG) provides detailed instructions on how to deploy, configure and validate the NVIDIA® DOCA™ Firefly Time Synchronization Service within a Kubernetes cluster using the DOCA Platform Framework (DPF). This document is an extension of the RDG for DPF with OVN-Kubernetes and HBN Services (referred to as the Baseline RDG). It details the additional steps and modifications required to deploy the Firefly Time Sync Service into the environment established by the Baseline RDG.

This guide is designed for experienced System Administrators, System Engineers, and Solution Architects seeking to implement high-precision time synchronization in high-performance Kubernetes clusters using NVIDIA BlueField DPU and DPF. Familiarity with the Baseline RDG is required.

Warning

  • This reference implementation, as the name implies, is a specific, opinionated deployment example designed to address the use case described above.
  • While other approaches may exist to implement similar solutions, this document provides a detailed guide for this particular method.

Abbreviations and Acronyms

Term Definition Term Definition
BC Boundary Clock NTP Network Time Protocol
BFB BlueField Bootstream OC Ordinary Clock
BGP Border Gateway Protocol OVN Open Virtual Network
CNI Container Network Interface PHC PTP Hardware Clock
DOCA Data Center Infrastructure-on-a-Chip Architecture PRTC Primary Reference Time Clock (e.g., ITU-T G.8272)
DPF DOCA Platform Framework PTP Precision Time Protocol (IEEE 1588)
DPU Data Processing Unit RDG Reference Deployment Guide
DTS DOCA Telemetry Service RDMA Remote Direct Memory Access
G.8275.1 ITU-T Recommendation for PTP Profile (Full Timing Support) SF Scalable Function
GM Grandmaster Clock SFC Service Function Chaining
HBN Host-Based Networking SR-IOV Single Root Input/Output Virtualization
IPAM IP Address Management TAI International Atomic Time
ITU-T International Telecommunication Union - Telecommunication Standardization Sector TOR Top of Rack
K8S Kubernetes UTC Coordinated Universal Time
MAAS Metal as a Service

Introduction

Accurate time synchronization is critical for various modern data center applications, including distributed databases, real-time analytics, precise event ordering, and detailed telemetry. While Network Time Protocol (NTP) is commonly used and provides millisecond-level time accuracy, which is sufficient for many legacy applications, emerging applications—particularly in fields such as artificial intelligence (AI) and high-performance computing—require time synchronization with precision levels far beyond what NTP can offer. These applications often necessitate time accuracy in the range of tens of nanoseconds to microseconds.

The Firefly Time Sync Service, deployed via the NVIDIA DOCA Platform Framework (DPF), leverages the Precision Time Protocol (PTP) capabilities of NVIDIA BlueField® DPU and NVIDIA Spectrum™ switches to deliver highly accurate time synchronization across the cluster.

Firefly runs the PTP stack directly on the DPU's Arm cores, synchronizing the DPU's PTP Hardware Clock (PHC). It then facilitates the synchronization of the DPU's system clock and the host server's system clock with this precise PHC. This architecture offloads the time synchronization task from the host CPU and provides a robust, OS-agnostic solution. This combined approach enables the full utilization of the DPU for precise timekeeping (sub-microsecond accuracy), supporting time-sensitive applications and enhancing overall data center synchronization.

The guide details the steps required to achieve highly accurate, PTP-based time synchronization across cluster nodes equipped with NVIDIA® BlueField® DPU, interconnected via NVIDIA® Spectrum® switches running Cumulus Linux. Leveraging NVIDIA's DPF, administrators can provision and manage DPU resources while deploying and orchestrating the Firefly Time Sync Service alongside other essential infrastructure components, like accelerated OVN-Kubernetes and Host-Based Networking (HBN).

This document extends the capabilities of the DPF-managed Kubernetes cluster described in the RDG for DPF with OVN-Kubernetes and HBN Services (referred to as the Baseline RDG) by deploying the NVIDIA DOCA Firefly Time Sync Service within the existing DPF deployment (which includes OVN-Kubernetes and HBN services) to achieve a comprehensive, accelerated, and precisely synchronized infrastructure.

References

This section supplements the "References" section of the Baseline RDG. Refer to the Baseline RDG (Section "References") for other relevant references.

Solution Architecture

The overall solution architecture remains consistent with the Baseline RDG (Section "Solution Architecture"), with the addition of components and configurations for time synchronization using the Firefly Time Sync Service.

Key Components and Technologies

This section highlights the key technologies involved in the time synchronization solution, supplementing those described in the Baseline RDG (Section "Solution Architecture", Subsection "Key Components and Technologies").

  • Precision Time Protocol (PTP) (defined by IEEE 1588) is a protocol used to synchronize clocks throughout a computer network. It is designed to achieve sub-microsecond accuracy, making it suitable

for demanding applications in telecommunications, finance, industrial automation, and high-performance computing clusters. PTP relies on a master-slave hierarchy of clocks and uses hardware timestamping to minimize latency and jitter introduced by network components and software stacks.

  • NVIDIA DOCA™ Firefly Time Sync Service is an NVIDIA DOCA service that enables high-precision time synchronization for NVIDIA BlueField DPU and connected hosts. It leverages the PTP capabilities of the DPU hardware to achieve sub-microsecond accuracy. The Firefly service supports multiple deployment modes, configuration profiles, and third-party providers to deliver time synchronization services to DPU and connected hosts.

Solution Design

Solution Logical Design

The logical design described in the Baseline RDG (Section "Solution Architecture", Subsection "Solution Design", Sub-subsection "Solution Logical Design") is augmented with the PTP Grandmaster node and the time synchronization components.

Additions for Firefly:

  • PTP Grandmaster Node is added:
    • A bare-metal server equipped with an NVIDIA ConnectX-7 NIC.
    • Connected to the high-speed switch (e.g., SN3700).
  • The SN3700 switch acts as a PTP Boundary Clock.
  • Firefly Time Sync Services are deployed on both K8s tenant hosts and DPU nodes:
    • The Firefly Time Sync Service on the DPU acts as a PTP client, synchronizing the PHCs from the SN3700, and then the DPU's Arm system clock.
    • The Firefly Time Sync Service on the host synchronizes the host system clock to the DPU's PHC.

image-2025-5-29_16-19-42-1.png

K8s Cluster Logical Design

The K8s cluster logical design remains the same as described in the Baseline RDG (Section "Solution Architecture", Subsection "Solution Design", Sub-subsection "K8s Cluster Logical Design").

DPF is responsible for deploying the Firefly DPUServices—both DPU and host components—onto the respective DPU K8s worker nodes and their hosts.

Timing Network Design

This section details the time synchronization architecture.

Key Design Considerations

  • The PTP profile demonstrated utilizes Layer 2 transport. It aligns closely with the ITU-T G.8275.1 telecom profile, which defines PTP for phase/time synchronization with full timing support from the network. This profile maps PTP messages directly over Ethernet using a specific EtherType and employs non-forwardable, link-local multicast MAC addresses (e.g., 01-80-C2-00-00-0E) for PTP message communication between peer ports. The solution also incorporates Boundary Clock (BC) functionality on the NVIDIA Spectrum switch.

  • The PTP time source (Grandmaster) used in this reference setup is a Linux server configured as a PTP Grandmaster for demonstration purposes and may not meet formal PTP Grandmaster clock performance standards (like ITU-T G.8272 PRTC). Setting up the Grandmaster node itself (OS installation, basic configuration) is not be demonstrated in detail; however, its PTP "master" configuration files are provided as examples.

    Note: For a UTC-traceable and accurate reference, a PRTC: ITU-T G.8272-compliant Grandmaster connected to GPS/GNSS can be used.

  • The setup described is a reference deployment and does not encompass all considerations required for a production-grade, highly available, and fully redundant time synchronization infrastructure, such as multiple Grandmaster deployment or complex failover scenarios (except for basic PTP interface redundancy on the Firefly Time Sync Service).

  • NTP Considerations:

    • The cluster is expected to be deployed with NTP (Network Time Protocol) initially, as per the Baseline RDG.
    • Control-plane nodes will continue to use NTP and are not part of the PTP synchronization domain in this guide.
    • NTP service should be disabled on Worker Nodes and DPU once the Firefly Time Sync Service is operational and PTP synchronization is established. This is typically handled by the DPF's DPUFlavor for the DPU and is the user's responsibility for the host.
Core Synchronization Elements
  • PTP Grandmaster (GM) Node: A dedicated server (bare-metal recommended) acting as the primary time source for the PTP domain. In this RDG, a Linux server with a ConnectX-7 NIC is configured to function as a PTP Grandmaster. For production environments, a dedicated, commercially available PTP Grandmaster appliance compliant with standards such as ITU-T G.8272 (PRTC-A or PRTC-B) is recommended for higher stability and accuracy.
  • NVIDIA Spectrum 交换机 (as PTP Boundary Clocks): The existing Spectrum switches (e.g., SN3700) are configured to act as PTP Boundary Clocks (BCs). They synchronize to an upstream PTP clock (either the GM or another BC) and provide PTP time to downstream devices (DPU or other BCs).
  • NVIDIA BlueField-3 DPU (as PTP Ordinary Clock–Slave/Client): The DPU on the worker nodes run the Firefly Time Sync Service. The DPU's PTP client synchronizes its PTP Hardware Clock (PHC) to the PTP time provided by the connected switch (BC).
  • DOCA Platform Framework (DPF): As in the Baseline RDG, DPF orchestrates the deployment and lifecycle management of DPUServices, now including the Firefly Time Sync Service components.
PTP Network Hierarchy
  1. PTP Grandmaster (GM): The authoritative time source for the PTP domain.
    • In this RDG: A Linux server with ConnectX-7, configured as a PTP master.
  2. PTP Boundary Clock (BC): The SN3700 Cumulus Linux switch.
    • It synchronizes its clock to the PTP GM (acting as a PTP slave towards the GM).
    • It provides PTP time to the DPU (acting as a PTP master towards the DPU).
  3. PTP Ordinary Clock (OC) - Slave: The BlueField-3 DPU running the Firefly Time Sync Service.
    • The DPU's PTP client synchronizes its PHC from the PTP time provided by the switch (BC).
Clock Types and Standards (Targeted)
  • PTP Grandmaster (Conceptual): Aims for PRTC-like behavior (ITU-T G.8272).
  • Switch (Boundary Clock): Configured to meet ITU-T G.8273.2 Class C T-BC requirements (without SyncE).
  • DPU (Ordinary Clock - Slave): Configured to meet ITU-T G.8273.2 Class C T-TSC (Telecom Time Slave Clock) requirements (without SyncE).

Warning: Reference PTP configurations for the DPU (via Firefly DPUServiceConfiguration CR), the Switch (Cumulus Linux commands), and the PTP Grandmaster (linuxptp configuration files) are provided in the relevant subsections of the 'Deployment and Configuration' section of this RDG.

Firefly Time Sync Service Design

  1. Firefly DPU Service (firefly-dpu-dpuservice). The Firefly DPU service is orchestrated as a DPU Service deployed on the BlueField DPU's Arm cores and is responsible for the primary PTP client operations and DPU time synchronization.
    • PTP Service: Utilizes PTP4L program as a third-party provider for PTP time synching service
    • OS Time Calibration: Utilizes PHC2SYS program as a third-party provider for OS time calibration service on the DPU Arm OS
    • Service Interface (Trusted Scalable Function): Utilizes a Trusted Scalable Function (SF) as its network interface to the fabric. This is crucial for achieving the high-precision timestamping functionality required by Firefly. The Trusted SF is configured and provisioned using DPUFlavor and potentially DPUServiceNAD (Network Attachment Definition) DPF Custom Resources.
    • Redundant PTP Interfaces: Supports configuration of two service interfaces (Trusted SFs) for PTP link redundancy. This allows the service to maintain PTP lock in case one of the physical links or paths to the PTP Boundary Clock fails.
    • PTP Profile Configuration: The PTP client within the Firefly DPU service is configured to align with the ITU-T G.8275.1 telecom profile, utilizing L2 transport and specific PTP message parameters.
    • Custom Flows for PTP Control Traffic: DPF facilitates the setup of custom OVS flows to steer the specific PTP control traffic (non-forwardable L2 multicast) between the physical port and the Firefly service's SF. This ensures PTP packets are correctly handled and not misrouted.
    • PTP Monitor Server: DPU Firefly service acts as a

server exposing PTP monitoring data to a PTP Monitor Client (Firefly Host Monitor Service)

  • Communication with Host Service: Exposes a DPU Cluster NodePort service, which allows the Firefly Host Monitor Service running on the x86 host to communicate with the DPU service for retrieving PTP monitoring information.

  • Firefly Host Monitor Service (firefly-host-monitor-dpuservice): The Firefly Host Monitor service is orchestrated as a DPUService deployed on the X86 tenant cluster hosts and is responsible for PTP state monitoring and host time synchronization.

    • OS Time Calibration: Utilizes PHC2SYS program as a third-party provider for OS time calibration service on the Host OS
    • Network Interface (VF): The service utilizes a Virtual Function (VF) injected into its pod by the OVN-Kubernetes CNI (via Multus and the SRIOV Network Operator). This VF shares the underlying PTP Hardware Clock (PHC) with the DPU, allowing the Firefly Host Monitor service to accurately fetch the DPU's synchronized PHC time.
    • PTP Monitor Client: Host Firefly service acts as a client registered for consuming PTP monitoring data from a PTP Monitor Server (Firefly DPU Service)
    • Communication with DPU Service: The Firefly DPU service (running on the DPU) exposes a DPU Cluster NodePort Kubernetes service. The Firefly Host Monitor service (running on the host) in turn exposes a tenant cluster Kubernetes service, which facilitates its connection to the local DPU's NodePort service. This communication channel is primarily used to monitor the DPU's PTP synchronization state and verify its health.

image-2025-5-29_16-23-55-1.png

Time Synchronization Flow

  1. The PTP Grandmaster node generates and distributes PTP timing messages.
  2. The SN3700 switch (PTP BC) receives these messages on its PTP slave port connected to the GM, synchronizes its internal clock, and regenerates PTP messages on its PTP master ports connected to the DPU.
  3. The BlueField-3 DPU (running Firefly's PTP client) receives PTP messages from the switch on its PTP slave port(s) and disciplines its PTP Hardware Clock (PHC). Firefly Time Sync Service supports using two DPU ports for PTP slave for link redundancy.
  4. The Firefly DPU service synchronizes the DPU's Arm OS system clock to its disciplined PTP Hardware Clock (PHC).
  5. The Firefly Host Monitor service, running on the host, monitors the PTP synchronization state on the DPU.
  6. The Firefly Host Monitor service then synchronizes the host's OS system clock to the DPU's precise PHC.

image-2025-5-29_16-24-21-1.png

Service Function Chaining (SFC) Design

The Firefly Time Sync Service deployment leverages the Service Function Chaining (SFC) capabilities inherent in the DPF system, as described in the Baseline RDG (refer to HBN and OVN-Kubernetes SFC discussions in the Baseline RDG, Section "DPF Installation", Subsection "DPU Provisioning and Service Installation"). However, the introduction of Firefly for PTP traffic necessitates specific considerations and alterations to the traffic flow:

  • The deployment of the Firefly Time Sync Service modifies the existing Service Function Chain (SFC). The original SFC, designed for HBN and OVN-Kubernetes services, now takes the form of a branched structure. This "T-shaped" chain allows the Firefly service, residing on a dedicated branch, to directly communicate with the physical network interface for PTP message exchange.
  • Concurrently, DPF orchestrates a custom flow mechanism specifically for PTP's non-forwardable L2 multicast traffic (e.g., packets to 01-80-C2-00-00-0E). This mechanism ensures that these specialized PTP packets are handled distinctly from the primary workload data path, being precisely redirected only between the wire and the Firefly service on the DPU. Such isolation prevents the propagation of link-local PTP packets to other services in the chain, thereby maintaining the integrity of both PTP communication and general workload traffic.

image-2025-5-29_16-24-45-1.png

Firewall Design

The firewall design remains as described in the Baseline RDG (Section "Solution Architecture", Subsection "Solution Design", Sub-subsection "Firewall Design").

The PTP GM node is connected to both the High-Speed and Management networks, as shown in the diagram with the worker nodes.

PTP traffic for this internal cluster synchronization does not traverse the main firewall providing external connectivity.

Software Stack Components

This section updates the software stack from the Baseline RDG (Section "Solution Architecture", Subsection "Software Stack Components") with Firefly-specific components.

image-2025-6-18_12-57-4-1.png

Error: Make sure to use the exact same versions for the software stack as described above and in the Baseline RDG.

Bill of Materials

This section updates the Bill of Materials (BOM) from the Baseline RDG (Section "Solution Architecture", Subsection "Bill of Materials"). All other components remain as per the Baseline RDG.

image-2025-5-29_16-25-42-1.png

Deployment and Configuration

This section details the deployment and configuration steps, referencing the Baseline RDG where procedures are unchanged and detailing new or modified steps for Firefly Time Sync Service integration.

Node and Switch Definitions

These are the definitions and parameters used for deploying the demonstrated fabric:

Refer to the "Node and Switch Definitions" in the Baseline RDG (Section "Deployment and Configuration", Subsection "Node and Switch Definitions").

The following provides the definition for the new PTP Grandmaster Node switch port:

Switch Port Usage
Hostname Rack ID Ports
hs-switch 1 swp1,11-14,20
mgmt-switch 1 swp1-4
Hosts
Rack Server Type Server Name Switch Port IP and NICs Default Gateway

| Rack1 | PTP GM Node | ptp-gm | mgmt-switch: swp4 hs-switch: swp20 | eno4: 10.0.110.8/24 ens1f1np1: n/a | 10.0.110.254 |

Wiring

参考基线RDG（第"部署与配置"节，"布线"小节，包括"Hypervisor节点"和"K8s Worker节点"子小节）了解Hypervisor和K8s Worker节点的布线。

PTP GM Node

  • 基本布线类似于Worker节点（具有单个高速端口）
  • 将ptp-gm服务器的管理接口连接到mgmt-switch（例如SN2201）。
  • 将ptp-gm服务器的ConnectX-7接口（用于PTP）连接到hs-switch（例如SN3700）。从交换机的角度来看，该端口将作为PTP从端口，从GM接收时间。

image-2025-6-25_13-3-5-1.png

Fabric Configuration

Updating Cumulus Linux

与基线RDG（第"部署与配置"节，"Fabric配置"小节，"更新Cumulus Linux"子小节）相比无变化。确保交换机处于推荐的Cumulus Linux版本。

Configuring the Cumulus Linux Switch

本节详细介绍了交换机配置（hs-switch，例如SN3700）的修改，以启用PTP边界时钟功能。基线RDG（第"部署与配置"节，"Fabric配置"小节，"配置Cumulus Linux交换机"子小节）中关于BGP和基本L3网络的配置仍然是基础。以下是PTP的附加配置：

nv set service ptp 1 enable on
nv set service ptp 1 multicast-mac non-forwarding
nv set service ptp 1 current-profile default-itu-8275-1
nv set interface swp20 link state up
nv set interface swp20 type swp
nv set interface 11-14,20 ptp enable on
nv config apply -y

添加PTP GM节点后，SN2201交换机（mgmt-switch）配置如下：

nv set interface swp4 link state up
nv set interface swp4 type swp
nv set interface swp1-4 bridge domain br_default
nv config apply -y

Host Configuration

与基线RDG（第"部署与配置"节，"主机配置"小节）相比无变化。

Hypervisor Installation and Configuration

与基线RDG（第"部署与配置"节，"Hypervisor安装与配置"小节）相比无变化。

Prepare Infrastructure Servers

与基线RDG（第"部署与配置"节，"准备基础设施服务器"小节）中关于Firewall VM、Jump VM、MaaS VM的内容相比无变化。

Provision Master VMs and Worker Nodes Using MaaS

与基线RDG（第"部署与配置"节，"使用MaaS配置Master VM和Worker节点"小节）相比无变化。

PTP Grandmaster节点是本RDG中单独手动配置的节点。

PTP GrandMaster Server Configuration

如前所述，本RDG不涵盖Grandmaster节点的详细OS安装和基本服务器配置。本参考部署中的GM假定为安装了linuxptp包的Linux服务器，使用其ConnectX-7 NIC进行PTP。

以下是本RDG中PTP Grandmaster节点使用的参考ptp4l.conf配置文件。该文件通常应放置在GM服务器的/etc/linuxptp/ptp4l-master.conf。在此示例中，连接到高速交换机的接口为"ens1f1np1"。

[global]
#
domainNumber                    24
serverOnly                      1
verbose                         1
logging_level                   6
dataset_comparison                       G.8275.x
G.8275.defaultDS.localPriority                128
maxStepsRemoved                               255
logAnnounceInterval                            -3
logSyncInterval                                -4
logMinDelayReqInterval                         -4
G.8275.portDS.localPriority                   128
clockClass 6
ptp_dst_mac                     01:80:C2:00:00:0E
network_transport                              L2
fault_reset_interval                            1
hybrid_e2e 0

[ens1f1np1]

K8s Cluster Deployment and Configuration

Kubespray Deployment and Configuration

使用Kubespray对Master节点进行初始Kubernetes集群部署以及后续验证的过程与基线RDG（第"K8s集群部署与配置"节，子节："Kubespray部署与配置"、"使用Kubespray Ansible Playbook部署集群"、"K8s部署验证"）相比保持不变。

警告： 与基线RDG一样，Worker节点将在DPF和加速CNI的先决条件组件安装完成后添加。

DPF Installation

DPF安装过程（Operator、系统组件）基本遵循基线RDG。主要修改发生在"DPU配置和服务安装"期间，以部署Firefly时间同步服务配置。

Software Prerequisites and Required Variables

有关软件先决条件（如helm、envsubst）以及在export_vars.env中定义的必要环境变量，请参考基线RDG（第"DPF安装"节，"软件先决条件和必要变量"小节）。

CNI Installation

与基线RDG（第"DPF安装"节，"CNI安装"小节）相比无变化。

DPF Operator Installation

与基线RDG（第"DPF安装"节，"DPF Operator安装"小节）相比无变化。

DPF System Installation

与基线RDG（第"DPF安装"节，"DPF系统安装"小节）相比无变化。

Install Components to Enable Accelerated CNI Nodes

与基线RDG（第"DPF安装"节，"安装组件以启用加速CNI节点"小节）相比无变化。

DPU Provisioning and Service Installation

本节详细介绍了Firefly时间同步服务的部署。该过程涉及创建专用的Firefly自定义资源（CR）并配置必要的DPF对象，以便在DPU配置阶段同时部署该服务。

虽然部署DPUService（如OVN、HBN、DTS和BlueMan）的一般方法在基线RDG（第"DPF安装"节，"DPU配置和服务安装"小节）中已有介绍，但本节专门关注如何将Firefly服务与OVN和HBN核心服务一起部署。

  1. 在部署manifests/05-dpudeployment-installation目录下的对象之前，需要做一些调整以包含Firefly服务并获得更好的性能结果，如基线RDG所述。

#f60">  Create a new DPUFlavor using the following YAML: Per Baseline RDG: The parameter NUM_VF_MSIX is configured to 48 in the provided example, which is suited for the HP servers that were used in this RDG. Set this parameter to the physical number of cores in the NUMA node where the NIC is located.  A special annotation is used for creating Trusted SFs required by Firefly The Real Time Clock required by Firefly is enabled using the parameter: REAL_TIME_CLOCK_ENABLE THe NTP service is disabled on the DPU, as required by Firefly running phc2sys manifests/05-dpudeployment-installation/dpuflavor_perf_firefly.yaml Expand source --- apiVersion: provisioning.dpu.nvidia.com/v1alpha1 kind: DPUFlavor metadata: annotations: provisioning.dpu.nvidia.com/num-of-trusted-sfs: "3"   name: dpf-provisioning-hbn-ovn-performance-firefly namespace: dpf-operator-system spec: bfcfgParameters:

  • UPDATE_ATF_UEFI=yes
  • UPDATE_DPU_OS=yes
  • WITH_NIC_FW_UPDATE=yes configFiles:
  • operation: override path: /etc/mellanox/mlnx-bf.conf permissions: "0644" raw: | ALLOW_SHARED_RQ="no" IPSEC_FULL_OFFLOAD="no" ENABLE_ESWITCH_MULTIPORT="yes"
  • operation: override path: /etc/mellanox/mlnx-ovs.conf permissions: "0644" raw: | CREATE_OVS_BRIDGES="no"
  • operation: override path: /etc/mellanox/mlnx-sf.conf permissions: "0644" raw: "" grub: kernelParameters:
    • console=hvc0
    • console=ttyAMA0
    • earlycon=pl011,0x13010000
    • fixrttc
    • net.ifnames=0
    • biosdevname=0
    • iommu.passthrough=1
    • cgroup_no_v1=net_prio,net_cls
    • hugepagesz=2048kB
    • hugepages=8072 nvconfig:
  • device: '*' parameters:

    • PF_BAR2_ENABLE=0

    • PER_PF_NUM_SF=1

    • PF_TOTAL_SF=20

    • PF_SF_BAR_SIZE=10

    • NUM_PF_MSIX_VALID=0

    • PF_NUM_PF_MSIX_VALID=1

    • PF_NUM_PF_MSIX=228

    • INTERNAL_CPU_MODEL=1

    • INTERNAL_CPU_OFFLOAD_ENGINE=0

    • SRIOV_EN=1

    • NUM_OF_VFS=46

    • LAG_RESOURCE_ALLOCATION=1

    • NUM_VF_MSIX=48

    • REAL_TIME_CLOCK_ENABLE=1   ovs: rawConfigScript: | _ovs-vsctl() { ovs-vsctl --no-wait --timeout 15 "$@" }

      _ovs-vsctl set Open_vSwitch . other_config:doca-init=true _ovs-vsctl set Open_vSwitch . other_config:dpdk-max-memzones=50000 _ovs-vsctl set Open_vSwitch . other_config:hw-offload=true _ovs-vsctl set Open_vSwitch . other_config:pmd-quiet-idle=true _ovs-vsctl set Open_vSwitch . other_config:max-idle=20000 _ovs-vsctl set Open_vSwitch . other_config:max-revalidator=5000 _ovs-vsctl --if-exists del-br ovsbr1 _ovs-vsctl --if-exists del-br ovsbr2 _ovs-vsctl --may-exist add-br br-sfc _ovs-vsctl set bridge br-sfc datapath_type=netdev _ovs-vsctl set bridge br-sfc fail_mode=secure _ovs-vsctl --may-exist add-port br-sfc p0 _ovs-vsctl set Interface p0 type=dpdk _ovs-vsctl set Interface p0 mtu_request=9216 _ovs-vsctl set Port p0 external_ids:dpf-type=physical _ovs-vsctl --may-exist add-port br-sfc p1 _ovs-vsctl set Interface p1 type=dpdk _ovs-vsctl set Interface p1 mtu_request=9216 _ovs-vsctl set Port p1 external_ids:dpf-type=physical

      _ovs-vsctl set Open_vSwitch . external-ids:ovn-bridge-datapath-type=netdev _ovs-vsctl --may-exist add-br br-ovn _ovs-vsctl set bridge br-ovn datapath_type=netdev _ovs-vsctl set Interface br-ovn mtu_request=9216 _ovs-vsctl --may-exist add-port br-ovn pf0hpf _ovs-vsctl set Interface pf0hpf type=dpdk _ovs-vsctl set Interface pf0hpf mtu_request=9216

    cat <<EOT > /etc/netplan/99-dpf-comm-ch.yaml network: renderer: networkd version: 2 ethernets: pf0vf0: mtu: 9000 dhcp4: no bridges: br-comm-ch: dhcp4: yes interfaces: - pf0vf0 EOT

  # When running Firefly with phc2sys on the DPU, NTP must be disabled
  systemctl disable ntpsec --now </pre>
   </div>
  </div>
  <p></p>
 </li>
 <li>
  <p>Adjust&nbsp;<code>dpudeployment.yaml</code> to reference the<strong> DPUFlavor&nbsp;</strong>suited for performance/Firefly (This component provisions DPU on the worker nodes and defines a set of <strong>DPUServices</strong> and <strong>DPUServiceChain</strong> to run on those DPU. The DTS and BlueMan services are removed):
   <br></p>
  <div class="code panel pdl" style="border-width: 1px;">
   <div class="codeHeader panelHeader pdl hide-border-bottom" style="border-bottom-width: 1px;">
    <b class="code-title">manifests/05-dpudeployment-installation/dpudeployment.yaml</b><span class="collapse-source expand-control" style="display:none;"><span class="expand-control-icon icon">&nbsp;</span><span class="expand-control-text">Expand source</span></span><span class="collapse-spinner-wrapper"></span>
   </div>
   <div class="codeContent panelContent pdl hide-toolbar">
    <pre class="syntaxhighlighter-pre" data-syntaxhighlighter-params="brush: python; gutter: false; theme: Midnight; collapse: true" data-theme="Midnight">---

apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUDeployment metadata: name: ovn-hbn-firefly namespace: dpf-operator-system spec: dpus: bfb: bf-bundle flavor: dpf-provisioning-hbn-ovn-performance-firefly dpuSets: - nameSuffix: "dpuset1" nodeSelector: matchLabels: feature.node.kubernetes.io/dpu-enabled: "true" services: ovn: serviceTemplate: ovn serviceConfiguration: ovn hbn: serviceTemplate: hbn serviceConfiguration: hbn firefly-dpu: serviceConfiguration: firefly-dpu serviceTemplate: firefly-dpu firefly-host: serviceConfiguration: firefly-host serviceTemplate: firefly-host serviceChains: switches: - ports: - serviceInterface: matchLabels: uplink: p0 - service: name: hbn interface: p0_if - service: interface: firefly_if name: firefly-dpu - ports: - serviceInterface: matchLabels: uplink: p1 - service: name: hbn interface: p1_if - service: interface: firefly2_if name: firefly-dpu - ports: - serviceInterface: matchLabels: port: ovn - service: name: hbn interface: pf2dpu2_if Set the mtu to 8940 for the OVN DPUServiceConfig (to deploy the OVN Kubernetes workloads on the DPU with the same MTU as in the host): manifests/05-dpudeployment-installation/dpuserviceconfig_ovn.yaml Expand source --- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceConfiguration metadata: name: ovn namespace: dpf-operator-system spec: deploymentServiceName: "ovn" serviceConfiguration: helmChart: values: k8sAPIServer: https://$TARGETCLUSTER_API_SERVER_HOST:$TARGETCLUSTER_API_SERVER_PORT podNetwork: $POD_CIDR/24 serviceNetwork: $SERVICE_CIDR mtu: 8940 dpuManifests: kubernetesSecretName: "ovn-dpu" # user needs to populate based on DPUServiceCredentialRequest vtepCIDR: "10.0.120.0/22" # user needs to populate based on DPUServiceIPAM hostCIDR: $TARGETCLUSTER_NODE_CIDR # user needs to populate ipamPool: "pool1" # user needs to populate based on DPUServiceIPAM ipamPoolType: "cidrpool" # user needs to populate based on DPUServiceIPAM ipamVTEPIPIndex: 0 ipamPFIPIndex: 1 Create a new DPUServiceNAD to allow FIreFly to consume a network with Trusted SF resources and without IPAM:  manifests/05-dpudeployment-installation/dpuservicenad_firefly.yaml Expand source --- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceNAD metadata: name: mybrsfc-firefly namespace: dpf-operator-system annotations: dpuservicenad.svc.dpu.nvidia.com/use-trusted-sfs: "" spec: resourceType: sf ipam: false bridge: "br-sfc" mtu: 1500 Create a new DPUServiceConfig (references to firefly DPUServiceNAD network) and DPUServiceTemplate for the  Firefly DPU service:  YAML --- apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceConfiguration metadata: name: firefly-dpu namespace: dpf-operator-system spec: deploymentServiceName: firefly-dpu interfaces: - name: firefly_if network: mybrsfc-firefly - name: firefly2_if network: mybrsfc-firefly serviceConfiguration: configPorts: ports: - name: monitor port: 25600

protocol: TCP serviceType: ClusterIP serviceDaemonSet: labels: svc.dpu.nvidia.com/custom-flows: firefly helmChart: values: exposedPorts: ports: monitor: true ptpConfig: ptp.conf ptpInterfaces: firefly_if config: content: ptp.conf: | [global] domainNumber 24 clientOnly 1 verbose 1 logging_level 6 dataset_comparison G.8275.x G.8275.defaultDS.localPriority 128 maxStepsRemoved 255 logAnnounceInterval -3 logSyncInterval -4 logMinDelayReqInterval -4 G.8275.portDS.localPriority 128 ptp_dst_mac 01:80:C2:00:00:0E network_transport L2 fault_reset_interval 1 hybrid_e2e 0

      [firefly_if]
      [firefly2_if]

apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceTemplate metadata: name: firefly-dpu namespace: dpf-operator-system spec: deploymentServiceName: firefly-dpu helmChart: source: chart: doca-firefly repoURL: $HELM_REGISTRY_REPO_URL version: 1.1.5 values: config: isLocalPath: false containerImage: nvcr.io/nvidia/doca/doca_firefly:1.7.1-doca3.0.0 enableTXPortTimestampOffloading: true hostNetwork: false monitorState: 0.0.0.0 phc2sysArgs: -a -r -l 6 resourceRequirements: memory: 512Mi

apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceConfiguration metadata: name: firefly-host namespace: dpf-operator-system spec: deploymentServiceName: firefly-host upgradePolicy: applyNodeEffect: false serviceConfiguration: deployInCluster: true helmChart: values: monitorStateFromDPUService: firefly-dpu

apiVersion: svc.dpu.nvidia.com/v1alpha1 kind: DPUServiceTemplate metadata: name: firefly-host namespace: dpf-operator-system spec: deploymentServiceName: firefly-host helmChart: source: chart: doca-firefly repoURL: $HELM_REGISTRY_REPO_URL version: 1.1.6 values: containerImage: nvcr.io/nvidia/doca/doca_firefly:1.7.2-doca3.0.0-host hostNetwork: false monitorClientPhc2sysInterface: eth0 monitorClientType: phc2sys phc2sysState: disable ppsDevice: disable ppsState: do_nothing ptpState: disable tolerations: - effect: NoSchedule key: k8s.ovn.org/network-unavailable operator: Exists resourceRequirements: memory: 512Mi

The rest of the configuration files remain the same, including:

  • BFB to download BlueField Bitstream to a shared volume.

    # manifests/05-dpudeployment-installation/bfb.yaml
---
apiVersion: provisioning.dpu.nvidia.com/v1alpha1
kind: BFB
metadata:
  name: bf-bundle
  namespace: dpf-operator-system
spec:
  url: $BLUEFIELD_BITSTREAM

  • OVN DPUServiceTemplate to deploy OVN Kubernetes workloads to the DPU.

    # manifests/05-dpudeployment-installation/dpuservicetemplate_ovn.yaml
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "ovn"
  helmChart:
    source:
      repoURL: $OVN_KUBERNETES_REPO_URL
      chart: ovn-kubernetes-chart
      version: $TAG
    values:
      commonManifests:
        enabled: true
      dpuManifests:
        enabled: true
      leaseNamespace: "ovn-kubernetes"
      gatewayOpts: "--gateway-interface=br-ovn --gateway-uplink-port=puplinkbrovn"

  • HBN DPUServiceConfig and DPUServiceTemplate to deploy HBN workloads to the DPU.

    # manifests/05-dpudeployment-installation/dpuserviceconfig_hbn.yaml
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: hbn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "hbn"
  serviceConfiguration:
    serviceDaemonSet:
      annotations:
        k8s.v1.cni.cncf.io/networks: |-
          [
          {"name": "iprequest", "interface": "ip_lo", "cni-args": {"poolNames": ["loopback"], "poolType": "cidrpool"}},
          {"name": "iprequest", "interface": "ip_pf2dpu2", "cni-args": {"poolNames": ["pool1"], "poolType": "cidrpool", "allocateDefaultGateway": true}}
          ]
    helmChart:
      values:
        configuration:
          perDPUValuesYAML: |
            - hostnamePattern: "*"
              values:
                bgp_peer_group: hbn
            - hostnamePattern: "worker1*"
              values:
                bgp_autonomous_system: 65101
            - hostnamePattern: "worker2*"
              values:
                bgp_autonomous_system: 65201
          startupYAMLJ2: |
            - header:
                model: BLUEFIELD
                nvue-api-version: nvue_v1
                rev-id: 1.0
                version: HBN 2.4.0
            - set:
                interface:
                  lo:
                    ip:
                      address:
                        {{ ipaddresses.ip_lo.ip }}/32: {}
                    type: loopback
                  p0_if,p1_if:
                    type: swp
                    link:
                      mtu: 9000
                  pf2dpu2_if:
                    ip:
                      address:
                        {{ ipaddresses.ip_pf2dpu2.cidr }}: {}
                    type: swp
                    link:
                      mtu: 9000
                router:
                  bgp:
                    autonomous-system: {{ config.bgp_autonomous_system }}
                    enable: on
                    graceful-restart:
                      mode: full
                    router-id: {{ ipaddresses.ip_lo.ip }}
                vrf:
                  default:
                    router:
                      bgp:
                        address-family:
                          ipv4-unicast:
                            enable: on
                            redistribute:
                              connected:
                                enable: on
                          ipv6-unicast:
                            enable: on
                            redistribute:
                              connected:
                                enable: on
                        enable: on
                        neighbor:
                          p0_if:
                            peer-group: {{ config.bgp_peer_group }}
                            type: unnumbered
                          p1_if:
                            peer-group: {{ config.bgp_peer_group }}
                            type: unnumbered
                        path-selection:
                          multipath:
                            aspath-ignore: on
                        peer-group:
                          {{ config.bgp_peer_group }}:
                            remote-as: external

  interfaces:
    ## NOTE: Interfaces inside the HBN pod must have the `_if` suffix due to a naming convention in HBN.
  - name: p0_if
    network: mybrhbn
  - name: p1_if
    network: mybrhbn
  - name: pf2dpu2_if
    network: mybrhbn

    # manifests/05-dpudeployment-installation/dpuservicetemplate_hbn.yaml
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
  name: hbn
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "hbn"
  helmChart:
    source:
      repoURL: $HBN_REPO_URL
      chart: hbn-chart
      version: $TAG
    values:
      ...

dpf-operator-system
spec:
  deploymentServiceName: "hbn"
  helmChart:
    source:
      repoURL: $HELM_REGISTRY_REPO_URL
      version: 1.0.2
      chart: doca-hbn
    values:
      image:
        repository: $HBN_NGC_IMAGE_URL
        tag: 3.0.0-doca3.0.0
      resources:
        memory: 6Gi
        nvidia.com/bf_sf: 3

  • OVN DPUServiceCredentialRequest 用于允许跨集群通信。

    ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceCredentialRequest
metadata:
  name: ovn-dpu
  namespace: dpf-operator-system
spec:
  serviceAccount:
    name: ovn-dpu
    namespace: dpf-operator-system
  duration: 24h
  type: tokenFile
  secret:
    name: ovn-dpu
    namespace: dpf-operator-system
  metadata:
    labels:
      dpu.nvidia.com/image-pull-secret: ""

  • DPUServiceInterfaces 用于 DPU 上的物理端口。

    ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: p0
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            uplink: "p0"
        spec:
          interfaceType: physical
          physical:
            interfaceName: p0
---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: p1
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            uplink: "p1"
        spec:
          interfaceType: physical
          physical:
            interfaceName: p1

  • OVN DPUServiceInterface 用于定义连接到 DPU 上 OVN 工作负载的端口。

    ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceInterface
metadata:
  name: ovn
  namespace: dpf-operator-system
spec:
  template:
    spec:
      template:
        metadata:
          labels:
            port: ovn
        spec:
          interfaceType: ovn

  • DPUServiceIPAM 用于在 DPUCluster 上设置 IP 地址管理。

    ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
  name: pool1
  namespace: dpf-operator-system
spec:
  ipv4Network:
    network: "10.0.120.0/22"
    gatewayIndex: 3
    prefixSize: 29

  • DPUServiceIPAM 用于 HBN 中的环回接口。

    ---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceIPAM
metadata:
  name: loopback
  namespace: dpf-operator-system
spec:
  ipv4Network:
    network: "11.0.0.0/24"
    prefixSize: 32

  1. 使用以下命令应用上述所有 YAML 文件：

    Jump Node Console

    $ cat manifests/05-dpudeployment-installation/*.yaml | envsubst | kubectl apply -f -

  2. 通过确保 DPUServices 已创建并已协调、DPUServiceIPAMs 已协调、DPUServiceInterfaces 已协调以及 DPUServiceChains 已协调，验证 DPUService 安装：

    注意：这些验证命令可能需要多次运行以确保满足条件。

    Jump Node Console

    $ kubectl wait --for=condition=ApplicationsReconciled --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_ovn-hbn-firefly
dpuservice.svc.dpu.nvidia.com/firefly-dpu-4v26p condition met
dpuservice.svc.dpu.nvidia.com/firefly-host-d5c97 condition met
dpuservice.svc.dpu.nvidia.com/hbn-77jcn condition met
dpuservice.svc.dpu.nvidia.com/ovn-6xnbh condition met

$ kubectl wait --for=condition=DPUIPAMObjectReconciled --namespace dpf-operator-system dpuserviceipam --all
dpuserviceipam.svc.dpu.nvidia.com/loopback condition met
dpuserviceipam.svc.dpu.nvidia.com/pool1 condition met

$ kubectl wait --for=condition=ServiceInterfaceSetReconciled --namespace dpf-operator-system dpuserviceinterface --all
dpuserviceinterface.svc.dpu.nvidia.com/firefly-dpu-firefly-if-v8r7j condition met
dpuserviceinterface.svc.dpu.nvidia.com/firefly-dpu-firefly2-if-h6hhd condition met
dpuserviceinterface.svc.dpu.nvidia.com/hbn-p0-if-6jprb condition met
dpuserviceinterface.svc.dpu.nvidia.com/hbn-p1-if-fh2w6 condition met
dpuserviceinterface.svc.dpu.nvidia.com/hbn-pf2dpu2-if-wks6w condition met
dpuserviceinterface.svc.dpu.nvidia.com/ovn condition met
dpuserviceinterface.svc.dpu.nvidia.com/p0 condition met
dpuserviceinterface.svc.dpu.nvidia.com/p1 condition met

$ kubectl wait --for=condition=ServiceChainSetReconciled --namespace dpf-operator-system dpuservicechain --all
dpuservicechain.svc.dpu.nvidia.com/ovn-hbn-firefly-d7vtb condition met

K8s Cluster Scale-out

Add Worker Nodes to the Cluster

向集群添加工作节点的过程与基线 RDG（“K8s Cluster Scale-out”部分，“Add Worker Nodes to the Cluster”小节）保持不变。

  • 参考基线 RDG：“K8s Cluster Scale-out”部分，“Add Worker Nodes to the Cluster”小节。
  • 添加新工作节点时，DPF 将配置其 DPU 并将所有已配置的 DPUService（包括新添加的 Firefly DPU 和 Host Monitor 服务）部署到这些节点/DPU 上。

错误：部署 Firefly Host Service 后，请确保在工作节点上禁用 NTP。

恭喜——DPF 系统已成功安装！

Verification

本节详细说明如何验证整体部署。常规 DPF 系统验证（DPU 就绪状态、核心组件如 Multus、SR-IOV、OVN 在主机/DPU 上的 DaemonSet 状态）与基线 RDG（“Verification”部分）保持一致。

Infrastructure Latency & Bandwidth Validation

Infrastructure Latency & Bandwidth Validation

No changes from the Baseline RDG (Section "Verification", Subsection "Infrastructure Latency & Bandwidth Validation"). This RDG does not include new performance tests or validation beyond time synchronization.

Time Sync Service Verification

PTP State Monitoring from Tenant K8s Host

The Firefly host-monitor service should provide logs or status indicating the PTP synchronization state of the DPU that it is monitoring.

  • Verify that a Firefly pod is running on each host and retrieve its name:
$ kubectl get pod -n dpf-operator-system -o wide | grep firefly
doca-firefly-dgnmf                                            1/1     Running     1 (2m33s ago)   2m40s   10.233.68.22   worker1   <none>           <none>
doca-firefly-pkxsm                                            1/1     Running     1 (2m33s ago)   2m40s   10.233.67.12   worker2   <none>           <none>
  • View logs of a specific pod:

Jump Node Console

$ kubectl logs -n dpf-operator-system doca-firefly-dgnmf

  • In the logs, look for output similar to the example below, which indicates the PTP and host synchronization status. Key fields to observe include, among others:

    • gmIdentity: The identity of the current Grandmaster clock.
    • port_state: Should indicate Active for the DPU's PTP ports when synchronized.
    • master_offset: Shows the average, maximum, and root mean square (rms) offset from the master clock in nanoseconds. Lower, stable values are desirable.
    • ptp_stable: Should indicate Yes or Recovered when PTP synchronization is stable.
    • ptp_time (TAI) and system_time (UTC) (under DPU information): These should reflect the current PHC time and the DPU's system time.
    • ptp_ports: Lists the state of the DPU's PTP ports (e.g., one Slave and other Listening if redundant ports are configured).

  • PTP Monitor log example:

PTP Monitor Logs

gmIdentity:                B8:3F:D2:FF:FE:6A:E7:67 (b83fd2.fffe.6ae767)
portIdentity:              F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-1)
port_state:                Active
domainNumber:              24
master_offset:             avg: 20      max:    30      rms:    4
gmPresent:                 true
ptp_stable:                Yes
UtcOffset:                 37
timeTraceable:             0
frequencyTraceable:        0
grandmasterPriority1:      128
gmClockClass:              6
gmClockAccuracy:           0xfe
grandmasterPriority2:      128
gmOffsetScaledLogVariance: 0xffff
ptp_time (TAI):            Tue May 27 12:09:27 2025
ptp_time (UTC adjusted):   Tue May 27 12:08:50 2025
system_time (UTC):         Tue May 27 12:08:50 2025
ptp_ports:                 F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-1) - Slave
                           F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-2) - Listening

Host information:
system_time (UTC):    Tue May 27 12:08:50 2025
phc_time (TAI):       Tue May 27 12:09:27 2025
  • For additional PTP Monitor information, refer to the DOCA Firefly Service Guide (References list).

Automatic Host System Clock Sync Verification

Warning: Make sure NTP is disabled on the Worker Nodes once the Firefly Host Service is deployed.

As mentioned in this RDG, the Firefly Host Monitor service is also responsible for syncing the host OS system clock to the PHC, and using the PHC2SYS program as a third-party OS time calibration provider.

  • Connect to one of the tenant K8s worker node hosts and verify that NTP services are inactive/disabled.
  • Check the following log created by the service on the host filesystem:

Worker Host Console

worker1:~# tail -f /var/log/doca/firefly/monitor_client_phc2sys.log
phc2sys[1112425.357]: CLOCK_REALTIME phc offset        14 s2 freq   +8045 delay    521
phc2sys[1112426.357]: CLOCK_REALTIME phc offset         1 s2 freq   +8036 delay    498
phc2sys[1112427.357]: CLOCK_REALTIME phc offset        19 s2 freq   +8055 delay    513
phc2sys[1112428.357]: CLOCK_REALTIME phc offset        -9 s2 freq   +8032 delay    520
phc2sys[1112429.358]: CLOCK_REALTIME phc offset        -7 s2 freq   +8032 delay    521
phc2sys[1112430.358]: CLOCK_REALTIME phc offset       -11 s2 freq   +8025 delay    511
phc2sys[1112431.358]: CLOCK_REALTIME phc offset        -9 s2 freq   +8024 delay    520
phc2sys[1112432.358]: CLOCK_REALTIME phc offset       -11 s2 freq   +8019 delay    520
phc2sys[1112433.359]: CLOCK_REALTIME phc offset         4 s2 freq   +8031 delay    523
phc2sys[1112434.378]: CLOCK_REALTIME phc offset         3 s2 freq   +8031 delay    520
phc2sys[1112435.379]: CLOCK_REALTIME phc offset       -13 s2 freq   +8016 delay    521
  • The log should be actively updating, indicating that PHC2SYS is running and periodically comparing and adjusting the host's CLOCK_REALTIME (system clock) against the DPU's PHC.
  • Stable frequency/delay values and consistently small offset values are good indicators for close and stable synchronization between the host clock and the DPU PHC.

The Monitoring information presented by the Firefly Host Monitor service also provides indications of the host's current system time under the "Host information" section.

  • PTP Monitor log example–DPU information:

PTP Monitor Logs

ptp_time (TAI):            Tue May 27 14:48:59 2025
ptp_time (UTC adjusted):   Tue May 27 14:48:22 2025
system_time (UTC):         Tue May 27 14:48:22 2025
  • PTP Monitor log example–Host information:

PTP Monitor Logs

Host information:
system_time (UTC):    Tue May 27 14:48:22 2025
phc_time (TAI):       Tue May 27 14:48:59 2025

  • Host information:

    • phc_time (TAI): Current PHC time detected by Firefly host service, should match the PHC time presented by DPU (ptp_time TAI)
    • system_time (UTC): Host's system clock, should be synchronized to the DPU's PHC, after accounting for the UTC offset (e.g., 37 seconds for TAI to UTC). These times should be very closely aligned. Host system time (UTC) should match the DPU's system_time (UTC) as both services are using PHC2SYS to sync the system time to a shared PHC.

  • Issue "date" command on the host to verify the current system time matches the one shown in the PTP Monitor log. You can compare it to a known accurate time source (e.g., the PTP GM's system clock). The drift should be minimal and within expected PTP accuracy.

Worker Host Console

worker1:~# date --iso-8601=ns
2025-05-27T14:48:25,011078256+00:00

Link Failure/Recovery (PTP Failover on DPU)

The Firefly Host-monitor service should provide logs or status indicating the PTP synchronization state of the DPU it's monitoring information.

  • Simulate Link Failure–Administratively bring down the link for the active PTP port on one of the DPU from the switch side.

SN3700 Switch Console

nv set interface swp11 link state down
nv config apply -y
  • Observe failover via PTP Monitor logs on the relevant worker host—look for "State Recovered", an increased error count, and the second port acquiring the "Slave" role.

PTP Monitor Logs

gmIdentity:                B8:3F:D2:FF:FE:6A:E7:67 (b83fd2.fffe.6ae767)
portIdentity:              F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-2)
port_state:                Active
domainNumber:              24
master_offset:             avg: 47      max:    151     rms:    54
gmPresent:                 true
ptp_stable:                Recovered
UtcOffset:                 37
timeTraceable:             0
frequencyTraceable:        0
grandmasterPriority1:      128
gmClockClass:              6
gmClockAccuracy:           0xfe
grandmasterPriority2:      128
gmOffsetScaledLogVariance: 0xffff
ptp_time (TAI):            Tue May 27 15:02:10 2025
ptp_time (UTC adjusted):   Tue May 27 15:01:33 2025
system_time (UTC):         Tue May 27 15:01:33 2025
ptp_ports:                 F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-1) - Listening
                           F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-2) - Slave
error_count:               1
last_err_time (UTC):       Tue May 27 15:01:04 2025

Host information:
system_time (UTC):    Tue May 27 15:01:33 2025
phc_time (TAI):       Tue May 27 15:02:10 2025

  • Simulate Link Recovery–Administratively bring down the network link for the active PTP port on the switch.

    SN3700 Switch Console

    nv set interface swp11 link state up
nv config apply -y

  • Observe Recovery via PTP Monitor logs–look for "State Recovered", an increased error count, and the first port reqcquiring the "Slave" role .

    PTP Monitor Logs

    gmIdentity:                B8:3F:D2:FF:FE:6A:E7:67 (b83fd2.fffe.6ae767)
portIdentity:              F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-1)
port_state:                Active
domainNumber:              24
master_offset:             avg: 0       max:    11      rms:    5
gmPresent:                 true
ptp_stable:                Recovered
UtcOffset:                 37
timeTraceable:             0
frequencyTraceable:        0
grandmasterPriority1:      128
gmClockClass:              6
gmClockAccuracy:           0xfe
grandmasterPriority2:      128
gmOffsetScaledLogVariance: 0xffff
ptp_time (TAI):            Tue May 27 15:04:39 2025
ptp_time (UTC adjusted):   Tue May 27 15:04:02 2025
system_time (UTC):         Tue May 27 15:04:02 2025
ptp_ports:                 F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-1) - Slave
                           F2:77:76:FF:FE:1A:BE:19 (f27776.fffe.1abe19-2) - Listening
error_count:               2
last_err_time (UTC):       Tue May 27 15:03:52 2025

Host information:
system_time (UTC):    Tue May 27 15:04:02 2025
phc_time (TAI):       Tue May 27 15:04:39 2025

Authors
image-2025-9-15_10-4-30.png Itai LevyOver the past few years, Itai Levy has worked as a 解决方案 Architect and member of the NVIDIA Networking “解决方案 Labs” team. Itai designs and executes cutting-edge solutions around Cloud Computing, Software-Defined Networking, Storage and Security. His main areas of expertise include NVIDIA BlueField Data Processing Unit (DPU) solutions and accelerated K8s/OpenStack platforms.