基于DPF和DTS的集中式DPU监控解决方案参考部署指南

本参考部署指南（RDG）提供了在Kubernetes集群中使用DPF部署和管理NVIDIA BlueField-3 DPU上的DOCA Telemetry Service（DTS）实例，并设置集中式监控堆栈的详细说明。

文档目录

Created on May 22, 2025

范围

利用NVIDIA的DPF，管理员可以在Kubernetes集群中配置和管理DPU，同时部署和编排基础设施服务（如HBN和加速的OVN-Kubernetes）。结合DTS，这可以实现对DPU资源的广泛监控。该方法充分利用了NVIDIA DPU硬件加速和卸载能力，最大限度地提高了数据中心的负载效率和性能。

本文档面向希望部署高性能、支持DPU的Kubernetes集群并监控其DPU资源的经验丰富的系统管理员、系统工程师和解决方案架构师。

注意

本RDG假设用户已经安装了DPF，成功配置了DPU，并部署了不同的DOCA服务（其中包括DOCA Telemetry Service）。

有关DPF安装的更多信息，请参阅：RDG for DPF with OVN-Kubernetes and HBN Services - NVIDIA Docs。

警告

本参考实现，顾名思义，是一个特定的、有倾向性的部署示例，旨在解决上述用例。

虽然可能存在其他方法来实现类似的解决方案，但本文档提供了此特定方法的详细指南。

缩写和首字母缩略词

术语	定义	术语	定义
DOCA	Data Center Infrastructure-on-a-Chip Architecture	K8S	Kubernetes
DPF	DOCA Platform Framework	OVN	Open Virtual Network
DPU	Data Processing Unit	PVC	Persistent Volume Claim
DTS	DOCA Telemetry Service	RDG	Reference Deployment Guide
HBN	Host Based Networking	TSDB	Time Series Database

引言

DOCA Platform Framework (DPF) 是一个在Kubernetes集群中配置和编排NVIDIA BlueField DPU及DPU服务的系统。

DPF通过Kubernetes API提供编排，处理DPU配置和生命周期管理，并支持在DPU上高效部署和编排基础设施服务，从而简化了DPU管理。

其中一项服务是DOCA Telemetry Service (DTS)，它从内置提供程序和外部遥测应用程序收集数据。DTS支持多种导出机制，包括可由Prometheus服务器抓取的Prometheus端点。使用Grafana作为收集数据的可视化平台，用户可以方便地监控其DPU资源。

在由DPF配置和管理的大型DPU集群中，运行着相关的DTS服务，因此需要一种自动化且可扩展的方法来监控这些DTS实例，以避免给集群和系统管理员带来过重负担。

通过利用DPF编排能力、Kubernetes原生工具和Prometheus服务发现，可以实现高效的监控解决方案。

本指南提供了此类解决方案的实用示例，演示了如何实现集中式DPU监控。

参考资料

解决方案架构

关键组件和技术

NVIDIA BlueField® Data Processing Unit (DPU) NVIDIA® BlueField® 数据处理单元 (DPU) 为现代数据中心和超级计算集群带来了前所未有的创新。凭借其强大的计算能力和集成的软件定义硬件加速器（用于网络、存储和安全），BlueField 为任何环境中的任何工作负载创建了安全且加速的基础设施，开启了加速计算和人工智能的新时代。
NVIDIA DOCA Software Framework NVIDIA DOCA™ 释放了 NVIDIA® BlueField® 网络平台的潜力。通过利用 BlueField DPU 和 SuperNIC 的强大功能，DOCA 能够快速创建卸载、加速和隔离数据中心工作负载的应用程序和服务。它使开发人员能够创建软件定义的、云原生的、DPU 和 SuperNIC 加速的服务，并具有零信任保护，满足现代数据中心的性能和安全需求。
NVIDIA ConnectX SmartNICs 10/25/40/50/100/200 和 400G 以太网网卡业界领先的 NVIDIA® ConnectX® 智能网卡系列提供先进的硬件卸载和加速功能。 NVIDIA 以太网适配器为超大规模、公有云和私有云、存储、机器学习、AI、大数据和电信平台提供了最高的 ROI 和最低的总拥有成本。
NVIDIA LinkX Cables NVIDIA® LinkX® 线缆和收发器产品系列提供了业界最完整的 10、25、40、50、100、200 和 400GbE 以太网以及 100、200 和 400Gb/s InfiniBand 产品线，适用于云、HPC、超大规模、企业、电信、存储和人工智能数据中心应用。

NVIDIA Spectrum 以太网交换机

Flexible form-factors with 16 to 128 physical ports, supporting 1GbE through 400GbE speeds.

Based on a ground-breaking silicon technology optimized for performance and scalability, NVIDIA Spectrum switches are ideal for building high-performance, cost-effective, and efficient Cloud Data Center Networks, Ethernet Storage Fabric, and Deep Learning Interconnects.

NVIDIA combines the benefits of NVIDIA Spectrum™ switches, based on an industry-leading application-specific integrated circuit (ASIC) technology, with a wide variety of modern network operating system choices, including NVIDIA Cumulus® Linux, SONiC and NVIDIA Onyx®.

NVIDIA Cumulus Linux

NVIDIA® Cumulus® Linux is the industry's most innovative open network operating system that allows you to automate, customize, and scale your data center network like no other.

Kubernetes

Kubernetes is an open-source container orchestration platform for deployment automation, scaling, and management of containerized applications.

OVN-Kubernetes

OVN-Kubernetes (Open Virtual Networking - Kubernetes) is an open-source project that provides a robust networking solution for Kubernetes clusters with OVN (Open Virtual Networking) and Open vSwitch (Open Virtual Switch) at its core. It is a Kubernetes networking conformant plugin written according to the CNI (Container Network Interface) specifications.

Solution Design

The solution design is based on RDG for DPF with OVN-Kubernetes and HBN Services - Solution Design.

K8s Cluster Logical Design - Monitoring Stack

The following K8s logical design illustration demonstrates the main components of the monitoring stack in this solution:

1 x Prometheus server pod - scrapes metrics from instrumented jobs.
1 x Grafana pod - provides visualization for the collected data.
1 x Alertmanager pod - handles alerts sent by the Prometheus server.

The entire monitoring stack is deployed and managed using the kube-prometheus-stack Helm chart. Each pod is deployed as a StatefulSet, which also manages the PVCs providing persistent storage. Using service discovery (DNS based in this example), and by configuring the DTS DPUService to expose its Prometheus endpoint port to the host cluster, the Prometheus server can automatically detect every DTS instance in the DPU K8s cluster and pull metrics from it.

Warning

A more detailed explanation about the solution is provided in the following sections of the RDG.

The DPU K8s control plane is omitted from this scheme to simplify the view. For further details about the K8s cluster design, refer to RDG for DPF with OVN-Kubernetes and HBN Services - K8s Cluster Logical Design.

Software Stack Components

Danger Make sure to use the exact same versions for the software stack as described above.

Bill of Materials

The bill of materials is based upon the same hardware as demonstrated in RDG for DPF with OVN-Kubernetes and HBN Services - Bill of Materials.

Deployment and Configuration

DTS Upgrade to Configure ConfigPorts

The following section explains how to leverage the DNS capabilities provided by Kubernetes to obtain a dynamic list of all the replicas for a service. This allows Prometheus to be kept informed automatically about which DTS instances it needs to scrape, without needing to statically reconfigure it every time an additional DTS instance is added to the K8s cluster.

To achieve this, several configurations are required:

A headless service on the host cluster, which in turn will create an SRV record, allowing Prometheus to utilize its DNS-based service discovery feature.
The DTS Prometheus endpoint port (9100 by default) needs to be exposed to the host cluster via a NodePort service.
An EndPointSlice to back the headless service on the host cluster with the DPU IPs as its endpoints and the nodePorts values as its port.

Fortunately, all of these can be configured using the ConfigPorts field for the DTS DPUService. For more information on this feature, refer to: dpuservice-configPorts.

The following illustration demonstrates the explanation provided above:

Proceed with the following configuration:

Upgrade the DTS DPUService using the following configuration:

manifests/05-dpudeployment-installation/dpuserviceconfig_dts.yaml

---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceConfiguration
metadata:
  name: dts
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "dts"
  serviceConfiguration:
    configPorts:
      serviceType: None
      ports:
        - name: httpserverport
          protocol: TCP
          port: 9100

manifests/05-dpudeployment-installation/dpuservicetemplate_dts.yaml

---
apiVersion: svc.dpu.nvidia.com/v1alpha1
kind: DPUServiceTemplate
metadata:
  name: dts
  namespace: dpf-operator-system
spec:
  deploymentServiceName: "dts"
  helmChart:
    source:
      repoURL: $HELM_REGISTRY_REPO_URL
      version: 1.0.6
      chart: doca-telemetry
    values:
      exposedPorts:
        ports:
          httpserverport: true

Run the following command:
Warning
- The following command assumes the user has cloned the doca-platform Git repository, changed to the necessary directory inside it and defined the variables required for the DPF installation.
- For more information, check: RDG for DPF with OVN-Kubernetes and HBN Services - DPF Installation Software Prerequisites and Required Variables.
Jump Node Console
```
$ cat manifests/05-dpudeployment-installation/*dts.yaml | envsubst | kubectl apply -f -
```

Verify that the DTS DPUService is in the ready state and that a headless service has been created on the host cluster:

Warning The following verification commands may need to be run multiple times to ensure the condition is met.

Jump Node Console

$ kubectl wait --for=condition=ApplicationsReady --namespace dpf-operator-system dpuservices -l svc.dpu.nvidia.com/owned-by-dpudeployment=dpf-operator-system_ovn-hbn | grep dts
dpuservice.svc.dpu.nvidia.com/dts-mk55x condition met

$ kubectl get svc -n dpf-operator-system | grep dts
dts-mk55x                                           ClusterIP   None            <none>        9100/TCP                     2m24s

Verify that the SRV record is resolvable from the host cluster:
1. In this example the master1 node is used (since it has an IP in the pod subnet). SSH into the respective node:
  
  Jump Node Console
```
depuser@jump:~$ ssh master1
```
2. Resolve the headless service SRV record, which should return all DTS endpoint SRV records (2 in this example):
  
  Warning Replace dts-mk55x with your service name.
  
  Master1 Console
```
depuser@master1:~# dig srv _httpserverport._tcp.dts-mk55x.dpf-operator-system.svc.cluster.local +short
0 50 30342 worker1-0000-89-00.dts-mk55x.dpf-operator-system.svc.cluster.local.
0 50 30342 worker2-0000-89-00.dts-mk55x.dpf-operator-system.svc.cluster.local.
```

Setup Centralized Monitoring Stack

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Grafana is an open-source software which allows users to query, visualize, alert on, and explore their metrics, logs, and traces wherever they are stored and turn TSDBs data into insightful graphs and visualizations.

In this RDG, the monitoring stack will be installed using the kube-prometheus-stack Helm chart. This chart installs the core components of the kube-prometheus stack, including a collection of Kubernetes manifests, Grafana dashboards, Prometheus rules, and documentation and scripts. Together they provide an easy-to-operate, end-to-end monitoring solution for Kubernetes cluster using Prometheus Operator.

Add the Prometheus-Community repository and update it:

Jump Node Console

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update

The following kube-prometheus-stack.yaml values file will be applied:

Warning

The Prometheus server is already configured with DNS-based service discovery to automatically discover all DTS instances in the cluster (DNS points to the headless service SRV record created earlier).

The Prometheus server, Grafana, and Alertmanager StatefulSets are backed by PVCs using the local-path StorageClass, each with a size of 10Gi. By default, the PVCs are retained in case of StatefulSet deletion or scale-down.

All of the services in the stack are deployed with a service of type NodePort for easy access to their UIs from a browser in the jump host.

All of the pods are configured to run on the control plane nodes with an anti-affinity for better load sharing.

kube-prometheus-stack.yaml

alertmanager:
  service:
    type: NodePort
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
    nodeSelector:
      node-role.kubernetes.io/control-plane: ""
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                      - prometheus
                      - grafana
              topologyKey: kubernetes.io/hostname
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
grafana:
  persistence:
    enabled: true
    storageClassName: "local-path"
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                    - prometheus
                    - alertmanager
            topologyKey: kubernetes.io/hostname
  service:
    type: NodePort
  useStatefulSet: true
kubeStateMetrics:
  enabled: false
nodeExporter:
  enabled: false
prometheusOperator:
  admissionWebhooks:
    patch:
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
          effect: NoSchedule
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Exists
      effect: NoSchedule
prometheus:
  service:
    type: NodePort
  prometheusSpec:
    tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
    nodeSelector:
      node-role.kubernetes.io/control-plane: ""
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app.kubernetes.io/name
                    operator: In
                    values:
                      - grafana
                      - alertmanager
              topologyKey: kubernetes.io/hostname
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-path
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
    additionalScrapeConfigs:
      - job_name: 'dts-metrics'
        dns_sd_configs:
          - names:
            - '_httpserverport._tcp.dts-mk55x.dpf-operator-system.svc.cluster.local'
        relabel_configs:
          - source_labels: [__address__]
            target_label: dpu_instance
            action: replace
            regex: '^([^.]+)\..*$'

Install the kube-prometheus-stack Helm chart using the following command:

Jump Node Console

$ helm install --create-namespace --namespace kube-prometheus-stack kube-prometheus-stack prometheus-community/kube-prometheus-stack --version v70.4.1 -f kube-prometheus-stack.yaml

Verify that all the pods in the kube-prometheus-stack namespace are in ready state:

Jump Node Console

$ kubectl wait --for=condition=ready --namespace kube-prometheus-stack pods --all
pod/alertmanager-kube-prometheus-stack-alertmanager-0 condition met
pod/kube-prometheus-stack-grafana-0 condition met
pod/kube-prometheus-stack-operator-584fccf98d-w8hnc condition met
pod/prometheus-kube-prometheus-stack-prometheus-0 condition met

Verify in the Prometheus UI that the DNS service discovery works well.
1. Enter an RDP session, open a web browser, and enter http://<TARGETCLUSTER_API_SERVER_HOST>:30090 to access the Prometheus web UI:
  Note:
  - By default, in the kube-prometheus-stack chart, port 30090 is the port for NodePort type service for Prometheus UI.
  - 10.0.110.10 is the IP address corresponding to the variable TARGETCLUSTER_API_SERVER_HOST in this RDG.
2. Navigate to Status → Service Discovery. You should see something similar to the following under dts-metrics job:
3. Navigate to Status → Target health to verify both DTS endpoints are in the 'UP' state under dts-metrics job:

Display metrics in Grafana

To view the metrics DTS exposed by Grafana and construct useful monitoring graphs, access the Grafana UI:

Find out the nodePort of the Grafana NodePort service:

Jump Node Console

$ kubectl get svc -n kube-prometheus-stack kube-prometheus-stack-grafana
NAME                            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kube-prometheus-stack-grafana   NodePort   10.233.29.146   <none>        80:31443/TCP   9m46s

In the RDP session, open a web browser and enter http://<TARGETCLUSTER_API_SERVER_HOST>: (10.0.110.10 and 31443 respectively in this example):

Enter the admin login credentials to access the Grafana home page:

To obtain the Grafana secret name in the kube-prometheus-stack operator namespace, run:

Jump Node Console

$ kubectl get secrets -n kube-prometheus-stack | grep grafana
NAME                                                                             TYPE                 DATA   AGE
kube-prometheus-stack-grafana                                                    Opaque               3      10m

Run the following command to obtain the admin username:

Jump Node Console

$ kubectl get secrets -n kube-prometheus-stack kube-prometheus-stack-grafana -o json | jq '.data."admin-user"' | cut -d '"' -f 2 | base64 --decode

Output example:

Jump Node Console
```
admin
```

Run the following command to obtain the admin password:

Jump Node Console

$ kubectl get secrets -n kube-prometheus-stack kube-prometheus-stack-grafana -o json | jq '.data."admin-password"' | cut -d '"' -f 2 | base64 --decode

Output example:

Jump Node Console
```
prom-operator
```
Return to the login page and enter the credentials previously obtained:

Navigate to the Dashboards page where pre-configured dashboards installed by the kube-prometheus-stack helm chart are already available:
Click on New → New Dashboard → Add visualization → Select Prometheus as Data Source. After that, start adding panels based on different DTS metrics. For instance:
1. Click Back to dashboard in the top-right corner of the new panel.
2. Click Settings in the top-right corner of the new dashboard.
3. Go to the Variables tab.
4. Add the following variables:
  1. "Select variable type": Data source, "General": ("Name": datasource), "Data source options": ("Type": Prometheus)
  2. "Select variable type": Query, "General": ("Name": dpu_instance, "Label": dpu_instance), "Query options": ("Data source": $(datasource), "Query": ("Query type": Label values, "Label": dpu_instance, "Metric": pf0vf0_eth_rx_bytes, "Label filters": job =~ dts-metrics))
5. Return to the main dashboard page and click Edit on the previously added empty panel.
6. Configure the new panel as follows:
  1. Under the 1st query row (marked as 'A' by default), switch from Builder to Code.
  2. Enter the following query to display the average rate of received bits per second in the last 5 minutes:
    
    Network Received PromQL
```
rate(label_replace({__name__=~".*_eth_rx_bytes", job="dts-metrics", dpu_instance="$dpu_instance"},"name_label","$1","__name__", "(.+)")[5m:]) * 8
```
  3. On the right-side of the screen under "Panel options", configure:
    - Title: Network Received
    - Description: Network received (bits/s)
    - Unit: bits/sec(SI)
    - Min: 0
  4. Click Run queries to display the data in the panel.
7. Click Back to dashboard and then choose Add → Visualization.
8. Configure an additional panel:
  1. Run the following query to display the average rate of transmitted bits per second in the last 5 minutes:
    
    Network Transmitted PromQL
```
rate(label_replace({__name__=~".*_eth_tx_bytes", job="dts-metrics", dpu_instance="$dpu_instance"},"name_label","$1","__name__", "(.+)")[5m:]) * 8
```

rate(label_replace({__name__=~".*_eth_tx_bytes", job="dts-metrics", dpu_instance="$dpu_instance"},"name_label","$1","__name__", "(.+)")[5m:]) * 8

On the right-side of the screen under "Panel options", configure:
- Title: Network Transmitted
- Description: Network transmitted (bits/s)
- Unit: bits/sec(SI)
- Min: 0
Click Run queries to display the data in the panel.
Click Back to dashboard and align the panels:
Click Save dashboard in the top-right corner of the dashboard.

Authors


	Guy ZilbermanGuy Zilberman is a solution architect at NVIDIA's 网络解决方案 Labs, bringing extensive experience from several leadership roles in cloud computing. He specializes in designing and implementing solutions for cloud and containerized workloads, leveraging NVIDIA's advanced networking technologies. His work primarily focuses on open-source cloud infrastructure, with expertise in platforms such as Kubernetes (K8s) and OpenStack.

Guy ZilbermanGuy Zilberman is a solution architect at NVIDIA's 网络解决方案 Labs, bringing extensive experience from several leadership roles in cloud computing. He specializes in designing and implementing solutions for cloud and containerized workloads, leveraging NVIDIA's advanced networking technologies. His work primarily focuses on open-source cloud infrastructure, with expertise in platforms such as Kubernetes (K8s) and OpenStack.

范围

缩写和首字母缩略词

引言

参考资料

解决方案架构

关键组件和技术

Solution Design

K8s Cluster Logical Design - Monitoring Stack

Software Stack Components

Bill of Materials

Deployment and Configuration

Node and Switch Definitions

Wiring

Fabric Configuration

DTS Upgrade to Configure ConfigPorts

Setup Centralized Monitoring Stack

Display metrics in Grafana

Authors