RDG：基于Red Hat OSP13、Nuage加速SDN和NVIDIA ConnectX-5智能网卡的NFVi平台

本文档介绍了一个完整的参考设计指南（RDG），用于构建加速、高性能且超高效的基于OpenStack的网络功能虚拟化基础设施（NFVi）平台。

文档目录

简介

本文档介绍了一个完整的参考设计指南（RDG），用于构建加速、高性能且超高效的基于OpenStack的网络功能虚拟化基础设施（NFVi）平台。

该解决方案包含以下主要组件：

Red Hat OpenStack Platform (OSP)：领先的商业支持的OpenStack开源软件发行版
Nuage Software Define Networking (SDN)：领先的商业支持的SDN解决方案
Nvidia 互连解决方案：领先的智能数据中心互连供应商

该解决方案利用了OVS卸载这一开源技术，允许将SDN数据平面从服务器CPU（通常为x86）上运行卸载到NVIDIA® SmartNIC上。这使得虚拟网络功能（VNF）应用或其他应用能够享受高吞吐量、高消息速率以及低且可预测的延迟，同时释放CPU核心以处理主机网络栈。

本文档简要概述了相关技术，重点介绍了新功能，并指导用户如何部署一个完整的工作集群。

参考资料

组件概述

Nuage VSP（虚拟化服务平台）包含三个组件：

Nuage VSD - 虚拟化服务器目录，提供网络策略和分析引擎
Nuage VSC - 虚拟化服务控制器，提供实时网络控制平面
Nuage OVRS - 卸载虚拟路由和交换，是软件转发平面

Nvidia组件：

智能 NVIDIA® ConnectX®-5 网卡引入了新的加速引擎，以最大化高性能。这些网卡能够提供非常高的消息速率，为最苛刻的应用和市场提供最高性能和最灵活的解决方案。
ASAP² - 加速交换和数据包处理®解决方案结合了硬件的性能和效率与虚拟交换软件的灵活性，依赖于 eSwitch（网卡嵌入式交换机），负责支持OpenFlow转发规则的复杂数据包处理。
Nvidia交换机 SN2410 - 为当今数据中心日益增长的需求提供最可预测、最高性能的100GbE ToR交换平台。基于Spectrum® ASIC，配备48个25GbE端口，SN2410提供高达4Tb/s的吞吐量和5.95Bpps的处理能力。

Red Hat OSP（OpenStack平台）主要组件：

OpenStack Platform Director - 用于安装和管理完整OpenStack环境的工具集。
OpenStack Platform Controller - 控制器节点负责托管Red Hat OpenStack平台环境中的核心服务。
Server OpenStack Nova Compute - 负责构建磁盘镜像、启动实例、检查实例状态、附加持久存储以及终止实例。

解决方案概述

VSD组成网络策略范围，并通过XMPP消息协议将其发送给VSC。VSC将策略解释为OpenFlow协议，并传达给OVRS执行，从而产生基于流的数据平面。

这些转发数据平面流可以卸载到Nvidia SmartNIC，并由于ASAP²技术存储在硬件中，从而加速数据包转发并减少本地主机的CPU处理。所有这些都得到Red Hat OpenStack平台的支持和封装。

卸载基本架构与特性规格

工作原理：

第一个数据包命中慢路径，然后由Nuage SDN（VSD+VSC）层组成流，并通过控制平面使用OpenFlow协议插入OVRS-vSwitch。新学习的流同时卸载到Nvidia SmartNIC。后续数据包命中SmartNIC并匹配此特定卸载流时，会自动通过VF接口（蓝线）转发到客户虚拟机，绕过传统内核转发路径（红线）。这种独特的方式能够保持Nuage SDN层完整，同时为每个VNF（虚拟网络功能）实现高带宽和高速数据包率。

ASAP² VF-LAG

ASAP² VF-LAG功能通过将BOND功能卸载到Nvidia SmartNIC硬件，为单个VF接口提供透明冗余并加倍吞吐量。OVRS通过开放API无缝负责卸载BOND配置。LAG LACP（链路聚合控制协议）和LAG HA（高可用性）是卸载LAG支持的透明模式。

解决方案逻辑设计

多节点规模部署中，每个网络角色共享公共管理网络。租户网络主要包含用户数据流量以及计算节点OVRS到VSC服务器的带内控制消息（使用OpenFlow协议）。OpenStack控制器API通过专用网络维护和访问。

物料清单

Solution Physical Design / Wiring

Network subnets and IPs ranges are documented in network-environment.yaml.

VSD and VSC communicate over the Tenant network. In real production environment, it is recommended to share VSD and VSC communication with a dedicated management network.

The below is a Nvidia Switch configuration example:

Mellanox_SW_Config_Example.txt

Host Configuration

Ensure that VT-d and SR-IOV are enabled in BIOS per server.

Deployment Guide

Host Deployment and Configuration

Warning Before we begin

This post highlights and points only public Nuage documentation for installing all components covering Red Hat and Nvidia. Nuage 文档

In order to get the most updated licensed Nuage packages and documentation, please contact Nokia at: nuage-smartnic-support@list.nokia.com

To install OSP 13 director, the server needs to be registered and licensed under Red Hat account. For more information: Contact Red Hat

1. Install Openstack Director on the Undercloud System

Follow Nuage guidelines for Red Hat official documentation: Install openstack director on the undercloud system.

Warning Note: local_interface in undercloud.conf file stands for OSP Director "Control NIC 2" interface which is connected to "Provisioning / Control Plane" network, as shown in the "Solution physical design / wiring" figure.

The below is an undercloud.conf file example for this deployment guide:

[DEFAULT]

# IP information for the interface on the Undercloud that will be
# handling the PXE boots and DHCP for Overcloud instances.  The IP
# portion of the value will be assigned to the network interface
# defined by local_interface, with the netmask defined by the prefix
# portion of the value. (string value)
local_ip = 192.168.24.1/24

# Virtual IP or DNS address to use for the public endpoints of
# Undercloud services. Only used with SSL. (string value)
# Deprecated group/name - [DEFAULT]/undercloud_public_vip
undercloud_public_host = 192.168.24.2

# Virtual IP or DNS address to use for the admin endpoints of
# Undercloud services. Only used with SSL. (string value)
# Deprecated group/name - [DEFAULT]/undercloud_admin_vip
undercloud_admin_host = 192.168.24.3

# DNS nameserver(s) to use for the undercloud node. (list value)
undercloud_nameservers = 10.7.77.192,10.7.77.135

# List of ntp servers to use. (list value)
undercloud_ntp_servers = 192.114.62.250

# Network interface on the Undercloud that will be handling the PXE
# boots and DHCP for Overcloud instances. (string value)
local_interface = enp132s0f0

# Whether to install the TripleO UI. (boolean value)
enable_ui = false

# Whether to install requirements to run the TripleO validations.
# (boolean value)
enable_validations = true

# Maximum number of attempts the scheduler will make when deploying
# the instance. You should keep it greater or equal to the number of
# bare metal nodes you expect to deploy at once to work around
# potential race condition when scheduling. (integer value)
# Minimum value: 1
scheduler_max_attempts = 10

[auth]

[ctlplane-subnet]

#
# From instack-undercloud
#

# Network CIDR for the Neutron-managed subnet for Overcloud instances.
# (string value)
# Deprecated group/name - [DEFAULT]/network_cidr
cidr = 192.168.24.0/24

# Start of DHCP allocation range for PXE and DHCP of Overcloud
# instances on this network. (string value)
# Deprecated group/name - [DEFAULT]/dhcp_start
dhcp_start = 192.168.24.5

# End of DHCP allocation range for PXE and DHCP of Overcloud instances
# on this network. (string value)
# Deprecated group/name - [DEFAULT]/dhcp_end
dhcp_end = 192.168.24.24

# Temporary IP range that will be given to nodes on this network
# during the inspection process. Should not overlap with the range
# defined by dhcp_start and dhcp_end, but should be in the same ip
# subnet. (string value)
# Deprecated group/name - [DEFAULT]/inspection_iprange
inspection_iprange = 192.168.24.100,192.168.24.120

# Network gateway for the Neutron-managed network for Overcloud
# instances on this network. (string value)
# Deprecated group/name - [DEFAULT]/network_gateway
gateway = 192.168.24.1

# The network will be masqueraded for external access. (boolean value)
masquerade = true

2. Download Nuage Source Code

Follow the instructions in Downloading Nuage source code.

Warning Note: Make sure to set git branch to version 13.605.1.

3. Pull Nuage and Red Hat Docker Containers

Follow the instructions in Prepare the Containers, sections 3.1 and 3.2

Warning Notes:

Make sure you have followed the steps from Red Hat OSP13 user guide to register containers in local repository in Obtaining container images and configuring local registry Section 5.5: "Using the undercloud as a local registry"

The local registry IP is 192.168.24.1 in this example deployment

In nuage_container_config.yaml set Nuage release to: 6-0-5 and 'latest' tag.

4. Register and Inspect the Bare Metal Nodes

Follow the instructions in Register and inspect the bare metal nodes

Warning Note: Make sure you have followed the steps from Red Hat OSP13 user guide to register bare metal nodes Registering Nodes for the Overcloud, Section 6.1: "Registering Nodes for the Overcloud".

The below is an instackenv.json file example for this deployment guide:

{
    "nodes": [
        {
            "pm_password": "add_your_password_here",
            "name": "control-0",
            "pm_addr": "10.7.214.205",
            "capabilities": "profile:control,boot_option:local",
            "pm_type": "pxe_ipmitool",
            "pm_user": "add_your_user_here"
        },
        {
            "pm_password": "add_your_password_here",
            "name": "compute-0",
            "pm_addr": "10.7.214.208",
            "capabilities": "profile:computeovrs,boot_option:local",
            "pm_type": "pxe_ipmitool",
            "pm_user": "add_your_user_here"
        },
        {
            "pm_password": "add_your_password_here",
            "name": "compute-1",
            "pm_addr": "10.7.214.209",
            "capabilities": "profile:computeovrs,boot_option:local",
            "pm_type": "pxe_ipmitool",
            "pm_user": "add_your_user_here"
        }
    ]
}

5. 下载 Nuage VSP RPM 包并创建 Yum 仓库

按照下载 Nuage VSP RPM 包并创建 Yum 仓库中的说明操作，以确定所需的软件包。

以下是按组划分的所有必需软件包的完整列表：

Nuage 通用软件包： nuage-bgp, nuage-metadata-agent, nuage-openstack-neutronclient, nuage-puppet-modules, python-openvswitch-nuage

Nuage OVRS 软件包： mstflint, nuage-openvswitch-ovrs

额外软件包（来自 Red Hat 官方仓库）： boost-filesystem, boost-regex, boost-system, python3, python3-libs, python3-pip, python3-setuptools, createrepo, deltarpm, libconfig, lldpad, perl-JSON, perl-Sys-Syslog, protobuf-c, python-deltarpm, glibc, keyutils-libs, krb5-libs, libcom_err, libgcc, libselinux, libsepol, libstdc++, libverto, nss-softokn-freebl, openssl-libs, pcre, zlib

注意： 如需获取最新的授权 Nuage 软件包，请联系 Nokia： nuage-smartnic-support@list.nokia.com

为 Nuage 软件包创建本地仓库：

为每个软件包组创建两个目录，如下所示：
- /var/www/html/nuage_osp13_6.0.5/nuage_common/
- /var/www/html/nuage_osp13_6.0.5/nuage_ovrs
根据所需软件包列表，将每个软件包复制到其对应的组目录中。
创建一个名为 nuage_ospd13.repo 的文件，并将其放置在： /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts 本节提供了 nuage_ospd13.repo 文件的示例。
安装 createrepo 工具。

什么是 createrepo？ 更多信息，请参见 http://yum.baseurl.org/wiki/RepoCreate.html
对步骤 1 中创建的每个目录执行 createrepo。

以下是本部署指南的 nuage_ospd13.repo 文件示例：

########################################################################################################################
############## Repositories which user want to enable by default on the overcloud image while patching #################
########################################################################################################################

# This repo should contain following packages:
# nuage-bgp
# nuage-puppet-modules
# nuage-openstack-neutronclient
# python-openvswitch-nuage
# nuage-metadata-agent

[nuage_common]
name=nuage_osp13_6.0.5
baseurl=http://192.168.24.1/nuage_osp13_6.0.5/nuage_common
enabled=1
gpgcheck=0

########################################################################################################################
############## Repositories which we recommend user to disable by default ##############################################
########################################################################################################################

# This repo should contain following packages:
# nuage-openvswitch-ovrs
# mstflint

[nuage_ovrs]
name=nuage_osp13_6.0.5
baseurl=http://192.168.24.1/nuage_osp13_6.0.5/nuage_ovrs
enabled=1
gpgcheck=0

6. 修改 Overcloud 镜像

在完成本节中的前置设置后，按照修改 overcloud 镜像中的说明操作。

注意： 确保将 overcloud 镜像从 ~/images/overcloud-full.qcow2 复制到： /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts

前置设置

编辑 /home/stack/nuage-ospdirector/image-patching/nuage_image_patching_scripts/nuage_patching_config.yaml 文件，添加以下内容：
- RepoFile: 'nuage_ospd13.repo'
- DeploymentType: ["ovrs"]
为了下载额外软件包，编辑 nuage_patching_config.yaml 文件并添加 Red Hat 账户信息。这样 overcloud 镜像就可以从 Red Hat 订阅的仓库中获取软件包。以下是所需参数的示例：
- RhelUserName: 'abc'
- RhelPassword: '***'
- RhelPool: '1234567890123445'

7. 创建数据平面角色并更新节点配置文件

按照创建数据平面角色并更新节点配置文件中的说明操作。

按如下方式生成 nuage_roles_data.yaml 文件：

openstack overcloud roles generate --roles-path /home/stack/nuage-tripleo-heat-templates/roles -o /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml Controller ComputeOvrs

以下是本部署指南的 node-info.yaml 文件示例：

parameter_defaults:
  OvercloudControllerFlavor: control
  ControllerCount: 1
  OvercloudComputeOvrsFlavor: computeovrs
  ComputeOvrsCount: 2

8. 为 OpenStack 安装生成 CMS ID

按照为 OpenStack 部署生成 CMS ID 中的说明操作。

注意：

确保您的 OSP Director 能够通过公共网卡 1 与 VSD 服务器建立网络连接，如解决方案物理设计/布线部分所述。

生成 CMS-ID 的命令行示例：
python configure_vsd_cms_id.py --server 10.7.215.50:8443 --serverauth csproot:csproot --organization csp --auth_resource /me --serverssl True --base_uri /nuage/api/v5_0

9. 自定义环境文件

按照自定义环境文件中的说明操作。

注意： 基于解决方案物理设计/布线部分。

在本部署中，租户网卡 1 的 VSD IP 为 192.168.10.101

在本部署中，租户网卡 1 的 VSC IP 为 192.168.10.102

以下是本部署指南的 neutron-nuage-config.yaml 和 nova-nuage-config.yaml 文件示例：

# A Heat environment file which can be used to enable a
# a Neutron Nuage backend on the controller, configured via puppet
resource_registry:
  OS::TripleO::Services::NeutronDhcpAgent: OS::Heat::None
  OS::TripleO::Services::NeutronL3Agent: OS::Heat::None
  OS::TripleO::Services::NeutronMetadataAgent: OS::Heat::None
  OS::TripleO::Services::NeutronOvsAgent: OS::Heat::None
  OS::TripleO::Services::ComputeNeutronOvsAgent: OS::Heat::None

  # Override the NeutronMl2PluginBase to use Nuage inside Docker container
  OS::TripleO::Docker::NeutronMl2PluginBase: ../puppet/services/neutron-plugin-ml2-nuage.yaml
  OS::TripleO::Services::NeutronCorePlugin: ../docker/services/neutron-plugin-ml2-nuage.yaml

parameter_defaults:
  NeutronNuageNetPartitionName: 'default_name'
  NeutronNuageVSDIp: '192.168.10.101:8443'

RDG：基于Red Hat OSP13、Nuage加速SDN和NVIDIA ConnectX-5智能网卡的NFVi平台

10. 分配Nuage Controller角色

按照 nuage controller role controller 中的说明操作。

警告：要提取控制器节点UUID，请使用以下命令：
openstack baremetal node list

11. 分配Nuage Offloaded VRS节点角色

按照 offload vrs role computeovrs 中的说明操作。

注意：
要提取计算节点UUID，请运行以下命令：
openstack baremetal node list
关于ConnectX固件，请参见固件下载。

下载固件bin文件后，对固件目录 /var/www/html/FW_<VERSION> 执行 createrepo。

本部署使用固件版本 16.26.1040。

本部署使用两个NVIDIA Bond接口。端口在 mellanox-environment.yaml 文件中指定。接口名称必须与目标服务器的操作系统命名一致。

以下是本部署指南的 mellanox-environment.yaml 文件示例：

resource_registry:
  OS::TripleO::ComputeSriov::NodeUserData: ../firstboot/mellanox_fw_update.yaml

parameter_defaults:
  ################
  # Nic's params #
  ################
  MellanoxTenantPort1: "ens2f0"
  MellanoxTenantPort2: "ens2f1"
  MellanoxInfraPort1: "ens1f0"
  MellanoxInfraPort2: "ens1f1"

  ########################
  # FIRST Boot FW config #
  ########################

  BIN_DIR_URL: "http://192.168.24.1/FW_16_26_1040/"
  NUM_OF_VFS: 16
  SRIOV_EN: True
  ESWITCH_IPV4_TTL_MODIFY_ENABLE: True
  PRIO_TAG_REQUIRED_EN: True

12. 网络隔离

按照网络隔离部分中的说明操作："VF lag with VLANs for Nvidia ConnectX-5 NICs"。

注意：
BondInterfaceOvsOptions 在 network-environment.yaml 中设置为 bond LACP。

ovs-hw-offload.yaml 包含 computeovrs 服务器的NVIDIA租户接口名称。
确保 ovs-hw-offload.yaml 包含以下行（参见文件示例）：
resource_registry:
  OS::TripleO::Services::NuageComputeOvrs: ../puppet/services/nuage-compute-ovrs.yaml
ovs-hw-offload.yaml 中的 NodeRootPassword 设置为默认的OS root密码。

以下是本部署指南的 computeovrs.yaml、controller.yaml、ovs-hw-offload.yaml 和 network-environment.yaml 文件示例：

parameter_defaults:
  # This section is where deployment-specific configuration is done
  # CIDR subnet mask length for provisioning network
  ControlPlaneSubnetCidr: '24'
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.168.24.1
  EC2MetadataIp: 192.168.24.1  # Generally the IP of the Undercloud
  # Customize the IP subnets to match the local environment
  StorageNetCidr: '172.16.1.0/24'
  StorageMgmtNetCidr: '172.16.3.0/24'
  InternalApiNetCidr: '172.16.2.0/24'
  TenantNetCidr: '192.168.10.0/24'
  ExternalNetCidr: '172.16.4.0/24'
  ManagementNetCidr: '10.0.1.0/24'
  # Customize the VLAN IDs to match the local environment
  StorageNetworkVlanID: 30
  StorageMgmtNetworkVlanID: 40
  InternalApiNetworkVlanID: 20
  TenantNetworkVlanID: 50
  ExternalNetworkVlanID: 10
  ManagementNetworkVlanID: 60
  StorageAllocationPools: [{'start': '172.16.1.200', 'end': '172.16.1.250'}]
  StorageMgmtAllocationPools: [{'start': '172.16.3.200', 'end': '172.16.3.250'}]
  InternalApiAllocationPools: [{'start': '172.16.2.200', 'end': '172.16.2.250'}]
  TenantAllocationPools: [{'start': '192.168.10.50', 'end': '192.168.10.80'}]
  # Leave room if the external network is also used for floating IPs
  ExternalAllocationPools: [{'start': '172.16.4.100', 'end': '172.16.4.150'}]
  ManagementAllocationPools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}]
  # Gateway routers for routable networks
  ExternalInterfaceDefaultRoute: '172.16.4.1'
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["8.8.8.8","10.7.77.192"]
  # The tunnel type for the tenant network (vxlan or gre). Set to '' to disable tunneling.
  NeutronTunnelTypes: 'vxlan'
  # Customize bonding options, e.g. "mode=4 lacp_rate=1 updelay=1000 miimon=100"
  # for Linux bonds w/LACP, or "bond_mode=active-backup" for OVS active/backup.
  BondInterfaceOvsOptions: "mode=4 miimon=100 lacp_rate=1 xmit_hash_policy=layer3+4"
  TenantPortEthtoolOptions: "-L ${DEVICE} combined 40; -X ${DEVICE} equal 38; -U ${DEVICE} flow-type tcp4 src-port 6633 action 39 loc 1; -U ${DEVICE} flow-type tcp4 src-port 7406 action 39 loc 2; -U ${DEVICE} flow-type tcp4 src-port 39090 action 39 loc 3;-U ${DEVICE} flow-type tcp4 src-port 179 action 39 loc 4"

heat_template_version: queens
description: >
  Software Config to drive os-net-config with 2 bonded nics on a bridge with VLANs attached for the ComputeSriov role.
parameters:
  ControlPlaneIp:
    default: ''
    description: IP address/subnet on the ctlplane network
    type: string
  StorageIpSubnet:
    default: ''
    description: IP address/subnet on the storage network
    type: string
  StorageMgmtIpSubnet:
    default: ''
    description: IP address/subnet on the storage_mgmt network
    type: string
  InternalApiIpSubnet:
    default: ''
    description: IP address/subnet on the internal_api network
    type: string
  TenantIpSubnet:
    default: ''
    description: IP address/subnet on the tenant network
    type: string
  ExternalIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  ManagementIpSubnet:
    default: ''
    description: IP address/subnet on the management network
    type: string
  BondInterfaceOvsOptions:
    default: bond_mode=active-backup
    description: 'The ovs_options or bonding_options string for the bond
      interface. Set things like lacp=active and/or bond_mode=balance-slb
      for OVS bonds or like mode=4 for Linux bonds using this option.'
    type: string
  StorageNetworkVlanID:
    default: 30
    description: Vlan ID for the storage network traffic.
    type: number
  StorageMgmtNetworkVlanID:
    default: 40
    description: Vlan ID for the storage_mgmt network traffic.
    type: number
  InternalApiNetworkVlanID:
    default: 20
    description: Vlan ID for the internal_api network traffic.
    type: number
  TenantNetworkVlanID:
    default: 50
    description: Vlan ID for the tenant network traffic.
    type: number
  ExternalNetworkVlanID:
    default: 10
    description: Vlan ID for the external network traffic.
    type: number
  ManagementNetworkVlanID:
    default: 60
    description: Vlan ID for the management network traffic.
    type: number
  ExternalInterfaceDefaultRoute:
    default: '172.16.4.1'
    description: default route for the external network
    type: string
  DnsServers:
    default: ['8.8.8.8', '10.7.77.192']
    description: DNS servers
    type: comma_delimited_list
  NeutronTunnelTypes:
    default: 'vxlan'
    description: Tunnel types for tenant network
    type: string
  BondInterfaceOvsOptions:
    default: 'mode=4 miimon=100 lacp_rate=1 xmit_hash_policy=layer3+4'
    description: Bonding options
    type: string
  TenantPortEthtoolOptions:
    default: '-L ${DEVICE} combined 40; -X ${DEVICE} equal 38; -U ${DEVICE} flow-type tcp4 src-port 6633 action 39 loc 1; -U ${DEVICE} flow-type tcp4 src-port 7406 action 39 loc 2; -U ${DEVICE} flow-type tcp4 src-port 39090 action 39 loc 3;-U ${DEVICE} flow-type tcp4 src-port 179 action 39 loc 4'
    description: ethtool options for tenant ports
    type: string

40

description: Vlan ID for the storage_mgmt network traffic.
type: number

InternalApiNetworkVlanID: default: 20 description: Vlan ID for the internal_api network traffic. type: number TenantNetworkVlanID: default: 50 description: Vlan ID for the tenant network traffic. type: number ExternalNetworkVlanID: default: 10 description: Vlan ID for the external network traffic. type: number ManagementNetworkVlanID: default: 60 description: Vlan ID for the management network traffic. type: number ControlPlaneDefaultRoute: # Override this via parameter_defaults description: The default route of the control plane network. type: string ExternalInterfaceDefaultRoute: default: '192.168.24.2' description: default route for the external network type: string ControlPlaneSubnetCidr: # Override this via parameter_defaults default: '24' description: The subnet CIDR of the control plane network. type: string DnsServers: # Override this via parameter_defaults default: [] description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf. type: comma_delimited_list EC2MetadataIp: # Override this via parameter_defaults description: The IP address of the EC2 metadata server. type: string MellanoxTenantPort1: description: Mellanox Tenant Port1 type: string MellanoxTenantPort2: description: Mellanox Tenant Port2 type: string MellanoxInfraPort1: description: Mellanox Infra Port1 type: string MellanoxInfraPort2: description: Mellanox Infra Port2 type: string TenantPortEthtoolOptions: description: Port ethtool options for Control Plane Protectionn type: string resources: OsNetConfigImpl: type: OS::Heat::SoftwareConfig properties: group: script config: str_replace: template: get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh params: $network_config: network_config: - type: interface name: eno1 use_dhcp: false addresses: - ip_netmask: list_join: - / - - get_param: ControlPlaneIp - get_param: ControlPlaneSubnetCidr routes: - ip_netmask: 169.254.169.254/32 next_hop: get_param: EC2MetadataIp - default: true next_hop: get_param: ControlPlaneDefaultRoute # Linux bond for Offload VRS Compute nodes - type: linux_bond name: infra-bond dns_servers: get_param: DnsServers bonding_options: get_param: BondInterfaceOvsOptions members: - type: interface name: get_param: MellanoxInfraPort1 - type: interface name: get_param: MellanoxInfraPort2 - type: vlan device: infra-bond vlan_id: get_param: StorageNetworkVlanID addresses: - ip_netmask: get_param: StorageIpSubnet - type: vlan device: infra-bond vlan_id: get_param: InternalApiNetworkVlanID addresses: - ip_netmask: get_param: InternalApiIpSubnet - type: linux_bond name: tenant-bond dns_servers: get_param: DnsServers bonding_options: get_param: BondInterfaceOvsOptions members: - type: sriov_pf name: get_param: MellanoxTenantPort1 link_mode: switchdev numvfs: 8 promisc: true use_dhcp: false primary: true ethtool_opts: get_param: TenantPortEthtoolOptions - type: sriov_pf name: get_param: MellanoxTenantPort2 link_mode: switchdev numvfs: 8 promisc: true use_dhcp: false ethtool_opts: get_param: TenantPortEthtoolOptions - type: vlan device: tenant-bond vlan_id: get_param: TenantNetworkVlanID addresses: - ip_netmask: get_param: TenantIpSubnet outputs: OS::stack_id: description: The OsNetConfigImpl resource. value: get_resource: OsNetConfigImpl


```yaml
heat_template_version: queens
description: &gt;
  Software Config to drive os-net-config with 2 bonded nics on a bridge with VLANs attached for the Controller role.
parameters:
  ControlPlaneIp:
    default: ''
    description: IP address/subnet on the ctlplane network
    type: string
  StorageIpSubnet:
    default: ''
    description: IP address/subnet on the storage network
    type: string
  StorageMgmtIpSubnet:
    default: ''
    description: IP address/subnet on the storage_mgmt network
    type: string
  InternalApiIpSubnet:
    default: ''
    description: IP address/subnet on the internal_api network
    type: string
  TenantIpSubnet:
    default: ''
    description: IP address/subnet on the tenant network
    type: string
  ExternalIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  ManagementIpSubnet:
    default: ''
    description: IP address/subnet on the management network
    type: string
  BondInterfaceOvsOptions:
    default: bond_mode=active-backup
    description: 'The ovs_options or bonding_options string for the bond
      interface. Set things like lacp=active and/or bond_mode=balance-slb
      for OVS bonds or like mode=4 for Linux bonds using this option.'
    type: string
  StorageNetworkVlanID:
    default: 30
    description: Vlan ID for the storage network traffic.
    type: number
  StorageMgmtNetworkVlanID:
    default: 40
    description: Vlan ID for the storage_mgmt network traffic.
    type: number
  InternalApiNetworkVlanID:
    default: 20
    description: Vlan ID for the internal_api network traffic.
    type: number
  TenantNetworkVlanID:
    default: 50
    description: Vlan ID for the tenant network traffic.
    type: number
  ExternalNetworkVlanID:
    default: 10
    description: Vlan ID for the external network traffic.
    type: number
  ManagementNetworkVlanID:
    default: 60
    description: Vlan ID for the management network traffic.
    type: number
  ControlPlaneDefaultRoute: # Override this via parameter_defaults
    description: The default route of the control plane network.
    type: string
  ExternalInterfaceDefaultRoute:
    default: '192.168.24.2'
    description: default route for the external network
    type: string
  ControlPlaneSubnetCidr: # Override this via parameter_defaults
    default: '24'
    description: The subnet CIDR of the control plane network.
    type: string
  DnsServers: # Override this via parameter_defaults
    default: []
    description: A list of DNS servers (2 max for some implementations) that will be added to resolv.conf.
    type: comma_delimited_list
  EC2MetadataIp: # Override this via parameter_defaults
    description: The IP address of the EC2 metadata server.
    type: string
  MellanoxInfraPort1:
    description: Mellanox Infra Port1
    type: string
  MellanoxInfraPort2:
    description: Mellanox Infra Port2
    type: string

resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:
              - type: interface
                name: eno1
                use_dhcp: false
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                - ip_netmask: 169.254.169.254/32
                  next_hop:
                    get_param: EC2MetadataIp
              # Linux bond for Controller nodes
              - type: linux_bond
                name: infra-bond
                dns_servers:
                  get_param: DnsServers
                bonding_options:
                  get_param: BondInterfaceOvsOptions
                members:
                - type: interface
                  name:
                    get_param: MellanoxInfraPort1
                - type: interface
                  name:
                    get_param: MellanoxInfraPort2
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: StorageNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: StorageMgmtNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageMgmtIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: TenantNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: TenantIpSubnet
              - type: vlan
                device: infra-bond
                vlan_id:
                  get_param: ExternalNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: ExternalIpSubnet
                routes:
                - default: true
                  next_hop:
                    get_param: ExternalInterfaceDefaultRoute

outputs:
  OS::stack_id:
    description: The OsNetConfigImpl resource.
    value:
      get_resource: OsNetConfigImpl

# A Heat environment file that enables OVS Hardware Offload in the overcloud.
resource_registry:
  OS::TripleO::Services::NuageComputeOvrs: ../puppet/services/nuage-compute-ovrs.yaml

parameter_defaults:

  NovaSchedulerDefaultFilters: ['RetryFilter','AvailabilityZoneFilter','RamFilter','ComputeFilter','ComputeCapabilitiesFilter','ImagePropertiesFilter','ServerGroupAntiAffinityFilter','ServerGroupAffinityFilter','PciPassthroughFilter']
  NovaSchedulerAvailableFilters: ["nova.scheduler.filters.all_filters","nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter"]
  NodeRootPassword: "add_your_password_here"

  # Kernel arguments for ComputeOvrs node
  ComputeOvrsParameters:
    KernelArgs: "intel_iommu=on iommu=pt pci=realloc"
    #NOTE: By default TunedProfileName is set to "cpu-partitioning" in ovrs-role.yaml.
    # If IsolCpusList is not set in your environment, then leave TunedProfileName below to set to

RDG：基于Red Hat OSP13、Nuage加速SDN和NVIDIA ConnectX-5智能网卡的NFVi平台

引言

本文档提供了一份完整的参考设计指南（RDG），用于构建基于Red Hat OpenStack Platform 13、Nuage加速SDN和NVIDIA ConnectX-5智能网卡的加速、高性能、超高效NFVi平台。

部署Overcloud

12. 配置ComputeOVRS角色

编辑 /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml 文件，在 ComputeOVRS 角色下添加以下参数：

    # 如果计划在环境中设置 IsolCpusList
    #   1. 可以注释以下行，将 TunedProfileName 设置为 "cpu-partitioning" 或
    #   2. 可以传递自定义的 Tuned Profile 以应用到主机
    TunedProfileName: ""
    OvsHwOffload: True
    # SR-IOV PF接口到neutron physical_network的映射。
    # 对于Vxlan/GRE，physical_network应为null。
    # 对于flat/vlan，physical_network应与neutron中配置的一致。
    NovaPCIPassthrough:
      - devname: "ens2f0"
        physical_network: null
      - devname: "ens2f1"
        physical_network: null

13. 部署Overcloud

按照部署Overcloud 中的说明，第3节：“For VF lag with VLANs for CX-5 NICs”。

openstack overcloud deploy --templates -r /home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml -e /home/stack/templates/node-info.yaml -e /home/stack/templates/overcloud_images.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/nuage_overcloud_images.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/network-environment.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/net-bond-with-vlans.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/neutron-nuage-config.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/nova-nuage-config.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/mellanox-environment.yaml -e /home/stack/nuage-tripleo-heat-templates/environments/ovs-hw-offload.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml --ntp-server 192.168.24.1

为方便起见，所有模板文件已附上：

yaml_config.tar.gz

文件OSP目录位置：

/home/stack/nuage-tripleo-heat-templates/templates/nuage_roles_data.yaml
/home/stack/templates/node-info.yaml
/home/stack/templates/overcloud_images.yaml
/home/stack/nuage-tripleo-heat-templates/environments/nuage_overcloud_images.yaml
/home/stack/nuage-tripleo-heat-templates/environments/network-environment.yaml
/home/stack/nuage-tripleo-heat-templates/environments/net-bond-with-vlans.yaml
/home/stack/nuage-tripleo-heat-templates/environments/neutron-nuage-config.yaml
/home/stack/nuage-tripleo-heat-templates/environments/nova-nuage-config.yaml
/home/stack/nuage-tripleo-heat-templates/environments/mellanox-environment.yaml
/home/stack/nuage-tripleo-heat-templates/environments/ovs-hw-offload.yaml
/home/stack/nuage-tripleo-heat-templates/network/config/bond-with-vlans/controller.yaml
/home/stack/nuage-tripleo-heat-templates/network/config/bond-with-vlans/computeovrs.yaml

应用部署与配置

本文档不涵盖VSD和VSC服务器的安装与配置。假设这些已根据Nuage文档正常运行。

请在进行下一步前查看此例外：

确保在neutron-nuage-config.yaml中定义为'NeutronNuageVSDUsername'的VSD用户也作为CMS组的一部分在VSD中定义。否则会导致虚拟机创建失败。

为此，从neutron-nuage-config.yaml中获取用户名（本例中用户名为'csproot'），打开VSD WebUI的“Platform Configuration”，将用户移动到“Assigned Users”一侧。

打开VSD WebUI
点击Platform Configuration图标（右上角）
点击Settings
点击左侧的Groups图标，选择CMS Group，然后点击Assign按钮
选择所需用户，点击箭头将其移至右侧，然后点击Submit。

最终应得到如下结果：

配置基本Flavor

source overcloudrc
openstack flavor create m1.small --id 1 --ram 4096 --disk 20 --vcpus 4

在VSD中创建托管网络

进入OpenStack WebUI。

成功部署后，可访问OpenStack仪表板。

/home/stack/overcloudrc 文件包含用户和密码登录凭据。
在OpenStack WebUI的Project → Network选项卡下，点击“create network”：
命名并设置所需复选框：
选择子网类型为“VSD Managed Subnet (auto)”
从WebUI选项中选择Organization、Domain、Zone和Subnet。假设这些参数已在VSD中配置，并包含活跃策略。

VSD与活动策略一起。

[mediaInline].3

子网字段自动填充。点击下一步，网络创建完成。

[mediaInline].4

Capture.3JPG.JPG

创建'switchdev'端口

从CLI创建一个具有'switchdev'能力的端口，并将其附加到上一节所述创建的所需网络。

openstack port create direct1 --vnic-type=direct --no-security-group --network Net1 --binding-profile '{"capabilities":["switchdev"]}'
openstack port create direct2 --vnic-type=direct --no-security-group --network Net1 --binding-profile '{"capabilities":["switchdev"]}'

在独立虚拟机管理程序上启动虚拟机

从CLI启动虚拟机，并插入每个虚拟机的端口ID和镜像名称：

openstack server create  --flavor m1.small --image <image_name> --nic port-id=<port1_ID> --availability-zone nova:overcloud-computesriov-0.localdomain vm1
openstack server create  --flavor m1.small --image <image_name> --nic port-id=<port2_ID> --availability-zone nova:overcloud-computesriov-1.localdomain vm2

部署验证

在两个运行中的虚拟机之间发起ping请求，查看是否收到回复。

设置连续ping请求，并切换到虚拟机管理程序CLI以检查卸载机制。

此示例显示了传出和返回ICMP请求/返回数据包的卸载流。

VXLAN接口持有传入流，代表端口持有传出流。

转储OVRS卸载规则

[root@overcloud-computesriov-0 ~]sudo ovs-dpctl dump-flows type=offloaded --name
in_port(ens2f0_6),eth(dst=fa:16:3e:81:a2:b2),eth_type(0x8100),vlan(vid=0,pcp=0),encap(eth_type(0x0800),ipv4(src=10.44.0.0/255.255.0.0,dst=0.0.0.0/1.0.0.0,proto=1,tos=0/0x3)), packets:2808, bytes:438048, used:0.120s, actions:set(tunnel(tun_id=0x8cfcb6,dst=192.168.10.52,tp_dst=4789,flags(key))),vxlan_sys_4789
tunnel(tun_id=0x8cfcb6,src=192.168.10.52,dst=192.168.10.50,ttl=64,flags(+key)),in_port(vxlan_sys_4789),eth(dst=fa:16:3e:01:78:98),eth_type(0x8100),vlan(vid=0,pcp=0),encap(eth_type(0x0800),ipv4(dst=0.0.0.0/1.0.0.0,proto=1)), packets:2808, bytes:286416, used:0.130s, actions:ens2f0_6

转储TC VXLAN接口卸载规则

[root@overcloud-computesriov-0 ~]tc filter show dev vxlan_sys_4789: parent ffff:
filter protocol 802.1Q pref 4 flower chain 0 handle 0x8fb
  vlan_id 0
  vlan_prio 0
  vlan_ethtype ip
  dst_mac 68:54:ed:00:0d:0b
  src_mac c6:17:f1:d8:4e:32
  eth_type ipv4
  ip_proto udp
  ip_tos 0x0/ff
  ip_ttl 0x3d/ff
  dst_ip 17.0.11.108
  src_ip 49.0.11.108
  dst_port 10001
  src_port 1000
  in_hw in_hw_count 1
        action order 1: tunnel_key  set
        src_ip 0.0.0.0
        dst_ip 3.1.1.102
        key_id 7708303
        dst_port 4789
        nocsum
        ttl 64 pipe
         index 9472 ref 1 bind 1

        action order 2:  pedit action pipe keys 5
         index 9472 ref 1 bind 1
         key #0  at ipv4+8: val 3c000000 mask 00ffffff
         key #1  at eth+4: val 0000fa7b mask ffff0000
         key #2  at eth+8: val aea37243 mask 00000000
         key #3  at eth+0: val 00000301 mask 00000000
         key #4  at eth+4: val 01660000 mask 0000ffff

        action order 3: csum (iph, udp) action pipe
        index 9472 ref 1 bind 1

        action order 4: mirred (Egress Redirect to device vxlan_sys_4789) stolen
        index 9472 ref 1 bind 1
        cookie c81b9f7c3c4cac8437509ba179c02c83

转储TC代表接口卸载规则

[root@overcloud-computesriov-0 ~]tc filter show dev ens2f0_6: parent ffff:
filter protocol 802.1Q pref 2 flower chain 0
 filter protocol 802.1Q pref 4 flower chain 0
 filter protocol 802.1Q pref 5 flower chain 0
 filter protocol 802.1Q pref 6 flower chain 0
 filter protocol 802.1Q pref 7 flower chain 0
 filter protocol 802.1Q pref 7 flower chain 0 handle 0x1
   vlan_id 0
   vlan_ethtype ip
   dst_mac fa:16:3e:81:a2:b2
   eth_type ipv4
   ip_proto icmp
   ip_tos 0/3
   dst_ip 10.44.27.17/1.0.0.0
   src_ip 10.44.27.20/16
   in_hw
         action order 1: tunnel_key set
         src_ip 0.0.0.0
         dst_ip 192.168.10.52
         key_id 9239734
         dst_port 4789
         nocsum
         ttl 64 pipe
         index 1 ref 1 bind 1

         action order 2: mirred (Egress Redirect to device vxlan_sys_4789) stolen
         index 1 ref 1 bind 1

性能测试

性能测试部署在单个计算节点和XIA服务器上，两者都连接到NVIDIA交换机。

计算节点是待测设备，运行TestPMD over两个VF，并连接到bond VF-LAG接口。

IXIA向NVIDIA交换机注入平坦UDP数据包，交换机在逻辑LACP bond之上的VXLAN隧道中封装数据包。

VF-LAG bond拦截数据包并将其转发到连接到testPMD虚拟机的VF1和VF2，虚拟机将数据包从VF1转发到VF2，再从VF2转发回VF1到IXIA服务器。

流数量由源和目标IP的范围决定。

测试结果

下表显示了带宽和每秒数据包数，每列在硬件中维护不同数量的流，从64到400,000。

关于作者


	Amir Zeidner 多年来，Amir一直担任解决方案架构师，主要专注于电信领域，领导先进解决方案以满足5G、NFV和SDN网络基础设施需求。Amir在数据平面加速技术（如ASAP²和DPDK）方面的专业知识，加上对开源云基础设施的深入了解，使他能够在电信领域推广和交付独特的端到端NVIDIA网络解决方案。

简介

参考资料

组件概述

解决方案概述

卸载基本架构与特性规格

ASAP² VF-LAG

解决方案逻辑设计

物料清单

Solution Physical Design / Wiring

Host Configuration

Deployment Guide

Host Deployment and Configuration

1. Install Openstack Director on the Undercloud System

2. Download Nuage Source Code

3. Pull Nuage and Red Hat Docker Containers

4. Register and Inspect the Bare Metal Nodes

5. 下载 Nuage VSP RPM 包并创建 Yum 仓库

6. 修改 Overcloud 镜像

前置设置

7. 创建数据平面角色并更新节点配置文件

8. 为 OpenStack 安装生成 CMS ID

9. 自定义环境文件

RDG：基于Red Hat OSP13、Nuage加速SDN和NVIDIA ConnectX-5智能网卡的NFVi平台

10. 分配Nuage Controller角色

11. 分配Nuage Offloaded VRS节点角色

12. 网络隔离

40

RDG：基于Red Hat OSP13、Nuage加速SDN和NVIDIA ConnectX-5智能网卡的NFVi平台

引言

部署Overcloud

12. 配置ComputeOVRS角色

13. 部署Overcloud

应用部署与配置

配置基本Flavor

在VSD中创建托管网络

[mediaInline].3

子网字段自动填充。点击下一步，网络创建完成。

[mediaInline].4

创建'switchdev'端口

在独立虚拟机管理程序上启动虚拟机

部署验证

转储OVRS卸载规则

转储TC VXLAN接口卸载规则

转储TC代表接口卸载规则

关于作者

相关文档