RDG：在 Red Hat OpenShift Container Platform v4.1 上通过 InfiniBand 加速 ML 和 DL 工作负载
创建于 2019 年 11 月 1 日。简介 Red Hat、NVIDIA 和 NVIDIA 正在合作，为 HPC、人工智能和机器学习工作负载提供高性能平台。
文档目录
创建于 2019 年 11 月 1 日
简介
Red Hat、NVIDIA 和 NVIDIA 正在合作，为 HPC、人工智能和机器学习工作负载提供高性能平台。
这是一份参考部署指南 (RDG)，用于在裸机用户置备基础设施 (UPI) 上部署 Red Hat OpenShift Container Platform (RH OCP) v4.1，该平台专为通过 NVIDIA InfiniBand 网络进行 RDMA 加速的机器学习 (ML) 和深度学习 (DL) 应用而设计。
在本文档中，我们将介绍以下内容：
- 如何在运行 RHEL 7.6 的裸机 GPU 节点上部署 RH OCP v4.1
- 如何通过高性能 InfiniBand 网络运行基于 Horovod 框架的分布式 TensorFlow 基准测试。
注意： 用于 RH OCP 的高性能 InfiniBand 网络目前是技术预览功能。 Red Hat 不建议在生产环境中使用技术预览功能。这些功能不受 Red Hat 生产服务级别协议 (SLA) 支持，可能不完整或功能不稳定。这些功能提供对即将推出的产品功能的早期访问，使客户能够在开发过程中测试功能并提供反馈。 有关更多信息，请参阅 Red Hat 技术预览功能支持范围。
参考文献
- OpenShift 4.1 Bare Metal Install Quickstart
- OCP4 UPI Helper Node Playbook
- RH OCP 4.1. Installing a cluster on bare metal
- QM8700 Series - NVIDIA Quantum™ HDR 200Gb/s InfiniBand Smart 交换机
- ConnectX®-5 Single/Dual-Port Adapter supporting 100Gb/s with VPI
- MLNX-OS® InfiniBand/VPI Switch-based Operating System
- MLNX_OFED
- NVIDIA OFED GPUDirect RDMA
- TensorFlow
- Uber Horovod github
- TensorFlow benchmarks
- Machine Learning Toolkit for Kubernetes
- MPI_Operator - Kubernetes Operator for Allreduce-style Distributed Training
组件概述
-
NVIDIA GPU NVIDIA GPU 专为 HPC、AI 和深度学习应用中最苛刻的工作负载而设计。GPU 加速服务器计算能力，同时降低成本。GPU 加速的深度学习框架提供了设计和训练自定义深度神经网络的灵活性。 每个主要的深度学习框架（如 TensorFlow、PyTorch 等）都已实现 GPU 加速，数据科学家和研究人员无需对 GPU 进行编程即可立即投入生产。
-
Red Hat OpenShift Container Platform (RH OCP) Red Hat OpenShift Container Platform (RH OCP) 为开发人员和 IT 组织提供了一个混合云应用平台，用于在安全、可扩展的资源上部署新的和现有的应用程序，且配置和管理开销最小。 基于 Red Hat Enterprise Linux 平台和 Kubernetes，OCP 为当今的企业级应用提供了更安全、可扩展的多租户操作系统，同时提供了集成的应用程序运行时和库。
-
Kubeflow Kubeflow 是一个用于机器学习应用的云原生平台，基于 Google 的内部机器学习流水线。更多信息请访问 kubeflow.org。
-
Horovod Horovod 是一个用于 TensorFlow、Keras、PyTorch 和 MXNet 的分布式训练框架。Horovod 的目标是使分布式深度学习快速且易于使用。
-
TensorFlow TensorFlow 是一个开源软件库，由 Google Brain 团队开发，用于进行机器学习和深度神经网络研究。该库通过使用数据流图进行数值计算，图中的节点表示数学运算，图边表示在节点之间通信的多维数据数组（张量）。TensorFlow 支持 Cuda 和 cuDNN（需注册）。 本指南使用 TensorFlow 网站 的源文件，以便于安装。
-
Kubernetes RDMA device plugin RDMA 设备插件提供对 Kubernetes Worker 节点的访问，以在 Kubernetes Worker 节点上运行的多个 Pod 之间共享单个 RDMA 设备 (HCA)。
-
GPUDirect RDMA GPUDirect RDMA 实现了同一主机或不同主机上的 GPU 之间直接 P2P（点对点）数据交换路径，可直接与使用 RDMA 协议的 NVIDIA 设备进行数据交换。这显著降低了 GPU 到 GPU 的通信延迟，并完全卸载了 CPU，使其从网络中的所有 GPU 到 GPU 通信中移除。
GPUDirect RDMA 技术与 NVIDIA ConnectX®-4 网卡（及后续版本）无缝协作。
解决方案概述
设备
本解决方案使用以下硬件规格：
逻辑设计
逻辑设计包括以下层：
- 两个独立的网络层：
- 管理网络
- 高速 InfiniBand 网络
- 计算层：
- UPI 网络网关节点
- OCP4 UPI Helper 节点
- Bootstrap 节点
- 3 个 Master 节点
- Worker0 节点（无 GPU）
- 4 个 Worker 节点，配备 NVIDIA Tesla P100 GPU 卡和 NVIDIA ConnectX-5 网卡。
GPU 节点逻辑设计
下图展示了基于 GPU 的工作节点的组件：
物料清单
下表列出了本部署指南中使用的硬件组件：
注意： 本部署指南不涉及服务器/虚拟机管理程序虚拟化安装和虚拟机创建步骤。
服务器布线
在此基于 GPU 的服务器设置中，每个 HCA 仅第一个端口通过 EDR 线缆连接到 InfiniBand 交换机：
网络与结构配置
网络配置
每个基于 GPU 的服务器通过 EDR InfiniBand 铜缆连接到 NVIDIA QM8700 InfiniBand 交换机。
下表详细列出了服务器和交换机名称及网络配置：
| 服务器/交换机类型 | 服务器/交换机名称 | InfiniBand 网络 | 管理网络 |
|---|---|---|---|
| 网关节点 | clx-ocp-gwc | 无 | eno0: 静态 (Wan)eno1: 静态 (UPI Lan) |
| OCP4 UPI 节点 | ocp-helper | 无 | eno0: 静态 (UPI Lan) |
| Master 节点 1-3 | master[0-2] | 无 | eno0: 来自 DHCP (由 UPI 保留) |
| Worker 节点 0 | worker0 | 无 | eno0: 来自 DHCP (由 UPI 保留) |
| Worker 节点 1 | worker-p1 | ib0: 自动ib1: 自动 | eno0: 来自 DHCP (由 UPI 保留) |
| Worker 节点 2 | worker-p2 | ib0: 自动ib1: 自动 | eno0: 来自 DHCP (由 UPI 保留) |
| Worker 节点 3 | worker-p3 | ib0: 自动ib1: 自动 | eno0: 来自 DHCP (由 UPI 保留) |
| Worker 节点 4 | worker-p4 | ib0: 自动ib1: 自动 | eno0: 来自 DHCP (由 UPI 保留) |
| InfiniBand 交换机 | swx-mld-s01 | 无 | mgmt0: 来自 DHCP (由 UPI 保留) |
注意： IBx 接口（ib0, ib1）无需额外配置。
InfiniBand 结构网络拓扑
单交换机解决方案的初始设置
在此部署场景中，您最多可连接 20 台服务器，使用 NVIDIA Quantum™ HDR 200Gb/s QM8700 InfiniBand 智能交换机。
两层胖树拓扑的扩展设置
在此部署场景中，您最多可扩展到 20 台 Spine 交换机 和 40 台 Leaf 交换机（Spine 与 Leaf 交换机之间单连接），支持最多 400 台服务器。
注意： 对于扩展设置，建议使用 NVIDIA Unified Fabric Manager (UFM®)。
InfiniBand 结构配置
以下是配置过程中的重要建议和前提条件：
- 请参考 MLNX-OS 用户手册以熟悉交换机软件（位于 enterprise-support.nvidia.com/s/）
- 将交换机升级到最新的 MLNX-OS 版本
- 需要 InfiniBand 子网管理器 (SM) 来正确配置 InfiniBand 结构
在 InfiniBand 结构中运行 InfiniBand 子网管理器 (SM) 有三种方式：
- 在一个或多个托管交换机上启动 SM。这是一种非常方便且快速的操作，可实现更简单的 InfiniBand“即插即用”。
- 通过执行
/etc/init.d/opensmd命令在一个或多个服务器上运行 OpenSM 守护进程。如果节点数达到 648 或更多，建议在服务器上运行 SM。 - 使用 Unified Fabric Management (UFM®)。NVIDIA 的统一...
Fabric Manager (UFM®) 是一个强大的可扩展计算平台，它消除了结构管理的复杂性，提供了对流量的深度可见性，并优化了结构性能。
在本指南中，我们将使用在 InfiniBand 交换机上启动 InfiniBand SM 的方法。
以下是所选方法的配置步骤。
要在其中一台受管交换机上启用 SM，请执行以下操作：
登录交换机并输入以下配置命令（swx-mld-s01 是我们的交换机名称）：
Mellanox MLNX-OS Switch Management
switch login: admin
Password:
swx-mld-s01 [standalone: master] > enable
swx-mld-s01 [standalone: master] # configure terminal
swx-mld-s01 [standalone: master] (config) # ib smnode swx-mld-s01 enable
swx-mld-s01 [standalone: master] (config) # ib smnode swx-mld-s01 sm-priority 0
swx-mld-s01 [standalone: master] (config) # ib sm virt enable
swx-mld-s01 [standalone: master] (config) # write memory
swx-mld-s01 [standalone: master] (config) # reload
交换机重启后，检查交换机配置。它应如下所示：
Mellanox MLNX-OS Switch Management
switch login: admin
Password:
swx-mld-s01 [standalone: master] > enable
swx-mld-s01 [standalone: master] # configure terminal
swx-mld-s01 [standalone: master] (config) # show running-config
##
## Running database "initial"
## Generated at 2019/03/19 17:58:53 +0200
## Hostname: swx-mld-s01
##
##
## Running-config temporary prefix mode setting
##
no cli default prefix-modes enable
##
## Subnet Manager configuration
##
ib sm virt enable
##
## Other IPv6 configuration
##
no ipv6 enable
##
## AAA remote server configuration
##
# ldap bind-password ********
# radius-server key ********
# tacacs-server key ********
##
## Network management configuration
##
# web proxy auth basic password ********
clock timezone Asia Middle_East Jerusalem
no ntp server 192.114.62.250 disable
ntp server 192.114.62.250 keyID 0
no ntp server 192.114.62.250 trusted-enable
ntp server 192.114.62.250 version 4
##
## X.509 certificates configuration
##
#
# Certificate name system-self-signed, ID 0cd5b6a0da88a0e68b8f3b49408b361afc73289d
# (public-cert config omitted since private-key config is hidden)
##
## IB nodename to GUID mapping
##
ib smnode swx-mld-s01 create
ib smnode swx-mld-s01 enable
ib smnode swx-mld-s01 sm-priority 0
##
## Persistent prefix mode setting
##
cli default prefix-modes enable
部署步骤
网关节点配置
对于网关节点，我们将使用具有两个 NIC 和 CentOS 7 作为操作系统的虚拟机。
将 CentOS 7 配置为 NAT 路由器的步骤：
-
按如下方式配置 NIC：
-
ens224 - 公共，使用 DHCP
-
ens192 - 静态 UPI 局域网
# cat /etc/sysconfig/network-scripts/ifcfg-ens192 TYPE=Ethernet BOOTPROTO=static NAME=ens192 DEVICE=ens192 ONBOOT=yes IPADDR=192.168.7.1 NETMASK=255.255.255.0 DNS1=192.168.7.254
-
-
启用 IP 转发：
# sysctl -w net.ipv4.ip_forward=1 # echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/ip_forward.conf # sysctl -p -
启用 NAT：
# firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o ens224 -j MASQUERADE -s 192.168.7.0/24 # firewall-cmd --change-interface=ens224 --zone=external --permanent # firewall-cmd --change-interface=ens192 --zone=internal --permanent # firewall-cmd --set-default-zone=internal # firewall-cmd --complete-reload -
检查配置：
# firewall-cmd --get-active-zones internal interfaces: ens192 external interfaces: ens224
OCP4 UPI Helper 节点配置
OCP4 UPI Helper 节点部署需要一个具有互联网访问权限的单独网络，并包含以下组件：
- DNS 服务器
- 2x 负载均衡器
- Web 服务器
- DHCP 服务器
- PXE 服务器
- TFTP 服务器
- NFSv4 服务器
- 堡垒主机
OCP4 UPI Helper 节点的配置步骤如下：
- 安装操作系统
- 检查先决条件
- 准备 UPI
- 创建 Ignition 配置
安装 OCP4 Helper 节点操作系统
UPI Helper 节点安装指南 推荐使用带有 EPEL 仓库的 CentOS 7 操作系统。
对于我们的设置，OCP4 UPI Helper 节点需要 RHEL 7.6 操作系统。这将使我们能够添加基于裸机 GPU 的节点并扩展 OpenShift 集群。
对于 RHEL 7.6，您需要启用以下仓库：rhel-7-server-rpms、rhel-7-server-extras-rpms、rhel-7-server-ansible-2.7-rpms 和 rhel-7-server-ose-4.1-rpms。更多信息请参考 OpenShift 用户指南。
检查 OCP4 UPI 先决条件
克隆 github 仓库并从 https://github.com/christianh814/ocp4-upi-helpernode 安装额外的软件包。
# yum -y install ansible git
# git clone https://github.com/christianh814/ocp4-upi-helpernode
# cd ocp4-upi-helpernode
准备 UPI
完成准备步骤后，您的工作目录将是 ocp4-upi-helpernode。
从 docs/examples 文件夹复制 vars.yaml 文件，并根据您的网络配置进行修改。
以下是 vars.yaml 文件的示例：
---
disk: sda
helper:
name: "ocp-helper"
ipaddr: "192.168.7.254"
networkifacename: "ens192"
dns:
domain: "ocp.labs.mlnx"
clusterid: "ocp4"
forwarder1: "8.8.8.8"
forwarder2: "8.8.4.4"
dhcp:
router: "192.168.7.1"
bcast: "192.168.7.255"
netmask: "255.255.255.0"
poolstart: "192.168.7.10"
poolend: "192.168.7.30"
ipid: "192.168.7.0"
netmaskid: "255.255.255.0"
bootstrap:
name: "bootstrap"
ipaddr: "192.168.7.20"
macaddr: "00:0c:29:cc:87:b6"
masters:
- name: "master0"
ipaddr: "192.168.7.21"
macaddr: "00:0c:29:82:0f:6c"
- name: "master1"
ipaddr: "192.168.7.22"
macaddr: "00:0c:29:f0:f5:11"
- name: "master2"
ipaddr: "192.168.7.23"
macaddr: "00:0c:29:19:75:42"
workers:
- name: "worker0"
ipaddr: "192.168.7.11"
macaddr: "00:0c:29:c9:b9:c6"
- name: "worker1"
ipaddr: "192.168.7.12"
macaddr: "ac:1f:6b:25:1f:f0"
- name: "worker2"
ipaddr: "192.168.7.13"
macaddr: "ac:1f:6b:25:85:ec"
- name: "worker3"
ipaddr: "192.168.7.14"
macaddr: "ac:1f:6b:25:20:12"
- name: "worker4"
ipaddr: "192.168.7.15"
macaddr: "ac:1f:6b:25:1f:dc"
在 vars/main.yml 中查看并设置所需的 OCP 安装组件。例如：
---
staticips: false
force_ocp_download: true
ocp_bios: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/4.1.0/rhcos-4.1.0-x86_64-metal-bios.raw.gz"
ocp_initramfs: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/4.1.0/rhcos-4.1.0-x86_64-installer-initramfs.img"
ocp_install_kernel: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/4.1.0/rhcos-4.1.0-x86_64-installer-kernel"
ocp_client: "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest-4.1/openshift-client-linux-4.1.20.tar.gz"
ocp_installer: "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest-4.1/openshift-install-linux-4.1.20.tar.gz"
运行 Ansible Playbook 来设置您的 OCP UPI Helper 节点：
# ansible-playbook -e @vars.yaml tasks/main.yml
要验证 OCP UPI Helper 节点，请运行
使用以下参数运行 /usr/local/bin/helpernodecheck 命令：{dns-masters|dns-workers|dns-etcd|install-info|haproxy|services|nfs-info}。例如：
[root@ocp-helper ocp4-upi-helpernode]# /usr/local/bin/helpernodecheck dns-workers
======================
DNS Config for Workers
======================
; Create entries for the worker hosts
worker0 IN A 192.168.7.11
worker1 IN A 192.168.7.12
worker2 IN A 192.168.7.13
worker3 IN A 192.168.7.14
worker4 IN A 192.168.7.15
======================
DNS Lookup for Workers
======================
worker0.ocp4.ocp.labs.mlnx
-------------------------------------------------
IP: 192.168.7.11
Reverse: worker0.ocp4.ocp.labs.mlnx.
worker1.ocp4.ocp.labs.mlnx
-------------------------------------------------
IP: 192.168.7.12
Reverse: worker1.ocp4.ocp.labs.mlnx.
worker2.ocp4.ocp.labs.mlnx
-------------------------------------------------
IP: 192.168.7.13
Reverse: worker2.ocp4.ocp.labs.mlnx.
worker3.ocp4.ocp.labs.mlnx
-------------------------------------------------
IP: 192.168.7.14
Reverse: worker3.ocp4.ocp.labs.mlnx.
worker4.ocp4.ocp.labs.mlnx
-------------------------------------------------
IP: 192.168.7.15
Reverse: worker4.ocp4.ocp.labs.mlnx.
创建 Ignition 配置文件
创建 Ignition 配置文件 install-config.yaml 是 RH OCP 安装所必需的。
要创建 Ignition 配置文件，我们首先创建一个安装文件夹：
mkdir ~/ocp4
cd ~/ocp4
对于 install-config.yaml 的完整配置，我们需要两个额外的参数：pullSecret 和 sshKey。
pullSecret 可以从 cloud.redhat.com 获取。
- 使用您的 Red Hat 帐户登录
- 点击“Bare Metal”
- 点击“Download Pull Secret”或“Copy Pull Secret”
sshKey 是您的公共 SSH 密钥（例如 ~/.ssh/id_rsa.pub）。
以下是 install-config.yaml 文件的示例：
apiVersion: v1
baseDomain: ocp.labs.mlnx
compute:
- hyperthreading: Enabled
name: worker
replicas: 1
controlPlane:
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: ocp4
networking:
clusterNetworks:
- cidr: 10.254.0.0/16
hostPrefix: 24
networkType: OpenShiftSDN
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":....}}}'
sshKey: 'ssh-rsa AAAA... root@ocp-helper'
通过运行以下命令生成 ignition 配置：
# openshift-install create ignition-configs
现在将 ignition 文件复制到 Web 服务器的 ignition 目录：
# cd ~/ocp4/
# cp *.ign /var/www/html/ignition/
# restorecon -vR /var/www/html/
OCP4 UPI Helper 节点现在已准备好进行 RH OCP 安装过程。
RH OCP 部署
创建 OpenShift Container Platform 集群
在以下步骤中，我们将在基于 RHEL CoreOS 的虚拟机上安装 OCP bootstrap、OCP 管理和 OCP 监控组件。
在开始安装过程之前，请确保在 OCP4 UPI Helper 节点上配置了 ssh-agent。您可以按照此指南进行逐步配置。
按以下顺序使用 PXE 启动我们准备好的虚拟机：
- Bootstrap
- Masters
- Workers
有关在裸机上安装 RH OCP 4.1 的更多信息，请参阅 OCP 4.1 安装指南。
要监控安装过程，请使用以下命令：
- bootstrap 阶段：
openshift-install wait-for bootstrap-complete --log-level debug
# openshift-install wait-for bootstrap-complete --log-level debug
DEBUG OpenShift Installer v4.1.20-201910102034-dirty
DEBUG Built from commit e4708ece20e3f03947e9f5f460f1d5cbcd401249
INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4.ocp.labs.mlnx:6443...
INFO API v1.13.4+520769a up
INFO Waiting up to 30m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
- 从负载均衡器配置
/etc/haproxy/haproxy.cfg中移除 bootstrap 资源，并重启haproxy服务：
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
listen stats
bind :9000
mode http
stats enable
stats uri /
monitor-uri /healthz
frontend openshift-api-server
bind *:6443
default_backend openshift-api-server
mode tcp
option tcplog
backend openshift-api-server
balance source
mode tcp
# server bootstrap 192.168.7.20:6443 check # remark after finish bootstarp
server master0 192.168.7.21:6443 check
server master1 192.168.7.22:6443 check
server master2 192.168.7.23:6443 check
frontend machine-config-server
bind *:22623
default_backend machine-config-server
mode tcp
option tcplog
backend machine-config-server
balance source
mode tcp
# server bootstrap 192.168.7.20:22623 check # remark after finish bootstarp
server master0 192.168.7.21:22623 check
server master1 192.168.7.22:22623 check
server master2 192.168.7.23:22623 check
frontend ingress-http
bind *:80
default_backend ingress-http
mode tcp
option tcplog
backend ingress-http
balance source
mode tcp
server worker0-http-router0 192.168.7.11:80 check
server worker1-http-router1 192.168.7.12:80 check
server worker2-http-router2 192.168.7.13:80 check
server worker3-http-router3 192.168.7.14:80 check
server worker4-http-router4 192.168.7.15:80 check
frontend ingress-https
bind *:443
default_backend ingress-https
mode tcp
option tcplog
backend ingress-https
balance source
mode tcp
server worker0-https-router0 192.168.7.11:443 check
server worker1-https-router1 192.168.7.12:443 check
server worker2-https-router2 192.168.7.13:443 check
server worker3-https-router3 192.168.7.14:443 check
server worker4-https-router4 192.168.7.15:443 check
# service haproxy restart
-
完成集群安装：
openshift-install wait-for install-complete --log-level debug# openshift-install wait-for install-complete --log-level debug DEBUG OpenShift Installer v4.1.20-201910102034-dirty DEBUG Built from commit e4708ece20e3f03947e9f5f460f1d5cbcd401249 INFO Waiting up to 30m0s for the cluster at https://api.ocp4.ocp.labs.mlnx:6443 to initialize... DEBUG Cluster is initialized INFO Waiting up to 10m0s for the openshift-console route to be created... DEBUG Route found in openshift-console namespace: console DEBUG Route found in openshift-console namespace: downloads DEBUG OpenShift console route is created INFO Install complete! INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/install/auth/kubeconfig' INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4.ocp.labs.mlnx INFO Login to the console with user: kubeadmin, password: *****-*****-*****-*****
OpenShift Cluster Scale-up using RHEL Compute Machine
请在 OCP4 UPI 辅助节点上安装所需软件包以运行集群扩容 playbook，包括 Openshift-Ansible：
# yum install openshift-ansible openshift-clients jq
接下来的步骤适用于 Kubernetes 工作节点 - worker1、worker2、worker3 和 worker4。
Preparing a GPU-based RHEL compute node
OpenShift Container Platform 环境中的 Red Hat Enterprise Linux (RHEL) 计算节点或工作节点必须满足硬件规格和系统级要求。基础操作系统需要 RHEL 7.6 "Minimal" 安装选项。
注意： OpenShift Container Platform 4.1 仅支持 RHEL 7.6。请勿将计算节点升级到 RHEL 8。
仅启用 OpenShift Container Platform 4.1 所需的仓库：
# subscription-manager repos \
--enable="rhel-7-server-rpms" \
--enable="rhel-7-server-extras-rpms" \
--enable="rhel-7-server-ose-4.1-rpms"
停止并禁用主机上的防火墙：
# systemctl disable firewalld.service
# systemctl stop firewalld.service
安装任何其他必需的软件包并锁定内核版本：
# yum -y install yum-plugin-versionlock
# yum versionlock kernel-3.10.0-1062.1.2.el7
# yum -y update kernel
# yum -y install perl gtk2 atk cairo tcl gcc-gfortran tcsh tk pciutils lsof
# reboot
注意： NVIDIA GPU 驱动程序的安装仅针对
kernel-3.10.0-1062.1.2.el7进行了验证。
禁用 nouveau 内核模块：
# echo 'blacklist nouveau' > /etc/modprobe.d/blacklist-nouveau.conf
# echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf
# dracut --force
# reboot
重启后，确保 nouveau 模块未出现在以下列表中：
# lsmod | grep nouveau
Installing NVIDIA OFED
有两种方法可以为上述指定内核版本安装 NVIDIA OFED。
-
从 NVIDIA 网站 下载 NVIDIA OFED v4.7-1.0.0.1。下载安装包并运行命令
mlnx_add_kernel_support.sh以添加对内核的支持。请参阅此用户指南获取说明。
-
或者，您可以从此处下载预配置的 NVIDIA OFED 镜像，并将其复制到计算节点的
root文件夹。此镜像内置了对kernel-3.10.0-1062.1.2.el7的支持。
安装步骤：
获取镜像后，运行：
# mkdir /mnt/iso
# mount -o loop /root/MLNX_OFED_LINUX-4.7-1.0.0.1-rhel7.6-x86_64-ext.iso /mnt/iso
# /mnt/iso/mlnxofedinstall --force
# reboot
安装带有 InfiniBand 补丁的 SELinux。
从附件归档文件（infiniband.zip）中提取 infiniband.*，并将其复制到每个计算节点，然后在本地文件夹中执行：
# semodule -i infiniband.pp
现在计算节点已准备好加入 OpenShift 集群。
Adding GPU-based RHEL Compute Nodes to the OpenShift Cluster
有关将 RHEL 计算节点添加到 OpenShift 集群的更多信息，请参阅 OCP 安装指南中的添加 RHEL 计算节点部分。
要使用 RHEL 计算节点扩容 OpenShift 集群：
- 使用
ssh-copy-id从 OCP4 UPI 辅助节点将 SSH 密钥安装到计算节点上，作为无密码身份验证的授权密钥。 - 提取 OpenShift 集群的 "pull secret"。
- 创建一个名为
hosts的 Ansible 清单文件，定义计算节点和所需变量。 - 运行用于 RHEL 计算节点集群扩容的 Ansible playbook。
- 批准 RHEL 计算节点的 CSR。
以下是用于 OpenShift 集群扩容的 hosts 文件示例：
[all:vars]
ansible_user=root
#ansible_become=True
openshift_kubeconfig_path="~/.kube/config"
openshift_pull_secret_path="~/pull-secret.txt"
[workers]
worker0.ocp4.ocp.labs.mlnx
[new_workers]
worker1.ocp4.ocp.labs.mlnx
worker2.ocp4.ocp.labs.mlnx
worker3.ocp4.ocp.labs.mlnx
worker4.ocp4.ocp.labs.mlnx
运行扩容 playbook：
# cd /usr/share/ansible/openshift-ansible
# ansible-playbook -i ~/hosts playbooks/scaleup.yml
扩容 playbook 执行输出：
[root@ocp-helper openshift-ansible # ansible-playbook -i ~/hosts playbooks/scaleup.yml
PLAY [Pre-scaleup checks] ******************************************************************************************************************************************************************************************************************
TASK [fail] ********************************************************************************************************************************************************************************************************************************
Tuesday
29 October 2019 16:31:43 +0200 (0:00:00.068) 0:00:00.068 ******* skipping: [localhost]
PLAY [install nodes] ***********************************************************************************************************************************************************************************************************************
TASK [Gathering Facts] ********************************************************************************************************************************************************************************************************************* Tuesday 29 October 2019 16:31:43 +0200 (0:00:00.039) 0:00:00.107 ******* ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker2.ocp4.ocp.labs.mlnx]
TASK [openshift_node : include_tasks] ****************************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:31:45 +0200 (0:00:02.000) 0:00:02.107 ******* included: /usr/share/ansible/openshift-ansible/playbooks/roles/openshift_node/tasks/install.yml for worker1.ocp4.ocp.labs.mlnx, worker2.ocp4.ocp.labs.mlnx, worker3.ocp4.ocp.labs.mlnx, worker4.ocp4.ocp.labs.mlnx
TASK [openshift_node : Install openshift support packages] ********************************************************************************************************************************************************************************* Tuesday 29 October 2019 16:31:45 +0200 (0:00:00.642) 0:00:02.750 ******* lok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Install openshift packages] ***************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:34:47 +0200 (0:03:01.994) 0:03:04.744 ******* ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Enable the CRI-O service] ******************************************************************************************************************************************************************************************* Tuesday 29 October 2019 16:35:18 +0200 (0:00:30.724) 0:03:35.469 ******* ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx]
TASK [openshift_node : include_tasks] ****************************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:19 +0200 (0:00:00.820) 0:03:36.289 ******* included: /usr/share/ansible/openshift-ansible/playbooks/roles/openshift_node/tasks/config.yml for worker1.ocp4.ocp.labs.mlnx, worker2.ocp4.ocp.labs.mlnx, worker3.ocp4.ocp.labs.mlnx, worker4.ocp4.ocp.labs.mlnx
TASK [openshift_node : Disable swap] ******************************************************************************************************************************************************************************************************* Tuesday 29 October 2019 16:35:19 +0200 (0:00:00.461) 0:03:36.751 ******* ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : sysctl] ************************************************************************************************************************************************************************************************************* Tuesday 29 October 2019 16:35:20 +0200 (0:00:00.437) 0:03:37.188 ******* [WARNING]: The value 1 (type int) in a string field was converted to u'1' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.
ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Disable firewalld service] ****************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:20 +0200 (0:00:00.476) 0:03:37.665 ******* ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Setting sebool container_manage_cgroup] ***************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:21 +0200 (0:00:00.447) 0:03:38.112 ******* ok: [worker4.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker2.ocp4.ocp.labs.mlnx]
TASK [openshift_node : create temp directory] ********************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:21 +0200 (0:00:00.601) 0:03:38.714 ******* changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx] changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Wait for bootstrap endpoint to show up] ***************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:22 +0200 (0:00:00.432) 0:03:39.147 ******* ok: [worker4.ocp4.ocp.labs.mlnx] ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker1.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Fetch bootstrap ignition file locally] ****************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:23 +0200 (0:00:00.931) 0:03:40.078 ******* changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Copy pull secret in the directory] ********************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:23 +0200 (0:00:00.668) 0:03:40.747 ******* changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Get release image] ************************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:24 +0200 (0:00:00.986) 0:03:41.733 ******* changed: [worker1.ocp4.ocp.labs.mlnx -> localhost] changed: [worker3.ocp4.ocp.labs.mlnx -> localhost] changed: [worker4.ocp4.ocp.labs.mlnx -> localhost] changed: [worker2.ocp4.ocp.labs.mlnx -> localhost]
TASK [openshift_node : Set openshift_release_image fact] *********************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:25 +0200 (0:00:01.111) 0:03:42.845 ******* ok: [worker1.ocp4.ocp.labs.mlnx] ok: [worker2.ocp4.ocp.labs.mlnx] ok: [worker3.ocp4.ocp.labs.mlnx] ok: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Pull release image] ************************************************************************************************************************************************************************************************* Tuesday 29 October 2019 16:35:26 +0200 (0:00:00.246) 0:03:43.091 ******* changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx] changed: [worker1.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Get machine controller daemon image from release image] ************************************************************************************************************************************************************* Tuesday 29 October 2019 16:35:53 +0200 (0:00:27.708) 0:04:10.799 ******* changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Pull MCD image] ***************************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:35:56 +0200 (0:00:02.139) 0:04:12.939 ******* changed: [worker4.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx] changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker2.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Apply ignition manifest] ******************************************************************************************************************************************************************************************** Tuesday 29 October 2019 16:36:05 +0200 (0:00:09.036) 0:04:21.975 ******* changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx]
TASK [openshift_node : Reboot the host and wait for it to come back] *********************************************************************************************************************************************************************** Tuesday 29 October 2019 16:36:06 +0200 (0:00:01.052) 0:04:23.028 ******* changed: [worker1.ocp4.ocp.labs.mlnx] changed: [worker2.ocp4.ocp.labs.mlnx] changed: [worker4.ocp4.ocp.labs.mlnx] changed: [worker3.ocp4.ocp.labs.mlnx]
PLAY RECAP ********************************************************************************************************************************************************************************************************************************* localhost : ok=0 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 worker1.ocp4.ocp.labs.mlnx : ok=21 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 worker2.ocp4.ocp.labs.mlnx : ok=21 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 worker3.ocp4.ocp.labs.mlnx : ok=21 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 worker4.ocp4.ocp.labs.mlnx : ok=21 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Tuesday 29 October 2019 16:38:36 +0200 (0:02:30.434) 0:06:53.462 *******
openshift_node : Install openshift support packages ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 181.99s openshift_node : Reboot the host and wait for it to come back --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 150.43s openshift_node : Install openshift packages
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30.72s openshift_node : Pull release image ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 27.71s openshift_node : Pull MCD image ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9.04s openshift_node : Get machine controller daemon image from release image ------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.14s Gathering Facts --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2.00s openshift_node : Get release image -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.11s openshift_node : Apply ignition manifest -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.05s openshift_node : Copy pull secret in the directory ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.99s openshift_node : Wait for bootstrap endpoint to show up ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.93s openshift_node : Enable the CRI-O service ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.82s openshift_node : Fetch bootstrap ignition file locally ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.67s openshift_node : include_tasks ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.64s openshift_node : Setting sebool container_manage_cgroup ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.60s openshift_node : sysctl ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.48s openshift_node : include_tasks ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.46s openshift_node : Disable firewalld service ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.45s openshift_node : Disable swap ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.44s openshift_node : create temp directory ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.43s
执行 Scale-up playbook 后，批准为添加的每台机器生成的所有待处理证书签名请求 (CSR)：
```bash
# oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve
# oc get csr
NAME AGE REQUESTOR CONDITION
csr-2pn2r 44s system:node:worker2.ocp4.ocp.labs.mlnx Approved,Issued
csr-7fd8p 6m31s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-djzv6 6m30s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-h985k 6m21s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-j6rdh 6m32s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-lhvqq 46s system:node:worker3.ocp4.ocp.labs.mlnx Approved,Issued
csr-m52kp 49s system:node:worker1.ocp4.ocp.labs.mlnx Approved,Issued
csr-x47hg 55s system:node:worker4.ocp4.ocp.labs.mlnx Approved,Issued
csr-x8cgl 6m21s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
确认集群识别了这些机器：
# oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master0.ocp4.ocp.labs.mlnx Ready master 1d v1.13.4+a80aad556 192.168.7.21 <none> Red Hat Enterprise Linux CoreOS 410.8.20191011.0 (Ootpa) 4.18.0-80.11.2.el8_0.x86_64 cri-o://1.13.11-0.13.dev.rhaos4.1.gitbdeb2ca.el8-dev
master1.ocp4.ocp.labs.mlnx Ready master 1d v1.13.4+a80aad556 192.168.7.22 <none> Red Hat Enterprise Linux CoreOS 410.8.20191011.0 (Ootpa) 4.18.0-80.11.2.el8_0.x86_64 cri-o://1.13.11-0.13.dev.rhaos4.1.gitbdeb2ca.el8-dev
master2.ocp4.ocp.labs.mlnx Ready master 1d v1.13.4+a80aad556 192.168.7.23 <none> Red Hat Enterprise Linux CoreOS 410.8.20191011.0 (Ootpa) 4.18.0-80.11.2.el8_0.x86_64 cri-o://1.13.11-0.13.dev.rhaos4.1.gitbdeb2ca.el8-dev
worker0.ocp4.ocp.labs.mlnx Ready worker 1d v1.13.4+a80aad556 192.168.7.11 <none> Red Hat Enterprise Linux CoreOS 410.8.20191011.0 (Ootpa) 4.18.0-80.11.2.el8_0.x86_64 cri-o://1.13.11-0.13.dev.rhaos4.1.gitbdeb2ca.el8-dev
worker1.ocp4.ocp.labs.mlnx Ready worker 1h v1.13.4+a80aad556 192.168.7.12 <none> OpenShift Enterprise 3.10.0-1062.1.2.el7.x86_64 cri-o://1.13.11-0.11.dev.rhaos4.1.git3338d4d.el7
worker2.ocp4.ocp.labs.mlnx Ready worker 1h v1.13.4+a80aad556 192.168.7.13 <none> OpenShift Enterprise 3.10.0-1062.1.2.el7.x86_64 cri-o://1.13.11-0.11.dev.rhaos4.1.git3338d4d.el7
worker3.ocp4.ocp.labs.mlnx Ready worker 1h v1.13.4+a80aad556 192.168.7.14 <none> OpenShift Enterprise 3.10.0-1062.1.2.el7.x86_64 cri-o://1.13.11-0.11.dev.rhaos4.1.git3338d4d.el7
worker4.ocp4.ocp.labs.mlnx Ready worker 1h v1.13.4+a80aad556 192.168.7.15 <none> OpenShift Enterprise 3.10.0-1062.1.2.el7.x86_64 cri-o://1.13.11-0.11.dev.rhaos4.1.git3338d4d.el7
NVIDIA GPU 驱动和插件部署
部署的下一步是安装 OCP 的 NVIDIA 组件。
此步骤必须在 OCP4 UPI Helper 节点上执行。
部署节点特性发现 (NFD)
-
从 github 在 OpenShift 4.X 中部署 NFD：
# mkdir ~/install # cd ~/install # git clone https://github.com/openshift/cluster-nfd-operator # PULLPOLICY=Always make -C cluster-nfd-operator deploy -
验证 GPU 节点标签是否正确：
# oc describe nodes | grep 10de feature.node.kubernetes.io/pci-10de.present=true feature.node.kubernetes.io/pci-10de.present=true # oc describe nodes | grep kernel feature.node.kubernetes.io/kernel-version.full=3.10.0-XXXXX-x86_64 feature.node.kubernetes.io/kernel-version.major=3 feature.node.kubernetes.io/kernel-version.minor=10 feature.node.kubernetes.io/kernel-version.revision=0
特殊资源运算符 (SRO) 部署
在 OCP4 UPI Helper 节点上执行
-
从 github 部署 SRO：
# cd ~/install # git clone https://github.com/openshift-psap/special-resource-operator # cd special-resource-operator # git checkout release-4.2 # 适用于 OCP 4.0、4.1、4.2 # PULLPOLICY=Always make deploy -
验证 GPU 是否已启用，将看到扩展资源 GPU 和杂项 NVIDIA 特性：
# oc describe node worker1.ocp4.ocp.labs.mlnx | grep nvidia nvidia.com/cuda.driver.major=418 nvidia.com/cuda.driver.minor=87 nvidia.com/cuda.driver.rev=01 nvidia.com/cuda.runtime.major=10 nvidia.com/cuda.runtime.minor=1 nvidia.com/gfd.timestamp=1572712654 nvidia.com/gpu.compute.major=6 nvidia.com/gpu.compute.minor=0 nvidia.com/gpu.family=pascal nvidia.com/gpu.machine=SYS-4028GR-TR2 nvidia.com/gpu.memory=16280
nvidia.com/gpu.product=Tesla-P100-PCIE-16GB nvidia.com/gpu: 4
注意：如果 SRO 部署在 NVIDIA 驱动程序验证步骤中挂起，请在每个 GPU 节点上使用以下命令重启 CRI-O 服务：
# systemctl restart crio
# systemctl status crio
成功安装 SRO 后，输出如下所示：
# oc get pod -n openshift-sro -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cuda-vector-add 0/1 Completed 0 1h 10.254.6.13 worker2.ocp4.ocp.labs.mlnx <none> <none>
nvidia-dcgm-exporter-8w25q 2/2 Running 0 1h 192.168.7.15 worker4.ocp4.ocp.labs.mlnx <none> <none>
nvidia-dcgm-exporter-k7nkr 2/2 Running 0 1h 192.168.7.14 worker3.ocp4.ocp.labs.mlnx <none> <none>
nvidia-dcgm-exporter-pxb2b 2/2 Running 0 1h 192.168.7.12 worker1.ocp4.ocp.labs.mlnx <none> <none>
nvidia-dcgm-exporter-t5xtf 2/2 Running 0 1h 192.168.7.13 worker2.ocp4.ocp.labs.mlnx <none> <none>
nvidia-device-plugin-daemonset-52w7n 1/1 Running 0 1h 10.254.7.9 worker3.ocp4.ocp.labs.mlnx <none> <none>
nvidia-device-plugin-daemonset-7hpwk 1/1 Running 0 1h 10.254.5.14 worker1.ocp4.ocp.labs.mlnx <none> <none>
nvidia-device-plugin-daemonset-brk87 1/1 Running 0 1h 10.254.4.9 worker4.ocp4.ocp.labs.mlnx <none> <none>
nvidia-device-plugin-daemonset-zcsv7 1/1 Running 0 1h 10.254.6.14 worker2.ocp4.ocp.labs.mlnx <none> <none>
nvidia-device-plugin-validation 0/1 Completed 0 1h 10.254.7.10 worker3.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-2pmh5 1/1 Running 0 1h 10.254.4.8 worker4.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-5qzww 1/1 Running 0 1h 10.254.7.8 worker3.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-72bgb 1/1 Running 0 1h 10.254.5.11 worker1.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-qvsnj 1/1 Running 0 1h 10.254.6.9 worker2.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-validation 0/1 Completed 0 1h 10.254.5.13 worker1.ocp4.ocp.labs.mlnx <none> <none>
nvidia-feature-discovery-54xt5 1/1 Running 0 1h 10.254.4.10 worker4.ocp4.ocp.labs.mlnx <none> <none>
nvidia-feature-discovery-np6rj 1/1 Running 0 1h 10.254.5.16 worker1.ocp4.ocp.labs.mlnx <none> <none>
nvidia-feature-discovery-t5lpl 1/1 Running 0 1h 10.254.7.11 worker3.ocp4.ocp.labs.mlnx <none> <none>
启用 GPUDirect 内核模块
从 OCP4 UPI Helper 节点在每个基于 GPU 的节点上手动启动 nv_peer_memory 服务。
注意：GPUDirect 目前是技术预览功能。
Helper 节点将收到一个包含 NVIDIA GPU 驱动程序的 Pod 列表：
# oc get pod -n openshift-sro -o wide | grep nvidia-driver
nvidia-driver-daemonset-2pmh5 1/1 Running 0 1h 10.254.4.8 worker4.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-5qzww 1/1 Running 0 1h 10.254.7.8 worker3.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-72bgb 1/1 Running 0 1h 10.254.5.11 worker1.ocp4.ocp.labs.mlnx <none> <none>
nvidia-driver-daemonset-qvsnj 1/1 Running 0 1h 10.254.6.9 worker2.ocp4.ocp.labs.mlnx <none> <none>
对于 DaemonSet 中的每个 Pod，执行以下命令：
# oc -n openshift-sro rsh nvidia-driver-daemonset-5qzww
sh-4.2# bash
[root@nvidia-driver-daemonset-5qzww /]# modprobe nv_peer_mem
[root@nvidia-driver-daemonset-5qzww /]# lsmod | grep nv_peer_mem
[root@nvidia-driver-daemonset-5qzww /]# exit
注意：如果 Worker 节点已重启，则必须执行此步骤。
部署 InfiniBand 和 KubeFlow Kubernetes 组件
将 Openshift-rdma.zip 复制到 OCP4 UPI Helper 节点并解压文件。该压缩包包含以下文件：
- device-plugin.yaml – 用于部署共享 InfiniBand HCA 的 RDMA 设备插件的 DaemonSet
- mpijob-gpud.yaml - MPI 任务示例
- mpi-operator.yaml - KubeFlow/mpi-operator 完整安装（无需安装 KubeFlow）
- rdma-hca-node-config.yaml - RDMA 设备插件的 ConfigMap 配置文件
-
安装 RDMA 设备插件：
# oc apply -f rdma-hca-node-config.yaml # oc apply -f device-plugin.yaml -
KubeFlow MPI-operator 安装命令：
# oc apply -f mpi-operator.yaml
应用程序部署和配置
mpijob-gpud.yaml 文件中提供了应用程序部署示例。该示例描述了如何使用 KubeFlow MPI-Operator 通过高性能 InfiniBand 网络运行基于 Horovod 框架的分布式 TensorFlow 基准测试。
以下是 mpijob-gpud.yaml 文件中用于运行 TensorFlow 基准测试的环境变量设置：
-
TCP 模式
NCCL_IB_DISABLE=1NCCL_NET_GDR_LEVEL=0
-
不使用 GPUDirect
NCCL_IB_DISABLE=0NCCL_NET_GDR_LEVEL=0
-
使用 GPUDirect
NCCL_IB_DISABLE=0NCCL_NET_GDR_LEVEL=1
在 Helper 节点上运行以下命令部署应用程序：
# oc apply -f mpijob-gpud.yaml
性能测试
以下是使用 KubeFlow/mpi-operator 进行分布式 TensorFlow 基准测试的日志。
使用 GPUDirect（GDR）时，在我们的 POC 环境中（包括 4 台服务器，每台配备 4 块 P100 PCI GPU），性能将提升 6.26%。
对于更多服务器和更强大的 GPU，预期会有更高的性能提升。
以下是在 TCP 模式下使用 KubeFlow/mpi-operator 运行分布式 TensorFlow 基准测试的日志：
+ POD_NAME=tensorflow-benchmarks-worker-1
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-1 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "764542976" -mca ess_base_vpid 2 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[1:4]xnr8,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "764542976.0;tcp://10.254.5.30:36799" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "764542976.0;tcp://10.254.5.30:36799" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
+ POD_NAME=tensorflow-benchmarks-worker-3
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-3 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ;
DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "764542976" -mca ess_base_vpid 4 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[1:4]xnr8,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "764542976.0;tcp://10.254.5.30:36799" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "764542976.0;tcp://10.254.5.30:36799" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
+ POD_NAME=tensorflow-benchmarks-worker-0
+ + shift
POD_NAME=tensorflow-benchmarks-worker-2
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-0 -- /bin/sh+ -c/opt/kube/kubectl exec tensorflow-benchmarks-worker-2 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "764542976" -mca ess_base_vpid 1 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[1:4]xnr8,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "764542976.0;tcp://10.254.5.30:36799" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "764542976.0;tcp://10.254.5.30:36799" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "764542976" -mca ess_base_vpid 3 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[1:4]xnr8,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "764542976.0;tcp://10.254.5.30:36799" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "764542976.0;tcp://10.254.5.30:36799" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
2019-10-31 09:33:41.078094: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078419: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078447: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078269: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078241: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078241: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078247: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078278: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078279: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078698: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078761: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078744: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078571: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078689: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.078881: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:41.079080: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:33:42.823273: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x62b6630 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.823346: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.823358: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.823367: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.823376: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824362: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x50898f0 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.824414: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824427: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824436: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824445: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824714: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a3b870 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.824749: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824760: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824769: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.824777: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825649: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x561c1d0 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.825702: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825717: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825726: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825734: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825804: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4e2b4c0 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.825865: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825877: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825885: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.825893: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.827898: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x64b12f0 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.827962: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.827975: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.827984: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.827994: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.828199: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.828199: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.829567: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.830766: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.831848: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5b190c0 executing computations on platform Host. Devices:
2019-10-31 09:33:42.831879: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.831938: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5167140 executing computations on platform Host. Devices:
2019-10-31 09:33:42.831970: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.832258: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.832352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.832397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:42.832746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.832718: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.832784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:42.832751: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x56f9a10 executing
在平台主机上执行计算。设备：
2019-10-31 09:33:42.832785: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.833106: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5068d70 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.833193: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.833205: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.833214: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.833222: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.833201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.833249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:33:42.833835: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6393e60 executing computations on platform Host. Devices:
2019-10-31 09:33:42.833866: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.833940: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x48efc30 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.833980: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.833992: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.834000: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.834008: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.834212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.834244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:42.835262: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x658eb50 executing computations on platform Host. Devices:
2019-10-31 09:33:42.835297: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.835954: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4f08d30 executing computations on platform Host. Devices:
2019-10-31 09:33:42.835987: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.836263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.836294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:42.836375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.836416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:33:42.837365: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x659ea10 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.837434: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.837447: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.837457: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.837466: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.838824: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:33:42.839060: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x47f0990 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.839099: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839112: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839122: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839131: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839143: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x52bad20 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.839181: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839196: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839205: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839213: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.839459: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:33:42.840115: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4900010 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.840155: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840166: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840175: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840183: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840820: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5b34010 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.840855: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840866: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840875: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.840884: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.842132: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x51465a0 executing computations on platform Host. Devices:
2019-10-31 09:33:42.842164: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.842428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.842458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:33:42.842500: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x49cd490 executing computations on platform Host. Devices:
2019-10-31 09:33:42.842530: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.842959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.842997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:42.843562: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.843520: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x54497e0 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.843583: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.843605: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.843619: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.843627: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.843992: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:33:42.844902: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:33:42.845210: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.846111: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.846665: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5398550 executing computations on platform Host. Devices:
2019-10-31 09:33:42.846694: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.847238: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x667c220 executing computations on platform Host. Devices:
2019-10-31 09:33:42.847271: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.847787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.847818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:33:42.848053: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x49dd850 executing computations on platform Host. Devices:
2019-10-31 09:33:42.848090: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>,
2019-10-31 09:33:42.848369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.848410: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:33:42.848673: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5c11850 executing computations on platform Host. Devices:
2019-10-31 09:33:42.848705: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.849019: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x48ce1a0 executing computations on platform Host. Devices:
2019-10-31 09:33:42.849050: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.849469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.849498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:33:42.849226: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4845c50 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.849265: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.849278: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.849260: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.849287: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.849297: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.849810: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.849843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.849849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:33:42.849878: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:42.850540: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x577e2a0 executing computations on platform CUDA. Devices:
2019-10-31 09:33:42.850575: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.850595: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.850605: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.850613: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:33:42.852293: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5527030 executing computations on platform Host. Devices:
2019-10-31 09:33:42.852325: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.853152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.853196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:42.854239: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.855184: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:33:42.857681: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x49234b0 executing computations on platform Host. Devices:
2019-10-31 09:33:42.857711: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.858149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.858191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:42.858353: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x585baf0 executing computations on platform Host. Devices:
2019-10-31 09:33:42.858387: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:33:42.858874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:33:42.858942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:33:42.967915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.967994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:33:42.968008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:33:42.968318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:33:42.969387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.969425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:33:42.969437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:33:42.969687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:33:42.973332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.973402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:42.973416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:42.975511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:33:42.979394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.979472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:33:42.979486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:33:42.979557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.979628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:33:42.979642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:33:42.980554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.980595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:42.980608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:42.982272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:33:42.982341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:33:42.983720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.983805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:33:42.983819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:33:42.984121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.984161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:33:42.984174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:33:42.984355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:33:42.984174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:33:42.984643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.984682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:33:42.984695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:33:42.984881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:33:42.985547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.985590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:42.985602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:42.985879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess:
False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model 2019-10-31 09:33:42.980779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:42.980842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-10-31 09:33:42.980855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-10-31 09:33:42.981085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0) W1031 09:33:42.994619 140599664752384 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. W1031 09:33:42.978768 139735270668032 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. W1031 09:33:43.000526 139735270668032 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead. W1031 09:33:42.982844 139681104721664 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. 2019-10-31 09:33:42.982173: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:42.982206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:33:42.982217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:33:42.982395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) W1031 09:33:42.995758 140556463261440 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model W1031 09:33:43.005537 139681104721664 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead. TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model W1031 09:33:42.984108 140153269229312 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. W1031 09:33:43.007307 140153269229312 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.conv2d instead. 2019-10-31 09:33:42.982481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:42.982517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:33:42.982529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N 2019-10-31 09:33:42.982724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) W1031 09:33:42.991235 140519229642496 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. 2019-10-31 09:33:42.981610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:42.981648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:33:42.981659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:33:42.981865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) W1031 09:33:42.995507 139737117169408 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model 2019-10-31 09:33:42.979862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) W1031 09:33:42.993835 140336927958784 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. TensorFlow: 1.13 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 32 per device Num batches: 100 Num epochs: 0.04 Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: horovod
Generating training model W1031 09:33:42.997029 139829226764032 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled
自动由 placer 处理。
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:33:42.995286 140195360659200 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:33:42.994239 140271702304512 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:33:42.994754 139677737731840 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:33:43.013096 140519229642496 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.016547 139677737731840 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.016844 140336927958784 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.018044 140195360659200 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.018192 140271702304512 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.017959 140599664752384 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.018513 139737117169408 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.018520 140556463261440 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.020489 139829226764032 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:33:42.993818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.993855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:33:42.993867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:33:42.994212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
W1031 09:33:43.011168 140693077554944 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:33:43.003916 139625134778112 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-31 09:33:42.993204: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:42.993275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:33:42.993288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:33:42.993502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
W1031 09:33:43.006052 140530158384896 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:33:42.999568 140519047988992 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:33:43.022650 140519047988992 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.027833 139625134778112 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.028671 140530158384896 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.034708 140693077554944 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:33:43.045008 139735270668032 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
版本。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.054826 139681104721664 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.059210 140153269229312 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.059533 140519229642496 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.061650 139677737731840 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.064337 140195360659200 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.064880 140556463261440 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.065628 139737117169408 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.065797 140336927958784 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.067691 140599664752384 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.068659 139829226764032 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.071913 140519047988992 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.073436 140271702304512 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.074522 140530158384896 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.075871 139625134778112 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:43.082001 140693077554944 deprecation.py:323] 来自 /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d（来自 tensorflow.python.layers.pooling）已弃用，将在未来版本中移除。 更新说明： 请改用 keras.layers.max_pooling2d。 W1031 09:33:45.444138 139677737731840 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.460240 140153269229312 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.460233 140693077554944 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.461824 140195360659200 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.500418 139735270668032 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.500960 139625134778112 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.502329 140530158384896 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.504304 140519047988992 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.519560 140599664752384 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.531743 139681104721664 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.533167 140556463261440 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.546535 140519229642496 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.572159 140336927958784 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.576735 139829226764032 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.601868 139737117169408 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.606902 139677737731840 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.621120 140693077554944 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.625215 140271702304512 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.627010 140195360659200 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.627309 140153269229312 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.665405 139625134778112 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.667484 139735270668032 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.668427 140519047988992 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.668923 140530158384896 deprecation.py:323] 来自 /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32（来自 tensorflow.python.ops.math_ops）已弃用，将在未来版本中移除。 更新说明： 请改用 tf.cast。 W1031 09:33:45.684920 140599664752384 deprecation.py:323] 来自
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.711060 140556463261440 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.714397 139681104721664 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.717874 140519229642496 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.739548 140336927958784 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.746958 139829226764032 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.773261 139737117169408 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:33:45.799637 140271702304512 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph W1031 09:33:47.557883 139677737731840 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.560198 140693077554944 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.624082 140195360659200 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.636107 140153269229312 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.643582 139625134778112 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.651612 139735270668032 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.680140 140530158384896 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.724147 140599664752384 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.775033 140519229642496 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.785529 139829226764032 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.788232 139681104721664 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.798910 139737117169408 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.816028 140336927958784 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.823199 140556463261440 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:47.911291 140271702304512 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:33:48.041676 140519047988992 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession 2019-10-31 09:33:48.216494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:33:48.216593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:48.216609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:33:48.216620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N 2019-10-31 09:33:48.216880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) 2019-10-31 09:33:48.242691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:33:48.242809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:48.242824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:33:48.242834: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:33:48.243075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) 2019-10-31 09:33:48.323609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:33:48.323757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:48.323776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:33:48.323786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:33:48.324072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:33:48.340712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:33:48.340826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:48.340841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:33:48.340850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:33:48.341093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:33:48.351236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:33:48.351341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:48.351354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:33:48.351364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:33:48.351623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:33:48.381960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:33:48.382084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:33:48.382099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:33:48.382109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003]
2019-10-31 09:33:48.382446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:33:48.390172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:48.390323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.390339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:48.390349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:48.390635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:33:48.427686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:48.427803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.427818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:48.427828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:48.428213: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:33:48.490283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:48.490391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.490407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:48.490416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:48.490670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:33:48.507249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:48.507402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.507436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:33:48.507446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:33:48.507933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:33:48.556344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:33:48.556481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.556495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:33:48.556506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:33:48.556947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:33:48.574927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:48.575046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.575062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:33:48.575072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:33:48.575334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:33:48.576334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:33:48.576457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.576475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:33:48.576485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:33:48.577031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:33:48.621957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:33:48.622073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.622088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:33:48.622098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:33:48.622460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:33:48.648510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:33:48.648617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.648632: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:33:48.648642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:33:48.648908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:33:48.660298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:33:48.660445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:33:48.660466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:33:48.660477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:33:48.660728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
I1031 09:33:51.565660 139677737731840 session_manager.py:491] Running local_init_op.
I1031 09:33:51.679857 140693077554944 session_manager.py:491] Running local_init_op.
I1031 09:33:51.698377 140195360659200 session_manager.py:491] Running local_init_op.
I1031 09:33:51.745176 139677737731840 session_manager.py:493] Done running local_init_op.
I1031 09:33:51.749810 139625134778112 session_manager.py:491] Running local_init_op.
I1031 09:33:51.770334 139735270668032 session_manager.py:491] Running local_init_op.
I1031 09:33:51.828540 140153269229312 session_manager.py:491] Running local_init_op.
I1031 09:33:51.859666 139829226764032 session_manager.py:491] Running local_init_op.
I1031 09:33:51.862105 140530158384896 session_manager.py:491] Running local_init_op.
I1031 09:33:51.864490 140693077554944 session_manager.py:493] Done running local_init_op.
I1031 09:33:51.870752 140599664752384 session_manager.py:491] Running local_init_op.
I1031 09:33:51.883411 140195360659200 session_manager.py:493] Done running local_init_op.
I1031 09:33:51.935793 139625134778112 session_manager.py:493] Done running local_init_op.
I1031 09:33:51.940668 140519229642496 session_manager.py:491] Running local_init_op.
I1031 09:33:51.945004 139737117169408 session_manager.py:491] Running local_init_op.
I1031 09:33:51.967892 139735270668032 session_manager.py:493] Done running local_init_op.
I1031 09:33:51.979897 140336927958784 session_manager.py:491] Running local_init_op.
I1031 09:33:52.010399 140153269229312 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.041698 140519047988992 session_manager.py:491] Running local_init_op.
I1031 09:33:52.052779 139829226764032 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.060791 140530158384896 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.067558 140599664752384 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.077525 140556463261440 session_manager.py:491] Running local_init_op.
I1031 09:33:52.088519 139681104721664 session_manager.py:491] Running local_init_op.
I1031 09:33:52.091564 140271702304512 session_manager.py:491] Running local_init_op.
I1031 09:33:52.144148 140519229642496 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.144645 139737117169408 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.170006 140336927958784 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.232106 140519047988992 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.265110 140556463261440 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.277443 140271702304512 session_manager.py:493] Done running local_init_op.
I1031 09:33:52.277595 139681104721664 session_manager.py:493] Done running local_init_op.
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
Running warm up
2019-10-31 09:34:20.901503: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:20.901916: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:20.985326: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.108461: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.122823: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.298203: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.334801: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.336122: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.388912: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.416389: I
tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.620958: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.845133: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.907949: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.945960: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:21.994445: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-31 09:34:22.360291: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.19<0>
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.19<0>
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.19<0>
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.19<0>
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.27<0>
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.27<0>
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.27<0>
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.27<0>
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
NCCL version 2.4.2+cuda10.0
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.18<0>
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.18<0>
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.18<0>
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.18<0>
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.31<0>
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.31<0>
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.31<0>
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.31<0>
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO comm 0x7fcc343cd5e0 rank 0 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO comm 0x7f929c35d880 rank 6 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO comm 0x7f09183cac80 rank 9 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO comm 0x7fa1cc378dd0 rank 15 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO comm 0x7efc10383a70 rank 3 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO comm 0x7f1624349ad0 rank 14 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO comm 0x7ff4b835d910 rank 2 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO comm 0x7fcec83557f0 rank 1 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO comm 0x7f15b837ba20 rank 11 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO comm 0x7fcc3c33bb70 rank 13 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO comm 0x7f770837e260 rank 12 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO comm 0x7f2b985654d0 rank 4 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO comm 0x7f08503a4be0 rank 5 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO comm 0x7f80d438fcb0 rank 7 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO comm 0x7fd4e83c3530 rank 10 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO comm 0x7fdef84c17c0 rank 8 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO CUDA Dev 3[3], Socket NIC distance : SOC
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO CUDA Dev 2[2], Socket NIC distance : SOC
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO CUDA Dev 1[1], Socket NIC distance : SOC
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance : SOC
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO CUDA Dev 1[1], Socket NIC distance : SOC
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance : SOC
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO CUDA Dev 2[2], Socket NIC distance : SOC
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO CUDA Dev 3[3], Socket NIC distance : SOC
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO CUDA Dev 3[3], Socket NIC distance : SOC
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO CUDA Dev 2[2], Socket NIC distance : SOC
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance : SOC
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO CUDA Dev 1[1], Socket NIC distance : SOC
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO CUDA Dev 2[2], Socket NIC distance : SOC
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance : SOC
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO CUDA Dev 3[3], Socket NIC distance : SOC
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO CUDA Dev 1[1], Socket NIC distance : SOC
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Channel 00 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 7 -> 8 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Ring 00 : 11 -> 12 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Ring 00 : 15 -> 0 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Ring 00 : 3 -> 4 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Ring 00 : 13[1] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO Ring 00 : 14[2] -> 15[3] via P2P/IPC
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO Ring 00 : 6[2] -> 7[3] via P2P/IPC
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Ring 00 : 5[1] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Ring 00 : 4[0] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 00 : 2[2] -> 3[3] via P2P/IPC
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO Ring 00 : 1[1] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Ring 00 : 12[0] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO Ring 00 : 7 -> 8 [send] via NET/Socket/0
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 8[0] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO Ring 00 : 10[2] -> 11[3] via P2P/IPC
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Ring 00 : 9[1] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Ring 00 : 3 -> 4 [send] via NET/Socket/0
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO Ring 00 : 15 -> 0 [send] via NET/Socket/0
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO Ring 00 : 11 -> 12 [send] via NET/Socket/0
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO Ring 00 : 7[3] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Ring 00 : 3[3] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO Ring 00 : 15[3] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO Ring 00 : 11[3] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Trees [0] 2->3->-1/-1/-1
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO Trees [0] 14->15->-1/-1/-1
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 4 -> 8 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO Trees [0] 10->11->-1/-1/-1
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 00 : 2[2] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO Ring 00 : 1[1] -> 0[0] via P2P/IPC
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO Ring 00 : 14[2] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Ring 00 : 13[1] -> 12[0] via P2P/IPC
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Trees [0] 1->2->3/-1/-1
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO Ring 00 : 10[2] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Ring 00 : 5[1] -> 4[0] via P2P/IPC
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Ring 00 : 9[1] -> 8[0] via P2P/IPC
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO Trees [0] 6->7->-1/-1/-1
tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO Trees [0] 13->14->15/-1/-1
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Ring 00 : 4 -> 8 [send] via NET/Socket/0
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO Ring 00 : 6[2] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Trees [0] 4->5->6/-1/-1
tensorflow-benchmarks-worker-2:61:259 [3] NCCL INFO comm 0x7f15b837ba20 rank 11 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO Trees [0] 5->6->7/-1/-1
tensorflow-benchmarks-worker-1:60:261 [2] NCCL INFO comm 0x7f929c35d880 rank 6 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO Trees [0] 9->10->11/-1/-1
tensorflow-benchmarks-worker-2:60:260 [2] NCCL INFO comm 0x7fd4e83c3530 rank 10 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-1:61:259 [3] NCCL INFO comm 0x7f80d438fcb0 rank 7 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Trees [0] 8->9->10/-1/-1
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO comm 0x7f09183cac80 rank 9 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Ring 00 : 8 -> 4 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 12 -> 8 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 8 -> 0 [send] via NET/Socket/0
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO comm 0x7f08503a4be0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 0 -> 8 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 8 -> 4 [send] via NET/Socket/0
tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Ring 00 : 8 -> 12 [send] via NET/Socket/0
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO Trees [0] 8->4->5/-1/-1
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Trees [0] 12->13->14/-1/-1
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO comm 0x7fcc3c33bb70 rank 13 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-3:61:261 [3] NCCL INFO comm 0x7fa1cc378dd0 rank 15 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO comm 0x7efc10383a70 rank 3 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO Trees [0] 0->1->2/-1/-1
tensorflow-benchmarks-worker-0:58:260 [1] NCCL INFO comm 0x7fcec83557f0 rank 1 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Ring 00 : 8 -> 0 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Ring 00 : 0 -> 8 [send] via NET/Socket/0
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Trees [0] -1->0->1/8/-1
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Using 256 threads, Min Comp Cap 6, Trees enabled up to size 479999
tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO comm 0x7fcc343cd5e0 rank 0 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO comm 0x7ff4b835d910 rank 2 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-1:55:260 [0] NCCL INFO comm 0x7f2b985654d0 rank 4 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Ring 00 : 12 -> 8 [send] via NET/Socket/0
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Could not find real path of /sys/class/net/eth0/device
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO include/net.h:24 -> 2
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 3 >= 0)
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Ring 00 : 8 -> 12 [receive] via NET/Socket/0
tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO Trees [0]
8->12->13/-1/-1 tensorflow-benchmarks-worker-3:56:263 [0] NCCL INFO comm 0x7f770837e260 rank 12 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE tensorflow-benchmarks-worker-3:60:258 [2] NCCL INFO comm 0x7f1624349ad0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE tensorflow-benchmarks-worker-0:56:259 [0] NCCL INFO Launch mode Parallel tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO Trees [0] 0->8->9/4/12 tensorflow-benchmarks-worker-2:55:261 [0] NCCL INFO comm 0x7fdef84c17c0 rank 8 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.801 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 8.004 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 8.178 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.886 1 images/sec: 17.3 +/- 0.0 (jitter = 0.0) 7.729 1 images/sec: 17.3 +/- 0.0 (jitter = 0.0) 7.780 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.869 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.788 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.565 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.802 1 images/sec: 17.3 +/- 0.0 (jitter = 0.0) 7.806 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 8.090 1 images/sec: 17.4 +/- 0.0 (jitter = 0.0) 7.888 1 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.952 1 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.684 1 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.636 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.569 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.651 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.585 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.729 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.601 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.738 10 images/sec: 17.5 +/- 0.0 (jitter = 0.2) 7.696 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.547 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.879 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 8.061 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 8.019 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.723 10 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.627 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.714 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.731 10 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.839 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.591 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.617 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.645 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.505 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.639 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.702 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.593 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.661 20 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.767 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.730 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.756 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.601 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.814 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.580 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.423 20 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.555 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.609 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.679 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.558 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.614 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.722 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.654 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.851 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.626 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.861 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.539 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.639 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.513 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.762 30 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.889 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.560 30 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.547 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.613 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.563 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.683 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.570 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.455 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.594 40 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.625 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.509 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.672 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.411 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.389 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.605 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.414 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.686 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.588 40 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.592 50 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.581 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.572 50 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.593 50 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.643 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.531 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.526 50 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.584 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.487 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.552 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.607 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.526 50 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.468 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.433 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.512 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.561 50 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.533 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.523 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.493 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.499 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.484 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.466 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.476 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.351 60 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.602 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.460 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.718 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.592 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.530 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.527 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.387 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.481 60 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.484 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.508 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.475 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.500 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.561 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.528 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.542 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.406 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.521 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.523 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.468 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.482 70 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.617 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.523 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.522 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.490 70 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.572 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.454 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.464 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.474 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.409 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.402 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.527 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.470 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.401 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.533 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.439 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.451 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.481 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.485 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.559 80 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.541 80 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.437 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.394 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.500 90 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.425 90 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.489 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.531 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.444 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.404 90 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.478 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.525 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.533 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.505 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.459 90 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.434 90 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.473 90 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.494 90 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.479 100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.416
total images/sec: 280.34
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.449
total images/sec: 280.34
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.474
total images/sec: 280.36
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.545
total images/sec: 280.32
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.480
total images/sec: 280.32
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.571
total images/sec: 280.33
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.542
total images/sec: 280.32
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.644
total images/sec: 280.34
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.576
total images/sec: 280.31
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.508
total images/sec: 280.32
100 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.569
total images/sec: 280.34
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.466
total images/sec: 280.31
100 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.608
total images/sec: 280.32
100 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.479
total images/sec: 280.30
100 images/sec: 17.5 +/- 0.0 (jitter = 0.1) 7.457
total images/sec: 280.30
100 images/sec: 17.5 +/- 0.0 (jitter = 0.0) 7.603
total images/sec: 280.33
使用 KubeFlow/mpi-operator 运行分布式 TensorFlow 基准测试（未启用 GPUDirect）的日志：
+ POD_NAME=tensorflow-benchmarks-worker-0
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-0 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "943194112" -mca ess_base_vpid 1 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[2:99]pq5,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "943194112.0;tcp://10.254.7.17:57414" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "943194112.0;tcp://10.254.7.17:57414" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
+ POD_NAME=tensorflow-benchmarks-worker-2
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-2 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "943194112" -mca ess_base_vpid 3 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[2:99]pq5,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "943194112.0;tcp://10.254.7.17:57414" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "943194112.0;tcp://10.254.7.17:57414" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
+ POD_NAME=tensorflow-benchmarks-worker-1
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-1 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "943194112" -mca ess_base_vpid 2 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[2:99]pq5,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "943194112.0;tcp://10.254.7.17:57414" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "943194112.0;tcp://10.254.7.17:57414" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
+ POD_NAME=tensorflow-benchmarks-worker-3
+ shift
+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-3 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "943194112" -mca ess_base_vpid 4 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-[2:99]pq5,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "943194112.0;tcp://10.254.7.17:57414" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "943194112.0;tcp://10.254.7.17:57414" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"
2019-10-31 09:22:54.631963: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632290: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632290: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632337: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632513: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632527: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632533: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632586: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632550: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632322: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632709: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632709: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632441: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632877: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632446: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:54.632745: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:22:56.367912: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5cc7a70 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.367966: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.367978: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.367987: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.367995: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.369297: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x530d070 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.369376: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.369390: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.369400: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.369408: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373515: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5b49380 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.373552: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373563: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373973: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.373572: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373580: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373574: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4bf80a0 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.373627: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373641: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373650: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.373658: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.375236: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.376547: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5da52a0 executing computations on platform Host. Devices:
2019-10-31 09:22:56.376576: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.376964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.376995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:22:56.376762: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.377652: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.378087: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x53ea8c0 executing computations on platform Host. Devices:
2019-10-31 09:22:56.378116: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.378480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.378519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:22:56.379111: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5c26bd0 executing computations
on platform Host. Devices:
2019-10-31 09:22:56.379146: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.379513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.379556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:22:56.380668: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4cd5910 executing computations on platform Host. Devices:
2019-10-31 09:22:56.380699: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.382276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.382317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:22:56.382339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x61aa7c0 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.382406: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.382422: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.382431: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.382439: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.383896: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x529ddf0 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.383958: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.383970: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.383979: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.383988: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.386278: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4c07240 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.386353: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.386366: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.386375: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.386383: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.387767: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.388707: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.390507: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6288000 executing computations on platform Host. Devices:
2019-10-31 09:22:56.390536: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.391321: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x537b630 executing computations on platform Host. Devices:
2019-10-31 09:22:56.391351: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.391016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.391057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:22:56.391610: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5ef7e10 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.391685: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.391697: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.391707: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.391715: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.391836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.391866: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:22:56.392037: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.393860: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4f09e80 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.393941: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.393954: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.393963: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.393971: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.394344: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x60fc6c0 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.394384: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.394396: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.394406: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.394415: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.395273: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4ce4a90 executing computations on platform Host. Devices:
2019-10-31 09:22:56.395316: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.396847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.396893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:22:56.397416: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.397783: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x588fcd0 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.397826: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.397839: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.397848: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.397857: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.398054: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.399777: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:22:56.400107: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5fd5650 executing computations on platform Host. Devices:
2019-10-31 09:22:56.400143: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.400499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.400522: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x61d9ef0 executing computations on platform Host. Devices:
2019-10-31 09:22:56.400539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:22:56.400550: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.400889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.400930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:22:56.401425: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x602c630 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.401478: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.401491: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.401499: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.401508: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.401740: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.403194: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4fe76d0 executing computations on platform Host. Devices:
2019-10-31 09:22:56.403226: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.403824: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4ae4ee0 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.403884: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute
Capability 6.0
2019-10-31 09:22:56.403897: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.403906: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.403917: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.404350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.404381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:22:56.405827: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x596d4f0 executing computations on platform Host. Devices:
2019-10-31 09:22:56.405854: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.405909: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:22:56.406370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.406412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:22:56.406964: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4e72300 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.407025: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.407037: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.407046: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.407054: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.409017: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6109e60 executing computations on platform Host. Devices:
2019-10-31 09:22:56.409050: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.409434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.409474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:22:56.410072: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:22:56.411539: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:22:56.412921: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4bc2720 executing computations on platform Host. Devices:
2019-10-31 09:22:56.412950: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.413430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.413476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:22:56.413961: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4f4fb60 executing computations on platform Host. Devices:
2019-10-31 09:22:56.413995: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.416716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.416780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:22:56.417767: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5457b40 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.417842: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.417854: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.417863: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.417871: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.419904: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5abfd50 executing computations on platform CUDA. Devices:
2019-10-31 09:22:56.419943: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.419955: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.419964: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.419973: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:22:56.423879: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.424970: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:22:56.426821: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55353a0 executing computations on platform Host. Devices:
2019-10-31 09:22:56.426854: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.427604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.427637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:22:56.428632: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5b9d580 executing computations on platform Host. Devices:
2019-10-31 09:22:56.428663: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:22:56.429065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:22:56.429095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:22:56.514977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.515063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:22:56.515075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:22:56.515311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:22:56.521886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.521957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:22:56.521972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:22:56.522219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:22:56.523146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.523185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:22:56.523198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:22:56.523406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:22:56.524817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.524852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:22:56.524865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:22:56.525049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:22:56.527885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.527965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:22:56.527979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:22:56.528264: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.525635 139738184173312 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-31 09:22:56.530555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.530624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31
09:22:56.530637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:22:56.530876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:22:56.533018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.533090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:22:56.533104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:22:56.533667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.533777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:22:56.533792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:22:56.534101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:22:56.535090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:22:56.541262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.541337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:22:56.541351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:22:56.541767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.541804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:22:56.541818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:22:56.547678: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.547769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:22:56.547784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:22:56.548011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:22:56.549959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.550024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:22:56.550054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:22:56.550310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.540767 139688259663616 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.549264 139738184173312 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
2019-10-31 09:22:56.553952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.553995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:22:56.554008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:22:56.554209: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:22:56.555930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.555980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:22:56.555992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.536351 140029635917568 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.539637 140418172688128 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.544073 139855535986432 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:22:56.558225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
W1031 09:22:56.540405 140677311760128 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.559299 140029635917568 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.562481 140418172688128 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.563024 140677311760128 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.568738 139855535986432 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.563279 139688259663616 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.571310 140081232451328 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:22:56.542218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.542256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:22:56.542269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:22:56.543952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
W1031 09:22:56.566590 140056334747392 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.553285 140399337785088 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-31 09:22:56.543732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
W1031 09:22:56.561551 140037793744640 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:22:56.543864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
W1031 09:22:56.558358 140208799237888 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.575853 140313852450560 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-31 09:22:56.554008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:22:56.554068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:22:56.554081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:22:56.554325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
W1031 09:22:56.569016 140458139940608 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.560127 140119947085568 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.577635 140399337785088 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:22:56.559716 140509980870400 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.556494 140335919838976 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:22:56.581333 140208799237888 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.581812 140509980870400 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.582422 140335919838976 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.582604 140119947085568 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031
09:22:56.585775 140037793744640 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.592101 140458139940608 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.592391 140056334747392 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.595070 140081232451328 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.597523 139738184173312 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.601429 140313852450560 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:22:56.605411 140029635917568 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.608743 140677311760128 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.610531 140418172688128 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.615458 139688259663616 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.621234 139855535986432 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.626380 140399337785088 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.628083 140509980870400 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.628359 140119947085568 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.630022 140208799237888 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.634344 140335919838976 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.636383 140037793744640 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.638491 140458139940608 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.642560 140081232451328 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.644791 140056334747392 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:56.653468 140313852450560 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:22:58.918779 139738184173312 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:58.993772 139688259663616 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.006589 140509980870400 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.037595 140458139940608 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.040095 140677311760128 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.048176 140208799237888 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.066786 140056334747392 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.072578 140399337785088 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.074004 140418172688128 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.077747 139738184173312 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.084980 140081232451328 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.085981 140037793744640 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.091650 140029635917568 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.107937 140335919838976 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.113702 139855535986432 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.128616 140313852450560 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.157861 139688259663616 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.172701 140509980870400 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:22:59.176086 140119947085568 deprecation.py:323] From
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.203059 140677311760128 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.208686 140458139940608 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.214540 140208799237888 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.240148 140056334747392 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.241494 140418172688128 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.245378 140399337785088 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.253228 140037793744640 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.257912 140081232451328 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.264204 140029635917568 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.278820 140335919838976 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.285089 139855535986432 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.302716 140313852450560 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. W1031 09:22:59.361849 140119947085568 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph Initializing graph W1031 09:23:01.022134 139738184173312 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession Initializing graph W1031 09:23:01.126116 140509980870400 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.158821 139688259663616 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.161875 140458139940608 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.191108 140208799237888 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.192427 140677311760128 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.226984 140399337785088 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.229759 140418172688128 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.262723 140037793744640 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.277419 140335919838976 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.291485 140029635917568 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.304083 140081232451328 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.315738 140313852450560 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.316774 139855535986432 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.547414 140119947085568 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:23:01.630776 140056334747392 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession 2019-10-31 09:23:01.712294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:23:01.712394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:23:01.712408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:23:01.712418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N 2019-10-31 09:23:01.712657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) 2019-10-31 09:23:01.801686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:23:01.801803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:23:01.801819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:23:01.801829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:23:01.805672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:23:01.845457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:23:01.845549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:23:01.845563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:23:01.845573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:23:01.846712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) 2019-10-31 09:23:01.849217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:23:01.849313: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.849327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:23:01.849337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:23:01.849595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:23:01.873104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:23:01.873235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.873251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:23:01.873261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:23:01.873513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:23:01.875778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:23:01.875880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.875896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:23:01.875905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:23:01.876185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:23:01.915287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:23:01.915416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.915432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:23:01.915443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:23:01.915764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:23:01.929700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:23:01.929808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.929823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:23:01.929833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:23:01.930084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:23:01.949855: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:23:01.949970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.949986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:23:01.949996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:23:01.950398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:23:01.989288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:23:01.989427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:01.989445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:23:01.989455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:23:01.989732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:23:02.007311: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:23:02.007427: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:02.007441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:23:02.007451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:23:02.007695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:23:02.018650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:23:02.018750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:02.018764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:23:02.018773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:23:02.019026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:23:02.058512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:23:02.058620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:02.058636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:23:02.058645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:23:02.058895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:23:02.087659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:23:02.087769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:02.087786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:23:02.087796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:23:02.088065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:23:02.263137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:23:02.263242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:02.263257: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:23:02.263267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:23:02.263647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:23:02.311255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:23:02.311366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:23:02.311382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:23:02.311407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:23:02.311675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
I1031 09:23:05.001815 139738184173312 session_manager.py:491] Running local_init_op.
I1031 09:23:05.110615 140509980870400 session_manager.py:491] Running local_init_op.
I1031 09:23:05.175470 139738184173312 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.175794 139688259663616 session_manager.py:491] Running local_init_op.
I1031 09:23:05.194576 140458139940608 session_manager.py:491] Running local_init_op.
I1031 09:23:05.218982 140677311760128 session_manager.py:491] Running local_init_op.
I1031 09:23:05.245636 140208799237888 session_manager.py:491] Running local_init_op.
I1031 09:23:05.287847 140418172688128 session_manager.py:491] Running local_init_op.
I1031 09:23:05.289076 140509980870400 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.352205 140399337785088 session_manager.py:491] Running local_init_op.
I1031 09:23:05.366151 139688259663616 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.377948 140458139940608 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.383546 140335919838976 session_manager.py:491] Running local_init_op.
I1031 09:23:05.389096 140081232451328 session_manager.py:491] Running local_init_op.
I1031 09:23:05.403619 140677311760128 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.410839 140029635917568 session_manager.py:491] Running local_init_op.
I1031 09:23:05.417464 140037793744640 session_manager.py:491] Running local_init_op.
I1031 09:23:05.419884 140208799237888 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.487072 140313852450560 session_manager.py:491] Running local_init_op.
I1031 09:23:05.487062 140418172688128 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.528172 139855535986432 session_manager.py:491] Running local_init_op.
I1031 09:23:05.562064 140335919838976 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.564437 140399337785088 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.572010 140081232451328 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.591631 140029635917568 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.607434 140037793744640 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.667109 140056334747392 session_manager.py:491] Running local_init_op.
I1031 09:23:05.671607 140313852450560 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.704953 139855535986432 session_manager.py:493] Done running local_init_op.
I1031 09:23:05.706148
140119947085568 session_manager.py:491] Running local_init_op. I1031 09:23:05.865507 140056334747392 session_manager.py:493] Done running local_init_op. I1031 09:23:05.876705 140119947085568 session_manager.py:493] Done running local_init_op. Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up 2019-10-31 09:23:34.466658: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:34.697844: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:34.730197: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:34.755108: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:34.909374: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:34.920007: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:34.939428: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.025650: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.061560: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.124756: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.156766: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.232465: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.394387: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.593960: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:35.635274: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:23:36.004079: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.27<0> tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.16<0> tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.16<0> tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.27<0> tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.27<0> tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.27<0> tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.27<0> NCCL version 2.4.2+cuda10.0 tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.16<0> tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.16<0> tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.25<0> tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.25<0> tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.25<0> tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.25<0> tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.27<0> tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.27<0> tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.27<0> tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.16<0> tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.16<0> tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.25<0> tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.16<0> tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.16<0> tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.25<0> tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.25<0> tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.25<0> tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.16<0> tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.16<0> tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Setting affinity for GPU 0 to 0fff tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO comm 0x7f60783f8e30 rank 0 nranks 16 cudaDev 0 nvmlDev 0 tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Setting affinity for GPU 2 to 0fff tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO comm 0x7f5c283d9250 rank 2 nranks 16 cudaDev 2 nvmlDev 2 tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO Setting affinity for GPU 3 to 0fff tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO Setting affinity for GPU 1 to 0fff tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO comm 0x7fb0543bc410 rank 1 nranks 16 cudaDev 1 nvmlDev 1 tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO comm 0x7f83f83e3580 rank 3 nranks 16 cudaDev 3 nvmlDev 3 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Setting affinity for GPU 0 to 0fff tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO comm 0x7f6f4846aa90 rank 12 nranks 16 cudaDev 0 nvmlDev 0 tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO Setting affinity for GPU 3 to 0fff tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO comm 0x7f66443edf60 rank 7 nranks 16 cudaDev 3 nvmlDev 3 tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Setting affinity for GPU 0 to 0fff tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO comm 0x7f5a4048d110 rank 8 nranks 16 cudaDev 0 nvmlDev 0 tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Setting affinity for GPU 2 to 0fff tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO comm 0x7fbe043e2de0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO Setting affinity for GPU 1 to 0fff tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO comm 0x7fa1903dd250 rank 13 nranks 16 cudaDev 1 nvmlDev 1 tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO Setting affinity for GPU 3 to 0fff tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO comm 0x7fca18409ac0 rank 15 nranks 16 cudaDev 3 nvmlDev 3 tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Setting affinity for GPU 1 to 0fff tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Setting affinity for GPU 2 to 0fff tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO comm 0x7f16643ea9b0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Setting affinity for GPU 0 to 0fff tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO comm 0x7f9c6c47b310 rank 4 nranks 16 cudaDev 0 nvmlDev 0 tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO comm 0x7f0ac43ead00 rank 6 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO comm 0x7ff10c3ee8d0 rank 11 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO comm 0x7f31b83e57d0 rank 10 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO comm 0x7fb4b83eb6d0 rank 9 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Channel 00 : 0 1 3 6 4 5 7 10 8 9 11 14 12 13 15 2
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Channel 01 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 3 -> 6 [receive] via NET/IB/0
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 00 : 15 -> 2 [receive] via NET/IB/0
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Ring 00 : 11 -> 14 [receive] via NET/IB/0
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Ring 00 : 5[1] -> 7[3] via P2P/IPC
tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO Ring 00 : 1[1] -> 3[3] via P2P/IPC
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Ring 00 : 4[0] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 6[2] -> 4[0] via P2P/IPC
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO Ring 00 : 13[1] -> 15[3] via P2P/IPC
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 7 -> 10 [receive] via NET/IB/0
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 00 : 2[2] -> 0[0] via P2P/IPC
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Ring 00 : 14[2] -> 12[0] via P2P/IPC
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 00 : 12[0] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO Ring 00 : 7 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO Ring 00 : 15 -> 2 [send] via NET/IB/0
tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO Ring 00 : 3 -> 6 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 263 mtu 5 LID 12
tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 263 mtu 5 LID 16
tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 263 mtu 5 LID 9
tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO Ring 00 : 9[1] -> 11[3] via P2P/IPC
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Ring 00 : 8[0] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 10[2] -> 8[0] via P2P/IPC
tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO Ring 00 : 15[3] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO Ring 00 : 11 -> 14 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO Ring 00 : 13[1] -> 12[0] via P2P/IPC
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 263 mtu 5 LID 14
tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO Ring 00 : 7[3] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 00 : 12[0] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO Ring 00 : 3[3] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Ring 00 : 0[0] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO Ring 00 : 1[1] -> 0[0] via P2P/IPC
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Ring 00 : 4[0] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Ring 00 : 5[1] -> 4[0] via P2P/IPC
tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO Ring 01 : 13[1] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO Ring 00 : 11[3] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0)
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 00 : 10 -> 2 [receive] via NET/IB/0
tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO Ring 01 : 15 -> 0 [send] via NET/IB/0
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Ring 01 : 15 -> 0 [receive] via NET/IB/1
tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO Ring 01 : 1[1] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Ring 00 : 8[0] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO Ring 00 : 9[1] -> 8[0] via P2P/IPC
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0)
tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 266 mtu 5 LID 16
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Ring 01 : 0[0] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Ring 00 : 14 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO Ring 01 : 3 -> 4 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 11 -> 12 [receive] via NET/IB/1
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 12[0] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0)
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 6 -> 10 [receive] via NET/IB/0
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 0.
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0)
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Ring 01 : 7 -> 8 [receive] via NET/IB/1
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 14 -> 10 [receive] via NET/IB/0
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 267 mtu 5 LID 16
tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO Ring 01 : 9[1] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Ring 01 : 0 -> 12 [send] via NET/IB/1
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO Ring 01 : 11 -> 12 [send] via NET/IB/0
tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Ring 01 : 8[0] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Ring 01 : 3 -> 4 [receive] via NET/IB/1
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 6 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Ring 00 : 10 -> 14 [receive] via NET/IB/0
tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 266 mtu 5 LID 14
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 266 mtu 5 LID 12
tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 266 mtu 5 LID 9
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 10 -> 2 [send] via NET/IB/0
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 267 mtu 5 LID 14
tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Ring 01 : 5[1] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO Ring 01 : 7 -> 8 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Ring 01 : 4[0] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 267 mtu 5 LID 12
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 10 -> 6 [receive] via NET/IB/0
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0)
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 8 -> 12 [receive] via NET/IB/1
tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 00 : 2 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0)
tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 0 -> 12 [receive] via NET/IB/1
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 2[2] / HCA 0 (distance 0 >= 0)
tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 267 mtu 5 LID 8
tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 2 -> 10 [receive] via NET/IB/0
tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO
NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0) tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 269 mtu 5 LID 9 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 12 -> 4 [send] via NET/IB/1 tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Ring 01 : 8 -> 12 [send] via NET/IB/1 tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Ring 01 : 12 -> 4 [receive] via NET/IB/1 tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 270 mtu 5 LID 13 tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 01 : 2[2] -> 3[3] via P2P/IPC tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 10 -> 6 [send] via NET/IB/0 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 272 mtu 5 LID 10 tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 00 : 10 -> 14 [send] via NET/IB/0 tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 272 mtu 5 LID 14 tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO Ring 01 : 3[3] -> 2[2] via P2P/IPC tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 273 mtu 5 LID 14 tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO Ring 01 : 1[1] -> 0[0] via P2P/IPC tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0) tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO Trees [0] 1->3->-1/-1/-1 [1] 2->3->-1/-1/-1 tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Ring 01 : 2[2] -> 1[1] via P2P/IPC tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Ring 01 : 12 -> 0 [receive] via NET/IB/1 tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO Trees [0] 0->1->3/-1/-1 [1] 0->1->2/-1/-1 tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 01 : 6[2] -> 7[3] via P2P/IPC tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Ring 01 : 14[2] -> 15[3] via P2P/IPC tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 01 : 10[2] -> 11[3] via P2P/IPC tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO Ring 01 : 15[3] -> 14[2] via P2P/IPC tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO Ring 01 : 11[3] -> 10[2] via P2P/IPC tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO Ring 01 : 7[3] -> 6[2] via P2P/IPC tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Ring 01 : 5[1] -> 4[0] via P2P/IPC tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO Ring 01 : 13[1] -> 12[0] via P2P/IPC tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO Ring 01 : 9[1] -> 8[0] via P2P/IPC tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0) tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO Trees [0] 9->11->-1/-1/-1 [1] 10->11->-1/-1/-1 tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Ring 01 : 10[2] -> 9[1] via P2P/IPC tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO Trees [0] 13->15->-1/-1/-1 [1] 14->15->-1/-1/-1 tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Ring 01 : 12 -> 8 [receive] via NET/IB/1 tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO Trees [0] 8->9->11/-1/-1 [1] 8->9->10/-1/-1 tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO Trees [0] 5->7->-1/-1/-1 [1] 6->7->-1/-1/-1 tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Ring 01 : 14[2] -> 13[1] via P2P/IPC tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO Trees [0] 12->13->15/-1/-1 [1] 12->13->14/-1/-1 tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 01 : 6[2] -> 5[1] via P2P/IPC tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO Trees [0] 4->5->7/-1/-1 [1] 4->5->6/-1/-1 tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Trees [0] 10->6->4/-1/-1 [1] 5->6->7/-1/-1 tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO Trees [0] -1->2->0/10/-1 [1] 1->2->3/-1/-1 tensorflow-benchmarks-worker-0:60:261 [2] NCCL INFO comm 0x7f5c283d9250 rank 2 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE tensorflow-benchmarks-worker-0:61:259 [3] NCCL INFO comm 0x7f83f83e3580 rank 3 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE tensorflow-benchmarks-worker-0:59:260 [1] NCCL INFO comm 0x7fb0543bc410 rank 1 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Ring 01 : 4 -> 12 [send] via NET/IB/1 tensorflow-benchmarks-worker-1:58:258 [1] NCCL INFO comm 0x7f16643ea9b0 rank 5 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE tensorflow-benchmarks-worker-1:61:260 [3] NCCL INFO comm 0x7f66443edf60 rank 7 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO comm 0x7f0ac43ead00 rank 6 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 267 mtu 5 LID 11 tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO Trees [0] 6->4->5/-1/-1 [1] -1->4->5/12/-1 tensorflow-benchmarks-worker-1:57:267 [0] NCCL INFO comm 0x7f9c6c47b310 rank 4 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO Trees [0] 2->10->8/6/14 [1] 9->10->11/-1/-1 tensorflow-benchmarks-worker-2:60:258 [2] NCCL INFO comm 0x7f31b83e57d0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE tensorflow-benchmarks-worker-2:61:260 [3] NCCL INFO comm 0x7ff10c3ee8d0 rank 11 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE tensorflow-benchmarks-worker-2:59:261 [1] NCCL INFO comm 0x7fb4b83eb6d0 rank 9 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO Trees [0] 10->14->12/-1/-1 [1] 13->14->15/-1/-1 tensorflow-benchmarks-worker-3:60:259 [2] NCCL INFO comm 0x7fbe043e2de0 rank 14 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE tensorflow-benchmarks-worker-3:61:258 [3] NCCL INFO comm 0x7fca18409ac0 rank 15 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE tensorflow-benchmarks-worker-3:58:260 [1] NCCL INFO comm 0x7fa1903dd250 rank 13 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[0] / HCA 1 (distance 0 >= 0) tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 4 -> 12 [receive] via NET/IB/1 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 12 -> 8 [send] via NET/IB/1 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Ring 01 : 12 -> 0 [send] via NET/IB/1 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 277 mtu 5 LID 10 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 278 mtu 5 LID 10 tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Trees [0] 2->0->1/-1/-1 [1] 12->0->1/-1/-1 tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Using 256 threads, Min Comp Cap 6, Trees enabled up to size 479999 tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO Trees [0] 10->8->9/-1/-1 [1] 12->8->9/-1/-1 tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO comm 0x7f60783f8e30 rank 0 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE tensorflow-benchmarks-worker-0:58:258 [0] NCCL INFO Launch mode Parallel tensorflow-benchmarks-worker-2:57:259 [0] NCCL INFO comm 0x7f5a4048d110 rank 8 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO Trees [0] 14->12->13/-1/-1 [1] 4->12->13/8/0 tensorflow-benchmarks-worker-3:57:261 [0] NCCL INFO comm 0x7f6f4846aa90 rank 12 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss Done warm up Step Img/sec total_loss 1 images/sec: 186.8 +/- 0.0 (jitter = 0.0) 7.892 1 images/sec: 186.7 +/- 0.0 (jitter = 0.0) 7.650 1 images/sec: 186.5 +/- 0.0 (jitter = 0.0) 7.778 1 images/sec: 186.2 +/- 0.0 (jitter = 0.0) 7.585 1 images/sec: 186.4 +/- 0.0 (jitter = 0.0) 7.568 1 images/sec: 187.4 +/- 0.0 (jitter = 0.0) 7.965 1 images/sec: 186.5 +/- 0.0 (jitter = 0.0) 7.828 1 images/sec: 186.4 +/- 0.0 (jitter = 0.0) 8.063 1 images/sec: 186.5 +/- 0.0 (jitter = 0.0) 8.154 1 images/sec: 186.7 +/- 0.0 (jitter = 0.0) 7.896 1 images/sec: 186.1 +/- 0.0 (jitter = 0.0) 7.792 1 images/sec: 186.3 +/- 0.0 (jitter = 0.0) 7.707 1 images/sec: 186.6 +/- 0.0 (jitter = 0.0) 7.768 1 images/sec: 186.0 +/- 0.0 (jitter = 0.0) 7.985 1 images/sec: 186.6 +/- 0.0 (jitter = 0.0) 7.752 1 images/sec: 186.2 +/- 0.0 (jitter = 0.0) 7.902 10 images/sec: 185.8 +/- 0.5 (jitter = 0.7) 7.585 10 images/sec: 185.9 +/- 0.4 (jitter = 0.4) 7.623 10 images/sec: 186.0 +/- 0.4 (jitter = 0.3) 7.543 10 images/sec: 185.9 +/- 0.3 (jitter = 0.6) 7.645 10 images/sec: 185.7 +/- 0.5 (jitter = 0.8) 7.742 10 images/sec: 185.7 +/- 0.5 (jitter = 0.8) 7.718 10 images/sec: 185.7 +/- 0.5 (jitter = 0.9) 7.731 10 images/sec: 185.8 +/- 0.3 (jitter = 0.6) 7.557 10 images/sec: 185.7 +/- 0.4 (jitter = 1.0) 7.771 10 images/sec: 185.6 +/- 0.6 (jitter = 0.8) 7.869 10 images/sec: 185.8 +/- 0.4 (jitter = 0.6) 8.020 10 images/sec: 185.7 +/- 0.4 (jitter = 1.2) 7.594 10 images/sec: 185.7 +/- 0.5 (jitter = 0.3) 7.700 10 images/sec: 185.8 +/- 0.5 (jitter = 0.8) 7.648 10 images/sec: 185.8 +/- 0.3 (jitter = 0.7) 8.038 10 images/sec: 185.7 +/- 0.5 (jitter = 0.8) 7.800 20 images/sec: 186.1 +/- 0.3 (jitter = 0.5) 7.636 20 images/sec: 186.1 +/- 0.3 (jitter = 0.8) 7.773 20 images/sec: 186.1 +/- 0.2 (jitter = 0.5) 7.665 20 images/sec: 186.1 +/- 0.3 (jitter = 0.7) 7.570 20 images/sec: 186.2 +/- 0.2 (jitter = 0.4) 7.657 20 images/sec: 186.1 +/- 0.3 (jitter = 0.5) 7.419 20 images/sec: 186.1 +/- 0.3 (jitter = 0.8) 7.733 20 images/sec: 186.1 +/- 0.2 (jitter = 0.5) 7.520 20 images/sec: 186.1 +/- 0.3 (jitter = 0.7) 7.608 20 images/sec: 186.1 +/- 0.2 (jitter = 0.6) 7.645 20 images/sec: 186.1 +/- 0.2 (jitter = 0.4) 7.771 20 images/sec: 186.2 +/- 0.2 (jitter = 0.5) 7.590 20 images/sec: 186.1 +/- 0.2 (jitter = 0.5) 7.814 20 images/sec: 186.1 +/- 0.2 (jitter = 0.3) 7.683 20 images/sec: 186.1 +/- 0.3 (jitter = 0.5) 7.587 20 images/sec: 186.2 +/- 0.3 (jitter = 1.1) 7.582 30 images/sec: 186.0 +/- 0.2 (jitter = 0.6) 7.602 30 images/sec: 186.0 +/- 0.2 (jitter = 0.5) 7.587 30 images/sec: 186.0 +/- 0.2 (jitter = 0.6) 7.590 30 images/sec: 185.9 +/- 0.3 (jitter = 0.5) 7.559 30 images/sec: 185.9 +/- 0.2 (jitter = 0.6) 7.515 30 images/sec: 185.9 +/- 0.2 (jitter = 0.6) 7.636 30 images/sec: 185.9 +/- 0.2 (jitter = 0.8) 7.850 30 images/sec: 185.9 +/- 0.2 (jitter = 0.8) 7.856 30 images/sec: 185.9 +/- 0.3 (jitter = 0.8) 7.596 30 images/sec: 185.9 +/- 0.3 (jitter = 0.6) 7.750 30 images/sec: 185.9 +/- 0.3 (jitter = 0.9) 7.683 30 images/sec: 185.9 +/- 0.3 (jitter = 0.7) 7.593 30 images/sec: 185.9 +/- 0.3 (jitter = 0.8) 7.630 30 images/sec: 185.9 +/- 0.3 (jitter = 0.8) 7.518 30 images/sec: 185.9 +/- 0.3 (jitter = 0.9) 7.754 30 images/sec: 185.9 +/- 0.3 (jitter = 0.7) 7.876 40 images/sec: 185.6 +/- 0.2 (jitter = 0.7) 7.408 40 images/sec: 185.7 +/- 0.2 (jitter = 0.8) 7.657 40 images/sec: 185.6 +/- 0.2 (jitter = 0.8) 7.647 40 images/sec: 185.6 +/- 0.3 (jitter = 0.6) 7.562 40 images/sec: 185.7 +/- 0.3 (jitter = 0.6) 7.656 40 images/sec: 185.6 +/- 0.3 (jitter = 0.8) 7.656 40 images/sec: 185.7 +/- 0.3 (jitter = 0.9) 7.432 40 images/sec: 185.6 +/- 0.3 (jitter = 0.8) 7.660 40 images/sec: 185.7 +/- 0.2 (jitter = 0.9) 7.581 40 images/sec: 185.6 +/- 0.2 (jitter = 0.7) 7.613 40 images/sec: 185.7 +/- 0.2 (jitter = 0.7) 7.540 40 images/sec: 185.6 +/- 0.3 (jitter =
0.9)\t7.680\n40\timages/sec: 185.7 +/- 0.2 (jitter = 0.6)\t7.548\n40\timages/sec: 185.6 +/- 0.3 (jitter = 1.1)\t7.512\n40\timages/sec: 185.7 +/- 0.3 (jitter = 0.8)\t7.367\n40\timages/sec: 185.7 +/- 0.3 (jitter = 0.8)\t7.629\n50\timages/sec: 185.7 +/- 0.3 (jitter = 0.7)\t7.626\n50\timages/sec: 185.6 +/- 0.3 (jitter = 0.8)\t7.551\n50\timages/sec: 185.7 +/- 0.3 (jitter = 0.8)\t7.625\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.580\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.506\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.540\n50\timages/sec: 185.6 +/- 0.2 (jitter = 0.8)\t7.560\n50\timages/sec: 185.6 +/- 0.2 (jitter = 0.7)\t7.497\n50\timages/sec: 185.7 +/- 0.2 (jitter = 1.0)\t7.445\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.505\n50\timages/sec: 185.6 +/- 0.2 (jitter = 0.8)\t7.550\n50\timages/sec: 185.6 +/- 0.3 (jitter = 0.9)\t7.512\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.578\n50\timages/sec: 185.7 +/- 0.2 (jitter = 1.0)\t7.489\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.441\n50\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.583\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.6)\t7.435\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.7)\t7.470\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.473\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.6)\t7.435\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.376\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.6)\t7.497\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.9)\t7.574\n60\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.555\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.7)\t7.465\n60\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.562\n60\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.473\n60\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.632\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.6)\t7.509\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.7)\t7.473\n60\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.532\n60\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.550\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.471\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.461\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.517\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.493\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.585\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.462\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.520\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.530\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.535\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.426\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.560\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.475\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.464\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.511\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.503\n70\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.524\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.381\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.453\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.487\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.471\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.425\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.466\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.464\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.436\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.453\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.6)\t7.540\n80\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.503\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.479\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.384\n80\timages/sec: 185.7 +/- 0.1 (jitter = 0.7)\t7.501\n80\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.512\n80\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.410\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.518\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.459\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.506\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.395\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.444\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.447\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.428\n90\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.527\n90\timages/sec: 185.8 +/- 0.1 (jitter = 0.8)\t7.485\n90\timages/sec: 185.7 +/- 0.1 (jitter = 0.7)\t7.475\n90\timages/sec: 185.8 +/- 0.2 (jitter = 0.8)\t7.450\n90\timages/sec: 185.7 +/- 0.1 (jitter = 0.6)\t7.500\n90\timages/sec: 185.7 +/- 0.1 (jitter = 0.7)\t7.424\n90\timages/sec: 185.7 +/- 0.1 (jitter = 0.8)\t7.404\n90\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.528\n90\timages/sec: 185.8 +/- 0.1 (jitter = 0.8)\t7.553\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.489\n----------------------------------------------------------------\ntotal images/sec: 2969.87\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.606\n----------------------------------------------------------------\ntotal images/sec: 2969.87\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.9)\t7.510\n----------------------------------------------------------------\ntotal images/sec: 2969.87\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.8)\t7.630\n----------------------------------------------------------------\ntotal images/sec: 2969.83\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.7)\t7.440\n----------------------------------------------------------------\ntotal images/sec: 2969.91\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.614\n----------------------------------------------------------------\ntotal images/sec: 2970.02\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.420\n----------------------------------------------------------------\ntotal images/sec: 2969.89\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.2 (jitter = 0.8)\t7.548\n----------------------------------------------------------------\ntotal images/sec: 2969.84\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.7)\t7.416\n----------------------------------------------------------------\ntotal images/sec: 2969.87\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.8)\t7.449\n----------------------------------------------------------------\ntotal images/sec: 2970.00\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.8)\t7.457\n----------------------------------------------------------------\ntotal images/sec: 2969.87\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.8)\t7.480\n----------------------------------------------------------------\ntotal images/sec: 2969.98\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.6)\t7.553\n----------------------------------------------------------------\ntotal images/sec: 2969.87\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.8)\t7.476\n----------------------------------------------------------------\ntotal images/sec: 2969.95\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.7)\t7.432\n----------------------------------------------------------------\ntotal images/sec: 2969.89\n----------------------------------------------------------------\n100\timages/sec: 185.7 +/- 0.1 (jitter = 0.7)\t7.514\n----------------------------------------------------------------\ntotal images/sec: 2969.75\n----------------------------------------------------------------\n\n\n \n \n \n \n \n \n \n 使用 GPUDirect 运行分布式 TensorFlow 基准测试的日志（KubeFlow/mpi-operator）：\n \n \n \n + POD_NAME=tensorflow-benchmarks-worker-2\n+ shift\n+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-2 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "1488060416" -mca ess_base_vpid 3 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-zc[2:68]w,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "1488060416.0;tcp://10.254.5.28:54316" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "1488060416.0;tcp://10.254.5.28:54316" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"\n+ POD_NAME=tensorflow-benchmarks-worker-0\n+ shift\n+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-0 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "1488060416" -mca ess_base_vpid 1 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-zc[2:68]w,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "1488060416.0;tcp://10.254.5.28:54316" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "1488060416.0;tcp://10.254.5.28:54316" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"\n+ POD_NAME=tensorflow-benchmarks-worker-1\n+ shift\n+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-1 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "1488060416" -mca ess_base_vpid 2 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-zc[2:68]w,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "1488060416.0;tcp://10.254.5.28:54316" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "1488060416.0;tcp://10.254.5.28:54316" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"\n+ POD_NAME=tensorflow-benchmarks-worker-3\n+ shift\n+ /opt/kube/kubectl exec tensorflow-benchmarks-worker-3 -- /bin/sh -c PATH=/usr/local/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH ; export DYLD_LIBRARY_PATH ; /usr/local/bin/orted -mca ess "env" -mca ess_base_jobid "1488060416" -mca ess_base_vpid 4 -mca ess_base_num_procs "5" -mca orte_node_regex "tensorflow-benchmarks-launcher-zc[2:68]w,tensorflow-benchmarks-worker-[1:0-3]@0(5)" -mca orte_hnp_uri "1488060416.0;tcp://10.254.5.28:54316" -mca pml "ob1" -mca btl "^openib" -mca plm "rsh" --tree-spawn -mca orte_parent_uri "1488060416.0;tcp://10.254.5.28:54316" -mca plm_rsh_agent "/etc/mpi/kubexec.sh" -mca orte_default_hostfile "/etc/mpi/hostfile" -mca hwloc_base_binding_policy "none" -mca rmaps_base_mapping_policy "slot" -mca pmix "^s1,s2,cray,isolated"\n2019-10-31 09:28:00.800628: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n2019-10-31 09:28:00.800997: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n2019-10-31 09:28:00.800997: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this
TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801030: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801030: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801282: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801376: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801302: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801649: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801601: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801636: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801388: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801705: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801389: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801536: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:00.801461: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-10-31 09:28:02.552940: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x577afe0 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.553014: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.553027: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.553035: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.553043: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.558257: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:28:02.558331: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x571fd40 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.558380: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.558401: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.558409: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.558418: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.560581: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5858810 executing computations on platform Host. Devices:
2019-10-31 09:28:02.560613: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.560953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.560997: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:28:02.562364: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:28:02.565507: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x57fd570 executing computations on platform Host. Devices:
2019-10-31 09:28:02.565538: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.566019: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4feb450 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.566090: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.566103: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.566112: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.566120: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.566436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.566477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:28:02.568316: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x61a8df0 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.568370: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.568382: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.568390: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.568398: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569013: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x63f9d80 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.569074: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569088: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569097: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569105: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569386: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x61242b0 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.569435: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569448: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569457: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.569465: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.572749: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:28:02.573019: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x599ee80 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.573059: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.573071: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.573080: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.573088: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.575052: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.576129: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x50c8cb0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.576165: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.576414: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5c5e400 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.576478: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.576456: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.576467: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.576482: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.576490: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.576516: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.576594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.576640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:28:02.577872: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.578533: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x580b140 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.578627: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.578640: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.578649: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.578657: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.579048: I
tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6286630 executing computations on platform Host. Devices:
2019-10-31 09:28:02.579134: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.579233: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6201b00 executing computations on platform Host. Devices:
2019-10-31 09:28:02.579274: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.579465: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x64d75b0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.579501: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.579561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.579598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:28:02.579699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.579745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:28:02.579795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.579824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:28:02.580795: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a7c6d0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.580832: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.581184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.581222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:28:02.582929: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199930000 Hz
2019-10-31 09:28:02.584555: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x504b780 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.584604: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.584619: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.584627: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.584635: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.586058: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5d3bc70 executing computations on platform Host. Devices:
2019-10-31 09:28:02.586090: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.586869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.586912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:28:02.587132: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.588489: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x53cfb70 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.588534: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.588547: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.588556: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.588565: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.589801: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.589983: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x58e89b0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.590017: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.591058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.591099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:28:02.592938: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5128fc0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.592970: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.593754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.593793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:28:02.597195: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.600271: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5524250 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.600348: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.600359: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.600368: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.600376: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.600478: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x54ad3d0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.600511: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.601636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.601681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:28:02.604752: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.608007: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5601aa0 executing computations on platform Host. Devices:
2019-10-31 09:28:02.608058: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.608527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0e:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.608557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3
2019-10-31 09:28:02.609073: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x58ca930 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.609136: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.609150: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.609159: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.609168: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.610637: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4b03cf0 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.610699: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.610712: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.610721: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.610729: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.613057: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4ff4940 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.613092: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.613104: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.613114: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.613123: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.614904: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.616644: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.617349: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.617666: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x59a8170 executing computations on platform Host. Devices:
2019-10-31 09:28:02.617696: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.618133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.618177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:28:02.619870: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x50d2180 executing computations on platform Host. Devices:
2019-10-31 09:28:02.619901: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.620285: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x4be1520 executing computations on platform Host. Devices:
2019-10-31 09:28:02.620314: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.620599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:0c:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.620630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2
2019-10-31 09:28:02.620790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:04:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.620819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-10-31 09:28:02.621544: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6531630 executing computations on platform CUDA. Devices:
2019-10-31 09:28:02.621602: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.621615: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.621625: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.621634: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla P100-PCIE-16GB, Compute Capability 6.0
2019-10-31 09:28:02.629231: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199995000 Hz
2019-10-31 09:28:02.632316: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x660ee60 executing computations on platform Host. Devices:
2019-10-31 09:28:02.632347: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-10-31 09:28:02.633159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:06:00.0
totalMemory: 15.90GiB freeMemory: 14.90GiB
2019-10-31 09:28:02.633189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1
2019-10-31 09:28:02.694075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.694155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:28:02.694169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:28:02.694406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:28:02.701335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.701383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:28:02.701396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:28:02.723552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.723631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:28:02.723645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:28:02.725370: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.725442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:28:02.725455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:28:02.725764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.725830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:28:02.725844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:28:02.725697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:28:02.725870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
2019-10-31 09:28:02.726036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.726079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:28:02.726091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:28:02.726137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
2019-10-31 09:28:02.726303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
2019-10-31 09:28:02.726337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.726375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:28:02.726391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:28:02.726585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:28:02.730960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.731001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:28:02.731014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:28:02.731236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.738230 139856166496000 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-31 09:28:02.738561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.738615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:28:02.738628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:28:02.738826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:28:02.712095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.712132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:28:02.712144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:28:02.712327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
W1031 09:28:02.731517 139818715670272 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-10-31 09:28:02.701587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
W1031 09:28:02.727308 140163584431872 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled
自动由 placer 处理。
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:28:02.709091: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.709152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:28:02.709164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:28:02.709379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
W1031 09:28:02.732473 140586624427776 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.710931 140554276726528 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.735715 140554276726528 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
2019-10-31 09:28:02.744438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.744511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3
2019-10-31 09:28:02.744524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N
2019-10-31 09:28:02.744780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0)
W1031 09:28:02.754183 140163584431872 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.753435 139908811421440 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.745885 139841235764992 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.742853 139665852200704 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.737485 139662100997888 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.743636 139982112827136 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.756311 139818715670272 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.757796 140586624427776 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
2019-10-31 09:28:02.758983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.759041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:28:02.759055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:28:02.759335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
W1031 09:28:02.759724 139662100997888 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
2019-10-31 09:28:02.745103: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.745149: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-10-31 09:28:02.745162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-10-31 09:28:02.746129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0)
W1031 09:28:02.753757 140131597530880 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.766823 139665852200704 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.767652 139982112827136 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from
tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
2019-10-31 09:28:02.767762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.767823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2
2019-10-31 09:28:02.767837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N
2019-10-31 09:28:02.768104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0)
2019-10-31 09:28:02.769409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-31 09:28:02.769470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1
2019-10-31 09:28:02.769483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N
2019-10-31 09:28:02.769816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0)
W1031 09:28:02.770942 139841235764992 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.756439 139799656847104 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.761564 139856166496000 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.767498 140718879745792 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.777062 139853571868416 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.776549 140131597530880 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
TensorFlow: 1.13
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 512 global
32 per device
Num batches: 100
Num epochs: 0.04
Devices: ['horovod/gpu:0', 'horovod/gpu:1', 'horovod/gpu:2', 'horovod/gpu:3', 'horovod/gpu:4', 'horovod/gpu:5', 'horovod/gpu:6', 'horovod/gpu:7', 'horovod/gpu:8', 'horovod/gpu:9', 'horovod/gpu:10', 'horovod/gpu:11', 'horovod/gpu:12', 'horovod/gpu:13', 'horovod/gpu:14', 'horovod/gpu:15']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: horovod
==========
Generating training model
W1031 09:28:02.779106 139928895440640 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.779887 139908811421440 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.782153 139799656847104 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.787169 140554276726528 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.791988 140017035462400 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W1031 09:28:02.792617 140718879745792 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.801528 139853571868416 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.802262 139928895440640 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.805729 139662100997888 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.806548 140163584431872 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.807551 140586624427776 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.808814 139818715670272 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.808916 139856166496000 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.815933 139982112827136 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.817563 139665852200704 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.818260 140017035462400 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W1031 09:28:02.821896 139841235764992 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.826172 140131597530880 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.828052 139908811421440 deprecation.py:323] From
/examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.835323 139799656847104 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.841519 140718879745792 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.849115 139928895440640 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.852732 139853571868416 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:02.871250 140017035462400 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W1031 09:28:05.200214 140554276726528 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.218427 139662100997888 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.223022 140586624427776 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.231928 140131597530880 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.257417 139908811421440 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.267662 139982112827136 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.278401 140163584431872 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.293057 140718879745792 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.301071 139853571868416 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.304152 139665852200704 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.308290 139841235764992 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.311634 139856166496000 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.324692 139799656847104 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.337773 139928895440640 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.370465 140554276726528 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.386629 139662100997888 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.389393 140586624427776 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.395435 140131597530880 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.402461 140017035462400 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.422204 139908811421440 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.434711 139982112827136 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.452119 140163584431872 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.464861 139853571868416 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.472254 140718879745792 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.472842 139665852200704 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.478122 139856166496000 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.478706 139841235764992 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.488337 139818715670272 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.490337 139799656847104 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.516188 139928895440640 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.572691 140017035462400 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W1031 09:28:05.681944 139818715670272 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
Initializing graph
W1031 09:28:07.374773 140554276726528 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
W1031 09:28:07.388925 140131597530880 deprecation.py:323] From
/examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.394137 139662100997888 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.403966 139982112827136 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.430644 139853571868416 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.430839 139908811421440 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.458472 139799656847104 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.483218 139856166496000 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession Initializing graph W1031 09:28:07.488722 139841235764992 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.499745 140163584431872 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.507745 139665852200704 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.528688 140718879745792 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.568386 139928895440640 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.604121 140017035462400 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.769806 140586624427776 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession W1031 09:28:07.904350 139818715670272 deprecation.py:323] From /examples/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession 2019-10-31 09:28:08.072869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:28:08.072974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.072988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:28:08.072998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:28:08.073250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) 2019-10-31 09:28:08.081420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-10-31 09:28:08.081544: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.081559: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-10-31 09:28:08.081569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-10-31 09:28:08.081911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0) 2019-10-31 09:28:08.086161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:28:08.086268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.086283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:28:08.086292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:28:08.086530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) 2019-10-31 09:28:08.112899: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-10-31 09:28:08.113008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.113023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-10-31 09:28:08.113033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-10-31 09:28:08.113283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0) 2019-10-31 09:28:08.123983: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:28:08.124095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.124112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:28:08.124123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:28:08.124359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) 2019-10-31 09:28:08.165104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:28:08.165240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.165260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:28:08.165270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:28:08.165595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:28:08.194105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:28:08.194211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.194226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:28:08.194236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N 2019-10-31 09:28:08.194487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) 2019-10-31 09:28:08.198259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:28:08.198364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.198379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:28:08.198389: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:28:08.198660: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:28:08.208208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:28:08.208331: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.208346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:28:08.208356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:28:08.208692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) 2019-10-31 09:28:08.226755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:28:08.226854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.226869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:28:08.226893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1:
N 2019-10-31 09:28:08.227156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) 2019-10-31 09:28:08.228849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:28:08.228959: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.228974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:28:08.228984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N 2019-10-31 09:28:08.229243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) 2019-10-31 09:28:08.270501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-10-31 09:28:08.270605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.270621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-10-31 09:28:08.270630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-10-31 09:28:08.270879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0) 2019-10-31 09:28:08.318952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 1 2019-10-31 09:28:08.319104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.319121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 1 2019-10-31 09:28:08.319132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: N 2019-10-31 09:28:08.319487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 0000:06:00.0, compute capability: 6.0) 2019-10-31 09:28:08.328654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 2 2019-10-31 09:28:08.328751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.328766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 2 2019-10-31 09:28:08.328777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: N 2019-10-31 09:28:08.329034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 2, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0c:00.0, compute capability: 6.0) 2019-10-31 09:28:08.365747: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0 2019-10-31 09:28:08.365852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.365867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 2019-10-31 09:28:08.365876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N 2019-10-31 09:28:08.366110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:04:00.0, compute capability: 6.0) 2019-10-31 09:28:08.678376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 3 2019-10-31 09:28:08.678506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-10-31 09:28:08.678522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 3 2019-10-31 09:28:08.678532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: N 2019-10-31 09:28:08.679667: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14482 MB memory) -> physical GPU (device: 3, name: Tesla P100-PCIE-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0) I1031 09:28:11.440287 139982112827136 session_manager.py:491] Running local_init_op. I1031 09:28:11.446637 140554276726528 session_manager.py:491] Running local_init_op. I1031 09:28:11.481120 139853571868416 session_manager.py:491] Running local_init_op. I1031 09:28:11.490534 139662100997888 session_manager.py:491] Running local_init_op. I1031 09:28:11.541015 139908811421440 session_manager.py:491] Running local_init_op. I1031 09:28:11.559617 140131597530880 session_manager.py:491] Running local_init_op. I1031 09:28:11.591317 139799656847104 session_manager.py:491] Running local_init_op. I1031 09:28:11.606469 139841235764992 session_manager.py:491] Running local_init_op. I1031 09:28:11.627890 139982112827136 session_manager.py:493] Done running local_init_op. I1031 09:28:11.633262 140554276726528 session_manager.py:493] Done running local_init_op. I1031 09:28:11.646192 139665852200704 session_manager.py:491] Running local_init_op. I1031 09:28:11.661424 140718879745792 session_manager.py:491] Running local_init_op. I1031 09:28:11.665061 139853571868416 session_manager.py:493] Done running local_init_op. I1031 09:28:11.668709 139856166496000 session_manager.py:491] Running local_init_op. I1031 09:28:11.677753 139662100997888 session_manager.py:493] Done running local_init_op. I1031 09:28:11.721434 140163584431872 session_manager.py:491] Running local_init_op. I1031 09:28:11.736203 140017035462400 session_manager.py:491] Running local_init_op. I1031 09:28:11.739211 139908811421440 session_manager.py:493] Done running local_init_op. I1031 09:28:11.763015 140131597530880 session_manager.py:493] Done running local_init_op. I1031 09:28:11.778822 140586624427776 session_manager.py:491] Running local_init_op. I1031 09:28:11.791004 139799656847104 session_manager.py:493] Done running local_init_op. I1031 09:28:11.792444 139841235764992 session_manager.py:493] Done running local_init_op. I1031 09:28:11.840747 139665852200704 session_manager.py:493] Done running local_init_op. I1031 09:28:11.841266 139928895440640 session_manager.py:491] Running local_init_op. I1031 09:28:11.859958 140718879745792 session_manager.py:493] Done running local_init_op. I1031 09:28:11.861997 139856166496000 session_manager.py:493] Done running local_init_op. I1031 09:28:11.902634 140163584431872 session_manager.py:493] Done running local_init_op. I1031 09:28:11.913870 140017035462400 session_manager.py:493] Done running local_init_op. I1031 09:28:11.981170 140586624427776 session_manager.py:493] Done running local_init_op. I1031 09:28:12.039165 139928895440640 session_manager.py:493] Done running local_init_op. I1031 09:28:12.076092 139818715670272 session_manager.py:491] Running local_init_op. I1031 09:28:12.256900 139818715670272 session_manager.py:493] Done running local_init_op. Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up Running warm up 2019-10-31 09:28:40.740985: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:40.810479: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:40.975794: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.009451: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.137098: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.326322: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.340050: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.355355: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.369061: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.440759: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.484446: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.532443: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.560915: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.690003: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.727085: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally 2019-10-31 09:28:41.974115: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.26<0> tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.29<0> tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.29<0> tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.29<0> tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.5.29<0> tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.17<0> tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0. tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.17<0> tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so). tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.17<0>
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.4.17<0>
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.26<0>
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.26<0>
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.6.26<0>
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.26<0>
NCCL version 2.4.2+cuda10.0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NET/Socket : Using [0]eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 0.
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.26<0>
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.26<0>
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.6.26<0>
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.7.18<0>
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.29<0>
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.29<0>
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.29<0>
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.5.29<0>
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.17<0>
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.17<0>
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.17<0>
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/IB [1]mlx5_0:1/IB ; OOB eth0:10.254.4.17<0>
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO comm 0x7fdbf03f2980 rank 0 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO comm 0x7fd4683eab00 rank 2 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO comm 0x7f57503a6770 rank 9 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO comm 0x7f29243fcf30 rank 3 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO comm 0x7f79703ef4d0 rank 1 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO comm 0x7f04ac3dc080 rank 12 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO comm 0x7f42cc3d4910 rank 10 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO comm 0x7f31404045c0 rank 8 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO comm 0x7f3e203c5400 rank 11 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO comm 0x7f4f303a9880 rank 14 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO comm 0x7f058c3a3740 rank 13 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO comm 0x7f2e643dd590 rank 15 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO Setting affinity for GPU 1 to 0fff
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO comm 0x7f31dc403440 rank 5 nranks 16 cudaDev 1 nvmlDev 1
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Setting affinity for GPU 0 to 0fff
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO comm 0x7ffab84792e0 rank 4 nranks 16 cudaDev 0 nvmlDev 0
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO Setting affinity for GPU 3 to 0fff
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Setting affinity for GPU 2 to 0fff
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO comm 0x7f71fc4035a0 rank 6 nranks 16 cudaDev 2 nvmlDev 2
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO comm 0x7f24b43e5c10 rank 7 nranks 16 cudaDev 3 nvmlDev 3
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO CUDA Dev 2[2], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO CUDA Dev 3[3], IB NIC distance : PIX PHB
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance : PHB PIX
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Channel 00 : 0 1 3 6 4 5 7 10 8 9 11 14 12 13 15 2
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Channel 01 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Ring 00 : 11 -> 14 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Ring 00 : 15 -> 2 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 7 -> 10 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 3 -> 6 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO Ring 00 : 1[1] -> 3[3] via P2P/IPC
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Ring 00 : 8[0] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Ring 00 : 9[1] -> 11[3] via P2P/IPC
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Ring 00 : 13[1] -> 15[3] via P2P/IPC
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 00 : 12[0] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Ring 00 : 2[2] -> 0[0] via P2P/IPC
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO Ring 00 : 5[1] -> 7[3] via P2P/IPC
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Ring 00 : 4[0] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Ring 00 : 14[2] -> 12[0] via P2P/IPC
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 10[2] -> 8[0] via P2P/IPC
tensorflow-benchmarks-worker-1:60:259 [2]
NCCL INFO Ring 00 : 6[2] -> 4[0] via P2P/IPC
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO Ring 00 : 11 -> 14 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO Ring 00 : 15 -> 2 [send] via NET/IB/0
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Ring 00 : 3 -> 6 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO Ring 00 : 7 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 270 mtu 5 LID 9
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 270 mtu 5 LID 12
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 270 mtu 5 LID 16
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 276 mtu 5 LID 14
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO Ring 00 : 15[3] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO Ring 00 : 11[3] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Ring 00 : 3[3] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Ring 00 : 8[0] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Ring 00 : 13[1] -> 12[0] via P2P/IPC
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO Ring 00 : 1[1] -> 0[0] via P2P/IPC
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Ring 00 : 0[0] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO Ring 00 : 7[3] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Ring 00 : 9[1] -> 8[0] via P2P/IPC
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 00 : 12[0] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO Ring 00 : 5[1] -> 4[0] via P2P/IPC
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Ring 00 : 4[0] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 6 -> 10 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Ring 00 : 10 -> 2 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NCCL_NET_GDR_LEVEL set by environment to 1.
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Ring 00 : 14 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Ring 01 : 13[1] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO Ring 01 : 1[1] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO Ring 01 : 15 -> 0 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 11 -> 12 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Ring 01 : 15 -> 0 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Ring 01 : 3 -> 4 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 273 mtu 5 LID 12
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Ring 01 : 0[0] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 12[0] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Ring 01 : 7 -> 8 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 6 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO Ring 01 : 5[1] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO Ring 01 : 11 -> 12 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO Ring 01 : 7 -> 8 [send] via NET/IB/0
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 279 mtu 5 LID 14
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Ring 01 : 9[1] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Ring 01 : 3 -> 4 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 273 mtu 5 LID 9
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 280 mtu 5 LID 14
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Ring 01 : 8[0] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 14 -> 10 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO NET/IB: Dev 0 Port 1 qpn 273 mtu 5 LID 16
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 274 mtu 5 LID 12
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Ring 01 : 4[0] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 00 : 10 -> 6 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 10 -> 2 [send] via NET/IB/0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 274 mtu 5 LID 9
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Ring 01 : 0 -> 12 [send] via NET/IB/1
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Ring 00 : 10 -> 14 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Ring 01 : 12 -> 4 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 8 -> 12 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Ring 00 : 2 -> 10 [send] via NET/IB/0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 0 -> 12 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Ring 01 : 8 -> 12 [send] via NET/IB/1
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 283 mtu 5 LID 10
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 272 mtu 5 LID 8
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 12 -> 4 [send] via NET/IB/1
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 270 mtu 5 LID 11
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 2[2] / HCA 0 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 2 -> 10 [receive] via NET/IB/0/GDRDMA
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 276 mtu 5 LID 16
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 10 -> 6 [send] via NET/IB/0
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Ring 01 : 2[2] -> 3[3] via P2P/IPC
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 00 : 10 -> 14 [send] via NET/IB/0
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 279 mtu 5 LID 9
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Ring 01 : 3[3] -> 2[2] via P2P/IPC
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO NET/IB: Dev 0 Port 1 qpn 280 mtu 5 LID 9
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO Ring 01 : 1[1] -> 0[0] via P2P/IPC
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO Trees [0] 1->3->-1/-1/-1 [1] 2->3->-1/-1/-1
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Ring 01 : 2[2] -> 1[1] via P2P/IPC
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO Trees [0] 0->1->3/-1/-1 [1] 0->1->2/-1/-1
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Ring 01 : 14[2] -> 15[3] via P2P/IPC
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 01 : 6[2] -> 7[3] via P2P/IPC
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 01 : 10[2] -> 11[3] via P2P/IPC
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO Ring 01 : 15[3] -> 14[2] via P2P/IPC
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO Ring 01 : 11[3] -> 10[2] via P2P/IPC
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Ring 01 : 13[1] -> 12[0] via P2P/IPC
tensorflow-benchmarks-worker-0:61:258 [3] NCCL INFO comm 0x7f29243fcf30 rank 3 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO Ring 01 : 7[3] -> 6[2] via P2P/IPC
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO Trees [0] -1->2->0/10/-1 [1] 1->2->3/-1/-1
tensorflow-benchmarks-worker-0:60:260 [2] NCCL INFO comm 0x7fd4683eab00 rank 2 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-0:58:259 [1] NCCL INFO comm 0x7f79703ef4d0 rank 1 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Ring 01 : 12 -> 0 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Ring 01 : 9[1] -> 8[0] via P2P/IPC
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO Ring 01 : 5[1] -> 4[0] via P2P/IPC
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO Trees [0] 13->15->-1/-1/-1 [1] 14->15->-1/-1/-1
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Ring 01 : 14[2] -> 13[1] via P2P/IPC
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO Trees [0] 9->11->-1/-1/-1 [1] 10->11->-1/-1/-1
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Ring 01 : 10[2] -> 9[1] via P2P/IPC
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO Trees [0] 8->9->11/-1/-1 [1] 8->9->10/-1/-1
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO Trees [0] 5->7->-1/-1/-1 [1] 6->7->-1/-1/-1
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO Trees [0] 12->13->15/-1/-1 [1] 12->13->14/-1/-1
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO Trees [0] 10->14->12/-1/-1 [1] 13->14->15/-1/-1
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Ring 01 : 6[2] -> 5[1] via P2P/IPC
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO Trees [0] 4->5->7/-1/-1 [1] 4->5->6/-1/-1
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO Trees [0] 2->10->8/6/14 [1] 9->10->11/-1/-1
tensorflow-benchmarks-worker-2:60:259 [2] NCCL INFO comm 0x7f42cc3d4910 rank 10 nranks 16 cudaDev 2
nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-2:61:261 [3] NCCL INFO comm 0x7f3e203c5400 rank 11 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Ring 01 : 12 -> 8 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-2:58:258 [1] NCCL INFO comm 0x7f57503a6770 rank 9 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-3:61:260 [3] NCCL INFO comm 0x7f2e643dd590 rank 15 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB : GPU Direct RDMA Enabled for GPU 0[0] / HCA 1 (distance 0 < 1), read 0
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 4 -> 12 [receive] via NET/IB/1/GDRDMA
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 12 -> 8 [send] via NET/IB/1
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Ring 01 : 12 -> 0 [send] via NET/IB/1
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 275 mtu 5 LID 11
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 276 mtu 5 LID 11
tensorflow-benchmarks-worker-3:58:259 [1] NCCL INFO comm 0x7f058c3a3740 rank 13 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-3:60:261 [2] NCCL INFO comm 0x7f4f303a9880 rank 14 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-1:61:258 [3] NCCL INFO comm 0x7f24b43e5c10 rank 7 nranks 16 cudaDev 3 nvmlDev 3 - Init COMPLETE
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO Trees [0] 10->6->4/-1/-1 [1] 5->6->7/-1/-1
tensorflow-benchmarks-worker-1:60:259 [2] NCCL INFO comm 0x7f71fc4035a0 rank 6 nranks 16 cudaDev 2 nvmlDev 2 - Init COMPLETE
tensorflow-benchmarks-worker-1:58:260 [1] NCCL INFO comm 0x7f31dc403440 rank 5 nranks 16 cudaDev 1 nvmlDev 1 - Init COMPLETE
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Ring 01 : 4 -> 12 [send] via NET/IB/1
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO NET/IB: Dev 1 Port 1 qpn 277 mtu 5 LID 13
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO Trees [0] 6->4->5/-1/-1 [1] -1->4->5/12/-1
tensorflow-benchmarks-worker-1:57:261 [0] NCCL INFO comm 0x7ffab84792e0 rank 4 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO Trees [0] 14->12->13/-1/-1 [1] 4->12->13/8/0
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO Trees [0] 10->8->9/-1/-1 [1] 12->8->9/-1/-1
tensorflow-benchmarks-worker-3:57:258 [0] NCCL INFO comm 0x7f04ac3dc080 rank 12 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE
tensorflow-benchmarks-worker-2:57:260 [0] NCCL INFO comm 0x7f31404045c0 rank 8 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Trees [0] 2->0->1/-1/-1 [1] 12->0->1/-1/-1
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Using 256 threads, Min Comp Cap 6, Trees enabled up to size 479999
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO comm 0x7fdbf03f2980 rank 0 nranks 16 cudaDev 0 nvmlDev 0 - Init COMPLETE
tensorflow-benchmarks-worker-0:57:261 [0] NCCL INFO Launch mode Parallel
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
Done warm up
Step Img/sec total_loss
1 images/sec: 198.5 +/- 0.0 (jitter = 0.0) 7.672
1 images/sec: 198.9 +/- 0.0 (jitter = 0.0) 7.889
1 images/sec: 198.4 +/- 0.0 (jitter = 0.0) 7.782
1 images/sec: 198.5 +/- 0.0 (jitter = 0.0) 7.790
1 images/sec: 196.8 +/- 0.0 (jitter = 0.0) 7.873
1 images/sec: 197.4 +/- 0.0 (jitter = 0.0) 7.645
1 images/sec: 197.3 +/- 0.0 (jitter = 0.0) 8.005
1 images/sec: 197.5 +/- 0.0 (jitter = 0.0) 7.615
1 images/sec: 196.8 +/- 0.0 (jitter = 0.0) 7.785
1 images/sec: 196.7 +/- 0.0 (jitter = 0.0) 7.579
1 images/sec: 196.9 +/- 0.0 (jitter = 0.0) 7.843
1 images/sec: 197.9 +/- 0.0 (jitter = 0.0) 7.785
1 images/sec: 196.8 +/- 0.0 (jitter = 0.0) 8.085
1 images/sec: 198.0 +/- 0.0 (jitter = 0.0) 8.152
1 images/sec: 198.1 +/- 0.0 (jitter = 0.0) 8.017
1 images/sec: 198.5 +/- 0.0 (jitter = 0.0) 7.888
10 images/sec: 197.9 +/- 0.6 (jitter = 1.1) 7.675
10 images/sec: 198.1 +/- 0.5 (jitter = 0.6) 7.622
10 images/sec: 198.1 +/- 0.5 (jitter = 0.9) 7.662
10 images/sec: 197.9 +/- 0.6 (jitter = 1.3) 7.735
10 images/sec: 198.1 +/- 0.4 (jitter = 0.6) 7.623
10 images/sec: 198.1 +/- 0.4 (jitter = 0.7) 7.702
10 images/sec: 198.1 +/- 0.4 (jitter = 0.7) 7.734
10 images/sec: 198.1 +/- 0.4 (jitter = 0.8) 8.034
10 images/sec: 198.1 +/- 0.5 (jitter = 1.0) 8.060
10 images/sec: 197.9 +/- 0.6 (jitter = 1.3) 7.607
10 images/sec: 198.0 +/- 0.4 (jitter = 0.9) 7.706
10 images/sec: 197.9 +/- 0.5 (jitter = 1.4) 7.846
10 images/sec: 197.9 +/- 0.4 (jitter = 0.9) 7.738
10 images/sec: 197.9 +/- 0.4 (jitter = 0.9) 7.844
10 images/sec: 198.1 +/- 0.4 (jitter = 0.8) 7.562
10 images/sec: 197.6 +/- 0.7 (jitter = 1.6) 7.721
20 images/sec: 197.9 +/- 0.3 (jitter = 1.1) 7.660
20 images/sec: 197.9 +/- 0.3 (jitter = 0.8) 7.641
20 images/sec: 198.0 +/- 0.3 (jitter = 0.6) 7.684
20 images/sec: 197.9 +/- 0.3 (jitter = 1.4) 7.777
20 images/sec: 197.8 +/- 0.4 (jitter = 1.5) 7.606
20 images/sec: 197.9 +/- 0.3 (jitter = 1.4) 7.548
20 images/sec: 197.9 +/- 0.4 (jitter = 1.5) 7.615
20 images/sec: 197.9 +/- 0.4 (jitter = 1.4) 7.811
20 images/sec: 197.9 +/- 0.4 (jitter = 1.1) 7.711
20 images/sec: 197.9 +/- 0.4 (jitter = 0.9) 7.582
20 images/sec: 197.9 +/- 0.3 (jitter = 0.7) 7.611
20 images/sec: 197.8 +/- 0.2 (jitter = 1.1) 7.465
20 images/sec: 197.8 +/- 0.3 (jitter = 1.3) 7.557
20 images/sec: 197.8 +/- 0.4 (jitter = 1.1) 7.632
20 images/sec: 197.8 +/- 0.3 (jitter = 1.5) 7.757
20 images/sec: 197.8 +/- 0.3 (jitter = 1.0) 7.591
30 images/sec: 197.8 +/- 0.3 (jitter = 1.0) 7.578
30 images/sec: 197.9 +/- 0.3 (jitter = 1.5) 7.533
30 images/sec: 197.8 +/- 0.3 (jitter = 1.4) 7.672
30 images/sec: 197.8 +/- 0.3 (jitter = 1.3) 7.692
30 images/sec: 197.9 +/- 0.3 (jitter = 1.4) 7.661
30 images/sec: 197.8 +/- 0.3 (jitter = 1.1) 7.545
30 images/sec: 197.8 +/- 0.3 (jitter = 1.0) 7.874
30 images/sec: 197.8 +/- 0.2 (jitter = 1.4) 7.641
30 images/sec: 197.8 +/- 0.2 (jitter = 1.1) 7.705
30 images/sec: 197.8 +/- 0.3 (jitter = 1.1) 7.622
30 images/sec: 197.9 +/- 0.2 (jitter = 0.8) 7.624
30 images/sec: 197.8 +/- 0.3 (jitter = 0.9) 7.643
30 images/sec: 197.8 +/- 0.2 (jitter = 1.1) 7.876
30 images/sec: 197.8 +/- 0.3 (jitter = 1.0) 7.605
30 images/sec: 197.8 +/- 0.3 (jitter = 1.1) 7.880
30 images/sec: 197.8 +/- 0.3 (jitter = 1.0) 7.626
40 images/sec: 197.4 +/- 0.3 (jitter = 1.4) 7.569
40 images/sec: 197.4 +/- 0.3 (jitter = 1.9) 7.676
40 images/sec: 197.3 +/- 0.3 (jitter = 1.5) 7.450
40 images/sec: 197.4 +/- 0.4 (jitter = 1.5) 7.642
40 images/sec: 197.4 +/- 0.3 (jitter = 1.4) 7.400
40 images/sec: 197.4 +/- 0.4 (jitter = 1.7) 7.366
40 images/sec: 197.4 +/- 0.4 (jitter = 1.3) 7.579
40 images/sec: 197.3 +/- 0.3 (jitter = 1.7) 7.630
40 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.679
40 images/sec: 197.4 +/- 0.3 (jitter = 1.6) 7.562
40 images/sec: 197.3 +/- 0.3 (jitter = 1.6) 7.412
40 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.684
40 images/sec: 197.4 +/- 0.3 (jitter = 1.2) 7.548
40 images/sec: 197.4 +/- 0.3 (jitter = 1.1) 7.539
40 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.652
40 images/sec: 197.4 +/- 0.3 (jitter = 1.3) 7.646
50 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.481
50 images/sec: 197.3 +/- 0.3 (jitter = 1.3) 7.574
50 images/sec: 197.4 +/- 0.3 (jitter = 1.3) 7.538
50 images/sec: 197.4 +/- 0.3 (jitter = 1.2) 7.579
50 images/sec: 197.4 +/- 0.3 (jitter = 1.3) 7.576
50 images/sec: 197.4 +/- 0.3 (jitter = 1.7) 7.537
50 images/sec: 197.4 +/- 0.3 (jitter = 1.5) 7.528
50 images/sec: 197.4 +/- 0.3 (jitter = 1.2) 7.516
50 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.566
50 images/sec: 197.3 +/- 0.3 (jitter = 1.3) 7.477
50 images/sec: 197.4 +/- 0.3 (jitter = 1.1) 7.448
50 images/sec: 197.4 +/- 0.3 (jitter = 1.2) 7.606
50 images/sec: 197.4 +/- 0.3 (jitter = 1.6) 7.614
50 images/sec: 197.3 +/- 0.3 (jitter = 1.6) 7.643
50 images/sec: 197.3 +/- 0.3 (jitter = 1.6) 7.450
50 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.605
60 images/sec: 197.5 +/- 0.3 (jitter = 1.2) 7.633
60 images/sec: 197.4 +/- 0.3 (jitter = 1.2) 7.443
60 images/sec: 197.5 +/- 0.3 (jitter = 1.2) 7.491
60 images/sec: 197.5 +/- 0.3 (jitter = 1.1) 7.517
60 images/sec: 197.5 +/- 0.3 (jitter = 1.3) 7.413
60 images/sec: 197.5 +/- 0.3 (jitter = 1.3) 7.464
60 images/sec: 197.5 +/- 0.2 (jitter = 1.3) 7.496
60 images/sec: 197.5 +/- 0.2 (jitter = 1.0) 7.521
60 images/sec: 197.5 +/- 0.3 (jitter = 1.2) 7.455
60 images/sec: 197.5 +/- 0.3 (jitter = 1.5) 7.493
60 images/sec: 197.5 +/- 0.3 (jitter = 1.2) 7.586
60 images/sec: 197.4 +/- 0.3 (jitter = 1.5) 7.541
60 images/sec: 197.5 +/- 0.3 (jitter = 1.5) 7.501
60 images/sec: 197.5 +/- 0.3 (jitter = 1.2) 7.556
60 images/sec: 197.5 +/- 0.2 (jitter = 1.1) 7.412
60 images/sec: 197.5 +/- 0.3 (jitter = 1.4) 7.350
70 images/sec: 197.3 +/- 0.3 (jitter = 1.5) 7.535
70 images/sec: 197.3 +/- 0.3 (jitter = 1.6) 7.489
70 images/sec: 197.3 +/- 0.3 (jitter = 1.3) 7.553
70 images/sec: 197.3 +/- 0.3 (jitter = 1.2) 7.506
70 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.520
70 images/sec: 197.3 +/- 0.3 (jitter = 1.5) 7.498
70 images/sec: 197.3 +/- 0.2 (jitter = 1.2) 7.594
70 images/sec: 197.3 +/- 0.2 (jitter = 1.4) 7.428
70 images/sec: 197.3 +/- 0.3 (jitter = 1.3) 7.550
70 images/sec: 197.3 +/- 0.3 (jitter = 1.3) 7.542
70 images/sec: 197.3 +/- 0.2 (jitter = 1.2) 7.498
70 images/sec: 197.3 +/- 0.3 (jitter = 1.4) 7.477
70 images/sec: 197.3 +/- 0.3 (jitter = 1.3) 7.528
70 images/sec: 197.3 +/- 0.3 (jitter = 1.5) 7.506
70 images/sec: 197.3 +/- 0.3 (jitter = 1.5) 7.552
70 images/sec: 197.3 +/- 0.3 (jitter = 1.5) 7.500
80 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.454
80 images/sec: 197.3 +/- 0.2 (jitter = 1.4) 7.504
80 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.388
80 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.528
80 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.426
80 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.422
80 images/sec: 197.4 +/- 0.2 (jitter = 1.7) 7.467
80 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.447
80 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.459
80 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.460
80 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.471
80 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.472
80 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.513
80 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.517
80 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.409
80 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.527
90 images/sec: 197.5 +/- 0.2 (jitter = 1.2) 7.434
90 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.472
90 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.498
90 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.432
90 images/sec: 197.5 +/- 0.2 (jitter = 1.4) 7.462
90 images/sec: 197.5 +/- 0.2 (jitter = 1.3) 7.451
90 images/sec: 197.5 +/- 0.2 (jitter = 1.4) 7.456
90 images/sec: 197.5 +/- 0.2 (jitter = 1.6) 7.374
90 images/sec: 197.4 +/- 0.2 (jitter = 1.6) 7.439
90 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.499
90 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.519
90 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.467
90 images/sec: 197.4 +/- 0.2 (jitter = 1.1) 7.366
90 images/sec: 197.5 +/- 0.2 (jitter = 1.2) 7.441
90 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.527
90 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.459
100 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.483
----------------------------------------------------------------
total images/sec: 3156.37
----------------------------------------------------------------
100 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.469
total images/sec: 3156.67
100 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.441
total images/sec: 3156.58
100 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.486
total images/sec: 3156.68
100 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.421
total images/sec: 3156.25
100 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.500
total images/sec: 3156.61
100 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.592
total images/sec: 3156.35
100 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.418
total images/sec: 3156.62
100 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.563
total images/sec: 3156.42
100 images/sec: 197.4 +/- 0.2 (jitter = 1.2) 7.506
total images/sec: 3156.61
100 images/sec: 197.4 +/- 0.2 (jitter = 1.5) 7.466
total images/sec: 3156.46
100 images/sec: 197.4 +/- 0.2 (jitter = 1.6) 7.532
total images/sec: 3156.47
100 images/sec: 197.4 +/- 0.2 (jitter = 1.4) 7.452
total images/sec: 3156.24
100 images/sec: 197.4 +/- 0.2 (jitter = 1.6) 7.613
total images/sec: 3156.26
100 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.470
total images/sec: 3156.22
100 images/sec: 197.4 +/- 0.2 (jitter = 1.3) 7.569
total images/sec: 3156.20
附录
- 定制化 NVIDIA OFED 镜像（适用于 kernel-3.10.0-1062.1.2.el7）
- SELinux InfiniBand 补丁：infiniband.zip
- 其他 OCP 组件：Openshift-rdma.zip
相关文档
- TensorFlow Docker MultiGPU Training
- 16.11 Containers
- 16.12 Containers
- 17.01 Containers
- 17.02 Containers
- 17.03 Containers
- 17.04 Containers
- TensorFlow build using Bazel on TX2
- Tensorflow build using CMake
- 17.05 Containers
- 17.06 Containers
- Native TensorFlow build using Bazel on PX2
- 17.07 Containers
- 17.08 Containers
- 17.09 Containers