Setup Infiniband in ESX 5.5

by Andrew Calcutt last modified 2015-01-22T18:42:14+00:00

Reference Documents (most of these commands came from here, but several have been fixed) (translated to English)


Remove Mellanox 1.9.7 driver from ESXi 5.5

esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core

Install Mellanox 1.61 drivers, OFED, and OpenSM, MFT Tools

Copy files from *Downloads* to /tmp
cd /tmp
--- Install 1.6.1 driver ---
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -d /tmp/ --no-sig-check
--- Install OFED ---
esxcli software vib install -d /tmp/ --no-sig-check
--- Install OpenSM ---
esxcli software vib install -v /tmp/ib-opensm-x64-3.3.15-6.x86_64.vib --no-sig-check
--- Optional - Install MFT Tools ---
esxcli software vib install -v /tmp/mlnx-mft-2.7.1-1.vib --no-sig-check
esxcli software vib install -d /tmp/ --no-sig-check
--- Reboot to apply newly installed packages

Set MTU and Configure OpenSM

--- Enable 4k MTU ---
esxcli system module parameters set -m=mlx4_core -p=mtu_4k=1
--- Set up opensm ---
cd /tmp
echo "Default=0x7fff,ipoib,mtu=5:ALL=full;" > partitions.conf
cp partitions.conf  /scratch/opensm/<adapter_1_hca>/
cp partitions.conf  /scratch/opensm/<adapter_2_hca>/

Optional - Load / Unload the MFT Tools mst driver


--- load the mst driver ---
vmkload_mod mst
--- unload the mst driver ---
vmkload_mod -u mst



Document Actions