Setup Infiniband in ESX 5.5
Reference Documents
http://www.bussink.ch/?p=1306 (most of these commands came from here, but several have been fixed)
http://www.hypervisor.fr/?p=4662 (translated to English)
https://vibsdepot.v-front.de/wiki/index.php/Ib-opensm
http://www.mellanox.com/page/products_dyn?product_family=36&menu_section=34
http://www.mellanox.com/page/management_tools
http://www.mellanox.com/pdf/MFT/MFT_VMware_readme_2_7_1.pdf
Downloads
ib-opensm-x64-3.3.15-6.x86_64.zip
mlnx-mft-2.7.1-1.zip
MLNX-MST-ESX-2.0.0.0.zip
MLNX-OFED-ESX-1.8.2.4-10EM-500.0.0.472560.zip
mlx4_en-mlnx-1.6.1.2-471530.zip
Remove Mellanox 1.9.7 driver from ESXi 5.5
esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
reboot
Install Mellanox 1.61 drivers, OFED, and OpenSM, MFT Tools
Copy files from *Downloads* to /tmp
cd /tmp
--- Install 1.6.1 driver ---
unzip mlx4_en-mlnx-1.6.1.2-471530.zip
esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -d /tmp/mlx4_en-mlnx-1.6.1.2-offline_bundle-471530.zip --no-sig-check
--- Install OFED ---
esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.8.2.4-10EM-500.0.0.472560.zip --no-sig-check
--- Install OpenSM ---
esxcli software vib install -v /tmp/ib-opensm-x64-3.3.15-6.x86_64.vib --no-sig-check
--- Optional - Install MFT Tools ---
esxcli software vib install -v /tmp/mlnx-mft-2.7.1-1.vib --no-sig-check
esxcli software vib install -d /tmp/MLNX-MST-ESX-2.0.0.0.zip --no-sig-check
--- Reboot to apply newly installed packages
reboot
Set MTU and Configure OpenSM
--- Enable 4k MTU --- esxcli system module parameters set -m=mlx4_core -p=mtu_4k=1
--- Set up opensm ---
cd /tmp
echo "Default=0x7fff,ipoib,mtu=5:ALL=full;" > partitions.conf
cp partitions.conf /scratch/opensm/<adapter_1_hca>/
cp partitions.conf /scratch/opensm/<adapter_2_hca>/