In this episode, I’m playing with Single Root I/O Virtualization (SR-IOV) in Proxmox Virtual Environment (PVE). I’ve heard ruomors that it will be anything from a minor to major improvement in IO performance for my VMs, so I wanted to do some testing on my own system to be sure.

Please don’t take my results as final, I’m not comfortable saying that I’ve removed all of the background tasks and load from the measurements. My system is also relatively underpowered, so this may show as a thread-bound performance limit on some of these drivers where your system may not have this issue.

Contents⌗

Video⌗

Thumbnail

Enable SR-IOV⌗

Prerequisites:

  • Your card supports SR-IOV
  • You have enabled SR-IOV in the card firmware if required (See MSTFLINT below for MLX5)
  • You have enabled IOMMU and PCIe ACS in your BIOS/UEFI firmware
  • You have enabled IOMMU in the kernel (intel_iommu=on or amd_iommu=on) and functioning correctly

Once all of that is done, to enable SR-IOV, you just need to set a particular sysctl which corresponds to your network device. The sysctl is the same for all drivers (Intel/Mellanox). This is NOT persistent across reboots. I didn’t bother scripting a way to automate this for my tests, and since I’m not going to continue to use SR-IOV, I also didn’t write it here. Anyway:

echo 32 > /sys/class/net/enpXXsYfZ/device/sriov_num_vfs
#Obviously change the enpXX with the full name of your own device

#f0 is the physical function of the card and should already exist
#f1+ are the virtual functions

#At this point it will possibly take awhile and if you are unlucky
#and have system IOMMU / ACS issues it could hang here for 60+ seconds

LXC⌗

Here’s the LXC configuration I used. Proxmox renames the link on starting/stopping an LXC container, and sometimes it doesn’t rename back correctly. Be warned.

lxc.apparmor.profile: unconfined
lxc.net.2.type: phys
lxc.net.2.link: enp1s0f0v3
lxc.net.2.flags: up
lxc.net.2.ipv6.address: fc69::4/64

Mellanox MSTFLINT⌗

If you are using a Mellanox (now Nvidia) card, you may need to configure the firmware to enable SR-IOV and also set the number of VFs. This is persistent to the card, so if you bought your card used it may already be configured this way. In any case, here’s the commands:

#apt update if you haven't recently
apt install mstflint

#Find the PCI ID of the card
lspci
#My card was 01:00.0 and 01:00.1

#Query all of the info, if you are qurious what the card is configured as
mstconfig -d 01:00.0

#Configure the VFs and enable SR_IOV
mstconfig -d 01:00.0 set NUM_OF_VFS=32 SRIOV_EN=true

#Reboot