346 lines
12 KiB
Plaintext
346 lines
12 KiB
Plaintext
Paravirtualized RDMA Device (PVRDMA)
|
|
====================================
|
|
|
|
|
|
1. Description
|
|
===============
|
|
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
|
|
It works with its Linux Kernel driver AS IS, no need for any special guest
|
|
modifications.
|
|
|
|
While it complies with the VMware device, it can also communicate with bare
|
|
metal RDMA-enabled machines as peers.
|
|
|
|
It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).
|
|
|
|
It does not require the whole guest RAM to be pinned allowing memory
|
|
over-commit and, even if not implemented yet, migration support will be
|
|
possible with some HW assistance.
|
|
|
|
A project presentation accompany this document:
|
|
- https://blog.linuxplumbersconf.org/2017/ocw/system/presentations/4730/original/lpc-2017-pvrdma-marcel-apfelbaum-yuval-shaia.pdf
|
|
|
|
|
|
|
|
2. Setup
|
|
========
|
|
|
|
|
|
2.1 Guest setup
|
|
===============
|
|
Fedora 27+ kernels work out of the box, older distributions
|
|
require updating the kernel to 4.14 to include the pvrdma driver.
|
|
|
|
However the libpvrdma library needed by User Level Software is still
|
|
not available as part of the distributions, so the rdma-core library
|
|
needs to be compiled and optionally installed.
|
|
|
|
Please follow the instructions at:
|
|
https://github.com/linux-rdma/rdma-core.git
|
|
|
|
|
|
2.2 Host Setup
|
|
==============
|
|
The pvrdma backend is an ibdevice interface that can be exposed
|
|
either by a Soft-RoCE(rxe) device on machines with no RDMA device,
|
|
or an HCA SRIOV function(VF/PF).
|
|
Note that ibdevice interfaces can't be shared between pvrdma devices,
|
|
each one requiring a separate instance (rxe or SRIOV VF).
|
|
|
|
|
|
2.2.1 Soft-RoCE backend(rxe)
|
|
===========================
|
|
A stable version of rxe is required, Fedora 27+ or a Linux
|
|
Kernel 4.14+ is preferred.
|
|
|
|
The rdma_rxe module is part of the Linux Kernel but not loaded by default.
|
|
Install the User Level library (librxe) following the instructions from:
|
|
https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
|
|
|
|
Associate an ETH interface with rxe by running:
|
|
rxe_cfg add eth0
|
|
An rxe0 ibdevice interface will be created and can be used as pvrdma backend.
|
|
|
|
|
|
2.2.2 RDMA device Virtual Function backend
|
|
==========================================
|
|
Nothing special is required, the pvrdma device can work not only with
|
|
Ethernet Links, but also Infinibands Links.
|
|
All is needed is an ibdevice with an active port, for Mellanox cards
|
|
will be something like mlx5_6 which can be the backend.
|
|
|
|
|
|
2.2.3 QEMU setup
|
|
================
|
|
Configure QEMU with --enable-rdma flag, installing
|
|
the required RDMA libraries.
|
|
|
|
|
|
|
|
3. Usage
|
|
========
|
|
|
|
|
|
3.1 VM Memory settings
|
|
======================
|
|
Currently the device is working only with memory backed RAM
|
|
and it must be mark as "shared":
|
|
-m 1G \
|
|
-object memory-backend-ram,id=mb1,size=1G,share \
|
|
-numa node,memdev=mb1 \
|
|
|
|
|
|
3.2 MAD Multiplexer
|
|
===================
|
|
MAD Multiplexer is a service that exposes MAD-like interface for VMs in
|
|
order to overcome the limitation where only single entity can register with
|
|
MAD layer to send and receive RDMA-CM MAD packets.
|
|
|
|
To build rdmacm-mux run
|
|
# make rdmacm-mux
|
|
|
|
Before running the rdmacm-mux make sure that both ib_cm and rdma_cm kernel
|
|
modules aren't loaded, otherwise the rdmacm-mux service will fail to start.
|
|
|
|
The application accepts 3 command line arguments and exposes a UNIX socket
|
|
to pass control and data to it.
|
|
-d rdma-device-name Name of RDMA device to register with
|
|
-s unix-socket-path Path to unix socket to listen (default /var/run/rdmacm-mux)
|
|
-p rdma-device-port Port number of RDMA device to register with (default 1)
|
|
The final UNIX socket file name is a concatenation of the 3 arguments so
|
|
for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2
|
|
will be created.
|
|
|
|
pvrdma requires this service.
|
|
|
|
Please refer to contrib/rdmacm-mux for more details.
|
|
|
|
|
|
3.3 Service exposed by libvirt daemon
|
|
=====================================
|
|
The control over the RDMA device's GID table is done by updating the
|
|
device's Ethernet function addresses.
|
|
Usually the first GID entry is determined by the MAC address, the second by
|
|
the first IPv6 address and the third by the IPv4 address. Other entries can
|
|
be added by adding more IP addresses. The opposite is the same, i.e.
|
|
whenever an address is removed, the corresponding GID entry is removed.
|
|
The process is done by the network and RDMA stacks. Whenever an address is
|
|
added the ib_core driver is notified and calls the device driver add_gid
|
|
function which in turn update the device.
|
|
To support this in pvrdma device the device hooks into the create_bind and
|
|
destroy_bind HW commands triggered by pvrdma driver in guest.
|
|
|
|
Whenever changed is made to the pvrdma port's GID table a special QMP
|
|
messages is sent to be processed by libvirt to update the address of the
|
|
backend Ethernet device.
|
|
|
|
pvrdma requires that libvirt service will be up.
|
|
|
|
|
|
3.4 PCI devices settings
|
|
========================
|
|
RoCE device exposes two functions - an Ethernet and RDMA.
|
|
To support it, pvrdma device is composed of two PCI functions, an Ethernet
|
|
device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The
|
|
Ethernet function can be used for other Ethernet purposes such as IP.
|
|
|
|
|
|
3.5 Device parameters
|
|
=====================
|
|
- netdev: Specifies the Ethernet device function name on the host for
|
|
example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet
|
|
device used to create it.
|
|
- ibdev: The IB device name on host for example rxe0, mlx5_0 etc.
|
|
- mad-chardev: The name of the MAD multiplexer char device.
|
|
- ibport: In case of multi-port device (such as Mellanox's HCA) this
|
|
specify the port to use. If not set 1 will be used.
|
|
- dev-caps-max-mr-size: The maximum size of MR.
|
|
- dev-caps-max-qp: Maximum number of QPs.
|
|
- dev-caps-max-cq: Maximum number of CQs.
|
|
- dev-caps-max-mr: Maximum number of MRs.
|
|
- dev-caps-max-pd: Maximum number of PDs.
|
|
- dev-caps-max-ah: Maximum number of AHs.
|
|
|
|
Notes:
|
|
- The first 3 parameters are mandatory settings, the rest have their
|
|
defaults.
|
|
- The last 8 parameters (the ones that prefixed by dev-caps) defines the top
|
|
limits but the final values is adjusted by the backend device limitations.
|
|
- netdev can be extracted from ibdev's sysfs
|
|
(/sys/class/infiniband/<ibdev>/device/net/)
|
|
|
|
|
|
3.6 Example
|
|
===========
|
|
Define bridge device with vmxnet3 network backend:
|
|
<interface type='bridge'>
|
|
<mac address='56:b4:44:e9:62:dc'/>
|
|
<source bridge='bridge1'/>
|
|
<model type='vmxnet3'/>
|
|
<address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0' multifunction='on'/>
|
|
</interface>
|
|
|
|
Define pvrdma device:
|
|
<qemu:commandline>
|
|
<qemu:arg value='-object'/>
|
|
<qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/>
|
|
<qemu:arg value='-numa'/>
|
|
<qemu:arg value='node,memdev=mb1'/>
|
|
<qemu:arg value='-chardev'/>
|
|
<qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/>
|
|
<qemu:arg value='-device'/>
|
|
<qemu:arg value='pvrdma,addr=10.1,ibdev=rxe0,netdev=bridge0,mad-chardev=mads'/>
|
|
</qemu:commandline>
|
|
|
|
|
|
|
|
4. Implementation details
|
|
=========================
|
|
|
|
|
|
4.1 Overview
|
|
============
|
|
The device acts like a proxy between the Guest Driver and the host
|
|
ibdevice interface.
|
|
On configuration path:
|
|
- For every hardware resource request (PD/QP/CQ/...) the pvrdma will request
|
|
a resource from the backend interface, maintaining a 1-1 mapping
|
|
between the guest and host.
|
|
On data path:
|
|
- Every post_send/receive received from the guest will be converted into
|
|
a post_send/receive for the backend. The buffers data will not be touched
|
|
or copied resulting in near bare-metal performance for large enough buffers.
|
|
- Completions from the backend interface will result in completions for
|
|
the pvrdma device.
|
|
|
|
|
|
4.2 PCI BARs
|
|
============
|
|
PCI Bars:
|
|
BAR 0 - MSI-X
|
|
MSI-X vectors:
|
|
(0) Command - used when execution of a command is completed.
|
|
(1) Async - not in use.
|
|
(2) Completion - used when a completion event is placed in
|
|
device's CQ ring.
|
|
BAR 1 - Registers
|
|
--------------------------------------------------------
|
|
| VERSION | DSR | CTL | REQ | ERR | ICR | IMR | MAC |
|
|
--------------------------------------------------------
|
|
DSR - Address of driver/device shared memory used
|
|
for the command channel, used for passing:
|
|
- General info such as driver version
|
|
- Address of 'command' and 'response'
|
|
- Address of async ring
|
|
- Address of device's CQ ring
|
|
- Device capabilities
|
|
CTL - Device control operations (activate, reset etc)
|
|
IMG - Set interrupt mask
|
|
REQ - Command execution register
|
|
ERR - Operation status
|
|
|
|
BAR 2 - UAR
|
|
---------------------------------------------------------
|
|
| QP_NUM | SEND/RECV Flag || CQ_NUM | ARM/POLL Flag |
|
|
---------------------------------------------------------
|
|
- Offset 0 used for QP operations (send and recv)
|
|
- Offset 4 used for CQ operations (arm and poll)
|
|
|
|
|
|
4.3 Major flows
|
|
===============
|
|
|
|
4.3.1 Create CQ
|
|
===============
|
|
- Guest driver
|
|
- Allocates pages for CQ ring
|
|
- Creates page directory (pdir) to hold CQ ring's pages
|
|
- Initializes CQ ring
|
|
- Initializes 'Create CQ' command object (cqe, pdir etc)
|
|
- Copies the command to 'command' address
|
|
- Writes 0 into REQ register
|
|
- Device
|
|
- Reads the request object from the 'command' address
|
|
- Allocates CQ object and initialize CQ ring based on pdir
|
|
- Creates the backend CQ
|
|
- Writes operation status to ERR register
|
|
- Posts command-interrupt to guest
|
|
- Guest driver
|
|
- Reads the HW response code from ERR register
|
|
|
|
4.3.2 Create QP
|
|
===============
|
|
- Guest driver
|
|
- Allocates pages for send and receive rings
|
|
- Creates page directory(pdir) to hold the ring's pages
|
|
- Initializes 'Create QP' command object (max_send_wr,
|
|
send_cq_handle, recv_cq_handle, pdir etc)
|
|
- Copies the object to 'command' address
|
|
- Write 0 into REQ register
|
|
- Device
|
|
- Reads the request object from 'command' address
|
|
- Allocates the QP object and initialize
|
|
- Send and recv rings based on pdir
|
|
- Send and recv ring state
|
|
- Creates the backend QP
|
|
- Writes the operation status to ERR register
|
|
- Posts command-interrupt to guest
|
|
- Guest driver
|
|
- Reads the HW response code from ERR register
|
|
|
|
4.3.3 Post receive
|
|
==================
|
|
- Guest driver
|
|
- Initializes a wqe and place it on recv ring
|
|
- Write to qpn|qp_recv_bit (31) to QP offset in UAR
|
|
- Device
|
|
- Extracts qpn from UAR
|
|
- Walks through the ring and does the following for each wqe
|
|
- Prepares the backend CQE context to be used when
|
|
receiving completion from backend (wr_id, op_code, emu_cq_num)
|
|
- For each sge prepares backend sge
|
|
- Calls backend's post_recv
|
|
|
|
4.3.4 Process backend events
|
|
============================
|
|
- Done by a dedicated thread used to process backend events;
|
|
at initialization is attached to the device and creates
|
|
the communication channel.
|
|
- Thread main loop:
|
|
- Polls for completions
|
|
- Extracts QEMU _cq_num, wr_id and op_code from context
|
|
- Writes CQE to CQ ring
|
|
- Writes CQ number to device CQ
|
|
- Sends completion-interrupt to guest
|
|
- Deallocates context
|
|
- Acks the event to backend
|
|
|
|
|
|
|
|
5. Limitations
|
|
==============
|
|
- The device obviously is limited by the Guest Linux Driver features implementation
|
|
of the VMware device API.
|
|
- Memory registration mechanism requires mremap for every page in the buffer in order
|
|
to map it to a contiguous virtual address range. Since this is not the data path
|
|
it should not matter much. If the default max mr size is increased, be aware that
|
|
memory registration can take up to 0.5 seconds for 1GB of memory.
|
|
- The device requires target page size to be the same as the host page size,
|
|
otherwise it will fail to init.
|
|
- QEMU cannot map guest RAM from a file descriptor if a pvrdma device is attached,
|
|
so it can't work with huge pages. The limitation will be addressed in the future,
|
|
however QEMU allocates Guest RAM with MADV_HUGEPAGE so if there are enough huge
|
|
pages available, QEMU will use them. QEMU will fail to init if the requirements
|
|
are not met.
|
|
|
|
|
|
|
|
6. Performance
|
|
==============
|
|
By design the pvrdma device exits on each post-send/receive, so for small buffers
|
|
the performance is affected; however for medium buffers it will became close to
|
|
bare metal and from 1MB buffers and up it reaches bare metal performance.
|
|
(tested with 2 VMs, the pvrdma devices connected to 2 VFs of the same device)
|
|
|
|
All the above assumes no memory registration is done on data path.
|