Virtio on Linux

Introduction

Virtio is an open standard that defines a protocol for communication between drivers and devices of different types, see Chapter 5 ("Device Types") of the virtio spec ([1]). Originally developed as a standard for paravirtualized devices implemented by a hypervisor, it can be used to interface any compliant device (real or emulated) with a driver.

For illustrative purposes, this document will focus on the common case of a Linux kernel running in a virtual machine and using paravirtualized devices provided by the hypervisor, which exposes them as virtio devices via standard mechanisms such as PCI.

Device - Driver communication: virtqueues

Although the virtio devices are really an abstraction layer in the hypervisor, they're exposed to the guest as if they are physical devices using a specific transport method -- PCI, MMIO or CCW -- that is orthogonal to the device itself. The virtio spec defines these transport methods in detail, including device discovery, capabilities and interrupt handling.

The communication between the driver in the guest OS and the device in the hypervisor is done through shared memory (that's what makes virtio devices so efficient) using specialized data structures called virtqueues, which are actually ring buffers [1] of buffer descriptors similar to the ones used in a network device:

struct vring_desc

Virtio ring descriptors, 16 bytes long. These can chain together via next.

Definition:

struct vring_desc {
    __virtio64 addr;
    __virtio32 len;
    __virtio16 flags;
    __virtio16 next;
};

Members

addr

buffer address (guest-physical)

len

buffer length

flags

descriptor flags

next

index of the next descriptor in the chain, if the VRING_DESC_F_NEXT flag is set. We chain unused descriptors via this, too.

All the buffers the descriptors point to are allocated by the guest and used by the host either for reading or for writing but not for both.

Refer to Chapter 2.5 ("Virtqueues") of the virtio spec ([1]) for the reference definitions of virtqueues and "Virtqueues and virtio ring: How the data travels" blog post ([2]) for an illustrated overview of how the host device and the guest driver communicate.

The vring_virtqueue struct models a virtqueue, including the ring buffers and management data. Embedded in this struct is the virtqueue struct, which is the data structure that's ultimately used by virtio drivers:

struct virtqueue

a queue to register buffers for sending or receiving.

Definition:

struct virtqueue {
    struct list_head list;
    void (*callback)(struct virtqueue *vq);
    const char *name;
    struct virtio_device *vdev;
    unsigned int index;
    unsigned int num_free;
    unsigned int num_max;
    bool reset;
    void *priv;
};

Members

list

the chain of virtqueues for this device

callback

the function to call when buffers are consumed (can be NULL).

name

the name of this virtqueue (mainly for debugging)

vdev

the virtio device this queue was created for.

index

the zero-based ordinal number for this queue.

num_free

number of elements we expect to be able to fit.

num_max

the maximum number of elements supported by the device.

reset

vq is in reset state or not.

priv

a pointer for the virtqueue implementation to use.

Description

A note on num_free: with indirect buffers, each buffer needs one element in the queue, otherwise a buffer will need one element per sg element.

The callback function pointed by this struct is triggered when the device has consumed the buffers provided by the driver. More specifically, the trigger will be an interrupt issued by the hypervisor (see vring_interrupt()). Interrupt request handlers are registered for a virtqueue during the virtqueue setup process (transport-specific).

irqreturn_t vring_interrupt(int irq, void *_vq)

notify a virtqueue on an interrupt

Parameters

int irq

the IRQ number (ignored)

void *_vq

the struct virtqueue to notify

Description

Calls the callback function of _vq to process the virtqueue notification.

Device discovery and probing

In the kernel, the virtio core contains the virtio bus driver and transport-specific drivers like virtio-pci and virtio-mmio. Then there are individual virtio drivers for specific device types that are registered to the virtio bus driver.

How a virtio device is found and configured by the kernel depends on how the hypervisor defines it. Taking the QEMU virtio-console device as an example. When using PCI as a transport method, the device will present itself on the PCI bus with vendor 0x1af4 (Red Hat, Inc.) and device id 0x1003 (virtio console), as defined in the spec, so the kernel will detect it as it would do with any other PCI device.

During the PCI enumeration process, if a device is found to match the virtio-pci driver (according to the virtio-pci device table, any PCI device with vendor id = 0x1af4):

/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
static const struct pci_device_id virtio_pci_id_table[] = {
        { PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
        { 0 }
};

then the virtio-pci driver is probed and, if the probing goes well, the device is registered to the virtio bus:

static int virtio_pci_probe(struct pci_dev *pci_dev,
                            const struct pci_device_id *id)
{
        ...

        if (force_legacy) {
                rc = virtio_pci_legacy_probe(vp_dev);
                /* Also try modern mode if we can't map BAR0 (no IO space). */
                if (rc == -ENODEV || rc == -ENOMEM)
                        rc = virtio_pci_modern_probe(vp_dev);
                if (rc)
                        goto err_probe;
        } else {
                rc = virtio_pci_modern_probe(vp_dev);
                if (rc == -ENODEV)
                        rc = virtio_pci_legacy_probe(vp_dev);
                if (rc)
                        goto err_probe;
        }

        ...

        rc = register_virtio_device(&vp_dev->vdev);

When the device is registered to the virtio bus the kernel will look for a driver in the bus that can handle the device and call that driver's probe method.

At this point, the virtqueues will be allocated and configured by calling the appropriate virtio_find helper function, such as virtio_find_single_vq() or virtio_find_vqs(), which will end up calling a transport-specific find_vqs method.

References

[1] Virtio Spec v1.2: https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html

[2] Virtqueues and virtio ring: How the data travels https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels

Footnotes

[1]

that's why they may be also referred to as virtrings.