[Documentation to add for drvbhyve]

Networking (NAT)

• Load the pf module and enable the service.

• Set the forwarding knob to 1 (net.inet.ip.forwarding)

• Copy over libvirt's default configuration (see pkg info -D libvirt). Then restart libvirtd.

• This should get you virbr0 in ifconfig. If not, you need to use virsh to turn it on.

• Tap devices should be created automatically.

/etc/pf.conf:

xxxxxxxxxx
scrub all

nat on em0 from {192.168.122.0/24} to any -> (em0)

nat-anchor "libvirt\*"
anchor "libvirt\*"

pass all

• pfctl to load the new rules.

Configurations in libvirt that work:

xxxxxxxxxx
<interface type="bridge">
  <source bridge="virbr0"/>
  <model type="virtio"/>
</interface>

<=>

xxxxxxxxxx
<interface type='network'>
  <source network='default'/>
  <model type='virtio'/>
</interface>

E1000 emulation also works

xxxxxxxxxx
<interface type='bridge'>
  <source bridge='virbr0'/>
  <model type='e1000'/>
</interface>

Misc:

• Proxy ARP works

• Static routing should work.

• Bridged networking works. Just define a network for virsh with forward mode = 'bridge'

• Couldn't test out SR-IOV

Misc weird things:

• nmdm consoles work, but if using vm-bhyve, to quit it you need to press ~ and then ctrl+d

• In libvirt, you can just press ctrl + ]

• virtio-p9fs exists on libvirt, in process of updating virt-manager to use it

Things that need work:

• qcow2 isn't supported. Also can't convert it to native bhyve format.

Where to begin with VirtIO?

• So, we have virtio-sock and virtio-fs implementations (https://www.freebsd.org/status/report-2024-07-2024-09/vsock/), (https://lists.freebsd.org/archives/freebsd-hackers/2024-July/003403.html), but neither are merged.

• We're still using 0.9 - https://www.illumos.org/issues/17767 - port over.

• If looking at the 1.4 spec, we essentially only have support for drivers over PCI transport. So no virtio-mmio and virtio-ccw.

• (More important) things that are completely missing: virtio-crypto, virtio-iommu, virtio-sound, virtio-mem and virtio-input

• (Less important) " " " " " : virtio-pmem, virtio-gpio

• NB: @aokblast has packed virtqueues. That should land after virtio 1.1 is done (soon).

• It would be nice to have support for virtio-blk VTBLK_F_MQ, VTBLK_F_SCSI. VTBLK_F_RO should be easy to implement with camcontrol APIs?

On virtio-balloon:

• https://github.com/freebsd/freebsd-src/pull/2116 should probably be merged.

• Currently, we only support VIRTIO_BALLOON_F_MUST_TELL_HOST.

• It would be nice to have support for VIRTIO_BALLOON_F_FREE_PAGE_HINT, VIRTIO_BALLOON_F_PAGE_POISON and especially VIRTIO_BALLOON_F_PAGE_REPORTING.

• https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=292570#c7

Non-virtio unsupported features

• genid

• sev

• sgx

• MSR/TSC enumeration

• host passthrough on cpu

• host model on cpu

• custom cpu

• aarch64 is still experimental

• network backed NVRAM. We should be able to support iSCSI without much effort. Don't know about RDB and Gluster. It seems both are partially (?) supported.

• Not sure how many statistics listed in https://libvirt.org/formatdomain.html#id28 bhyvectl supports getting.

• SPICE server

Things like nested virtualization are also unsupported, but this may be because bhyve was meant to be lightweight (?)