Bridged networking for libvirt with NetworkManager: 2014 / Fedora 21

I'm not quite sure how, but I got sucked into spending the whole of today poking at various aspects of handling bridged networking with NetworkManager.

One of the most common uses of bridged networking is for virtualization: you set up a bridge for the host's connection to your router and configure virtual machines to use that bridge, which allows them to connect to the router just as if they were real physical machines that were plugged into it, they'll grab their configuration from your router's DHCP server, and virtual and metal systems can all talk to each other.

This is rather more convenient than libvirt's default setup where the VM host more or less acts like a NAT router for all the virtual machines running on it. This works out of the box, but has limitations. The VMs can connect out to the internet and to other systems on the same network as their host, but those systems and systems outside the local network can't connect in to the guests without some messy manual intervention. It's sort of the same situation you have with relation to the public internet when you're sitting behind your NAT router - you have to fiddle with stuff in the router settings in order to allow outside systems to connect in to servers running on your machine.

Since more or less time immemorial, one of the first things you see in any set of instructions you happen to find for configuring bridging for libvirt is "disable NetworkManager, because it doesn't work with bridges".

Every few months I ask the NM devs what the status of this is, and get a sort of handwavy reply, and move on to something else.

But no more! Today I decided to actually poke about at it and see how it works.

Executive summary: yes, you actually can configure bridging for libvirt purposes using NetworkManager, and have it work properly. It's not even that difficult. But there are some really evil gotchas.

Setting up a bridge with NetworkManager

If you have a clean Fedora 20+ (I think - I tested with 21 and 22, but from reports I've read I think this all applies to F20 as well) system, and you just want to make it work, here is what you should do.

  1. Turn off or delete your existing wired network connection. In GNOME, run the Control Center 'Network' panel and slide the slider to Off. Edit its properties You can do it with nmcli or ifdown or KDE or whatever as well. If you only turn it off, best to also set it not to start at boot: in GNOME, open its properties (with the weird cog icon), click Identity, and uncheck Connect automatically.
  2. Create a new bridge connection profile. In GNOME 'Network', click the +, then click Bridge. With nm-connection-editor, click Add, set the dropdown to Bridge, and click Create...
  3. If you use DHCP, leave everything in the Editing Bridge connection X window you see at default, except click Add next to the empty pane labelled "Bridged connections:", leave the dropdown box at Ethernet, and click Create.... If you need to customize your configuration at all - for static IP addressing, or whatever - do it here, in the bridge's properties, in the IPv4 Settings and IPv6 Settings tabs, before you click Add. If you forget to do it now, don't worry, you can always come back and edit the bridge's properties later.
  4. In the Editing bridgeX slave Y window you see, select your ethernet adapter in the drop-down labelled "Device MAC address:". Go to the General tab and check Automatically connect to this network when it is available.
  5. Click Save... (in Editing bridgeX slave Y)
  6. Click Save... (in Editing Bridge connection X)
  7. Open a terminal, and run as root: nmcli con show. You should see a connection whose name matches the 'bridgeX slave Y' profile you just created. Copy the UUID of that connection.
  8. Run as root nmcli con up (UUID), using the UUID you just copied. If you look at GNOME's "Network" applet again, you'll see a new connection suddenly appeared under "Wired". The fact that it didn't show up before and we had to use nmcli to turn it on is a bug. At this point, your network should come back up again, maybe after a 30 second or so delay. If you look in ifconfig or ip addr you'll see the IP address is tied to the bridge interface. If you look at brctl show you should see the bridge you created, with your network interface listed in the right-hand column.
  9. Configure your virtual machines to use the bridge as their Network source (in virt-manager, it's one of the properties of the VM's NIC, in the VM details page)
  10. Profit!
  11. If everything works, and it still works after a reboot, you might want to delete the original profile for your ethernet interface, to stop it confusing things.

During Fedora 21 development, two extra steps were needed between 9 and 10:

  1. Run as root sysctl -p /usr/lib/sysctl.d/00-system.conf
  2. Create a file /etc/udev/rules.d/99-bridge.rules containing just this line: ACTION=="add", SUBSYSTEM=="module", KERNEL=="bridge", RUN+="/usr/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/bridge". What we just did in these last two steps is deal with another, rather notorious, bug, which network.service has a workaround for, but NetworkManager does not.

but I have just tested that these are no longer needed with Fedora 21 Final, as a fix for that bug was included.

Setting up a bridge with virt-manager

You may be able to successfully create a bridged network using virt-manager, with some care and a following wind, but I don't recommend it, as there are some bugs that could leave you in a slightly messy state (though probably nothing a reboot wouldn't solve). But if you really want to try it, here's what I recommend:

  1. Run virt-manager
  2. Right click on your host, and click Details
  3. Go to the Network Interfaces tab
  4. Click + to add an interface
  5. Leave the type at Bridge and click Forward
  6. Set the Start mode: to onboot
  7. Do NOT check Activate now: ! Don't do it!
  8. Check the tick mark for your network interface in the "Choose interface(s) to bridge:" pane
  9. Change IP settings: to manually configured, and set the appropriate configuration for your network (DON'T try copying the settings from the existing wired connection, that seems to be really broken)
  10. Click Finish
  11. Do steps 9, 10 and 11 from the NetworkManager instructions above
  12. Now you can try running nmcli con show as root and bringing up the bridge and slave profiles with nmcli con up (UUID) for each, or you could try rebooting. Again, the network should come back up for the host when you get both the bridge and slave profiles active.
  13. Do steps 11, 12 and 13 from the NetworkManager instructions.

Don't try actually activating the bridge from virt-manager. It uses ifup commands, and I ran into various bugs with that (listed later in the post). If you're going to use virt-manager, just use it to create the configs, but use nmcli or ifup manually to actually interact with the connections (and see the bugs linked later).

Reduce the startup delay

By default the bridge will probably take 30 seconds to become active, each time it comes up (actually each time a slave connection comes up, it delays for 30 seconds). This is apparently intended for complex networks where 'routing loops' are possible if traffic is routed wrongly - the connection observes the network traffic flow for a while to see what it should do.

This is part of a protocol called STP which is apparently meant for complex enterprise-y networks with multiple bridges between network segments. It's probably safe to simply turn it off. To configure this, edit the settings for the Bridge connection from the network configuration tool, and on the Bridge tab, uncheck Enable STP (Spanning Tree Protocol).

Alternatively, you can reduce the delay to the minimum. To configure this, edit the settings for the Bridge connection from the network configuration tool, and on the Bridge tab, set Forward delay and Hello time to 2. (Don't try and set them to 0, or you may run into a bug).

Background and details

So what's actually going on here? Well, NetworkManager's way of handling bridges is actually not very different from the way the old network service handled them. In fact, at the config file level it's identical. If you already have correct configuration for a simple bridge in /etc/sysconfig/network-scripts you should be able to drop any NM_CONTROLLED=no lines, stop network.service, start NetworkManager.service, and find that NM brings up your bridge successfully. You'll need to do steps 9 through 11 from the NetworkManager instructions to make traffic flow correctly from the VMs, though.

A simple config with one bridge to one ethernet adapter basically consists of these files in /etc/sysconfig/network-scripts (on RH-ish distros):

ifcfg-br1

DEVICE=br1
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=dhcp
IPV6INIT=yes
IPV6_AUTOCONF=no
DHCPV6=no
STP=yes
DELAY=2
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME="Bridge br1"
UUID=xxx
BRIDGING_OPTS=priority=32768
PEERDNS=yes
PEERROUTES=yes

ifcfg-em1 (or whatever your adapter is called, though the name of the file doesn't actually matter any more)

DEVICE=em1
HWADDR=f4:6d:04:9a:1d:45
ONBOOT=yes
BRIDGE=br1

All the 'normal' config options go in the bridge connection's config, note (most of the settings aren't strictly necessary, those are just typical ones from a Fedora install on my system; the UUID= line obviously will be some UUID or other that NM generated for the connection, not xxx). The STP and DELAY options control the bits discussed under "Reduce the startup delay" above.

The interface's config just identifies the interface, says to start it on boot (assuming that's what you want), and says it's a bridge slave interface.

To NetworkManager these are two connections, and you want both of them to be active for the bridge to be running. If you set them both to ONBOOT=yes then the whole thing should just come up at boot time, or you can use nmcli con up to bring them up. Remember that NetworkManager can cope with the concept of there being multiple connections for a single device: you may well still have your original connection for the ethernet interface knocking around, and it may confuse things. If you have issues look in nmcli con show and see if you have more than one connection that's for the ethernet interface, and if the wrong one is active. You probably should just get rid of the non-bridge connection once you have the bridge working. At least set it ONBOOT=no.

If you create the setup using NetworkManager as described above, the bridge connection's name will likely be "Bridge connection 1", and the slave interface connection's name will likely be "bridge0 slave 1", or similar. In /etc/sysconfig/network-scripts they'll be named ifcfg-Bridge_connection_1 and ifcfg-bridge0_slave_1. If you use virt-manager, it'll use old-style interface names, and will actually overwrite the existing connection for your physical device (so at least you won't have two knocking around and confusing things).

As mentioned above, when everything's working, the bridge connection / interface should have the IP address, and 'brctl show' should list the bridge with the slave interface in the column on the right. And sysctl -a | grep bridge should show:

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

and your VMs should get IPs in the normal range for your router, and you should be able to connect between VMs and 'regular' systems.

Bugs I found along the way

  1. sysctl.conf / sysctl.conf.d settings not read when modules are loaded
  2. Network control center panel does not show non-active bridge slave profiles (consequently, cannot activate bridges properly)
  3. virt-manager errors in 'ifup br1' when creating and activating a bridged connection on NetworkManager system
  4. Cannot bring up a bridge via ifup without causing an error ('waiting for slaves before proceeding')
  5. Set up bridged connection, active slave connection, activate bridge -> active profile for interface switches from slave connection to 'Wired connection' profile
  6. Fails copying a simple interface configuration to a bridge

Comments

Bob wrote on 2014-07-24 06:08:
Hi, Thank you for the info but using "routed network" VMs can talk to each other and to the host and to the internet through the host AND can talk to other systems on the same network as the host.
adamw wrote on 2014-07-24 06:45:
edit: d'oh, I just realized I didn't quite explain that right. Guests behind NAT do have some access to the hosts on the same network as the host, you're right. But by default they won't get the DNS config from the router, and you can't forward ports to them from the router (though you can do something rather messy to forward them from the VM host - http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections ). You also can't access the guests from the bare metal hosts on the network in a NAT'ed config, at least without some more manual configuration. Thanks for the correction, I'll update the post.
Pieter wrote on 2014-07-24 11:57:
Thank you for taking the time to write this up. I was looking at those NAT iptables scripts to forward ports and thought: why not just add a bunch of extra IP addresses on the Host and forward all ports from IP on Host to IP address of the Guest. Seems simple enough so I'll give that a try next time I put a fresh F20 on my laptop.
adamw wrote on 2014-07-24 16:04:
To me that seems less simple than just setting up bridging, but I was trying to reason out in my mind last night whether something along those lines would work, and I think at least in theory something along those lines would work. That's basically more or less how libvirt's default NAT setup works anyway - forwarding via iptables. You'd have to handle DNS somehow too, of course, if you didn't want to use the numeric IP addresses for all the connections to the guests...and you have to set up routing so that traffic for those IP addresses reaches the VM host machine in the first place, I guess. This is where I started waving my hands vigorously last night, it's a bit tricky to do it all on a mental whiteboard and see whether it'd work or it's totally insane.
adamw wrote on 2014-07-24 17:33:
The libvirt docs actually document something rather like what you're talking about: http://libvirt.org/formatnetwork.html#examplesRoute
Phil wrote on 2014-07-24 12:32:
Ok, now for the stupid question: If you are using a laptop, can you add both wired and wireless connections ot the bridge?
adamw wrote on 2014-07-24 16:00:
Not so stupid. It appears that for $REASONS you can't attach a bridge to a wifi interface unless it's running in master mode (i.e., more or less, running as an AP). I've seen several slightly different explanations of why this is, but it basically boils down to wireless adapters not passing spoofed MAC addresses. More: http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge#It_doesn.27t_work_with_my_Wireless_card.21 https://wiki.debian.org/BridgeNetworkConnections#Bridging_with_a_wireless_NIC https://answers.launchpad.net/ubuntu/+question/225430 https://superuser.com/questions/597834/bridging-wifi-to-ethernet-on-ubuntu-not-working (the first answer there has a very clear and convincing explanation of the problem, which I'm almost sure is correct :>) Since a late 2.6 release or so, the kernel has had code which will basically refuse to let you attach a bridge to a wifi interface, it'll return an 'operation denied'. There is an interesting workaround that uses ebtables documented here: https://wiki.debian.org/BridgeNetworkConnections#Bridging_with_a_wireless_NIC (the second link above), but I can't vouch for whether it works.
Lavie wrote on 2014-08-11 14:21:
Hi, As regards Step 12 in "Setting up a bridge with NetworkManager" - "Profit!": Is that an inside joke, or does it have some technical meaning?
adamw wrote on 2014-08-12 10:58:
It's just a joke, didn't you see that joke before? Step 1: something, Step 2: ???, Step 3: Profit!
laine wrote on 2014-08-23 22:19:
This meme originated in South Park Season 2 Episode 17 "Gnomes": http://en.wikipedia.org/wiki/Gnomes_%28South_Park%29
Lavie wrote on 2014-08-12 07:31:
Well, I have Fedora 20 installed with all latest YUM updates as of Aug 11, 2014. I tried all the steps of "Setting up a bridge with NetworkManager". When I did a reboot - the system wouldn't come up. After the BIOS screen messages I just got a blank screen, and the disk didn't seem to be doing much. Now what?
adamw wrote on 2014-08-12 10:59:
Well, I've no idea what you did, but nothing in the guide should touch anything that happens that early in the boot process. The worst I can see that you can possibly do is break networking. If you don't see the bootloader menu, it definitely sounds like something else, not related to this guide.
Lavie wrote on 2014-08-12 14:25:
My bad. The bootloader did appear. I chose the latest kernel and then got a dark screen. I was once reasonably knowledgeable about Linux, but took a 9 year break and am just returning to it - specifically for free virtual machines. Therefore, I am not sure what to do to diagnose why after selecting the kernel I get a blank screen. The machine in question is a lab machine, so it's not a problem to reinstall Linux. However, I can only assume that if I perform the same steps that I'll cause the same problem. TIA
adamw wrote on 2014-08-12 14:39:
No problem. Is this a Fedora system, or something else? If Fedora, can you remove the kernel parameters 'rhgb quiet' from the boot options and try again? You may see what's going wrong with the boot, if you do that.
Lavie wrote on 2014-08-15 16:01:
I came back a few hours later, and another poweroff/on solved it. Works like a charm now. Thanks!
Edgar Hoch wrote on 2014-08-28 19:29:
Hello Adam, I found this page through Fedora bug #634736#c22 . I think steps 1-8 can be done with less manual work. You can find my description in bug #1134569 https://bugzilla.redhat.com/show_bug.cgi?id=1134569 . Best regards Edgar
adamw wrote on 2014-08-28 22:48:
Sure, you can do it much more efficiently with text manipulation, as is usually the case :) The post was intended to be a GUI-ish guide - it goes to the console only where GNOME bugs mean you can't actually do it any other way. Thanks for the link, I'm sure it'll be useful for more 'advanced' folks.
Aron Griffis wrote on 2014-09-26 15:37:
Thanks for this writeup, Adam! I've previously spent hours (days) munging sysconfig files until bridged networking would work properly on boot. Your guide worked great on a fresh Fedora 21 alpha install.
Ben Breard wrote on 2014-11-01 15:24:
Worked great except for PXE. Manually adding DELAY=0 to /etc/sysconfig/network-scripts/ifcfg-Bridge_connection_1 fixes the problem. It would be nice if that was a default or configurable via networkmanager. Thanks for the great post.
dnl wrote on 2014-11-05 11:31:
Thanks very much for this Adam. I gave this a try, as I'm one of those folks who usually just turns off NetworkManager. However, it does seem to have a major flaw. In step 8 you mention a 30 second delay in obtaining an IP address. That's fine when first configuring, however I'm seeing that delay on every single boot! It's really quite frustrating when my box used to boot to login (with network) in about 15 seconds total! To get network access now takes 15 + 30 seconds each time. In fact a systemd-analyze plot shows "NetworkManager-wait-online.service" taking 32 seconds alone. Is this expected behaviour or a result of one of the bugs?
adamw wrote on 2014-11-05 16:22:
It's expected behaviour by default - if you check the logs you'll see it's a designed delay during which the bridge is in 'learning mode'. It's intended for complex network setups where bridges might actually have to do some traffic analysis to figure out correct routing behaviour. For a simple case like ours it's unnecessary, and it's intended that you can disable it with DELAY=0 in the bridge configuration file (ifcfg-br1 in our example). This seems to have stopped working for me lately with F21, which is probably an NM bug or something - I have that setting but the delay still happens. But for quite some time it worked fine, the delay only suddenly started happening again relatively recently, and I hadn't had time to look into it yet.
adamw wrote on 2014-11-05 17:17:
If DELAY=0 doesn't work, try DELAY=2: https://bugzilla.redhat.com/show_bug.cgi?id=1160815
dnl wrote on 2014-11-09 11:08:
Thanks Adam, that's what I was after!
HNK wrote on 2015-04-01 18:53:
Thank you very much Adam! Verified on a Debian (testing) machine (network-manager 0.9.10.0-6). I had succes (the second time ;-) following the instructions!) Pitty the documentation on NM is lacking. This has made me very happy. How wonderful of you to write this up and support it too. Have a nice day!
Russell wrote on 2015-07-11 16:18:
Thank you! The details on STP were exactly what I was looking for.
greg wrote on 2015-08-10 05:03:
This did not work for my F22 install. Failed to obtain an IP address and broke the previously working config (when I turned it back on). Fedora developers FF2 take note seriously undercooked release if you need to use virtualisation (as I do as i need windows for work). NAT'ing is just not a valid option, While I can connect to the internet (after rebuilding my network config) I can't connect to the local Windows network..... sorry massive FAIL again my the developers of Boxes and the Fedora network team.
adamw wrote on 2015-08-10 14:48:

I'm sorry it's not working for you, but I tested it on F22 and it works fine for me. It also works for several commenters, as you can see.

Stuart Gathman wrote on 2015-08-29 03:02:
Anothing reason to use bridging is for IPv6. Yes, you can create an IPv6 /80 subnet on the virbr0, and run radvd on the VM host to advertise the route. But as with IP4 (and the nasty NAT rules), it is so much simpler to just use the bridge. Oh, I spent 4 hours trying to get the bridge working with NM. I found your page toward the end, but still wasn't successful, but maybe I just wasn't up to starting again. So I disabled NM and created the ifcfg-* files. Sigh. I'm not sure what NM buys me on a desktop anyway. (NM is mostly for laptops that roam between networks.)
adamw wrote on 2015-08-29 03:34:

Not really, NM covers all sorts of other stuff. But if you only have a single simple ethernet connection, it doesn't matter a lot what you use to manage it, there isn't a whole lot of management to do.

lzap wrote on 2015-09-24 14:50:
Hey, I am testing this in Fedora 22 and it works, but something is weird with my routing, I got 50 % packet loss. Also I don't see the sysctl values anymore: [root@virt ~]# sysctl -a | grep bridge sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.br0.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.enp8s0.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" sysctl: reading key "net.ipv6.conf.virbr0.stable_secret" sysctl: reading key "net.ipv6.conf.virbr0-nic.stable_secret" sysctl: reading key "net.ipv6.conf.virbr1.stable_secret" sysctl: reading key "net.ipv6.conf.virbr1-nic.stable_secret" sysctl: reading key "net.ipv6.conf.vnet0.stable_secret" sysctl: reading key "net.ipv6.conf.vnet1.stable_secret" sysctl: reading key "net.ipv6.conf.vnet2.stable_secret" [root@virt ~]# sysctl -p /usr/lib/sysctl.d/00-system.conf sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-arptables: No such file or directory
Jamie Nguyen wrote on 2015-12-16 13:18:
On Linux kernel 3.18 or later (eg, on Fedora), the bridge-nf-call-* parameters have been moved to the "br_netfilter" kernel module. If you don't load that module then you don't need to set those parameters. See Bridged Network 0, which is part of my libvirt Networking Handbook 1.
dnl wrote on 2016-07-26 12:18:
Hey Adam, I don't suppose you'd be able to update this for Fedora 24, please? I've been unable to get the Slave interface to come up and I'm also getting odd SELinux denials from firewalld trying to write interface configuration files (the old systemV network interface configuration files - not network manager?) There's a recent Fedora Magazine article which still promotes using a different non-NetworkManager method. While I appreciate many servers won't be running NetworkManager, many workstations will and the software supports bridging - but seems unreliable. https://fedoramagazine.org/build-network-bridge-fedora/ Either way, it's not documented well at all in the wiki: https://fedoraproject.org/wiki/Networking/Bridging Once I understand this, if I'm permitted to, I'll create an account and update that wiki. Thanks in advance!
dnl wrote on 2016-07-29 11:42:
I had a chance to research this and experiment and came up with the way to configure it using nmcli, rather than the GUI. I can only imagine that a combination of options I was fiddling with in the GUI was causing issues bringing the brige-slave up - something to do with me fiddling with firewalld default options and that causing a SELinux denial. So, here's what worked for me in Fedora 24 with nmcli. This is a simple workstation with one interface which runs VMs through that interface. No STP required. Shut down primary network interface - "eno1" for me nmcli con down eno1 Prevent it from starting on boot nmcli -a connection modify eno1 connection.autoconnect no Add a virtual bridge with name "bridge0" nmcli connection add type bridge ifname bridge0 con-name bridge0 Disable Spanning Tree Protocol on "bridge0" (unless you need it!) nmcli connection modify bridge0 bridge.stp no Disable IPv6 on "bridge0" unless your systems are not using IPv4 (using both for most systems is crazy!) nmcli connection modify bridge0 ipv6.method ignore Bring up the interface slaved to the bridge nmcli connection up bridge-slave-eno1 To check everything nmcli con show
Laszlo Ersek wrote on 2016-10-19 11:22:
Thank you, Adam, for writing this up. My config was mostly okay, but it wouldn't work in the VM; the bridge wasn't looking good. After reading this blog post, I realized I was missing point #1, "Turn off or delete your existing wired network connection". How incredibly non-intuitive from NetworkManager. :/ Anyway, you saved the day. Thanks!
Julian Alarcon wrote on 2017-03-08 14:51:
Hi You should update this post, as the bug mentioned here GNOEM:733634 is already resolved
adamw wrote on 2017-04-05 01:32:

Well, this is a blog. Of course old posts on it are outdated. If I spent my life updating my blog archive I'd never get anything else done.

dnl wrote on 2017-10-31 23:13:
Good point that this is an old blog entry. I guess it shows a need which isn't met elsewhere in documentation. Should this be added to the Fedora wiki, or is that cutting out some users of NetworkManager?
serg wrote on 2018-11-05 04:52:
https://www.cyberciti.biz/faq/how-to-add-network-bridge-with-nmcli-networkmanager-on-linux/ Open the Terminal app Get info about the current connection: nmcli con show Add a new bridge: nmcli con add type bridge ifname br0 Create a slave interface: nmcli con add type bridge-slave ifname eno1 master br0 Turn on br0: nmcli con up br0