Networking Lab: Ethernet Broadcast Domains

In Ethernet, all the nodes forming one L2 segment constitute a broadcast domain. Such nodes should be able to communicate using their L2 addresses (MAC) or by broadcasting frames. A broadcast domain is a logical division of a computer network. Multiple physical (L1) segments can be bridged to form a single broadcast domain. Multiple L2 segments can also be bridged to create a bigger broadcast domain.

Prerequisite

We will use a Linux bridge to emulate a network switch and Linux network namespaces to emulate network nodes.

The following helper functions should be defined (copy and paste them to helpers.sh or download this gist):

Create Linux bridge

Creates an isolated network namespace with a Linux bridge inside:

create_bridge() {
    local nsname="$1"
    local ifname="$2"

    echo "Creating bridge ${nsname}/${ifname}"

    ip netns add ${nsname}
    ip netns exec ${nsname} ip link set lo up
    ip netns exec ${nsname} ip link add ${ifname} type bridge
    ip netns exec ${nsname} ip link set ${ifname} up
}

Create End Host

Creates an isolated network namespace and connects it using a veth pair to the specified bridge (in another namespace):

create_end_host() {
    local host_nsname="$1"
    local peer1_ifname="$2a"
    local peer2_ifname="$2b"
    local bridge_nsname="$3"
    local bridge_ifname="$4"

    echo "Creating end host ${host_nsname} connected to ${bridge_nsname}/${bridge_ifname} bridge"

    # Create end host network namespace.
    ip netns add ${host_nsname}
    ip netns exec ${host_nsname} ip link set lo up

    # Create a veth pair connecting end host and bridge namespaces.
    ip link add ${peer1_ifname} netns ${host_nsname} type veth peer \
                ${peer2_ifname} netns ${bridge_nsname}
    ip netns exec ${host_nsname} ip link set ${peer1_ifname} up
    ip netns exec ${bridge_nsname} ip link set ${peer2_ifname} up

    # Attach peer2 interface to the bridge.
    ip netns exec ${bridge_nsname} ip link set ${peer2_ifname} master ${bridge_ifname}
}

Connect two Linux bridges

Interconnects two Linux bridges (switches) potentially residing in different namespaces using an auxiliary veth pair:

connect_bridges() {
    local bridge1_nsname="$1"
    local bridge1_ifname="$2"
    local bridge2_nsname="$3"
    local bridge2_ifname="$4"
    local peer1_ifname="veth_${bridge2_ifname}"
    local peer2_ifname="veth_${bridge1_ifname}"

    echo "Connecting bridge ${bridge1_nsname}/${bridge1_ifname} to ${bridge2_nsname}/${bridge2_ifname} bridge using veth pair"

    # Create veth pair.
    ip link add ${peer1_ifname} netns ${bridge1_nsname} type veth peer \
                ${peer2_ifname} netns ${bridge2_nsname}
    ip netns exec ${bridge1_nsname} ip link set ${peer1_ifname} up
    ip netns exec ${bridge2_nsname} ip link set ${peer2_ifname} up

    # Connect bridges.
    ip netns exec ${bridge1_nsname} ip link set ${peer1_ifname} master ${bridge1_ifname}
    ip netns exec ${bridge2_nsname} ip link set ${peer2_ifname} master ${bridge2_ifname}
}

How to send Ethernet frames from command line

Most of the available networking tools operate on L3 or above. Hence, we need to make our own tool to manually transmit arbitrary data on the data link layer. To send Ethernet frames programmatically we can leverage packet sockets (AF_PACKET) operating in raw mode (SOCK_RAW). Thus, we'll need to build Ethernet frames in the code and then write them into the raw packet socket. Luckily Ethernet frames have a fairly simple structure.

We will use the following (hopefully self-explanatory) python code to create and send Ethernet frames (gist):

#!/usr/bin/env python3

# Usage: ethsend.py eth0 ff:ff:ff:ff:ff:ff 'Hello everybody!'
#        ethsend.py eth0 06:e5:f0:20:af:7a 'Hello 06:e5:f0:20:af:7a!'
#
# Note: CAP_NET_RAW capability is required to use SOCK_RAW

import fcntl
import socket
import struct
import sys

def send_frame(ifname, dstmac, eth_type, payload):
    # Open raw socket and bind it to network interface.
    s = socket.socket(socket.AF_PACKET, socket.SOCK_RAW)
    s.bind((ifname, 0))

    # Get source interface's MAC address.
    info = fcntl.ioctl(s.fileno(),
                       0x8927,
                       struct.pack('256s', bytes(ifname, 'utf-8')[:15]))
    srcmac = ':'.join('%02x' % b for b in info[18:24])

    # Build Ethernet frame
    payload_bytes = payload.encode('utf-8')
    assert len(payload_bytes) <= 1500  # Ethernet MTU

    frame = human_mac_to_bytes(dstmac) + \
            human_mac_to_bytes(srcmac) + \
            eth_type + \
            payload_bytes

    # Send Ethernet frame
    return s.send(frame)

def human_mac_to_bytes(addr):
    return bytes.fromhex(addr.replace(':', ''))

def main():
  ifname = sys.argv[1]
  dstmac = sys.argv[2]
  payload = sys.argv[3]
  ethtype = b'\x7A\x05'  # arbitrary, non-reserved
  send_frame(ifname, dstmac, ethtype, payload)

if __name__ == "__main__":
    main()

Broadcast domain example I: multiple hosts connected to single network switch

This example demonstrates the simplest possible scenario - a bunch of network nodes connected to a shared switch.

Use the following script to set it all up:

$ cat > demo1.sh <<EOF
#!/usr/bin/env bash

source helpers.sh

create_bridge netns_br0 br0
create_end_host netns_veth0 veth0 netns_br0 br0
create_end_host netns_veth1 veth1 netns_br0 br0
create_end_host netns_veth2 veth2 netns_br0 br0
EOF

$ sudo bash demo1.sh
Creating bridge netns_br0/br0
Creating end host netns_veth0 connected to netns_br0/br0 bridge
Creating end host netns_veth1 connected to netns_br0/br0 bridge
Creating end host netns_veth2 connected to netns_br0/br0 bridge

And here is a quick demo demonstrating that hosts connected to a shared switch (bridge) can communicate with each other using either their MAC addresses or a broadcast address (FF:FF:FF:FF:FF:FF):

# Host 1 (Terminal 1)
$ sudo nsenter --net=/var/run/netns/netns_veth0
$ python3 ethsend.py veth0a ff:ff:ff:ff:ff:ff 'Hello all!'
$ python3 ethsend.py veth0a 86:47:f2:ff:fd:f2 'Hello 86:47:f2:ff:fd:f2!'  # veth1
$ python3 ethsend.py veth0a 1a:12:2b:da:e1:37 'Hello 1a:12:2b:da:e1:37!'  # veth2

# Host 2 (Terminal 2)
$ sudo nsenter --net=/var/run/netns/netns_veth1
$ ip link show veth1a  # note MAC address
$ tcpdump -i veth1a ether proto 0x7a05

# Host 3 (Terminal 3)
$ sudo nsenter --net=/var/run/netns/netns_veth2
$ ip link show veth2a   # note MAC address
$ tcpdump -i veth2a ether proto 0x7a05

The demo also shows that hosts on a single L2 segment don't technically need any L3 configuration (assigning IP addresses or setting routing rules) to communicate with each other.

To clean up, just remove the created network namespaces:

$ sudo ip netns delete netns_br0
$ sudo ip netns delete netns_veth0
$ sudo ip netns delete netns_veth1
$ sudo ip netns delete netns_veth2

Broadcast domain example II: two interconnected network switches

Normally, there is a physical limitation on the number of ports per switch. Even the Linux bridge (virtual switch) has a limitation of 1024 ports. Thus, an interconnection of multiple switches may be required in situations when the number of end hosts exceeds the number of available ports per switch.

In this example we'll show that multiple interconnected network switches (bridges) still form a single broadcast domain:

Use the following sequence of commands to set it all up:

$ cat > demo2.sh <<EOF
#!/usr/bin/env bash

source helpers.sh

# First bridge with 2 connected hosts
create_bridge netns_br10 br10
create_end_host netns_veth10 veth10 netns_br10 br10
create_end_host netns_veth11 veth11 netns_br10 br10

# Second bridge with another 2 hosts
create_bridge netns_br20 br20
create_end_host netns_veth20 veth20 netns_br20 br20
create_end_host netns_veth21 veth21 netns_br20 br20

# Connect bridges with a patch cord (veth pair)
connect_bridges netns_br10 br10 netns_br20 br20
EOF

$ sudo bash demo2.sh
Creating bridge netns_br10/br10
Creating end host netns_veth10 connected to netns_br10/br10 bridge
Creating end host netns_veth11 connected to netns_br10/br10 bridge
Creating bridge netns_br20/br20
Creating end host netns_veth20 connected to netns_br20/br20 bridge
Creating end host netns_veth21 connected to netns_br20/br20 bridge
Connecting bridge netns_br10/br10 to netns_br20/br20 bridge using veth pair

To demonstrate that these hosts still form a single broadcast domain we can use the following demo:

# Host 1 (Terminal 1)
$ sudo nsenter --net=/var/run/netns/netns_veth10
$ python3 ethsend.py veth10a ff:ff:ff:ff:ff:ff 'Hello all!'
$ python3 ethsend.py veth10a 4a:8b:47:0c:43:8b 'Hello 4a:8b:47:0c:43:8b!'  # veth11a
$ python3 ethsend.py veth10a 4e:5e:a8:51:32:3b 'Hello 4e:5e:a8:51:32:3b!'  # veth21a

# Host 2 (Terminal 2)
$ sudo nsenter --net=/var/run/netns/netns_veth11
$ ip link show veth11a  # note MAC address
$ tcpdump -i veth11a ether proto 0x7a05

# Host 3 (Terminal 3)
$ sudo nsenter --net=/var/run/netns/netns_veth20
$ ip link show veth20a  # note MAC address
$ tcpdump -i veth20a ether proto 0x7a05

# Host 4 (Terminal 4)
$ sudo nsenter --net=/var/run/netns/netns_veth21
$ ip link show veth21a  # note MAC address
$ tcpdump -i veth21a ether proto 0x7a05

The demo shows that from the hosts' standpoint there is (logically) no difference between being connected to a single switch (bridge) or multiple interconnected switches. They still form one flat L2 segment and broadcast domain.

To clean up, just remove the created network namespaces:

$ sudo ip netns delete netns_br10
$ sudo ip netns delete netns_veth10
$ sudo ip netns delete netns_veth11

$ sudo ip netns delete netns_br20
$ sudo ip netns delete netns_veth20
$ sudo ip netns delete netns_veth21

Broadcast domain example III: hierarchical internetworking (simplified)

In a big enough setup, a flat interconnection of switches will lead to a high amount of transit traffic. A hierarchical interconnection of switches may provide better performance.

In this example we'll show that multi-level interconnection of switches (bridges) also form a single broadcast domain:

Use the following sequence of commands to set it all up:

$ cat > demo3.sh <<EOF
#!/usr/bin/env bash

source helpers.sh
# Lower-layer switch 1
create_bridge netns_br10 br10
create_end_host netns_veth10 veth10 netns_br10 br10
create_end_host netns_veth11 veth11 netns_br10 br10

# Lower-layer switch 2
create_bridge netns_br20 br20
create_end_host netns_veth20 veth20 netns_br20 br20
create_end_host netns_veth21 veth21 netns_br20 br20

# Higher-layer switch
create_bridge netns_br30 br30

# Connect both lower-layer switches to higher layer switch
connect_bridges netns_br10 br10 netns_br30 br30
connect_bridges netns_br20 br20 netns_br30 br30
EOF

$ sudo bash demo3.sh
Creating bridge netns_br10/br10
Creating end host netns_veth10 connected to netns_br10/br10 bridge
Creating end host netns_veth11 connected to netns_br10/br10 bridge
Creating bridge netns_br20/br20
Creating end host netns_veth20 connected to netns_br20/br20 bridge
Creating end host netns_veth21 connected to netns_br20/br20 bridge
Creating bridge netns_br30/br30
Connecting bridge netns_br10/br10 to netns_br30/br30 bridge using veth pair
Connecting bridge netns_br20/br20 to netns_br30/br30 bridge using veth pair

Use the following sequence of command to demonstrate that all the hosts still form a single broadcast domain:

# Host 1 (Terminal 1)
sudo nsenter --net=/var/run/netns/netns_veth10
python3 ethsend.py veth10a ff:ff:ff:ff:ff:ff 'Hello all!'
python3 ethsend.py veth10a 2e:5b:3f:48:b0:82 'Hello 2e:5b:3f:48:b0:82!'  # veth11a
python3 ethsend.py veth10a 7a:dc:9a:74:35:d6 'Hello 7a:dc:9a:74:35:d6!'  # veth21a

# Host 2 (Terminal 2)
sudo nsenter --net=/var/run/netns/netns_veth11
ip link show veth11a  # note MAC address
tcpdump -i veth11a ether proto 0x7a05

# Host 3 (Terminal 3)
sudo nsenter --net=/var/run/netns/netns_veth20
ip link show veth20a  # note MAC address
tcpdump -i veth20a ether proto 0x7a05

# Host 4 (Terminal 4)
sudo nsenter --net=/var/run/netns/netns_veth21
ip link show veth21a  # note MAC address
tcpdump -i veth21a ether proto 0x7a05

Once again, logically there is no difference between flat and hierarchical interconnections:

To clean up, just remove the created network namespaces:

$ sudo ip netns delete netns_br10
$ sudo ip netns delete netns_veth10
$ sudo ip netns delete netns_veth11

$ sudo ip netns delete netns_br20
$ sudo ip netns delete netns_veth20
$ sudo ip netns delete netns_veth21

$ sudo ip netns delete netns_br30

Further Reading

• Computer Networking Introduction: Ethernet and IP (Heavily Illustrated)
• Bridge vs Switch: What I Learned From a Data Center Tour
• Container Networking Is Simple!
• Networking Lab: Simple VLAN
• Networking Lab: L3 to L2 Segments Mapping
• Networking Lab: Proxy ARP
• Networking Lab: Simple VXLAN

See also

• "Fun with veth-devices" series (I II III IV V VI VII VIII) by Dr. Ralph Mönchmeyer