Networking Lab: L3 to L2 Segments Mapping

It's pretty common for an L2 segment to have a single IP subnet running atop. However, technically it's possible to configure multiple IP subnets over a single L2 broadcast domain. And although more complicated, configuring a single IP subnet over multiple disjoint L2 segments is also doable. In this lab, we'll cover the first two scenarios while the more advanced third case deserves its own lab - Proxy ARP.

Prerequisite

We will use a Linux bridge to emulate a network switch and Linux network namespaces to emulate network nodes.

The following helper functions should be defined (copy and paste them to helpers.sh or download this gist):

Create Linux bridge

Creates an isolated network namespace with a Linux bridge inside:

create_bridge() {
    local nsname="$1"
    local ifname="$2"

    echo "Creating bridge ${nsname}/${ifname}"

    ip netns add ${nsname}
    ip netns exec ${nsname} ip link set lo up
    ip netns exec ${nsname} ip link add ${ifname} type bridge
    ip netns exec ${nsname} ip link set ${ifname} up
}

Create End Host

Creates an isolated network namespace and connects it using a veth pair to the specified bridge (in another namespace):

create_end_host() {
    local host_nsname="$1"
    local peer1_ifname="$2a"
    local peer2_ifname="$2b"
    local peer1_ifaddr="$3"
    local bridge_nsname="$4"
    local bridge_ifname="$5"

    echo "Creating end host ${host_nsname} ${peer1_ifaddr} connected to ${bridge_nsname}/${bridge_ifname} bridge"

    # Create end host network namespace.
    ip netns add ${host_nsname}
    ip netns exec ${host_nsname} ip link set lo up

    # Create a veth pair connecting end host and bridge namespaces.
    ip link add ${peer1_ifname} netns ${host_nsname} type veth peer \
                ${peer2_ifname} netns ${bridge_nsname}
    ip netns exec ${host_nsname} ip link set ${peer1_ifname} up
    ip netns exec ${bridge_nsname} ip link set ${peer2_ifname} up

    # Setting host's IP address.
    ip netns exec ${host_nsname} ip addr add ${peer1_ifaddr} dev ${peer1_ifname}

    # Attach peer2 interface to the bridge.
    ip netns exec ${bridge_nsname} ip link set ${peer2_ifname} master ${bridge_ifname}
}

Single IP subnet over L2 broadcast domain

This example demonstrates the simplest possible scenario - an IP subnet 192.168.0.0/24 configured over a single L2 broadcast domain formed by a Linux bridge.

Use the following script to set it all up:

$ cat > demo1.sh <<EOF
#!/usr/bin/env bash

source helpers.sh

create_bridge netns_br0 br0

create_end_host netns_veth0 veth0 '192.168.0.5/24' netns_br0 br0
create_end_host netns_veth1 veth1 '192.168.0.6/24' netns_br0 br0
create_end_host netns_veth2 veth2 '192.168.0.7/24' netns_br0 br0
EOF

$ sudo bash demo1.sh
Creating bridge netns_br0/br0
Creating end host netns_veth0 192.168.0.5/24 connected to netns_br0/br0 bridge
Creating end host netns_veth1 192.168.0.6/24 connected to netns_br0/br0 bridge
Creating end host netns_veth2 192.168.0.7/24 connected to netns_br0/br0 bridge

Notice, that we used both the IP address and the network mask while configuring network interfaces. Having a network mask specified allows the Linux kernel to deduce the route to the 192.168.0.0/24 network automatically.

# Host 192.168.0.5 (Terminal 1)
$ sudo nsenter --net=/var/run/netns/netns_veth0
$ ip route show
192.168.0.0/24 dev veth0a proto kernel scope link src 192.168.0.5

# Host 192.168.0.6 (Terminal 2)
$ sudo nsenter --net=/var/run/netns/netns_veth1
$ ip route show
192.168.0.0/24 dev veth1a proto kernel scope link src 192.168.0.6

# Host 192.168.0.7 (Terminal 3)
$ sudo nsenter --net=/var/run/netns/netns_veth2
$ ip route show
192.168.0.0/24 dev veth2a proto kernel scope link src 192.168.0.7

These routes have a link scope, i.e. packets destined to the 192.168.0.0/24 network won't need to go to a router. Instead, they can be delivered using the physical link these nodes are directly connected to.

Here is a quick demo demonstrating that point-to-point communication via IP heavily relies on the L2 communication means (MAC addressability and broadcasting):

# Host 192.168.0.5 (Terminal 1)
$ sudo nsenter --net=/var/run/netns/netns_veth0
$ ping 192.168.0.7

# Host 192.168.0.6 (Terminal 2)
$ sudo nsenter --net=/var/run/netns/netns_veth1
$ tcpdump -i veth1a

# Host 192.168.0.7 (Terminal 3)
$ sudo nsenter --net=/var/run/netns/netns_veth2
$ tcpdump -i veth2a

Single IP subnet on top of a broadcast domain.

To clean up, just remove the created network namespaces:

$ sudo ip netns delete netns_br0
$ sudo ip netns delete netns_veth0
$ sudo ip netns delete netns_veth1
$ sudo ip netns delete netns_veth2

Multiple IP subnets over single L2 broadcast domain

This is a slightly different scenario - two non-intersecting IP subnets 192.168.0.0/24 and 192.168.1.0/24 configured over a single L2 broadcast domain formed by a Linux bridge.

Use the following script to set it all up:

$ cat > demo2.sh <<EOF
#!/usr/bin/env bash

source helpers.sh

create_bridge netns_br0 br0

# Subnet 1
create_end_host netns_veth0 veth0 192.168.0.5/24 netns_br0 br0
create_end_host netns_veth1 veth1 192.168.0.6/24 netns_br0 br0
create_end_host netns_veth2 veth2 192.168.0.7/24 netns_br0 br0

# Subnet 2
create_end_host netns_veth3 veth3 192.168.1.5/24 netns_br0 br0
create_end_host netns_veth4 veth4 192.168.1.6/24 netns_br0 br0
create_end_host netns_veth5 veth5 192.168.1.7/24 netns_br0 br0
EOF

$ sudo bash demo2.sh
Creating bridge netns_br0/br0
Creating end host netns_veth0 192.168.0.5/24 connected to netns_br0/br0 bridge
Creating end host netns_veth1 192.168.0.6/24 connected to netns_br0/br0 bridge
Creating end host netns_veth2 192.168.0.7/24 connected to netns_br0/br0 bridge
Creating end host netns_veth3 192.168.1.5/24 connected to netns_br0/br0 bridge
Creating end host netns_veth4 192.168.1.6/24 connected to netns_br0/br0 bridge
Creating end host netns_veth5 192.168.1.7/24 connected to netns_br0/br0 bridge

Here is a demo demonstrating that Ethernet frames (in particular, ARP requests) that belong to one subnet are actually visible by the hosts from another subnet:

# Host 192.168.0.5 (Terminal 1)
$ sudo nsenter --net=/var/run/netns/netns_veth0
$ ping 192.168.0.7

# Host 192.168.0.6 (Terminal 2)
$ sudo nsenter --net=/var/run/netns/netns_veth1
$ tcpdump -i veth1a arp or icmp

# Host 192.168.0.7 (Terminal 3)
$ sudo nsenter --net=/var/run/netns/netns_veth2
$ tcpdump -i veth2a arp or icmp

# Host 192.168.1.5 (Terminal 4)
$ sudo nsenter --net=/var/run/netns/netns_veth3
$ ping 192.168.1.7

# Host 192.168.1.6 (Terminal 5)
$ sudo nsenter --net=/var/run/netns/netns_veth4
$ tcpdump -i veth4a arp or icmp

# Host 192.168.1.7 (Terminal 6)
$ sudo nsenter --net=/var/run/netns/netns_veth5
$ tcpdump -i veth5a arp or icmp

Two IP subnets on top of a single broadcast domain.

First of all, the demo demonstrates that it's indeed possible to have multiple IP subnets over a shared L2 segment. However, it also shows that there is a lack of isolation and the traffic destined to one of the subnets might be visible by the nodes from another subnet. It happens because nodes from both subnets share the same broadcast domain. This may be undesirable, in particular, due to security concerns. In such a situation, configuring multiple VLANs on the shared bridge can bring the proper isolation.

To clean up, just remove the created network namespaces:

$ sudo ip netns delete netns_br0
$ sudo ip netns delete netns_veth0
$ sudo ip netns delete netns_veth1
$ sudo ip netns delete netns_veth2
$ sudo ip netns delete netns_veth3
$ sudo ip netns delete netns_veth4
$ sudo ip netns delete netns_veth5

Further Reading

• Computer Networking Introduction: Ethernet and IP (Heavily Illustrated)
• Bridge vs Switch: What I Learned From a Data Center Tour
• Container Networking Is Simple!
• Networking Lab: Ethernet Broadcast Domains
• Networking Lab: Simple VLAN
• Networking Lab: Proxy ARP
• Networking Lab: Simple VXLAN