Handing off data-centre IPs to servers behind two consumer ISPs

My servers sit in my own room, fed by two ordinary broadband lines. The public addresses I want on the VMs are routed to me in two data centres, where I keep one small router each. Something has to carry those addresses from the data centres to the VMs, keep working when an ISP has a bad day, and do it without NAT.

This post is the whole of that something, written down from the running configuration. Every hop in it is routed, including the ones that cross the internet.

The cast

Device	Where	What it does
`core`	My server room	MikroTik CCR2216. Terminates both ISPs and all four tunnels, holds every VLAN gateway, the firewall and the policy routing.
`dc1-gw`	DC1	MikroTik RB5009 on a 10G uplink. DC1 gives it a /30 for its own WAN and routes a second /30, `192.0.2.4/30`, to it. That second block is what gets handed off.
`dc2-gw`	DC2	RouterOS CHR, a router running as a VM. It has one address of its own and three more /32s, `198.51.100.3` to `.5`, routed to it.
CRS518	My server room	Switch between the core and the Proxmox hosts, on a 100G trunk. Layer 2 only: it carries VLANs and routes nothing.

The two lines into the server room are not alike, and most of the design follows from that.

ISP-A is 1 Gbit/s over PPPoE with a static address. Its BNG supports RFC 4638, so the PPPoE session runs at a full 1500-byte MTU with the ethernet port underneath set to 1508.
ISP-B is 500 Mbit/s through an ONT that insists on doing NAT. The core sits in the ONT’s DMZ on a private address. The public address in front of it is static.

Each data centre reaches the core through two tunnels that never share an ISP. Solid lines ride ISP-A, dashed lines ride ISP-B.

Why routed and not bridged

The common way to put a remote public address on a VM is to stretch layer 2 across a tunnel and bridge it at both ends. I went the other way for two reasons.

I have two tunnels per data centre, on purpose, over different ISPs. Put two layer 2 paths into one bridge and you have built a loop that needs bonding or spanning tree to keep it quiet, running over tunnels. Two routed paths are two OSPF neighbours with a cost each.

With routing, the firewall and the shaping for every public address also sit in one place, on the core, while each VM still holds its public IP directly on its own interface.

The price is that the data centre has to route the block to my router instead of putting it on a shared segment, and the network config inside the VM looks slightly odd. Both are covered below.

Four tunnels, each pinned to one ISP

Tunnel	Between	Rides on	Type	MTU
`gre-dc1`	core and dc1-gw	ISP-A	GRE inside IPsec transport mode, IKEv2	1434
`wg-dc1`	core and dc1-gw	ISP-B	WireGuard	1400
`wg-dc2`	core and dc2-gw	ISP-A	WireGuard	1440
`l2tp-dc2`	core and dc2-gw	ISP-B	L2TP over IPsec, NAT-T	1404

The mix of tunnel types is partly history. The one hard rule is that the two tunnels to a data centre never share an underlay, so one ISP failing leaves one tunnel to each data centre standing.

A tunnel that simply follows the default route is no use as a backup, because when the default route moves, the “backup” moves with it. So each tunnel is nailed to its ISP with a small routing table of its own:

/routing table add name=via-isp-a fib
/routing table add name=via-isp-b fib

/ip route add routing-table=via-isp-a dst-address=0.0.0.0/0 gateway=pppoe-isp-a
/ip route add routing-table=via-isp-b dst-address=0.0.0.0/0 gateway=192.168.1.254%sfp-isp-b check-gateway=ping
/ip route add routing-table=via-isp-b dst-address=192.168.1.0/24 gateway=sfp-isp-b

# WireGuard to DC2 always leaves through ISP-A, WireGuard to DC1 always through ISP-B
/ip firewall mangle add chain=output protocol=udp dst-address=198.51.100.2 dst-port=<redacted> action=mark-routing new-routing-mark=via-isp-a passthrough=no
/ip firewall mangle add chain=output protocol=udp dst-address=192.0.2.2 dst-port=<redacted> action=mark-routing new-routing-mark=via-isp-b passthrough=no

# anything sourced from the ONT-side address stays on ISP-B (this is what steers the L2TP client)
/routing rule add src-address=192.168.1.2/32 table=via-isp-b action=lookup-only-in-table

The GRE tunnel needs no mark. Its local address is the ISP-A static address, and the main table’s default route already points at ISP-A.

Two details in that block cost me an evening each.

The connected route for 192.168.1.0/24 inside via-isp-b looks redundant. Without it, check-gateway=ping on the default route deadlocks: the ping to the ONT is sourced from 192.168.1.2, the source rule sends it to via-isp-b only, and that table has no way to reach the gateway until the route that depends on the ping comes up.

WireGuard behind the ONT’s NAT works only in one direction of setup. In my testing RouterOS sends its own WireGuard packets without passing them through connection tracking, so a source NAT rule never touches them. The fix is to let the data-centre end dial in: dc1-gw has the ISP-B public address as its peer endpoint, the ONT forwards the packet to the core, and the core answers from the address the handshake arrived on.

The MTU arithmetic

Every tunnel MTU is the largest value that still fits the line underneath without fragmenting.

GRE in IPsec over ISP-A: 1434. The encrypted packet comes out at 1496 bytes. One more AES block would make it 1512, over the line’s 1500.
WireGuard over ISP-A: 1440. WireGuard adds 60 bytes on IPv4, so the outer packet is exactly 1500.
WireGuard over ISP-B: 1400. By the same sum 1440 should fit, and a 1500-byte ping with the don’t-fragment bit set passes. Real traffic at that size crawled anyway. At 1400 the tunnel runs at line rate, so 1400 it is.
L2TP in IPsec over ISP-B: 1404. NAT traversal wraps ESP in UDP, which costs 8 more bytes. At 1404 the outer packet is 1488. At 1408 it tips into the next cipher block and fragments.

OSPF and BFD across the tunnels

The three routers form one OSPF area. Each has a loopback, every tunnel is a point-to-point OSPF interface with a 5-second hello and a 20-second dead timer, and BFD runs on top at 200 ms with a multiplier of 5, so a dead tunnel is noticed in about a second.

The cost printed at each end of a link is what that router pays to send traffic into it. Lower wins.

For DC2 the costs are the same at both ends: 10 for WireGuard over ISP-A, 200 for the L2TP backup over ISP-B. Traffic uses WireGuard in both directions unless it is down.

DC1 is asymmetric on purpose. The core sends towards dc1-gw over WireGuard on ISP-B (10 beats 20). dc1-gw sends towards the core over GRE on ISP-A (5 beats 10). I tested both arrangements with real downloads and uploads, and each direction had a clear winner on a different line. OSPF costs are set per interface and per router, so splitting the directions costs nothing. Both tunnels end on the same two routers, which means the stateful firewall on the core still sees both halves of every connection.

# core
/routing ospf instance add name=edge router-id=10.255.0.1 redistribute=static out-filter-chain=ospf-out
/routing ospf area add name=backbone instance=edge
/routing ospf interface-template add area=backbone interfaces=lo passive
/routing ospf interface-template add area=backbone interfaces=gre-dc1 type=ptp cost=20 hello-interval=5s dead-interval=20s use-bfd=yes
/routing ospf interface-template add area=backbone interfaces=wg-dc1 type=ptp cost=10 hello-interval=5s dead-interval=20s use-bfd=yes
/routing ospf interface-template add area=backbone interfaces=wg-dc2 type=ptp cost=10 hello-interval=5s dead-interval=20s use-bfd=yes
/routing ospf interface-template add area=backbone interfaces=l2tp-dc2 type=ptp cost=200 hello-interval=5s dead-interval=20s use-bfd=yes

/routing bfd configuration add interfaces=gre-dc1 min-rx=200ms min-tx=200ms multiplier=5
/routing bfd configuration add interfaces=wg-dc1 min-rx=200ms min-tx=200ms multiplier=5

The WireGuard peers all carry allowed-address=0.0.0.0/0. WireGuard’s own idea of routing is switched off that way, and OSPF decides what goes into each tunnel.

What the core announces

The data-centre routers announce only their loopbacks. The core announces the public addresses, and a filter makes sure nothing else leaks out with them:

/routing filter rule add chain=ospf-out rule="if (dst==198.51.100.3/32 && dst-len==32) { accept }"
/routing filter rule add chain=ospf-out rule="if (dst==198.51.100.4/32 && dst-len==32) { accept }"
/routing filter rule add chain=ospf-out rule="if (dst==198.51.100.5/32 && dst-len==32) { accept }"
/routing filter rule add chain=ospf-out rule="if (dst==192.0.2.4/30 && dst-len==30) { accept } reject"

So dc1-gw learns “192.0.2.4/30 lives behind the core” from OSPF, over whichever tunnel is cheapest for it at that moment. Nobody maintains static routes for the public blocks on the data-centre side.

The handoff: a VLAN, a /32 gateway and host routes

On the core, each public block gets a VLAN. The gateway address on that VLAN is a private /32, and every public address is a /32 host route pointing at the VLAN interface:

/interface vlan add name=vlan1207-dc1-pub interface=trunk vlan-id=1207
/ip address add address=10.20.7.1/32 interface=vlan1207-dc1-pub

/ip route add dst-address=192.0.2.4/32 gateway=vlan1207-dc1-pub
/ip route add dst-address=192.0.2.5/32 gateway=vlan1207-dc1-pub
/ip route add dst-address=192.0.2.6/32 gateway=vlan1207-dc1-pub
/ip route add dst-address=192.0.2.7/32 gateway=vlan1207-dc1-pub

# the aggregate that OSPF redistributes; real traffic always matches a /32 above
/ip route add blackhole dst-address=192.0.2.4/30 distance=250

Because nothing on the VLAN is configured as a subnet, there is no network address, no broadcast address and no gateway eating into the block. All four addresses of the /30 belong to VMs.

Inside the VM, the public address is a /32 and the gateway is declared on-link:

# /etc/netplan/50-public.yaml
network:
  version: 2
  ethernets:
    ens18:
      addresses: [192.0.2.5/32]
      routes:
        - to: default
          via: 10.20.7.1
          on-link: true
      nameservers:
        addresses: [1.1.1.1]

Nothing translates the VM’s address on the way in or on the way out.

Sending replies out through the right data centre

Left alone, the core would send a VM’s outbound packets through its own default route, straight out of ISP-A with a source address that ISP-A has never heard of. They would be dropped at the first router that checks. Traffic sourced from a public block has to leave through the data centre that owns it.

/routing table add name=to-dc1 fib

/ip firewall mangle add chain=prerouting src-address=192.0.2.4/30 dst-address-list=!LOCAL-NETS action=mark-routing new-routing-mark=to-dc1 passthrough=no

# default route = "wherever OSPF says dc1-gw's loopback is right now"
/ip route add routing-table=to-dc1 dst-address=0.0.0.0/0 gateway=10.255.0.11@main target-scope=30
# static fallbacks if OSPF itself is gone but a tunnel still answers
/ip route add routing-table=to-dc1 dst-address=0.0.0.0/0 gateway=10.0.2.1 distance=30 check-gateway=ping
/ip route add routing-table=to-dc1 dst-address=0.0.0.0/0 gateway=10.0.1.1 distance=40 check-gateway=ping

/routing rule add routing-mark=to-dc1 table=to-dc1 action=lookup-only-in-table

# and no NAT for the block, ahead of the masquerade rules
/ip firewall nat add chain=srcnat src-address=192.0.2.4/30 action=accept

The first route is the part I like most. Its gateway is the loopback of dc1-gw, resolved in the main table, where OSPF keeps it pointed at the best tunnel. When a tunnel dies and OSPF reconverges, this route follows by itself. The policy routing never has to know which tunnel is up.

LOCAL-NETS is an address list of everything that lives at the core: the internal VLANs and the public blocks themselves. Traffic from a public VM to anything on that list skips the mark and is routed locally, so two VMs in different blocks talk across the core and not across two data centres.

DC2 uses the same mark and the same kind of table, pointed at the loopback of dc2-gw.

The lookup-only-in-table rule makes the DC1 table fail closed. If every route in it is gone, the packets are dropped. They never fall back to the main table and wander out of a broadband line wearing a data-centre address.

The matching safety net in the data centre

DC1’s upstream routes the /30 to dc1-gw, and dc1-gw forwards it on because OSPF says so. If both tunnels drop, the OSPF route disappears, and without help the packets would follow the default route back to the upstream, which would send them to dc1-gw again. One line stops the loop:

# dc1-gw
/ip route add blackhole dst-address=192.0.2.4/30 distance=250

At distance 250 it loses to OSPF’s 110 whenever a tunnel is up, and catches everything when none is.

One MSS for both tunnels

The core clamps TCP MSS on everything entering or leaving the tunnels. For DC1 the clamp is a fixed number:

/ip firewall mangle add chain=forward out-interface=gre-dc1 protocol=tcp tcp-flags=syn tcp-mss=1361-65535 action=change-mss new-mss=1360
/ip firewall mangle add chain=forward in-interface=gre-dc1 protocol=tcp tcp-flags=syn tcp-mss=1361-65535 action=change-mss new-mss=1360
/ip firewall mangle add chain=forward out-interface=wg-dc1 protocol=tcp tcp-flags=syn tcp-mss=1361-65535 action=change-mss new-mss=1360
/ip firewall mangle add chain=forward in-interface=wg-dc1 protocol=tcp tcp-flags=syn tcp-mss=1361-65535 action=change-mss new-mss=1360

I used clamp-to-pmtu here at first and it bit me. With asymmetric routing the SYN leaves through one tunnel and the SYN-ACK returns through the other, and each gets clamped to the MTU of the tunnel it happened to cross. The two ends then agree on an MSS that fits one tunnel and not the other, and one direction fragments. Uploads from a VM dropped to around a tenth of what the line could do. The rule I took from it: when tunnels have different MTUs and a connection can cross both, clamp to the smallest one, as a fixed value. 1360 is the 1400-byte WireGuard MTU minus 40 bytes of IP and TCP headers.

Firewall and shaping

The public blocks are closed by default. On the core, each block gets a short list of explicit allows and then a drop for anything new arriving from outside:

/ip firewall filter add chain=forward dst-address=192.0.2.4/30 protocol=icmp action=accept
/ip firewall filter add chain=forward dst-address=192.0.2.4 protocol=tcp dst-port=80,443 action=accept
/ip firewall filter add chain=forward dst-address=192.0.2.4/30 connection-state=new in-interface-list=!LAN action=drop

Each block also has one fq_codel queue set at or a little under the rate of the line it normally uses. It matches only traffic between the block and the internet, so VM-to-VM traffic inside the room is never shaped.

What fails, and what happens next

What breaks	DC1 block	DC2 block
ISP-A	GRE drops. `dc1-gw` switches its sending to WireGuard on ISP-B. The core was already sending that way.	WireGuard drops. Both directions move to L2TP on ISP-B.
ISP-B	WireGuard drops. The core switches its sending to GRE on ISP-A. `dc1-gw` was already sending that way.	Nothing moves. Only the backup is lost.
Both tunnels to one data centre	Offline. The core drops the block’s outbound traffic and `dc1-gw` blackholes the inbound.	Offline until a tunnel returns.
A data-centre router or its upstream	That block is offline.	The other block is unaffected.

Where BFD is up, the switch takes about a second. BFD over the L2TP backup has been unreliable for me: after a redial the session can sit in init while OSPF across the same tunnel is perfectly healthy. Until that is solved, the 20-second OSPF dead timer is the real failover time for that one path.

Following one packet each way

A request from the internet to a VM on 192.0.2.5:

DC1’s upstream has a route for 192.0.2.4/30 towards dc1-gw and delivers the packet there.
dc1-gw looks up the /30, finds the OSPF route and sends the packet into gre-dc1, its cheaper tunnel.
The packet crosses the internet inside IPsec and comes out on the core.
The core’s forward chain checks it against the allows for that block.
The /32 host route for 192.0.2.5 points at VLAN 1207. The core ARPs for the address and hands the packet to the VM.

The reply:

The VM sends it to its gateway, 10.20.7.1, with 192.0.2.5 as the source.
In prerouting, the core sees a source inside the DC1 block and a destination that is not local, and marks the packet for the to-dc1 table.
The default route in to-dc1 resolves through the loopback of dc1-gw, which OSPF currently reaches over wg-dc1.
The packet crosses the internet inside WireGuard and comes out on dc1-gw.
dc1-gw forwards it to DC1’s upstream by its ordinary default route. The source address was never touched.

The request came in over ISP-A and the reply went out over ISP-B, and neither end of the connection notices.

Things that bit me

Two kinds of IPsec between the same two addresses

I once ran L2TP/IPsec (IKEv1) and GRE/IPsec (IKEv2) between the same pair of public IPs. The routers negotiated the GRE traffic selector inside the L2TP security association, a duplicate policy appeared, the real one went invalid and the L2TP session died. Each address pair now carries only one kind of IPsec.

A passing ping

The 1500-byte DF ping across ISP-B succeeded while full-size WireGuard packets on the same path were being mishandled, so I now test MTU with the traffic I actually care about.

Bandwidth tests that under-report loss

Chasing a slow download, the router’s own bandwidth test reported a loss figure about a tenth of the real one. Counting packets at both ends gave the real number: IPsec SA counters on each side, WireGuard interface counters on each side, or a temporary passthrough rule in mangle.

The second blackhole

The loop described above is easy to miss, because it only exists while the tunnels are down, which is exactly when nobody is looking at traceroutes. A routed block needs a blackhole at both ends.

How fast is it

Router to router, the IPsec-wrapped GRE over the 1 Gbit/s line moved roughly 700 Mbit/s of TCP with no CPU core pinned on either router. The two tunnels on ISP-B run at that line’s 500 Mbit/s once their MTUs are right.

The short version

Four tunnels carry the packets: two per data centre, one on each ISP.
OSPF with BFD picks the tunnel, separately for each direction.
/32 host routes on a VLAN put the public address on the VM itself, with no NAT and no wasted addresses.
Policy routing sends each block’s replies back through the data centre it came from, and fails closed.

If you run something similar and spot a mistake in mine, I would like to hear about it.