Redundancy of communication channels: Part 3 — Between offices: VPN, MPLS, dark fiber and SD-WAN

When a company has multiple offices, warehouses or stores, they need connectivity between them: a shared accounting system, a file server, telephony, video surveillance. If a branch loses connection with the central office, it often cannot work at all. There are several ways to connect sites, and they offer different levels of failure protection.

Ways to connect offices

VPN over the Internet

The most common and cheapest option: each office is connected to the Internet, and an encrypted tunnel is established between sites.

• IPsec — a standard protocol that is supported by virtually all routers and firewalls. Suitable when equipment from different vendors is used at the sites.
• WireGuard — a protocol that is simpler to configure with a small codebase, built into the Linux kernel and supported, for example, by MikroTik RouterOS 7.

Minimal WireGuard configuration for a tunnel between office A (network 10.1.0.0/24) and office B (network 10.2.0.0/24), side A:

[Interface]
Address = 10.255.0.1/30
ListenPort = 51820
PrivateKey = <private key of office A>

[Peer]
PublicKey = <public key of office B>
Endpoint = 198.51.100.20:51820
AllowedIPs = 10.255.0.2/32, 10.2.0.0/24
PersistentKeepalive = 25

The configuration on side B is mirrored. Packet forwarding must be enabled on both gateways (net.ipv4.ip_forward = 1), and the firewall must allow traffic between the local network and the tunnel.

The VPN tunnel itself is not redundant: it depends on the Internet links of both sites. Redundancy is provided by two tunnels through different providers and a mechanism that selects the working one.

MPLS L3VPN from the provider

The service provider builds an isolated virtual network for the company on its infrastructure. The technology is described in RFC 4364 (BGP/MPLS IP VPN). For the client, this looks like a single network between offices: routing inside it is performed by the provider; the client hands over its routes to the provider or simply specifies which networks are located where.

Pros: predictable latency, a contract with guaranteed availability, traffic does not go over the public Internet. Cons: it’s more expensive than VPN over the Internet, provisioning takes a long time, and the whole network depends on a single provider. For redundancy, MPLS is often supplemented with a VPN tunnel over the Internet from another provider.

Dark fiber

The company leases the optical fiber entirely and installs its own equipment at both ends. This provides very high speed and full control, but is only available where spare fibers exist: typically within a city or between adjacent buildings. One fiber is one route, so for reliability you need a second line via a different route or a backup VPN.

SD-WAN

SD-WAN is a control layer over multiple links: Internet, MPLS, mobile. Devices at sites constantly measure latency, loss and availability of each link, and a central console sets policies: telephony — over the link with the lowest latency, backups — over the cheapest. SD-WAN solutions are mostly commercial and require equipment from a single vendor at all sites. For networks of two or three offices, usually two VPN tunnels and dynamic routing are enough.

How to switch between links

Having two tunnels is not enough: the router must detect that one of them is down and shift the traffic.

• Reachability checks and static routes. The router checks an address on the far side of the tunnel and, on failure, removes the route through it. Simple, but with many networks the configuration grows large.
• Dynamic routing (OSPF or BGP) over the tunnels. Routers exchange routes themselves and select an available path. This scales better: adding a network in one office automatically becomes known to others. Practical examples — “BGP on Keenetic via Entware and FRR” and “BGP and OSPF on Keenetic”.

With two providers at a site there is another nuance: the reply to an incoming connection must go out via the same provider through which the request arrived, otherwise the connection will not be established. How to configure this is discussed in the article “MikroTik: return traffic via the same gateway it came through”.

Gateway redundancy: VRRP

If an office has a single router, its failure disables the site regardless of the number of links. The solution is two routers and a shared virtual gateway address that office computers use as the default.

• VRRP — an open standard. The current version, VRRP version 3, is described in RFC 9568 (April 2024) and works with IPv4 and IPv6. Routers elect a master; if it stops sending advertisements (by default once per second), the virtual address moves to the backup.
• HSRP — Cisco’s analogous protocol, working only on its equipment.

VRRP is supported by MikroTik, most enterprise routers and Linux (keepalived). When layer-2 connectivity between two MikroTik devices is needed and where schemes other than VRRP may be appropriate, see the article “Virtual address between two MikroTik”.

What to check when building

• Links at each office are truly independent: different providers, different routes and technologies if possible.
• The office router is not the only single point of failure: there is a second one with VRRP or at least a spare with a ready configuration.
• The failover time is known and acceptable for telephony and the accounting system.
• Tunnel failures are covered by monitoring, not discovered through user complaints.

Common mistakes

• Two tunnels through the same provider. If the provider has an outage, both disappear.
• Identical address spaces in offices (for example, 192.168.1.0/24 everywhere). When merging networks you have to renumber or implement address translation.
• A tunnel without liveness checks. The WireGuard interface may be technically up while traffic does not flow. Without checks and dynamic routing the router won’t know about it.
• MPLS without redundancy relying on “the provider guarantees it.” The contract gives compensation, not connectivity during an outage.