Introduction

In a previous post, I introduced VyOS and Vultr and teased follow-up posts that would show the deployment of VyOS as an edge router and perimeter firewall in front of an internal network, which would amount to a novel solution for the VPS space, where nothing like that is typically considered.

The technical work is complete to proof-of-concept standards. My personal domain’s core services are now self-operated by me, at Vultr, behind paired VyOS edge routers, with firewalling, network segmentation, dual-stack IPv4 & IPv6, and DNSSEC signing. This includes authoritative DNS, email, and web service for this blog.

In each of my next several posts, I’ll pick one aspect of the solution and deep dive it, in order to stay true to my IT philosophy and my plan (see my “About” page) for this blog:

Just about anybody can slam something in quickly and haul butt. That’s not how to succeed in IT, and that’s not where I’m at in this stage of life. I want to set myself apart by doing it well, and by using the blog to document along the way.

Topics will include redundancy, firewall, network segmentation (WAN, DMZ and intranet), remote access VPN, dual-stack IPv4 and IPv6, provisioning automation, configuration management, logging, monitoring and alerting. Beyond the VyOS network devices themselves, afterward, I’m apt to move on to talk about the core Internet services, and how I operate these within the DMZ, in a Linux environment.

Today’s topic

The first deep dive topic centers around redundancy and fail-over. I will start with some background and guiding principles, then focus in on network device redundancy generally, talk about where VyOS and Vultr come in on this, then select, implement, and validate my solution for VyOS HA in Vultr.

Background

In professional IT, going to production involves addressing redundancy. You will see terms like active/active, active/standby, manual and automatic fail-over, and you will encounter redundancy deployed both within the data center as well as schemes involving two or more data centers.

Engineering for redundancy adds cost and introduces complexity, all of which must be weighed, prioritized, and expertly balanced.

Dos and Don’ts:

  1. Do have an idea of the SLAs and SLOs to be hit before proceeding to design.
  2. Do brainstorm likely failure scenarios to address in your initial design.
  3. Do pay special attention to failure modes where a server is merely off in the weeds, or has lost connectivity to a backend, not just the case where the server dies outright.
  4. Do test and validate your fail-over mechanisms to the best of your ability.
  5. Do incorporate alerting to be notified of the problem: if fail-over works, you won’t see an outage.
  6. Do understand and document any caveats of the failover mechanism, such as degraded UI, or if users will need to log back in. Be up-front with stakeholders.

  7. Do postmortems to discuss what worked, what didn’t, and where you find room to improve, do so.

  8. Don’t obsess over the first-pass design. What’s important is that you have some HA story to go to market. The rest will come from hard-won experience. “Why didn’t the fail-over work?” is a question executives will ask. “It was an edge case and we’re fixing it.”
  9. Don’t think about how clever you can look today. Think 9 months down the line when whatever you put in today will need to be manipulated and leveraged during an emergency. Can you be back up to speed with it in 5 minutes, and teach it to somebody else in 10 minutes?
  10. Don’t allow in excess complexity to the point it becomes counter-productive. You can trip over your own feet, causing the outage you had aimed to prevent.

Flashy isn’t the goal; adopting a solution that meets your company’s needs and that your team can effectively implement and manage is.

Network device redundancy

Enterprise-grade network gear has a reputation of being extremely reliable. Failures are not expected to occur within the useful life of a device. Because both the up-front and ongoing maintenance costs for enterprise network devices are considerable, and because these devices have to compete for attention with other, more-likely failure points in contingency planning, not all projects will deploy redundant network hardware.

That said, the failure of a core piece of network gear, apart from a redundant counterpart, will mean at least hours of site downtime, especially when equipment is placed in third-party data centers where external partners, namely remote-hands technicians and vendor service personnel, are relied upon to respond.

What is more, a secondary device stands by not only for the failure of the primary device, but also to cover during maintenance, permitting advisory-only (“no impact is anticipated”) maintenance window notifications. Finally, when unexpected problems do occur with maintenance, these can oftentimes be detected before progressing to the secondary firewall, avoiding outages in scenarios of bad updates or configuration mistakes.

Today’s challenge

The Virtual Router Redundancy Protocol (VRRP) became a standard approach for ensuring network redundancy in production environments during the mid-2000s. Through VRRP, primary and secondary routers use heartbeats to check each other’s status, facilitating automatic fail-over to ensure continuous network operation and redundancy.

Border Gateway Protocol (BGP) can work in tandem with VRRP to manage WAN IP addressing across edge routers. With VRRP establishing primary and secondary roles for routers, ensuring internal network redundancy, BGP handles the external routing, allowing the WAN IP to “float” or transition seamlessly between the edge routers during fail-over events.

The VyOS manual’s chapter on HA is built around VRRP entirely, and Vultr’s KB has an article called “High Availability on Vultr with Floating IP and BGP” with sample configuration that we should be able to port to VyOS, since VyOS will definitely have robust support for BGP.

Architecture of the solution

Each edge router will have a dedicated external IP unique to it, but my domain’s core Internet services will be advertised on separate floating “service” IP addresses that are routed to my edge devices on their primary IPs using BGP ECMP/anycast, but with my primary edge device prioritized using AS path prepend.

I will have two public IPs (a primary and a secondary) for each of my three core services (DNS, SMTP, HTTPS) for a total of six floating public IPs. I will prioritize the same edge router in the VRRP configuration and hope that BGP and VRRP stay in sync; roughly, they should. I will also use a conntrack sync mechanism in attempt to cover some of the gray area and to avoid session drops during failover.

VRRP is a first hop redundancy protocol, and does not address whether the active router is actually capable of forwarding packets further. A dynamic interior protocol like OSPF could potentially do a better job to integrate with BGP, but I don’t have a comfort level to push anything like that to production, and all of the community documentation, which I will rely on, says VRRP. I would rather accept risk of an unlikely-to-encounter edge case for nonconvergence than to put in a too-complex solution that I’m not the master of: it would be a security risk and counter-productive to stability.

OSPF is something I would love to play with in the near future and I will do that in a lab environment where I plan to operate a routing core in addition to edge routers.

BGP peering with Vultr

BGP prefix-lists

set policy prefix-list VULTR-NJ-v4 rule 10 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 10 prefix '45.76.4.167/32'
set policy prefix-list VULTR-NJ-v4 rule 20 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 20 prefix '45.76.6.22/32'
set policy prefix-list VULTR-NJ-v4 rule 30 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 30 prefix '45.76.10.33/32'
set policy prefix-list VULTR-NJ-v4 rule 40 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 40 prefix '45.76.6.7/32'
set policy prefix-list VULTR-NJ-v4 rule 50 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 50 prefix '45.76.6.121/32'
set policy prefix-list VULTR-NJ-v4 rule 60 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 60 prefix '45.76.11.196/32'
set policy prefix-list VULTR-NJ-v4 rule 70 action 'permit'
set policy prefix-list VULTR-NJ-v4 rule 70 prefix '45.63.21.196/32'
set policy prefix-list6 VULTR-NJ-v6 rule 10 action 'permit'
set policy prefix-list6 VULTR-NJ-v6 rule 10 prefix '2001:19f0:5:416::/64'
set policy prefix-list6 VULTR-NJ-v6 rule 20 action 'permit'
set policy prefix-list6 VULTR-NJ-v6 rule 20 prefix '2001:19f0:1000:6946::/64'
set policy prefix-list6 VULTR-NJ-v6 rule 30 action 'permit'
set policy prefix-list6 VULTR-NJ-v6 rule 30 prefix '2001:19f0:5:34cd::/64'

BGP route maps (don’t be that guy)

set policy route-map 64515v4-IN rule 10 action 'deny'
set policy route-map 64515v4-OUT rule 10 action 'permit'
set policy route-map 64515v4-OUT rule 10 match ip address prefix-list 'VULTR-NJ-v4'
set policy route-map 64515v6-IN rule 10 action 'deny'
set policy route-map 64515v6-OUT rule 10 action 'permit'
set policy route-map 64515v6-OUT rule 10 match ipv6 address prefix-list 'VULTR-NJ-v6'

Private AS peer to Vultr

set protocols bgp 4288000595 address-family ipv4-unicast network 45.63.21.196/32
set protocols bgp 4288000595 address-family ipv4-unicast network 45.76.4.167/32
set protocols bgp 4288000595 address-family ipv4-unicast network 45.76.6.7/32
set protocols bgp 4288000595 address-family ipv4-unicast network 45.76.6.22/32
set protocols bgp 4288000595 address-family ipv4-unicast network 45.76.6.121/32
set protocols bgp 4288000595 address-family ipv4-unicast network 45.76.10.33/32
set protocols bgp 4288000595 address-family ipv4-unicast network 45.76.11.196/32
set protocols bgp 4288000595 address-family ipv6-unicast network 2001:19f0:5:34cd::/64
set protocols bgp 4288000595 address-family ipv6-unicast network 2001:19f0:5:416::/64
set protocols bgp 4288000595 address-family ipv6-unicast network 2001:19f0:1000:6946::/64
set protocols bgp 4288000595 neighbor 169.254.169.254 address-family ipv4-unicast nexthop-self force
set protocols bgp 4288000595 neighbor 169.254.169.254 address-family ipv4-unicast remove-private-as
set protocols bgp 4288000595 neighbor 169.254.169.254 address-family ipv4-unicast route-map export '64515v4-OUT'
set protocols bgp 4288000595 neighbor 169.254.169.254 address-family ipv4-unicast route-map import '64515v4-IN'
set protocols bgp 4288000595 neighbor 169.254.169.254 ebgp-multihop '2'
set protocols bgp 4288000595 neighbor 169.254.169.254 password 'redactedP@ssw0rd'
set protocols bgp 4288000595 neighbor 169.254.169.254 remote-as '64515'
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 address-family ipv6-unicast nexthop-self force
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 address-family ipv6-unicast remove-private-as
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 address-family ipv6-unicast route-map export '64515v6-OUT'
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 address-family ipv6-unicast route-map import '64515v6-IN'
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 ebgp-multihop '2'
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 password 'redactedP@ssw0rd'
set protocols bgp 4288000595 neighbor 2001:19f0:ffff::1 remote-as '64515'
set protocols bgp 4288000595 parameters router-id '45.76.0.255'

VRRP configuration inside

set high-availability vrrp group vpc-nj-dmz-v4vip interface 'eth1'
set high-availability vrrp group vpc-nj-dmz-v4vip priority '200'
set high-availability vrrp group vpc-nj-dmz-v4vip virtual-address 10.76.2.1/24
set high-availability vrrp group vpc-nj-dmz-v4vip vrid '21'
set high-availability vrrp group vpc-nj-dmz-v6vip interface 'eth1'
set high-availability vrrp group vpc-nj-dmz-v6vip priority '200'
set high-availability vrrp group vpc-nj-dmz-v6vip virtual-address 2001:19f0:5:416::1/64
set high-availability vrrp group vpc-nj-dmz-v6vip vrid '22'
set high-availability vrrp group vpc-nj-intranet-v4vip interface 'eth2'
set high-availability vrrp group vpc-nj-intranet-v4vip priority '200'
set high-availability vrrp group vpc-nj-intranet-v4vip virtual-address 10.76.4.1/24
set high-availability vrrp group vpc-nj-intranet-v4vip vrid '41'
set high-availability vrrp group vpc-nj-intranet-v6vip interface 'eth2'
set high-availability vrrp group vpc-nj-intranet-v6vip priority '200'
set high-availability vrrp group vpc-nj-intranet-v6vip virtual-address 2001:19f0:5:34cd::1/64
set high-availability vrrp group vpc-nj-intranet-v6vip vrid '42'
set high-availability vrrp sync-group MAIN member 'vpc-nj-dmz-v4vip'
set high-availability vrrp sync-group MAIN member 'vpc-nj-dmz-v6vip'
set high-availability vrrp sync-group MAIN member 'vpc-nj-intranet-v4vip'
set high-availability vrrp sync-group MAIN member 'vpc-nj-intranet-v6vip'

conntrack sync

set service conntrack-sync failover-mechanism vrrp sync-group 'MAIN'
set service conntrack-sync interface eth1
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp

Under construction

Please excuse that I’m publishing this article before it’s totally finished. Right now I need to be able to show it as a work in progress. The configuration dump presented here does work fully and I plan to add more step-by-step explanation as well as a section detailing the validation. The configuration dump given is for the primary device; the secondary device’s configuration follows from that; I believe it is just the primary external and internal IPs that get substituted.