You deployed a load balancer so that any backend server could fail without anyone noticing. Traffic flows around the failure; health checks quietly remove the dead server; users see nothing. Excellent.
Now ask the uncomfortable question: what happens when the load balancer itself fails?
If the answer is "everything behind it goes dark", your carefully balanced pool has a single point of failure sitting right at its front door. A power supply, a kernel panic, or an unlucky reboot takes out every service at once. The fix is an active-passive high availability pair — and understanding how it works is the difference between configuring it confidently and cargo-culting it.
The architecture: two nodes, floating IPs
A Tula HA pair consists of two appliances: one active (master) handling all traffic, one passive (backup) standing by. The mechanism that binds them is VRRP — the Virtual Router Redundancy Protocol — implemented through keepalived.
The key concept is the floating IP. Clients never connect to either node’s physical address; they connect to virtual IPs that are shared between the pair. Under normal operation the active node holds these addresses and answers for them. When the active node fails, VRRP reassigns the floating IPs to the passive node — and because clients only ever knew the floating address, no client-side reconfiguration is needed. From the outside, the load balancer just kept working.
Each node keeps its own independent identity — hostname, physical interfaces, management IP — while sharing a common cluster configuration: VIPs, backend definitions, SSL certificates, health checks, the lot.
Both nodes must agree on the configuration
A standby is only useful if it can actually serve the traffic. Tula uses csync2 to synchronise the cluster configuration between nodes: change something on the active node, and the update propagates to the passive node automatically. That covers the HAProxy and nftlb configurations, SSL certificates, SNMP settings, and health check definitions. Node-specific settings — hostnames, interface assignments — stay local by design.
This matters more than it sounds. A depressingly common HA failure mode elsewhere is a standby that was perfect six months ago and has silently drifted: a certificate renewed only on the primary, a VIP added in a hurry. Automatic synchronisation removes that entire category of surprise.
What actually happens during failover
Failover is triggered by silence. The active node multicasts VRRP advertisements every second; the passive node listens. The sequence:
- The active node suffers a critical failure — hardware fault, kernel panic, network partition, or service crash.
- The advertisements stop. After three missed advertisements, the passive node concludes the master is gone.
- The passive node takes over. It transitions to the master state and sends a gratuitous ARP announcement, telling upstream switches and routers that the floating IPs now live at its MAC address.
- Traffic resumes. The floating IPs are active on the new master’s interfaces, and it begins accepting connections immediately.
Total time from failure to traffic flowing again is typically under three seconds.
Tula also monitors its own critical services — HAProxy, nftlb, keepalived — on each node, and reduces the local VRRP priority if one fails. So failover protects you not only from a dead machine but from a sick one: if the proxy process on the active node crashes, traffic moves to the healthier node even though the box itself is still up.
The split-brain problem
The nightmare scenario for any HA pair is split-brain: both nodes believing they are the master, both claiming the floating IPs, and the network descending into duplicate-address chaos.
Tula defends against this in layers. Both nodes share a VRRP virtual router ID and authenticate their advertisements, ensuring they participate in the same VRRP group rather than talking past each other. Keepalived’s preemption setting controls whether a recovered primary reclaims the master role, preventing failover oscillation. And the service-health priority adjustment ensures that when both nodes are reachable but one is degraded, traffic settles on the healthier one.
Your part of the bargain is the network: both nodes must sit on the same Layer 2 segment and be able to exchange VRRP multicast traffic. If a firewall or switch configuration silently eats those advertisements, each node will conclude the other is dead — the classic self-inflicted split-brain.
Setting up a pair
The full walkthrough lives in the HA pair guide, but the shape of it:
- Deploy two Tula nodes on the same Layer 2 network segment, each with its own management IP.
- On the node you have designated primary, navigate to System > High Availability and add the peer node by its management IP. Tula establishes the trust relationship and configures csync2.
- Assign floating IPs — the addresses that will migrate between nodes.
- Set VRRP priorities. Higher priority wins the master role; a common convention is 150 for the primary and 100 for the backup.
- Enable HA. The nodes begin exchanging advertisements, and the dashboard shows one as Master, the other as Backup.
Test it like you mean it
An untested failover is a hypothesis, not a safety net. Once the pair is up:
- Reboot the active node and watch the dashboard: the backup should transition to master, and a continuously running request loop against a VIP should barely stutter.
- Verify failback behaviour when the original primary returns, so you know whether traffic moves back automatically and are not surprised at 3 a.m.
- Re-test after significant changes — new VIPs, interface changes, switch replacements. HA configurations rot quietly.
You can also have the appliance tell you when failover happens: Tula sends SNMP traps on HA state transitions, so your monitoring platform hears about a failover even when users never notice it.
The bottom line
High availability at the load balancing tier is not exotic. Two nodes, VRRP, synchronised configuration, and a floating IP — a design that has protected serious infrastructure for decades, packaged so that the setup is an afternoon’s work rather than a project. If everything behind your load balancer is redundant but the load balancer is not, this is the highest-value afternoon available to you.
The concepts are covered in more depth in the active-passive HA documentation.
