Skip to content
Load Balancing

Reading the Signals: Load Balancer Metrics That Actually Matter

Every request to every service you run passes through your load balancer. That makes it more than a traffic director — it is the single best observation point in your infrastructure, seeing client behaviour, backend health, and application performance in one place, without an agent installed anywhere.

The question is what to actually look at. Tula collects a lot; here is what earns attention, what the patterns mean, and how to get it all into the monitoring platform you already run.

The four signals worth watching

Connections per second and active connections. The rate of new connections arriving at each VIP, and the number currently open. Together they describe demand. Watch them long enough and your service’s natural rhythm emerges — the daily peak, the weekend lull — and deviations from that rhythm become your earliest warning of trouble or unexpected success.

Response times. Average and peak response times from backend servers, per VIP and per backend. This is the metric your users feel. A single backend whose response time is drifting up while its siblings hold steady is degrading — failing disk, runaway process, noisy neighbour — and per-backend granularity is what lets you spot it before the health checks do.

HTTP status codes. For Layer 7 VIPs, the breakdown of 2xx/3xx/4xx/5xx responses returned by the backends. The 5xx line deserves an alert: elevated server errors at the load balancer almost always mean an application-level failure on the backends rather than a load balancing problem — the balancer is faithfully reporting what it is being handed.

Bytes in and out. Traffic volume in each direction, per VIP and per backend. Beyond capacity planning, the in/out ratio characterises your workload — and a sudden change in that ratio (a scrape, a bulk export, an attack) is visible here before almost anywhere else.

Patterns: reading combinations, not lines

Single metrics inform; combinations diagnose.

  • Connections rising, response times flat — healthy growth within capacity. Enjoy it, and feed the trend into capacity planning.
  • Response times rising, connections flat — the backends are struggling with a stable load. Look at the servers, not the traffic: something behind the load balancer got slower.
  • Uneven distribution across backends — with a fair algorithm this points at a misconfigured scheduling algorithm, stale weights, aggressive session persistence, or a health check problem quietly concentrating traffic on a subset of the pool.
  • Elevated 5xx with normal traffic — an application deployment or dependency failure. The load balancer is the messenger, not the culprit.

The dashboard under Monitoring > Statistics presents all of this in real time across both the Layer 4 (nftlb) and Layer 7 (HAProxy) engines, drilling from per-VIP aggregates down to individual backends. A real-time connection table shows every active connection — client address, VIP, assigned backend, duration, bytes — which is invaluable when verifying session persistence or hunting a stuck connection.

History without housekeeping

Real-time answers "what is happening"; incidents and capacity planning need "what is normal". Tula stores historical data in RRDtool round-robin databases, which consolidate older data into lower-resolution averages within a fixed footprint — meaning the statistics store never grows, no matter how long the appliance runs, and there is no retention housekeeping to do.

Four windows are kept: hourly (minute-by-minute, for incident forensics and verifying a change), daily (five-minute resolution, for the shape of a day), weekly (thirty-minute resolution, for trends), and monthly (two-hour resolution, for growth). The discipline worth building: after any configuration change, look at the hourly graphs; once a month, look at the monthly ones. Most capacity surprises were visible for weeks first.

Feeding your monitoring platform over SNMP

A built-in dashboard is necessary but not sufficient — alerts belong in the platform your team already watches. Tula speaks SNMP for exactly this purpose, exposing system metrics (CPU, memory, disk, interfaces) through standard MIBs and load balancer specifics — VIP status, per-VIP connection counts, backend health states, connection rates — through an enterprise MIB you can download from the appliance under System > SNMP > Download MIB.

Version choice is a security decision: SNMP v2c (community strings, cleartext) is acceptable on a trusted management network; SNMP v3 with authPriv adds authentication and encryption for anything less trusted. Either way, restrict queries to your monitoring servers’ addresses in the access list, and remember firewalls need UDP 161 open for polls and UDP 162 for traps.

Polling tells you the state; traps tell you the moment it changes. Tula sends traps on the events you would want to be woken for:

  • HA failover — the node changed between master and backup states.
  • Backend state change — a server entered or left the healthy pool.
  • Service restart — HAProxy, nftlb, or keepalived restarted.
  • Resource threshold — CPU, memory, or disk crossed a configured limit.

The integration recipes for Nagios/Icinga, Zabbix, PRTG, and LibreNMS are in the SNMP documentation — in each case, import the MIB and the platform’s discovery does most of the work.

A starting alert set

If you are configuring alerts from scratch, five rules cover the incidents that matter:

  1. Any VIP down — page immediately.
  2. Backend state change trap — notify; a pool losing members is how outages begin.
  3. 5xx rate above baseline on any L7 VIP — notify the application’s owners.
  4. Response time sustained above threshold per VIP — investigate.
  5. HA failover trap — always investigate, even though service continued; failovers have causes.

Resist alerting on raw connection counts — traffic varies, and thresholds chosen on a quiet Tuesday generate noise forever after. Alert on state changes and error rates; graph the rest.

Your load balancer already sees everything. The full detail is in the statistics documentation and the SNMP guide — the work is simply deciding to look.

CategoriesLoad Balancing

Alison Ivers

Leave a comment

Your email address will not be published. Required fields are marked *