GTM – Healthcheck Monitor Connections not being Established

Issue

You may observe GTM Monitors failing with a message of ‘state: timeout’ within the logs messages. On further investigation you find that though the GTM is trying to build the connection (i.e sending the SYN), there is no response (SYN-ACK) from the destination. Resulting in the probe attempt failing.

Reason

The reason for this is due to the GTM sending the connection request to a server that already has an established socket open from the GTM for the given source port.

Consider the following setup,

  • You have a single loadbalancer (such as a Brocade ADX) with 2 backend webservers (Server1 and Server2) serving multiple domains.
  • The domains use the IPs of www.domain1.com = 1.1.1.1 and www.domain2.com = 2.2.2.2.
  • The GTM is configured to monitor these domains using standard HTTP monitors for each.

Now lets consider the following scenario,

  1. The GTM builds a connection to 1.1.1.1 (www.domain1.com) using a source port of 12345.
  2. Traffic is sent onto Server1. The server responds to the HTTP request. The GTM receives the response then closes its side of the connection.
  3. Server 1 now has a socket in TIME_WAIT with a Local Address of 0.0.0.0:80 and a Foreign address of  <GTM IP>:12345.
  4. The GTM attempts to build a connection (i.e a SYN is sent) to 2.2.2.2 (www.domain2.com).
  5. Server 1 receives the SYN which has a source IP of the GTM and a src/dest port of 12345/80.
  6. Because of the current socket in TIME_WAIT Server 1 believes the connection has already been established and does not respond with a SYN-ACK.

Solution

Though there is an option within tmos to disable socket reuse this only effects the bigd daemon rather than the big3d that is used (along with gtmd) for healthcheck monitoring on the GTM.

modify sys db bigd.reusesocket value disable save /sys config

To resolve this issue you will need to rewrite the source IP and source port before it is sent to the backend server. To achieve this there are 2 options,

  1. Proxy – Configure the virtual server on the Loadbalancer to act as a proxy. 
  2. SNAT –  SNAT the traffic going from the virtual server to the backend servers.

 

Rick Donato
Latest posts by Rick Donato (see all)

Want to become an F5 Loadbalancers expert?

Here is our hand-picked selection of the best courses you can find online:
F5 BIG-IP 101 Certification Exam – Complete Course
F5 BIG-IP 201 Certification Exam – Complete Course
and our recommended certification practice exams:
AlphaPrep Practice Tests - Free Trial