fir3net
PPS-Firenetbanner-780.5x190-30-03-17

GTM - Healthcheck Monitor Connections not being Established

Issue

You may observe GTM Monitors failing with a message of 'state: timeout' within the logs messages. On further investigation you find that though the GTM is trying to build the connection (i.e sending the SYN), there is no response (SYN-ACK) from the destination. Resulting in the probe attempt failing.

Reason

The reason for this is due to the GTM sending the connection request to a server that already has an established socket open from the GTM for the given source port.

Consider the following setup,

  • You have a single loadbalancer (such as a Brocade ADX) with 2 backend webservers (Server1 and Server2) serving multiple domains.
  • The domains use the IPs of www.domain1.com = 1.1.1.1 and www.domain2.com = 2.2.2.2.
  • The GTM is configured to monitor these domains using standard HTTP monitors for each.

Now lets consider the following scenario,

  1. The GTM builds a connection to 1.1.1.1 (www.domain1.com) using a source port of 12345.
  2. Traffic is sent onto Server1. The server responds to the HTTP request. The GTM receives the response then closes its side of the connection.
  3. Server 1 now has a socket in TIME_WAIT with a Local Address of 0.0.0.0:80 and a Foreign address of  <GTM IP>:12345.
  4. The GTM attempts to build a connection (i.e a SYN is sent) to 2.2.2.2 (www.domain2.com).
  5. Server 1 receives the SYN which has a source IP of the GTM and a src/dest port of 12345/80.
  6. Because of the current socket in TIME_WAIT Server 1 believes the connection has already been established and does not respond with a SYN-ACK.

Solution

Though there is an option within tmos to disable socket reuse this only effects the bigd daemon rather than the big3d that is used (along with gtmd) for healthcheck monitoring on the GTM.

modify sys db bigd.reusesocket value disable
save /sys config

To resolve this issue you will need to rewrite the source IP and source port before it is sent to the backend server. To achieve this there are 2 options,

  1. Proxy - Configure the virtual server on the Loadbalancer to act as a proxy. 
  2. SNAT -  SNAT the traffic going from the virtual server to the backend servers.

 

Tags: TCP, Healthchecks, HTTP, GTM, TIME_WAIT

About the Author

RDonato

R Donato

Rick Donato is the Founder and Chief Editor of Fir3net.com. He currently works as a Principal Network Security Engineer and has a keen interest in automation and the cloud.

You can find Rick on Twitter @f3lix001