GTM - Healthcheck Monitor Connections not being Established
You may observe GTM Monitors failing with a message of 'state: timeout' within the logs messages. On further investigation you find that though the GTM is trying to build the connection (i.e sending the SYN), there is no response (SYN-ACK) from the destination. Resulting in the probe attempt failing.
The reason for this is due to the GTM sending the connection request to a server that already has an established socket open from the GTM for the given source port.
Consider the following setup,
- You have a single loadbalancer (such as a Brocade ADX) with 2 backend webservers (Server1 and Server2) serving multiple domains.
- The domains use the IPs of www.domain1.com = 220.127.116.11 and www.domain2.com = 18.104.22.168.
- The GTM is configured to monitor these domains using standard HTTP monitors for each.
Now lets consider the following scenario,
- The GTM builds a connection to 22.214.171.124 (www.domain1.com) using a source port of 12345.
- Traffic is sent onto Server1. The server responds to the HTTP request. The GTM receives the response then closes its side of the connection.
- Server 1 now has a socket in TIME_WAIT with a Local Address of 0.0.0.0:80 and a Foreign address of <GTM IP>:12345.
- The GTM attempts to build a connection (i.e a SYN is sent) to 126.96.36.199 (www.domain2.com).
- Server 1 receives the SYN which has a source IP of the GTM and a src/dest port of 12345/80.
- Because of the current socket in TIME_WAIT Server 1 believes the connection has already been established and does not respond with a SYN-ACK.
Though there is an option within tmos to disable socket reuse this only effects the bigd daemon rather than the big3d that is used (along with gtmd) for healthcheck monitoring on the GTM.
modify sys db bigd.reusesocket value disable save /sys config
To resolve this issue you will need to rewrite the source IP and source port before it is sent to the backend server. To achieve this there are 2 options,
- Proxy - Configure the virtual server on the Loadbalancer to act as a proxy.
- SNAT - SNAT the traffic going from the virtual server to the backend servers.