My environment consists of only 2 nodes at the moment. I’m currently experiencing an issue where my forwarder’s logs are being flooded with entries like:
4/10/2016 9:32:06 AMtime="2016-04-10T13:32:05Z" level=error msg="Error reading metadata version: Get http://rancher-metadata/2015-07-25/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: dial udp 169.254.169.250:53: connect: network is unreachable"
4/10/2016 9:32:07 AMtime="2016-04-10T13:32:07Z" level=error msg="Error reading metadata version: Get http://rancher-metadata/2015-07-25/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: dial udp 169.254.169.250:53: connect: network is unreachable"
4/10/2016 9:32:08 AMtime="2016-04-10T13:32:08Z" level=error msg="Error reading metadata version: Get http://rancher-metadata/2015-07-25/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: dial udp 169.254.169.250:53: connect: network is unreachable"
4/10/2016 9:32:10 AMtime="2016-04-10T13:32:10Z" level=error msg="Error reading metadata version: Get http://rancher-metadata/2015-07-25/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: dial udp 169.254.169.250:53: connect: network is unreachable"
4/10/2016 9:32:12 AMtime="2016-04-10T13:32:12Z" level=error msg="Error reading metadata version: Get http://rancher-metadata/2015-07-25/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: dial udp 169.254.169.250:53: connect: network is unreachable"
4/10/2016 9:32:57 AMtime="2016-04-10T13:32:57Z" level=info msg="Listening on 0.0.0.0:3307"
4/10/2016 9:32:57 AMtime="2016-04-10T13:32:57Z" level=info msg="Forwarding setup to: 10.42.180.134:3306"
4/10/2016 2:23:21 PMtime="2016-04-10T18:23:21Z" level=error msg="read tcp 10.42.48.243:49185->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:49173: read: connection reset by peer"
4/10/2016 2:37:55 PMtime="2016-04-10T18:37:55Z" level=error msg="read tcp 10.42.48.243:50509->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:50865: read: connection reset by peer"
4/10/2016 2:37:56 PMtime="2016-04-10T18:37:56Z" level=error msg="read tcp 10.42.48.243:50541->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:50876: read: connection reset by peer"
4/10/2016 2:46:16 PMtime="2016-04-10T18:46:16Z" level=error msg="read tcp 10.42.48.243:51205->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:51899: read: connection reset by peer"
4/10/2016 2:46:16 PMtime="2016-04-10T18:46:16Z" level=error msg="read tcp 10.42.48.243:51208->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:51905: read: connection reset by peer"
4/10/2016 3:02:15 PMtime="2016-04-10T19:02:15Z" level=error msg="read tcp 10.42.48.243:52163->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:53343: read: connection reset by peer"
4/10/2016 3:49:54 PMtime="2016-04-10T19:49:54Z" level=error msg="read tcp 10.42.48.243:57575->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:60707: read: connection reset by peer"
4/10/2016 4:32:58 PMtime="2016-04-10T20:32:58Z" level=error msg="read tcp 10.42.48.243:35045->10.42.180.134:3306: read tcp 10.42.48.243:3307->10.42.247.172:40365: read: connection reset by peer"
10.42.48.243 = mariadb-galera-2
10.42.247.172 = mariadb-load-balancer
pingdom.com is reporting that my web site is experiencing brief 5 minute downtime intervals sporadically every few hours. I’m guessing it’s related to these log entries but I’m having a hard time deciphering how to diagnose the issue. Any suggestions would be very much appreciated.