Failed to get connection to database after update

doddenino · March 29, 2016, 2:58pm

Hi! I have a Rancher Server online that I set up to do some test a while ago. It uses an external mysql database.

Today I tried updating it (a big jump from 0.5.1 to 1.0.0), but it fails to connect to the database. I haven’t change anything, and I can connect to the db with mysql workbench.
I haven’t used it in a while, so it’s possible that I’m making some very stupid mistake.

This is the command I use:

docker run -p 8080:8080 --restart=always -d -e CATTLE_DB_CATTLE_MYSQL_HOST=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com -e CATTLE_DB_CATTLE_MYSQL_NAME=db -e CATTLE_DB_CATTLE_USERNAME=user -e CATTLE_DB_CATTLE_PASSWORD=password rancher/server

And this is the exception I get:

[pollingConfigurationSource] ERROR i.c.p.a.sources.LazyJDBCSource - Failed to get connection to database, will retry for 5 minutes
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not connect to address=(host=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : connect timed out)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4]
	at io.cattle.platform.archaius.sources.LazyJDBCSource.checkInitial(LazyJDBCSource.java:50) [cattle-framework-archaius-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.archaius.sources.LazyJDBCSource.poll(LazyJDBCSource.java:34) [cattle-framework-archaius-0.5.0-SNAPSHOT.jar:na]
	at com.netflix.config.AbstractPollingScheduler$1.run(AbstractPollingScheduler.java:163) [archaius-core-0.5.14.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_95]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_95]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_95]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : connect timed out
	at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.Driver.connect(Driver.java:110) ~[mariadb-java-client-1.3.4.jar:na]
	at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ~[commons-dbcp-1.4.jar:1.4]
	... 12 common frames omitted
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not connect to address=(host=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : connect timed out
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:674) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:580) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.Driver.connect(Driver.java:105) ~[mariadb-java-client-1.3.4.jar:na]
	... 16 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_95]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_95]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_95]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_95]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_95]
	at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_95]
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:344) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:669) ~[mariadb-java-client-1.3.4.jar:na]
	... 18 common frames omitted

Thanks!

denise · March 29, 2016, 4:51pm

It looks like in your comand you are missing
-e CATTLE_DB_CATTLE_MYSQL_PORT=<port>

Here’s what it recommends in the docs. http://docs.rancher.com/rancher/installing-rancher/installing-server/#using-an-external-database

sudo docker run -d --restart=always -p 8080:8080 \
    -e CATTLE_DB_CATTLE_MYSQL_HOST=<hostname or IP of MySQL instance> \
    -e CATTLE_DB_CATTLE_MYSQL_PORT=<port> \
    -e CATTLE_DB_CATTLE_MYSQL_NAME=<Name of Database> \
    -e CATTLE_DB_CATTLE_USERNAME=<Username> \
    -e CATTLE_DB_CATTLE_PASSWORD=<Password> \
    rancher/server

Also, can you clarify if you’re using v0.56.1? v0.59.1? Just trying to help pinpoint where your jump is from.

doddenino · March 30, 2016, 8:55am

Hi, thanks for the answer!

I’m getting the same error by specifying the port (it was the default anyway).

I’m using the latest version right now. I don’t remeber exactly which one I was using before, but I think 0.5.1

denise · March 30, 2016, 5:46pm

If you were using v0.5.1, then you would have deployed something on the 5th week of 2015 (over a year ago). Prior to v1.0.0, the numbering of our versions were based on the week of the year.

This is why I asked if you were running either v0.56.1 or v0.59.1, there is also v0.51.0.

FYI, if it’s v0.51.0 to v1.0.0, then there are major networking changes that occurred between v0.51.0 to v0.56. The DB migration might take a lot longer.

It seems like you are having a timeout issue (not related to upgrading). Are you able to start up a new rancher/server to connect to the DB to test out your external DB connection?

vincent · March 30, 2016, 5:47pm

Pretty safe to assume v0.51.0, as there was no public v0.5.1 tag.

doddenino · March 31, 2016, 3:25pm

I haven’t tryed with a new rancher/server. I can conect to that db without problems with other tools, though

cloudnautique · March 31, 2016, 7:29pm

have you tried launching with just the IP instead of the name of the rds instance?

doddenino · April 4, 2016, 2:33pm

No, I will try as soon as I have some spare time!

ckreutz · July 19, 2016, 1:02pm

Just spend a while debugging the same problem. In my case I had a $ sign in the password. Removed it and now it works.

Topic		Replies	Views
Running Rancher with external MySQL Database	4	5827	June 13, 2016
Mysql Rancher cannot connect Rancher 1.x	1	2200	November 17, 2017
Rancher DB connection errors when accessing deployed service via the web Rancher 1.x	9	5687	August 27, 2015
Rancher on Azure external mysql db trying to use public IP Rancher 1.x	0	780	March 20, 2018
Can't connect Rancher to DB	1	1456	July 12, 2016

Failed to get connection to database after update

Related topics