Failed to get connection to database after update

Hi! I have a Rancher Server online that I set up to do some test a while ago. It uses an external mysql database.

Today I tried updating it (a big jump from 0.5.1 to 1.0.0), but it fails to connect to the database. I haven’t change anything, and I can connect to the db with mysql workbench.
I haven’t used it in a while, so it’s possible that I’m making some very stupid mistake.

This is the command I use:

docker run -p 8080:8080 --restart=always -d -e CATTLE_DB_CATTLE_MYSQL_HOST=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com -e CATTLE_DB_CATTLE_MYSQL_NAME=db -e CATTLE_DB_CATTLE_USERNAME=user -e CATTLE_DB_CATTLE_PASSWORD=password rancher/server

And this is the exception I get:

[pollingConfigurationSource] ERROR i.c.p.a.sources.LazyJDBCSource - Failed to get connection to database, will retry for 5 minutes
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not connect to address=(host=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : connect timed out)
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4]
	at io.cattle.platform.archaius.sources.LazyJDBCSource.checkInitial(LazyJDBCSource.java:50) [cattle-framework-archaius-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.archaius.sources.LazyJDBCSource.poll(LazyJDBCSource.java:34) [cattle-framework-archaius-0.5.0-SNAPSHOT.jar:na]
	at com.netflix.config.AbstractPollingScheduler$1.run(AbstractPollingScheduler.java:163) [archaius-core-0.5.14.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_95]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_95]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_95]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_95]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_95]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_95]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_95]
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : connect timed out
	at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.Driver.connect(Driver.java:110) ~[mariadb-java-client-1.3.4.jar:na]
	at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) ~[commons-dbcp-1.4.jar:1.4]
	at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ~[commons-dbcp-1.4.jar:1.4]
	... 12 common frames omitted
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Could not connect to address=(host=rancherserver.cfpar5g3f1gu.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : connect timed out
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:674) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:580) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.Driver.connect(Driver.java:105) ~[mariadb-java-client-1.3.4.jar:na]
	... 16 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.7.0_95]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[na:1.7.0_95]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[na:1.7.0_95]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[na:1.7.0_95]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.7.0_95]
	at java.net.Socket.connect(Socket.java:579) ~[na:1.7.0_95]
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:344) ~[mariadb-java-client-1.3.4.jar:na]
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:669) ~[mariadb-java-client-1.3.4.jar:na]
	... 18 common frames omitted

Thanks!

It looks like in your comand you are missing
-e CATTLE_DB_CATTLE_MYSQL_PORT=<port>

Here’s what it recommends in the docs. http://docs.rancher.com/rancher/installing-rancher/installing-server/#using-an-external-database

sudo docker run -d --restart=always -p 8080:8080 \
    -e CATTLE_DB_CATTLE_MYSQL_HOST=<hostname or IP of MySQL instance> \
    -e CATTLE_DB_CATTLE_MYSQL_PORT=<port> \
    -e CATTLE_DB_CATTLE_MYSQL_NAME=<Name of Database> \
    -e CATTLE_DB_CATTLE_USERNAME=<Username> \
    -e CATTLE_DB_CATTLE_PASSWORD=<Password> \
    rancher/server

Also, can you clarify if you’re using v0.56.1? v0.59.1? Just trying to help pinpoint where your jump is from. :slight_smile:

Hi, thanks for the answer!

I’m getting the same error by specifying the port (it was the default anyway).

I’m using the latest version right now. I don’t remeber exactly which one I was using before, but I think 0.5.1

If you were using v0.5.1, then you would have deployed something on the 5th week of 2015 (over a year ago). Prior to v1.0.0, the numbering of our versions were based on the week of the year.

This is why I asked if you were running either v0.56.1 or v0.59.1, there is also v0.51.0.

FYI, if it’s v0.51.0 to v1.0.0, then there are major networking changes that occurred between v0.51.0 to v0.56. The DB migration might take a lot longer.

It seems like you are having a timeout issue (not related to upgrading). Are you able to start up a new rancher/server to connect to the DB to test out your external DB connection?

Pretty safe to assume v0.51.0, as there was no public v0.5.1 tag.

I haven’t tryed with a new rancher/server. I can conect to that db without problems with other tools, though

have you tried launching with just the IP instead of the name of the rds instance?

No, I will try as soon as I have some spare time! :slight_smile:

Just spend a while debugging the same problem. In my case I had a $ sign in the password. Removed it and now it works.