Rancher Hosts removed automatically when AVI went down suddenly

Hello Team,
I am not able to understand how did all the rancher hosts got removed automatically from Rancher UI.

We received the following error from the rancher server container logs.
2019-06-08 00:54:04,298 ERROR [:] [] [] [] [p1020391880-129] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [http://rancher-url/v1/subscribe]. Error id: [a0bcc291-0680-458c-aec4-e025b1d9f490]. org.jooq.exception.DataAccessException: SQL [select credential.id, credential.name, credential.account_id, credential.kind, credential.uuid, credential.description, credential.state, credential.created, credential.removed, credential.remove_time, credential.data, credential.public_value, credential.secret_value, credential.registry_id from credential where (credential.state = ? and credential.public_value = ? and credential.kind in (?, ?, ?))]; Cannot get a connection, pool error Timeout waiting for idle object
at org.jooq.impl.Utils.translate(Utils.java:1287)
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326)
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:351)
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:341)
at org.jooq.impl.AbstractResultQuery.fetchOne(AbstractResultQuery.java:470)
at org.jooq.impl.SelectImpl.fetchOne(SelectImpl.java:2372)
at io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.getAccountByKeys(AuthDaoImpl.java:214)
at io.cattle.platform.iaas.api.auth.integration.internal.rancher.BasicAuthImpl.getAccount(BasicAuthImpl.java:62)
at io.cattle.platform.iaas.api.auth.impl.ApiAuthenticator.getAccount(ApiAuthenticator.java:149)
at io.cattle.platform.iaas.api.auth.impl.ApiAuthenticator.handle(ApiAuthenticator.java:68)
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:99)
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:100)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:110)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:225)
at com.sun.proxy.$Proxy43.prepareStatement(Unknown Source)
at org.jooq.impl.ProviderEnabledConnection.prepareStatement(ProviderEnabledConnection.java:112)
at org.jooq.impl.SettingsEnabledConnection.prepareStatement(SettingsEnabledConnection.java:76)
at org.jooq.impl.AbstractResultQuery.prepare(AbstractResultQuery.java:230)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296)
… 37 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
… 45 common frames omitted

the following is the rancher agent log:

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

INFO: Starting agent for 90703594F2E1B9C03CCC

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

ERROR: Credentials are no longer valid, please re-register this agent

Can you let us know what is the issue here and how can we reproduce the same?
We need to understand as to what has happened when the AVI when down.

Regards,
Yash BRAHMANI

Hello Team,
We are using Rancher version 1.6.18.
Also we observed that the infrastructure stacks where not coming up when a single host was added.
healthcheck and schedulers were not able to come up properly going into instable/unhealthy state.

Regards,
Yash BRAHMANI