Hello Team,
I am not able to understand how did all the rancher hosts got removed automatically from Rancher UI.
We received the following error from the rancher server container logs.
2019-06-08 00:54:04,298 ERROR [:] [] [] [] [p1020391880-129] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [http://rancher-url/v1/subscribe]. Error id: [a0bcc291-0680-458c-aec4-e025b1d9f490]. org.jooq.exception.DataAccessException: SQL [select credential.id, credential.name, credential.account_id, credential.kind, credential.uuid, credential.description, credential.state, credential.created, credential.removed, credential.remove_time, credential.data, credential.public_value, credential.secret_value, credential.registry_id from credential where (credential.state = ? and credential.public_value = ? and credential.kind in (?, ?, ?))]; Cannot get a connection, pool error Timeout waiting for idle object
at org.jooq.impl.Utils.translate(Utils.java:1287)
at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326)
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:351)
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:341)
at org.jooq.impl.AbstractResultQuery.fetchOne(AbstractResultQuery.java:470)
at org.jooq.impl.SelectImpl.fetchOne(SelectImpl.java:2372)
at io.cattle.platform.iaas.api.auth.dao.impl.AuthDaoImpl.getAccountByKeys(AuthDaoImpl.java:214)
at io.cattle.platform.iaas.api.auth.integration.internal.rancher.BasicAuthImpl.getAccount(BasicAuthImpl.java:62)
at io.cattle.platform.iaas.api.auth.impl.ApiAuthenticator.getAccount(ApiAuthenticator.java:149)
at io.cattle.platform.iaas.api.auth.impl.ApiAuthenticator.handle(ApiAuthenticator.java:68)
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:99)
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:100)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:110)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:225)
at com.sun.proxy.$Proxy43.prepareStatement(Unknown Source)
at org.jooq.impl.ProviderEnabledConnection.prepareStatement(ProviderEnabledConnection.java:112)
at org.jooq.impl.SettingsEnabledConnection.prepareStatement(SettingsEnabledConnection.java:76)
at org.jooq.impl.AbstractResultQuery.prepare(AbstractResultQuery.java:230)
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296)
… 37 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
… 45 common frames omitted
the following is the rancher agent log:
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
INFO: Starting agent for 90703594F2E1B9C03CCC
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
ERROR: Credentials are no longer valid, please re-register this agent
Can you let us know what is the issue here and how can we reproduce the same?
We need to understand as to what has happened when the AVI when down.
Regards,
Yash BRAHMANI