Duplicate Key Violations

mota_nginya · April 12, 2018, 4:36pm

Hello Maestros,

Full disclosure, I am extremely green on Rancher and I find it really promising. However, I am seeing too many constraint errors in the log. Is this normal? I have a fresh install and currently running v1.6.16. I am in the process of trying out a HA implementation on bare metal.

Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Duplicate entry 'service.v3.12.kubernetes.cert' for key 'idx_data_name'
Query is: insert into `data` (`name`, `visible`, `value`) values (?, ?, ?), parameters ['service.v3.12.kubernetes.cert',0,'UEsDBBQACAgIALaDjEwAAAAAAAAAAAAAAAAGAAAAY2EucGVtZZNbr6pIEIXf+RXzTnY2Ago8nIfupoEWGgRBlDeV+11Qufz6455MTjIz9VT5VrIqqVX19fUpiHVi/4Ww5xONIODjH/jFUELQvCIErmwGJgJBRoh1MXzhzF5uoJtU97I3u4jk77sNXGxDF0zYxwcKOB1sAsygbNIvvMbF+lxTKJ5VH4tUdWdaYp6WZHW07sPID5sc/w/7Y8L8uPzLhBBISmDDrHrkVaErEwc/kzUAHARcGfzoKDM/PQbCBikM9QMqm7QITTIcrSscC5C0EPu2SMata5hi9E4PMLGlvcO7Ajg0cAirJjxypvba3fWE6Q0kCdIci45BwYIv75Cqah/NTnJG7GtsF2GorZ2ouN9dkZ63LXc/mte5VfhUknMhJEy2K4vk0CaPksPV7kk1ShOSyDO7rU4H2i1J9R531zs7X4/1LXpiadrsPTiG4DTdnPzZM8l+4eNX2dPGgHto1SwsTuuLDtfYMI9RLL/eBslFm3KNU1RlaSvh06ybllTD/eVGT4UpYq3dq2tSS6Y/k0Y/G+XyViJuKHv25JYwyt2gDZuZjMHQBIMTWkIRUZwBETqiJxwZWeT4ErLh7oEucmf0ykRU4ALYLXQiPjjAzD4ZHoUglTH0wUczvikQf24gVicMme/pk8dEUK/+Nzrun+gw0NCSco86uivnAWUD9TzHHpJB0lfDYprNQ1GvyAw0ijZT2rF+C7T21KbY7WxVLKtVBmiYjlOEMuH6jVKrxf7KXkSNfSbPjcHc1OB25Y2l1gMnEdrqnmTTpTNuu1asmzqyzNL7vtw3qW0/PUMUt73CpWOITGPwR7khAsMGCg2zYDyqXstVWc9zJIjPvbEur2dQe2Rx09i3...
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:1115)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1047)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1002)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:588)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:399)
	... 118 common frames omitted

time="2018-04-12T16:29:45Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
2018-04-12 16:29:45,369 ERROR [:] [] [] [] [p1020391880-167] [.a.a.ServiceCertificateActionHandler] Failed to generate certificate for service [30] org.jooq.exception.DataAccessException: SQL [insert into `data` (`name`, `visible`, `value`) values (?, ?, ?)]; (conn:445) Duplicate entry 'service.v3.12.kubernetes.cert' for key 'idx_data_name'
Query is: insert into `data` (`name`, `visible`, `value`) values (?, ?, ?), parameters ['service.v3.12.kubernetes.cert',0,'UEsDBBQACAgIALaDjEwAAAAAAAAAAAAAAAAGAAAAY2EucGVtZZNbr6pIEIXf+RXzTnY2Ago8nIfupoEWGgRBlDeV+11Qufz6455MTjIz9VT5VrIqqVX19fUpiHVi/4Ww5xONIODjH/jFUELQvCIErmwGJgJBRoh1MXzhzF5uoJtU97I3u4jk77sNXGxDF0zYxwcKOB1sAsygbNIvvMbF+lxTKJ5VH4tUdWdaYp6WZHW07sPID5sc/w/7Y8L8uPzLhBBISmDDrHrkVaErEwc/kzUAHARcGfzoKDM/PQbCBikM9QMqm7QITTIcrSscC5C0EPu2SMata5hi9E4PMLGlvcO7Ajg0cAirJjxypvba3fWE6Q0kCdIci45BwYIv75Cqah/NTnJG7GtsF2GorZ2ouN9dkZ63LXc/mte5VfhUknMhJEy2K4vk0CaPksPV7kk1ShOSyDO7rU4H2i1J9R531zs7X4/1LXpiadrsPTiG4DTdnPzZM8l+4eNX2dPGgHto1SwsTuuLDtfYMI9RLL/eBslFm3KNU1RlaSvh06ybllTD/eVGT4UpYq3dq2tSS6Y/k0Y/G+XyViJuKHv25JYwyt2gDZuZjMHQBIMTWkIRUZwBETqiJxwZWeT4ErLh7oEucmf0ykRU4ALYLXQiPjjAzD4ZHoUglTH0wUczvikQf24gVicMme/pk8dEUK/+Nzrun+gw0NCSco86uivnAWUD9TzHHpJB0lfDYprNQ1GvyAw0ijZT2rF+C7T21KbY7WxVLKtVBmiYjlOEMuH6jVKrxf7KXkSNfSbPjcHc1OB25Y2l1gMnEdrqnmTTpTNuu1asmzqyzNL7vtw3qW0/PUMUt73CpWOITGPwR7khAsMGCg2zYDyqXstVWc9zJIjPvbEur2dQe2Rx09i3...
	at org.jooq.impl.Utils.translate(Utils.java:1287)
	at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:495)
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:326)
	at org.jooq.impl.UpdatableRecordImpl.storeInsert0(UpdatableRecordImpl.java:230)
	at org.jooq.impl.UpdatableRecordImpl.access$100(UpdatableRecordImpl.java:90)
	at org.jooq.impl.UpdatableRecordImpl$2.operate(UpdatableRecordImpl.java:201)
	at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123)
	at org.jooq.impl.UpdatableRecordImpl.storeInsert(UpdatableRecordImpl.java:196)
	at org.jooq.impl.UpdatableRecordImpl.insert(UpdatableRecordImpl.java:144)
	at io.cattle.platform.object.impl.JooqObjectManager$1.executeNoResult(JooqObjectManager.java:80)
	at io.cattle.platform.engine.idempotent.IdempotentExecutionNoReturn.execute(IdempotentExecutionNoReturn.java:7)
	at io.cattle.platform.engine.idempotent.IdempotentExecutionNoReturn.execute(IdempotentExecutionNoReturn.java:3)
	at io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88)
	at io.cattle.platform.object.impl.JooqObjectManager.insert(JooqObjectManager.java:77)
	at io.cattle.platform.object.impl.AbstractObjectManager.create(AbstractObjectManager.java:62)
	at io.cattle.platform.object.impl.AbstractObjectManager.create(AbstractObjectManager.java:72)
	at io.cattle.platform.core.dao.impl.DataDaoImpl$1.doWithLock(DataDaoImpl.java:53)
	at io.cattle.platform.core.dao.impl.DataDaoImpl$1.doWithLock(DataDaoImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40)
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13)
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37)
	at io.cattle.platform.core.dao.impl.DataDaoImpl.getOrCreate(DataDaoImpl.java:33)
	at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:280)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy58.getOrCreate(Unknown Source)
	at io.cattle.platform.servicediscovery.api.service.impl.ServiceDiscoveryApiServiceImpl.getServiceCertificate(ServiceDiscoveryApiServiceImpl.java:837)
	at io.cattle.platform.servicediscovery.api.action.ServiceCertificateActionHandler.perform(ServiceCertificateActionHandler.java:42)
	at io.cattle.platform.api.resource.AbstractObjectResourceManager.resourceActionInternal(AbstractObjectResourceManager.java:382)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.resourceAction(AbstractBaseResourceManager.java:414)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceStackStorageDriverFilter.resourceAction(ServiceStackStorageDriverFilter.java:85)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceStackNetworkDriverFilter.resourceAction(ServiceStackNetworkDriverFilter.java:85)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceUpgradeValidationFilter.resourceAction(ServiceUpgradeValidationFilter.java:76)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceRestartValidationFilter.resourceAction(ServiceRestartValidationFilter.java:67)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceRollbackValidationFilter.resourceAction(ServiceRollbackValidationFilter.java:55)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceAddRemoveLinkServiceValidationFilter.resourceAction(ServiceAddRemoveLinkServiceValidationFilter.java:59)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.resource.AbstractResourceManagerFilter.resourceAction(AbstractResourceManagerFilter.java:58)
	at io.cattle.platform.servicediscovery.api.filter.ServiceSetServiceLinksValidationFilter.resourceAction(ServiceSetServiceLinksValidationFilter.java:48)
	at io.github.ibuildthecloud.gdapi.request.resource.impl.FilteredResourceManager.resourceAction(FilteredResourceManager.java:86)
	at io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:44)
	at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14)
	at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:27)
	at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.write(DefaultReadWriteApiDelegate.java:22)
	at sun.reflect.GeneratedMethodAccessor580.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:280)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy73.write(Unknown Source)
	at io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:19)
	at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:99)
	at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:100)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
	at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:110)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLIntegrityConstraintViolationException: (conn:445) Duplicate entry 'service.v3.12.kubernetes.cert' for key 'idx_data_name'
Query is: insert into `data` (`name`, `visible`, `value`) values (?, ?, ?), parameters ['service.v3.12.kubernetes.cert',0,'UEsDBBQACAgIALaDjEwAAAAAAAAAAAAAAAAGAAAAY2EucGVtZZNbr6pIEIXf+RXzTnY2Ago8nIfupoEWGgRBlDeV+11Qufz6455MTjIz9VT5VrIqqVX19fUpiHVi/4Ww5xONIODjH/jFUELQvCIErmwGJgJBRoh1MXzhzF5uoJtU97I3u4jk77sNXGxDF0zYxwcKOB1sAsygbNIvvMbF+lxTKJ5VH4tUdWdaYp6WZHW07sPID5sc/w/7Y8L8uPzLhBBISmDDrHrkVaErEwc/kzUAHARcGfzoKDM/PQbCBikM9QMqm7QITTIcrSscC5C0EPu2SMata5hi9E4PMLGlvcO7Ajg0cAirJjxypvba3fWE6Q0kCdIci45BwYIv75Cqah/NTnJG7GtsF2GorZ2ouN9dkZ63LXc/mte5VfhUknMhJEy2K4vk0CaPksPV7kk1ShOSyDO7rU4H2i1J9R531zs7X4/1LXpiadrsPTiG4DTdnPzZM8l+4eNX2dPGgHto1SwsTuuLDtfYMI9RLL/eBslFm3KNU1RlaSvh06ybllTD/eVGT4UpYq3dq2tSS6Y/k0Y/G+XyViJuKHv25JYwyt2gDZuZjMHQBIMTWkIRUZwBETqiJxwZWeT4ErLh7oEucmf0ykRU4ALYLXQiPjjAzD4ZHoUglTH0wUczvikQf24gVicMme/pk8dEUK/+Nzrun+gw0NCSco86uivnAWUD9TzHHpJB0lfDYprNQ1GvyAw0ijZT2rF+C7T21KbY7WxVLKtVBmiYjlOEMuH6jVKrxf7KXkSNfSbPjcHc1OB25Y2l1gMnEdrqnmTTpTNuu1asmzqyzNL7vtw3qW0/PUMUt73CpWOITGPwR7khAsMGCg2zYDyqXstVWc9zJIjPvbEur2dQe2Rx09i3...
	at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:133)
	at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101)
	at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77)
	at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:224)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:411)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:361)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeUpdate(MariaDbServerPreparedStatement.java:350)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
	at org.jooq.tools.jdbc.DefaultPreparedStatement.executeUpdate(DefaultPreparedStatement.java:88)
	at org.jooq.impl.AbstractStoreQuery.execute(AbstractStoreQuery.java:346)
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:322)
	... 111 common frames omitted
Caused by: org.mariadb.jdbc.internal.util.dao.QueryException: Duplicate entry 'service.v3.12.kubernetes.cert' for key 'idx_data_name'
Query is: insert into `data` (`name`, `visible`, `value`) values (?, ?, ?), parameters ['service.v3.12.kubernetes.cert',0,'UEsDBBQACAgIALaDjEwAAAAAAAAAAAAAAAAGAAAAY2EucGVtZZNbr6pIEIXf+RXzTnY2Ago8nIfupoEWGgRBlDeV+11Qufz6455MTjIz9VT5VrIqqVX19fUpiHVi/4Ww5xONIODjH/jFUELQvCIErmwGJgJBRoh1MXzhzF5uoJtU97I3u4jk77sNXGxDF0zYxwcKOB1sAsygbNIvvMbF+lxTKJ5VH4tUdWdaYp6WZHW07sPID5sc/w/7Y8L8uPzLhBBISmDDrHrkVaErEwc/kzUAHARcGfzoKDM/PQbCBikM9QMqm7QITTIcrSscC5C0EPu2SMata5hi9E4PMLGlvcO7Ajg0cAirJjxypvba3fWE6Q0kCdIci45BwYIv75Cqah/NTnJG7GtsF2GorZ2ouN9dkZ63LXc/mte5VfhUknMhJEy2K4vk0CaPksPV7kk1ShOSyDO7rU4H2i1J9R531zs7X4/1LXpiadrsPTiG4DTdnPzZM8l+4eNX2dPGgHto1SwsTuuLDtfYMI9RLL/eBslFm3KNU1RlaSvh06ybllTD/eVGT4UpYq3dq2tSS6Y/k0Y/G+XyViJuKHv25JYwyt2gDZuZjMHQBIMTWkIRUZwBETqiJxwZWeT4ErLh7oEucmf0ykRU4ALYLXQiPjjAzD4ZHoUglTH0wUczvikQf24gVicMme/pk8dEUK/+Nzrun+gw0NCSco86uivnAWUD9TzHHpJB0lfDYprNQ1GvyAw0ijZT2rF+C7T21KbY7WxVLKtVBmiYjlOEMuH6jVKrxf7KXkSNfSbPjcHc1OB25Y2l1gMnEdrqnmTTpTNuu1asmzqyzNL7vtw3qW0/PUMUt73CpWOITGPwR7khAsMGCg2zYDyqXstVWc9zJIjPvbEur2dQe2Rx09i3...
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:1115)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1047)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1002)
	at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:588)
	at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:399)
	... 118 common frames omitted

mota_nginya · April 12, 2018, 4:38pm

More errors…

time="2018-04-12T16:29:46Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:29:55Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
2018-04-12 16:29:57,896 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38->instanceHostMap:34] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-12] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:29:57,896 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38] [instance.start->(InstanceStart)] [] [cutorService-12] [i.c.p.process.instance.InstanceStart] Failed [1/5] to Starting for instance [38] 
2018-04-12 16:29:57,958 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38->instanceHostMap:34] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-12] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:29:57,958 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38] [instance.start->(InstanceStart)] [] [cutorService-12] [i.c.p.process.instance.InstanceStart] Failed [2/5] to Starting for instance [38] 
time="2018-04-12T16:29:58Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:29:59Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:30:00Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:30:02Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:30:03Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
2018-04-12 16:30:14,543 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32->instanceHostMap:30] [instance.start->(InstanceStart)->instancehostmap.activate] [] [ecutorService-9] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:14,543 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32] [instance.start->(InstanceStart)] [] [ecutorService-9] [i.c.p.process.instance.InstanceStart] Failed [3/5] to Starting for instance [32] 
2018-04-12 16:30:14,588 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32->instanceHostMap:30] [instance.start->(InstanceStart)->instancehostmap.activate] [] [ecutorService-9] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:14,588 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32] [instance.start->(InstanceStart)] [] [ecutorService-9] [i.c.p.process.instance.InstanceStart] Failed [4/5] to Starting for instance [32] 
2018-04-12 16:30:25,724 ERROR [cd9a65a0-b97f-47aa-b421-d43d2cb38bbe:542] [instance:42] [instance.start->(InstanceStart)] [] [ecutorService-3] [i.c.p.process.instance.InstanceStart] Failed [Dependencies readiness error instance is not running] for instance [42] 
2018-04-12 16:30:29,768 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38->instanceHostMap:34] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-12] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:29,768 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38] [instance.start->(InstanceStart)] [] [cutorService-12] [i.c.p.process.instance.InstanceStart] Failed [3/5] to Starting for instance [38] 
2018-04-12 16:30:29,837 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38->instanceHostMap:34] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-12] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:29,838 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38] [instance.start->(InstanceStart)] [] [cutorService-12] [i.c.p.process.instance.InstanceStart] Failed [4/5] to Starting for instance [38] 
time="2018-04-12T16:30:32Z" level=info msg="Stack Create Event Received" eventId=c55c4b01-56d0-41bd-8417-60f2a7a34740 resourceId=1st18 
time="2018-04-12T16:30:32Z" level=info msg="Stack Create Event Done" eventId=c55c4b01-56d0-41bd-8417-60f2a7a34740 resourceId=1st18 
2018-04-12 16:30:46,387 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32->instanceHostMap:30] [instance.start->(InstanceStart)->instancehostmap.activate] [] [ecutorService-9] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:46,387 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32] [instance.start->(InstanceStart)] [] [ecutorService-9] [i.c.p.process.instance.InstanceStart] Failed [5/5] to Starting for instance [32] 
2018-04-12 16:30:46,408 ERROR [38cb87b1-671e-4115-b813-de0ae566cb3e:494] [instance:32] [instance.start] [] [ecutorService-9] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:46,461 ERROR [:] [] [] [] [ecutorService-9] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:30:47,199 ERROR [9103fbc4-d48b-405e-96cb-2a48f288c662:388] [service:28] [service.activate] [] [cutorService-15] [c.p.e.p.i.DefaultProcessInstanceImpl] Expected state running but got error: Timeout getting IP address 
2018-04-12 16:30:47,209 ERROR [:] [] [] [] [cutorService-15] [.e.s.i.ProcessInstanceDispatcherImpl] Expected state running but got error: Timeout getting IP address 
time="2018-04-12T16:30:57Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:31:00Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
2018-04-12 16:31:01,652 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38->instanceHostMap:34] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-12] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:31:01,652 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38] [instance.start->(InstanceStart)] [] [cutorService-12] [i.c.p.process.instance.InstanceStart] Failed [5/5] to Starting for instance [38] 
2018-04-12 16:31:01,672 ERROR [ba77bf5d-1888-4a30-92bf-f6a099abfe51:500] [instance:38] [instance.start] [] [cutorService-12] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:31:01,716 ERROR [:] [] [] [] [cutorService-12] [.e.s.i.ProcessInstanceDispatcherImpl] Agent error for [compute.instance.activate.reply;agent=2]: Timeout getting IP address 
2018-04-12 16:31:02,198 ERROR [74bca9ab-36bc-4797-8bb0-a724c291e0b0:542] [instance:42] [instance.start->(InstanceStart)] [] [ecutorService-2] [i.c.p.process.instance.InstanceStart] Failed [Dependencies readiness error instance is not running] for instance [42] 
time="2018-04-12T16:31:04Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
2018-04-12 16:31:07,228 ERROR [064df2a1-9d79-4f9a-994d-cff2c5fbc484:1320] [service:42] [service.update] [] [cutorService-15] [c.p.e.p.i.DefaultProcessInstanceImpl] Expected state running but got removed 
2018-04-12 16:31:07,229 ERROR [3d6bb559-b053-4c63-9fb5-72d935c59bcd:1318] [service:40] [service.update] [] [ecutorService-3] [c.p.e.p.i.DefaultProcessInstanceImpl] Expected state running but got removed 
2018-04-12 16:31:07,238 ERROR [:] [] [] [] [cutorService-15] [.e.s.i.ProcessInstanceDispatcherImpl] Expected state running but got removed 
2018-04-12 16:31:07,241 ERROR [:] [] [] [] [ecutorService-3] [.e.s.i.ProcessInstanceDispatcherImpl] Expected state running but got removed 
time="2018-04-12T16:31:12Z" level=error msg="Error looking up token." error="HTTP error: 404 Not Found, 404" 
time="2018-04-12T16:31:23Z" level=info msg="Stack Create Event Received" eventId=9e0f24bd-5a89-433b-b001-9b7bb076789b resourceId=1st20 
time="2018-04-12T16:31:23Z" level=info msg="Stack Create Event Done" eventId=9e0f24bd-5a89-433b-b001-9b7bb076789b resourceId=1st20 
time="2018-04-12T16:31:26Z" level=info msg="Stack Create Event Received" eventId=b9c18af2-0d83-462d-83e8-96cb86d235de resourceId=1st22 
time="2018-04-12T16:31:26Z" level=info msg="Stack Create Event Done" eventId=b9c18af2-0d83-462d-83e8-96cb86d235de resourceId=1st22

Maybe I should add that server actually started.

Topic		Replies	Views
Rancher flooding logs with errors Rancher	4	2238	March 28, 2023
Detected error on rancher-monitoring (duplicate YAML key) Rancher	0	317	October 16, 2021
ETCD - E0315 leaderelection.go:258] Failed to update lock: etcdserver: mvcc: database space exceeded Rancher	1	1231	March 15, 2019
Rancher provisioning error Rancher	0	1039	September 29, 2022
I need help with importing my cluster	2	1056	October 10, 2019

Duplicate Key Violations

Related topics