I’ve tried to create 4 new hosts with the new agent version, they became visible in the infrastructure view, so I’ve removed one host and containers moved to the new one (load balancer wasn’t still available in the new hosts, probably because it was in error state.
I then tried to delete another host, but now everything is stuck, frontend never loads (identities ajax request never returns a response), rancher server has just errors like
2016-12-01 10:15:34,686 ERROR [:] [] [] [] [tp1830712962-17] [.AgentQualifierAuthorizationProvider] Failed to determine the proper agent ID for subscription for account [731]
I’ve tried to restore the previous version, still the same issue with upgrading the load balancers
2016-12-01 15:52:21,314 ERROR [:] [] [] [] [ecutorService-3] [.e.s.i.ProcessInstanceDispatcherImpl] Expected state running but got error: Timeout getting IP address
2016-12-01 15:52:24,171 ERROR [:] [] [] [] [tp1830712962-17] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [http://10.0.1.161:8080/v1/configcontent/metadata-answers]. Error id: [ff584964-5e1c-4b60-ab4a-664023a3e4dc]. org.yaml.snakeyaml.error.YAMLException: org.eclipse.jetty.io.EofException
at org.yaml.snakeyaml.Yaml.dumpAll(Yaml.java:247) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.Yaml.dump(Yaml.java:221) ~[snakeyaml-1.15.jar:na]
at io.cattle.platform.configitem.context.impl.ServiceMetadataInfoFactory.writeMetadata(ServiceMetadataInfoFactory.java:100) ~[cattle-config-item-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.configitem.server.model.impl.MetadataConfigItem.handleRequest(MetadataConfigItem.java:42) ~[cattle-config-item-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.configitem.server.impl.ConfigItemServerImpl.handleDownload(ConfigItemServerImpl.java:97) ~[cattle-config-item-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.configitem.server.impl.ConfigItemServerImpl.handleRequest(ConfigItemServerImpl.java:40) ~[cattle-config-item-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.configitem.api.manager.ConfigContentManager.handle(ConfigContentManager.java:80) ~[cattle-config-item-api-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.configitem.api.manager.ConfigContentManager.getByIdInternal(ConfigContentManager.java:49) ~[cattle-config-item-api-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.resource.impl.AbstractBaseResourceManager.getById(AbstractBaseResourceManager.java:61) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.ResourceManagerRequestHandler.generate(ResourceManagerRequestHandler.java:50) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.handle(DefaultReadWriteApiDelegate.java:28) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.DefaultReadWriteApiDelegate.read(DefaultReadWriteApiDelegate.java:18) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at sun.reflect.GeneratedMethodAccessor428.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:280) ~[spring-tx-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at com.sun.proxy.$Proxy55.read(Unknown Source) ~[na:na]
at io.github.ibuildthecloud.gdapi.request.handler.write.ReadWriteApiHandler.handle(ReadWriteApiHandler.java:22) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:99) ~[cattle-framework-java-server-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:95) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:88) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) [jetty-security-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.Server.handle(Server.java:499) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408) ~[jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302) ~[jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129) ~[jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:684) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208) ~[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:480) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:768) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801) [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:378) ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.gzip.DeflatedOutputStream.deflate(DeflatedOutputStream.java:74) ~[jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.gzip.DeflatedOutputStream.write(DeflatedOutputStream.java:64) ~[jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.gzip.GzipOutputStream.write(GzipOutputStream.java:46) ~[jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at org.eclipse.jetty.servlets.gzip.AbstractCompressedStream.write(AbstractCompressedStream.java:226) ~[jetty-servlets-9.2.11.v20150529.jar:9.2.11.v20150529]
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) ~[na:1.8.0_72]
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) ~[na:1.8.0_72]
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) ~[na:1.8.0_72]
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) ~[na:1.8.0_72]
at java.io.Writer.write(Writer.java:127) ~[na:1.8.0_72]
at org.yaml.snakeyaml.emitter.Emitter.writeIndent(Emitter.java:1110) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.emitter.Emitter$ExpectBlockMappingKey.expect(Emitter.java:624) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.emitter.Emitter.emit(Emitter.java:216) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serializeNode(Serializer.java:181) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serializeNode(Serializer.java:205) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serializeNode(Serializer.java:191) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serializeNode(Serializer.java:206) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serializeNode(Serializer.java:206) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serializeNode(Serializer.java:206) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.serializer.Serializer.serialize(Serializer.java:112) ~[snakeyaml-1.15.jar:na]
at org.yaml.snakeyaml.Yaml.dumpAll(Yaml.java:243) ~[snakeyaml-1.15.jar:na]
... 56 common frames omitted
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) ~[na:1.8.0_72]
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) ~[na:1.8.0_72]
at sun.nio.ch.IOUtil.write(IOUtil.java:148) ~[na:1.8.0_72]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504) ~[na:1.8.0_72]
at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:172) ~[jetty-io-9.2.11.v20150529.jar:9.2.11.v20150529]
... 88 common frames omitted
2016-12-01 15:52:24,171 ERROR [:] [] [] [] [tp1830712962-17] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [http://10.0.1.161:8080/v1/configcontent/metadata-answers]. Error id: [ff584964-5e1c-4b60-ab4a-664023a3e4dc]. org.yaml.snakeyaml.error.YAMLException: org.eclipse.jetty.io.EofException
It is most likely caused either by system being overloaded in general, or metadata container being stopped.
@Daniel_Skinner the exception you are seeing, is a completely diff one from the first 2 reported by @alex88. If you still have setup around, can you check if your infrastructure services are up (ipsec/networkServices/healthcheck) are up and running?
We are going to look at all the issues reported in this post.
@alena it’s all up and running now but I do think some of the infrastructure services were failing too. After I’d got it all working by rebuilding the load balancers there were a few stopped containers for ipsec and healthcheck I think despite all those stacks being healthy by this point.
@alex88 removed entries should be left in the DB, so the query should be still relevant.
@Daniel_Skinner failing ipsec/health check can result in lb (and other services) being in a bad state as prereq for upgrade completion would be service.healthState=healthy. If health check service was having problems, it is quite possible that it impacted LB upgrade.
Note that this bit a number of users, but it also looks like there are a number of causes (boot2docker, etc). The defacto bug seems to be this one: https://github.com/rancher/rancher/issues/6858
@Daniel_Skinner , I also ran into errors like “Error response from daemon: Driver devicemapper failed to remove root filesystem”, and I worked around the error by rebooting the host. I’m using CentOS 7 with direct-lvm, which I thought solved this issue (Not the default LVM & Loop mode, which is not recommended for production).
@Stefan_Lasiewski thank you for the reference, system services not getting ip addresses can be very much likely caused by b2d.
@alex88 the bug that you hit - NPE on lb upgrade - may be caused by the fact that lbs-to-upgrade were created prior to inserviceupgrade feature being in. With that feature we’ve started versioning launch configs, and based on NPE it looks like the version is null for the launch config object. “Created” date would be the question to this answer.
@alena : Thanks. I don’t use boot2docker, and I’ve worked past our problems and Rancher seems to be working well now. My problems may have all been caused by this issue, which I’ll post for others: