Rancher v0.25.0
Cattle v0.58.0
User Interface v0.29.0
Rancher Compose latest
Hooked up to github authentication and behind an nginx proxy.
Can you provide more details on what steps you took to get this error? We’d like to try and reproduce it.
I didn’t take many steps except "install and github authenticate"
What is a little different about my setup is that I have an ELB TCP proxying to an nginx container on 443 that then terminates SSL and sends 8080 traffic to the rancher master container.
If you go to the main menu on the top right of Rancher and select “Host Registration”, is your address set to the URL that is used to publicly access your Rancher instance? That same public URL obviously also has to be the callback URL for Github auth.
@jeremy the instructions for SSL say to set that to a space. Authentication works but in the javascript console I see this:
https://rancher.nacho.space/github/users/tobowers Failed to load resource: the server responded with a status of 502 (Bad Gateway)
The host registration is the URL that the agents use to talk to the rancher/server container. It is usually, but does not necessarily need to be the same as the URL that is used for public access to the API/UI.
@vincent - so setting the host registration url to my internal URI allows me to have hosts that aren’t reaching through the public API. However, I still get the 502s in the javascript console.
Skipping over nginx I see rancher is returning:
{“id”:“5a754911-512d-4a85-a8f5-016e3522b0af”,“type”:“error”,“links”:{},“actions”:{},“status”:500,“code”:“Internal Server Error”,“message”:“Internal Server Error”,“detail”:null}
And for more info… I was able to add a team to an environment, but I cannot add individual users.
There’s a proxy in rancher to send requests to /github/ to their API and passes the response back to the UI, because it can’t talk directly to GitHub. So that’s used when you type in a name to find out if it exists and what the name/avatar/etc are. Teams work slightly differently and doesn’t require a hit to the proxy, which is why they work (there isn’t an avatar for a team, and they can’t be looked up by name in the API).
@vincent - cool… that makes sense. How do I fix it? I need to have the Host Registration set to my internal DNS for agents to join.
The Host registration URL is only used for…host registration. It had nothing to do with this issue.
The UI is missing an error check to the message you see on the screen comes from there, but ultimately the problem is it didn’t get back a 200 and a json blob about you. So to fix it we need to figure out why you’re getting back a 500 and where it’s coming from. It could be GitHub returning 500 and we’re passing it along, but more likely there’s an unhandled exception in cattle and that’s our generic 500 error. Is there anything like a stack trace in rancher/server logs after hitting the URL?
Ooo… indeed there is (I missed that before)
2015-07-09 19:00:51,797 ERROR [:] [] [] [] [qtp688652575-43] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [io.github.ibuildthecloud.gdapi.request.ApiRequest@2000547a] java.lang.IllegalStateException: Can not set a value on a null target
at io.cattle.platform.object.util.DataAccessor.getTargetMap(DataAccessor.java:175) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.object.util.DataAccessor.get(DataAccessor.java:147) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.auth.github.GithubUtils.getAccessToken(GithubUtils.java:263) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.iaas.api.request.handler.GithubProxy.generate(GithubProxy.java:37) ~[cattle-iaas-auth-logic-0.5.0-SNAPSHOT.jar:na]
at io.github.ibuildthecloud.gdapi.request.handler.AbstractResponseGenerator.handle(AbstractResponseGenerator.java:14) ~[gdapi-java-server-0.11.1.1.jar:na]
at io.github.ibuildthecloud.gdapi.servlet.ApiRequestFilterDelegate.doFilter(ApiRequestFilterDelegate.java:88) ~[gdapi-java-server-0.11.1.1.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter$1.runInContext(ApiRequestFilter.java:60) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
at io.cattle.platform.api.servlet.ApiRequestFilter.doFilter(ApiRequestFilter.java:53) [cattle-framework-api-0.5.0-SNAPSHOT.jar:na]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) [jetty-servlet-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) [jetty-servlets-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) [jetty-servlets-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) [jetty-servlet-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) [jetty-servlet-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) [jetty-security-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) [jetty-servlet-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.Server.handle(Server.java:370) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) [jetty-http-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) [jetty-http-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) [jetty-server-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) [jetty-io-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) [jetty-io-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) [jetty-util-8.1.11.v20130520.jar:8.1.11.v20130520]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) [jetty-util-8.1.11.v20130520.jar:8.1.11.v20130520]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]Looks like there is a path in cattle that is allowing you to hit the github proxy without an AccessToken for github stored/ loaded and as a result when it tries to retrieve the token it errors. Without having access to it locally to debug I cannot say for certain what the exact cause but that is what it seems to be based on the provided stacktrace.
Happy to help with a screenshare, etc
Could this be related?
WebSocket connection to ‘wss://#{redacted_host_name}/v1/stats/#{redacted_long_token}’ failed: HTTP > Authentication failed; no valid credentials available
Update for those playing along at home. Thanks to a screenshare with @tobowers we were able to track down the cause of this issue, which is the proxy_buffer_size, which the SSL setup instructions tell you to put into the nginx config. That particular buffer length is needed for host -> rancher/server connections to work correctly though. We’ll be working on resolving this so that SSL + GitHub + host connections can all work at the same time.