Keep getting logged out on latest rancher

Rancher 1.x
1

On my personal dedicated server I am running a Rancher server now, unfortunately it seems that every time I perform an action it logs me out right after and dumps me back to the login screen.

It also seems that pages aren’t updating properly to backend actions.

Rancher v0.46.0
Cattle v0.112.0
User Interface v0.65.0
Rancher Compose v0.5.2

Are there any gotchas I don’t know about yet?

2

There aren’t any known gotchas with v0.46.0. I have been using v0.46.0, but haven’t had those issues. I know it’s been a couple of days. Are you still facing this issue?

3

Yeah still happening. I can only seem to perform a few actions before I get hit with a login screen. Same behavior across OSX and Linux, firefox (iceweasel) and chrome.

4

Is the server or MySQL out of disk space maybe?

5

No, around 190G free for the mount space

6

chas@reactor:/var/tmp/mysql$ df -h | grep /dev/md2 /dev/md2 206G 5.6G 190G 3% / chas@reactor:/var/tmp/mysql$ sudo du -sh . 115M .

(bind mount /var/tmp/mysql:/var/lib/mysql)

7

hi,
try to increase memory on rancher server.
Iv expirenced similar things and it usually happens when memory is low and processes ar being queued up…and when rancher server has a oome.

Br hw

8

It has 16gb of ram available to it, and 24 cores :
It also has no restrictions on resource usage

9

OK,
just to make sure, you have put JAVA_OPTS/Xmx on rancher-server to be 16G?

br hw

10

Sorry long weekend.

No actually I haven’t touched the JAVA_OPTS at all, didn’t think to assuming the application would scale to whatever docker gave it.

11

I migh be wrong, but i think rancher server default has xmx1024M so then it would not matter if you actually have 16g on server. you might experience oom.

Try running an higher xmx fully commited.

Br hw

12

Is it available as an environment variable?

13

@Chas
Based on the documentation:

http://docs.rancher.com/rancher/faqs/troubleshooting/#rancher-server-was-working-and-has-now-slowed-down-tremendously-how-can-i-recover

It looks like it’s an environment variable for the memory.

14

Thank you I was having a hard time finding that. Going to update my POC node.

15

The issue is continuing even with greatly expanded resources.

Additionally I am now getting an error when I browse to the catalogs page. It says
"error
Reload to try again or log out
"

16

Yeah this is still full force. Getting logged out constantly in 0.47, this never happened before and I just verified that my old 0.46 was not doing this until I upgraded to 0.47.

These rancher instances are barely doing anything, one agent and stack each. 0.47 has a lot of new toys but it’s been extremely unstable for me. I find it hard to believe this is a resource problem.

17

You shouldn’t need gigs of ram to login…

  • What kind of auth is enabled (github, local, ldap)?
  • Are you using any proxies or anything special in front of rancher (e.g. SSL termination)

To get some idea what’s going on, can you:

  • Go to the login page
  • Open up the browser dev tools (Chrome’s are the best)
  • Go to the “Network” tab
  • Check “Preserve Log”
  • Login and do something that causes you to get kicked out.
  • Take a screenshot of the network tab (scrolled down to the bottom)
  • There will likely be one response that says status 401… click on that and screenshot the “Headers” and “Preview” tabs contents.

Or if you’re willing to give me access to the instance or do a screenshare, you can contact me directly on the forum or email (my first name at rancher dot com).

18

I would give you access but it sits on a private network presently (not natted for remote access).

Local auth, no proxies or reverse proxy (no LB etc)

I nabbed the catalog page error first since I can produce it at will

rancher_error01.png352×1442 46.6 KB

edit forgot this:

Screenshot-23.png1090×928 116 KB

19

Could you look into rancher-server logs to see if at the start of rancher-server, if rancher was able to clone the catalog? It looks as if Rancher Catalog might be causing your issues. Does the machine have outbound internet access to be able to git clone the catalog?

In the beginning of the logs after the steps to set up Rancher server are complete, you should see something like

2015-11-30 18:51:22,194 INFO    [main] [ConsoleStatus] [102/102] [33071ms] [0ms] Starting storage-simulator 
18:51:22.298 [main] INFO  ConsoleStatus - [DONE ] [39406ms] Startup Succeeded, Listening on port 8081
time="2015-11-30T18:51:24Z" level=info msg="Starting Rancher Catalog service" 
time="2015-11-30T18:51:24Z" level=info msg="Cloning the catalog from github url https://github.com/rancher/rancher-catalog.git" 
Cloning into './DATA'...
time="2015-11-30T18:51:24Z" level=info msg="Starting websocket proxy. Listening on [:8080], Proxying to cattle API at [localhost:8081], Monitoring parent pid [9]."
20

Ah yes, interesting:

time="2015-12-03T22:20:24Z" level=info msg="Starting Rancher Catalog service" 
time="2015-12-03T22:20:24Z" level=info msg="Cloning the catalog from github url https://github.com/rancher/rancher-catalog.git" 
fatal: destination path './DATA' already exists and is not an empty directory.

Additionally I am beginning to wonder if my log-out issues are related to IceWeasel since I was unable to reproduce in chrome today (although I have reproduced it in chrome on my home laptop running Chrome on OSX)

I dug around in the container but can’t tell where the relative reference for ./DATA is