Go Routine Panic error in Rancher Server container

Archive Rancher 1.x
1

Hello Team,
We are using Rancher version 1.6.18.
Docker Version:
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-104.git4ef4b30.el7.x86_64
Go version: go1.10.3
Git commit: 4ef4b30/1.13.1
Built: Tue Sep 24 18:53:48 2019
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-104.git4ef4b30.el7.x86_64
Go version: go1.10.3
Git commit: 4ef4b30/1.13.1
Built: Tue Sep 24 18:53:48 2019
OS/Arch: linux/amd64
Experimental: false

We are getting following error in Rancher Server Container and not able to login to Rancher. When we enter the creds to reloads to login page again:

2020/01/30 14:41:45 http: proxy error: net/http: request canceled
2020/01/30 14:41:45 http: proxy error: net/http: request canceled
time=“2020-01-30T14:41:47Z” level=info msg=“Starting go-machine-service…” gitcommit=v0.39.2 service=gms
time=“2020-01-30T14:41:47Z” level=info msg=“Waiting for handler registration (1/2)” service=gms
2020-01-30 14:41:47,763 ERROR [:] [] [] [] [tp1020391880-17] [i.g.i.g.r.handler.ExceptionHandler ] Exception in API for request [http://localhost:8080/v2-beta/externalhandlers/1eh12418]. Error id: [8bff340f-64ea-4abd-86dd-c7388299bae6]. java.lang.NullPointerException: null

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x48c97f]

goroutine 24 [running]:
panic(0xadc180, 0xc82000a0a0)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events.glob.func1.1(0x9860c0, 0x0, 0x0)
/go/src/github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events/external_handler.go:55 +0x10f
github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events.waitForTransition(0xc82015a6a0, 0x0, 0x0)
/go/src/github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events/util.go:15 +0x1db
github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events.glob.func1(0xc82088c480, 0xc820312000, 0x0, 0x0)
/go/src/github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events/external_handler.go:56 +0xe8
github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events.(*EventRouter).createExternalHandler(0xc8203b6140, 0x0, 0x0)
/go/src/github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events/external_handler.go:38 +0x3f3
github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events.(*EventRouter).Start(0xc8203b6140, 0xc8202e67e0, 0x0, 0x0)
/go/src/github.com/rancher/go-machine-service/vendor/github.com/rancher/event-subscriber/events/listener.go:82 +0x42
main.main.func1(0xc8200162db, 0x18, 0xc820016072, 0x14, 0xc820012112, 0x28, 0xc8202e67e0, 0xc82022cb40)
/go/src/github.com/rancher/go-machine-service/main.go:46 +0x405
created by main.main
/go/src/github.com/rancher/go-machine-service/main.go:49 +0x2cb

time=“2020-01-30T14:41:47Z” level=fatal msg=“Unable to start event router” error=“Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [baseType=error, code=Internal Server Error, message=Internal Server Error] from [http://localhost:8080/v2-beta/externalhandlers/1eh12418]”
2020-01-30 14:41:47,776 ERROR [8aa81a02-e889-4061-a59b-c83f427b33f4:44073150] [externalHandler:12419] [externalhandler.activate] [] [ecutorService-3] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.InvalidResultException: Exactly one row expected for refresh. Record does not exist in database.

@vincent @superseb: Could you please help us resolve this problem?

Thank you

Regards,
Yash

2

Following is the list of error:

2020-01-30 14:52:00,328 INFO [main] [ConsoleStatus] Starting [91/93]: VolumeStoragePoolMapRemove
2020-01-30 14:52:00,328 INFO [main] [ConsoleStatus] Starting [92/93]: WebhookServiceLauncher
2020-01-30 14:52:00,328 INFO [main] [ConsoleStatus] Starting [93/93]: project.template.reload
2020-01-30 14:52:00,377 INFO [main] [ConsoleStatus] [DONE ] [20638ms] Startup Succeeded, Listening on port 8081
2020-01-30 14:52:09,721 ERROR [23267b64-1953-4a99-8b4d-1fcc03ba09e7:44072410] [externalHandler:11959] [externalhandler.create] [] [ecutorService-2] [.p.p.c.g.GenericResourceProcessState] Resource is null, can’t find resource id [11959]
2020-01-30 14:52:09,721 ERROR [23267b64-1953-4a99-8b4d-1fcc03ba09e7:44072410] [externalHandler:11959] [externalhandler.create] [] [ecutorService-2] [.p.p.c.g.GenericResourceProcessState] Resource is null, can’t find resource id [11959]
2020-01-30 14:52:09,733 ERROR [1005b7ac-a076-4db1-814b-b512d5f8999d:44072414] [externalHandler:11963] [externalhandler.create] [] [ecutorService-3] [.p.p.c.g.GenericResourceProcessState] Resource is null, can’t find resource id [11963]
2020-01-30 14:52:09,742 ERROR [:] [] [] [] [ecutorService-2] [.e.s.i.ProcessInstanceDispatcherImpl] Unknown exception running process [externalhandler.create:44072410] on [11959] java.lang.IllegalStateException: Failed to find process instance for [44072410]

Regards,
Yash

3

Hello Team,
Also while setting access control in Rancehr UI, we are getting the following error:
" No auth provider configured "
Let me know if you need more information.
Thank you

Regards,
Yash

4

Hello Team,
The issue is resolved as the two databases(cluster) were simultaneously trying to access the same container.
After stopping two database and starting one database resolved the issue.
Thank you

Regards,
Yash