SP3 3.0.101-0.47.79 kernel deadlock (dump_trace+0x75/0x300)

[hernan_fernandez]

Hi friends,

I have a SLES sp3 kernel version “Linux sepcomtas01 3.0.101-0.47.79-default #1 SMP Fri Apr 22 06:30:00 UTC 2016 (9d3a4e8) x86_64 x86_64 x86_64 GNU/Linux”. It doesn’t seem to be different from any other sles that I have installed. Any ideas?

server installed on vmware

thanks!

Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540574] Call Trace:
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540593] [] dump_trace+0x75/0x300
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540603] [] dump_stack+0x69/0x6f
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540613] [] print_bad_pte+0x1b2/0x270
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540621] [] vm_normal_page+0x5d/0x70
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540627] [] follow_page+0x29e/0x4a0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540634] [] __get_user_pages+0x174/0x560
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540641] [] __access_remote_vm+0x100/0x1e0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540648] [] access_process_vm+0x59/0x90
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540657] [] proc_pid_cmdline+0x76/0x130
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540665] [] proc_info_read+0xb2/0xf0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540674] [] vfs_read+0xc7/0x130
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540682] [] sys_read+0x53/0xa0
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540690] [] system_call_fastpath+0x16/0x1b
Jul 28 15:55:16 sepcomtas01 kernel: [1667457.540716] [<00007f3c2669fee0>] 0x7f3c2669fedf
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900272] Pid: 20048, comm: sapcimb Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900276] Call Trace:
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900297] [] dump_trace+0x75/0x300
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900307] [] dump_stack+0x69/0x6f
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900317] [] print_bad_pte+0x1b2/0x270
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900325] [] vm_normal_page+0x5d/0x70
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900331] [] follow_page+0x29e/0x4a0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900338] [] __get_user_pages+0x174/0x560
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900345] [] __access_remote_vm+0x100/0x1e0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900353] [] access_process_vm+0x59/0x90
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900362] [] proc_pid_cmdline+0x76/0x130
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900370] [] proc_info_read+0xb2/0xf0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900379] [] vfs_read+0xc7/0x130
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900387] [] sys_read+0x53/0xa0
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900395] [] system_call_fastpath+0x16/0x1b
Jul 28 15:55:29 sepcomtas01 kernel: [1667470.900421] [<00007ff19ab017e0>] 0x7ff19ab017df
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817007] Pid: 20060, comm: ps Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817011] Call Trace:
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817031] [] dump_trace+0x75/0x300
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817041] [] dump_stack+0x69/0x6f
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817051] [] print_bad_pte+0x1b2/0x270
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817059] [] vm_normal_page+0x5d/0x70
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817065] [] follow_page+0x29e/0x4a0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817072] [] __get_user_pages+0x174/0x560
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817079] [] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817086] [] access_process_vm+0x59/0x90
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817095] [] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817103] [] proc_info_read+0xb2/0xf0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817112] [] vfs_read+0xc7/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817119] [] sys_read+0x53/0xa0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817128] [] system_call_fastpath+0x16/0x1b
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.817168] [<00007fcb4d03eee0>] 0x7fcb4d03eedf
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825429] Pid: 20070, comm: ps Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825433] Call Trace:
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825446] [] dump_trace+0x75/0x300
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825455] [] dump_stack+0x69/0x6f
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825463] [] print_bad_pte+0x1b2/0x270
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825470] [] vm_normal_page+0x5d/0x70
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825476] [] follow_page+0x29e/0x4a0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825483] [] __get_user_pages+0x174/0x560
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825490] [] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825497] [] access_process_vm+0x59/0x90
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825505] [] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825512] [] proc_info_read+0xb2/0xf0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825525] [] vfs_read+0xc7/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825550] [] sys_read+0x53/0xa0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825557] [] system_call_fastpath+0x16/0x1b
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.825577] [<00007fada5ef1ee0>] 0x7fada5ef1edf
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849429] Pid: 20076, comm: ps Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849433] Call Trace:
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849445] [] dump_trace+0x75/0x300
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849454] [] dump_stack+0x69/0x6f
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849461] [] print_bad_pte+0x1b2/0x270
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849468] [] vm_normal_page+0x5d/0x70
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849474] [] follow_page+0x29e/0x4a0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849481] [] __get_user_pages+0x174/0x560
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849487] [] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849494] [] access_process_vm+0x59/0x90
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849502] [] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849509] [] proc_info_read+0xb2/0xf0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849516] [] vfs_read+0xc7/0x130
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849524] [] sys_read+0x53/0xa0
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849530] [] system_call_fastpath+0x16/0x1b
Jul 28 15:56:01 sepcomtas01 kernel: [1667502.849545] [<00007f6f0c8afee0>] 0x7f6f0c8afedf
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606921] Pid: 6954, comm: saposcol Tainted: G B E X 3.0.101-0.47.79-default #1
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606924] Call Trace:
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606943] [] dump_trace+0x75/0x300
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606953] [] dump_stack+0x69/0x6f
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606964] [] print_bad_pte+0x1b2/0x270
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606971] [] vm_normal_page+0x5d/0x70
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606977] [] follow_page+0x29e/0x4a0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606984] [] __get_user_pages+0x174/0x560
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606992] [] __access_remote_vm+0x100/0x1e0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.606999] [] access_process_vm+0x59/0x90
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607009] [] proc_pid_cmdline+0x76/0x130
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607016] [] proc_info_read+0xb2/0xf0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607026] [] vfs_read+0xc7/0x130
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607034] [] sys_read+0x53/0xa0
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607042] [] system_call_fastpath+0x16/0x1b
Jul 28 15:56:16 sepcomtas01 kernel: [1667517.607067] [<00007f3c2669fee0>] 0x7f3c2669fedf

[Jens-U]

Hi hernan_fernandez,

if you have (long-term) support, it might be a good idea to open a service request to have some SUSE engineer pin-point the actual cause of the problem.

Do you have more in the logs than this (ok, the actual deadlock messages are missing, but I’m referring to further errors/warnings reported prior to the deadlock, since last reboot). Maybe the VMware log offers more details as well.

Other than that? I have nothing to point at. You might try to “fix” things by rebooting the VM (although 19 days isn’t much uptime, for sure) and/or verify the file systems and installed packages for corruption. But all that is nothing more than a shot in the dark, as would be asking for an extended hardware check to rule out memory errors.

Regards,
J