We are using “Logic Monitor” tool in our environment to monitoring “SUSE Linux Enterprise Server for SAP Applications 12 SP4” running on ec2 instance on AWS cloud. Sometimes we observed that monitoring tool not able to pull data from server due to snmp service getting stuck or hung on SUSE SAP server. We investigated and found that sometime we were not able to found any output for snmpwalk command. We must be restart snmp service to fix the issue. This issue observed almost on all server.
Whenever issue reported , We have to stop snmp service and kill the associated process then again start the snmp process, so we are suspecting this is just like a bug on OS side and we are looking the solution for same. Please share if anyone have any idea about the root cause for such issue and how to fix it.
@Ajay Hi and welcome to the Forum Is it always the same process that inhibits snmp service? Have you inspected the logs to see what is happening when the snmp process stops?
Hi Malcolm,
Thanks for your response. Below my observation. One important thing here why we required to kill process associated with snmp before restart. That I am not getting , please help me to understand if you have any idea.
By looking at the /var/log/messages file, it can be seen the events registered when SNMP service is stopped and started.
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-12T16:00:37.400912+00:00 vhuaidapsbo01 snmpd[12820]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-12T16:00:37.412400+00:00 vhuaidapsbo01 snmpd[12834]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-13T16:00:38.093488+00:00 vhuaidapsbo01 snmpd[14135]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-13T16:00:38.104751+00:00 vhuaidapsbo01 snmpd[14539]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T14:44:59.355601+00:00 vhuaidapsbo01 snmpd[23569]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T14:44:59.367064+00:00 vhuaidapsbo01 snmpd[23580]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T14:45:34.353424+00:00 vhuaidapsbo01 snmpd[23762]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T14:45:34.364951+00:00 vhuaidapsbo01 snmpd[23774]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T15:44:34.609910+00:00 vhuaidapsbo01 snmpd[5015]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T15:44:46.234906+00:00 vhuaidapsbo01 snmpd[5119]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T15:52:20.762388+00:00 vhuaidapsbo01 snmpd[11441]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T15:52:20.772777+00:00 vhuaidapsbo01 snmpd[11452]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T15:55:07.208628+00:00 vhuaidapsbo01 snmpd[13199]: Shutting down snmpd:…failed
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T15:56:02.961205+00:00 vhuaidapsbo01 snmpd[14253]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T16:00:33.265795+00:00 vhuaidapsbo01 snmpd[17460]: Shutting down snmpd:…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-14T16:00:33.312312+00:00 vhuaidapsbo01 snmpd[17468]: Starting snmpd…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-15T16:00:33.924712+00:00 vhuaidapsbo01 snmpd[28892]: Shutting down snmpd:…done
scc_vhuaidapsbo01_210615_1640/messages.txt:2021-06-15T16:00:33.971773+00:00 vhuaidapsbo01 snmpd[28900]: Starting snmpd…done
scc_vhuaidapsbo02_210615_1641/messages.txt:2021-06-15T16:01:04.647602+00:00 vhuaidapsbo02 snmpd[27297]: Shutting down snmpd:…done
scc_vhuaidapsbo02_210615_1641/messages.txt:2021-06-15T16:01:04.695167+00:00 vhuaidapsbo02 snmpd[27305]: Starting snmpd…done
scc_vhuaidapsbo03_210615_1642/messages.txt:2021-06-15T15:59:01.947627+00:00 vhuaidapsbo03 snmpd[29638]: Shutting down snmpd:…done
scc_vhuaidapsbo03_210615_1642/messages.txt:2021-06-15T15:59:01.992288+00:00 vhuaidapsbo03 snmpd[29646]: Starting snmpd…done
However, there are no more details or references about SNMP. This events seem to be logged to /var/log/net-snmpd.log, as shown by the process details:
root 28911 1 0.0 0.0 75224 8936 S 00:00:01 /usr/sbin/snmpd -r -A -LF n /var/log/net-snmpd.log -p /var/run/snmpd.pid
@Ajay Hi that output is from the host your monitoring? What does the net-snmpd.log show? Can you check the output from journalctl looking back from then the snmpd service stopped?
Is it always the same process that inhibits snmp service? Have you inspected the logs to see what is happening when the snmp process stops?