sles11 suddently drops all services.....

SUSE Product Topics SLES Configure-Administer
1

Hello All,

i faced the following bizzare situation,

suddently 2 of our zVM based linux systems stoped responding via shh,ping was ok

after accessing them via console i noticed that ALL services were down…

/var/log/messages reported this:

Sep 16 12:35:09 linux-8 kdm_config[39122]: Multiple occurrences of key 'StaticServers' in section [General] of /usr/share/kde4/config/kdm/kdmrc
Sep 16 12:35:09 linux-8 kdm_config[39122]: Multiple occurrences of key 'UseTheme' in section [X-*-Greeter] of /usr/share/kde4/config/kdm/kdmrc
Sep 16 12:36:24 linux-8 kdm: 192.0.1.142:3[39126]: Cannot connect to 192.0.1.142:3, giving up
Sep 16 12:36:24 linux-8 kdm[9405]: Display 192.0.1.142:3 cannot be opened
Sep 16 12:37:35 linux-8 kernel: svc: 10.2.8.92, port=984: unknown version (43690890 for prog 100003, nfsd)
Sep 16 12:38:01 linux-8 kdm_config[39270]: Multiple occurrences of key 'StaticServers' in section [General] of /usr/share/kde4/config/kdm/kdmrc
Sep 16 12:38:01 linux-8 kdm_config[39270]: Multiple occurrences of key 'UseTheme' in section [X-*-Greeter] of /usr/share/kde4/config/kdm/kdmrc
Sep 16 12:38:03 linux-8 kdm: 192.0.1.142:3[39274]: Received unknown or unexpected command -2 from greeter
Sep 16 12:38:03 linux-8 kdm: 192.0.1.142:3[39274]: Abnormal termination of greeter for display 192.0.1.142:3, code 1, signal 0
Sep 16 12:38:29 linux-8 syslog-ng[8700]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/console)=0', dropped='udp(AF_INET(10.2.2.84:514))=0', processed='center(queued)=50890', processed='center(received)=25325', processed='destination(messages)=787', processed='destination(mailinfo)=427', processed='destination(mailwarn)=66', processed='destination(logserver)=24832', processed='destination(authlog)=24045', processed='destination(localmessages)=0', processed='destination(newserr)=0', processed='destination(mailerr)=0', processed='destination(warn)=168', processed='destination(console)=36', processed='destination(netmgm)=0', processed='destination(mail)=493', processed='destination(xconsole)=36', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='destination(newscrit)=0', processed='destination(newsnotice)=0', processed='source(src)=25325'
Sep 16 12:38:30 linux-8 kdm_config[39300]: Multiple occurrences of key 'StaticServers' in section [General] of /usr/share/kde4/config/kdm/kdmrc
Sep 16 12:38:30 linux-8 kdm_config[39300]: Multiple occurrences of key 'UseTheme' in section [X-*-Greeter] of /usr/share/kde4/config/kdm/kdmrc
Sep 16 12:38:30 linux-8 syslog-ng[8700]: Termination requested via signal, terminating;
Sep 16 12:38:30 linux-8 kernel: Kernel logging (proc) stopped.
Sep 16 12:38:30 linux-8 kernel: Kernel log daemon terminating.
Sep 16 12:38:30 linux-8 syslog-ng[8700]: syslog-ng shutting down; version='2.0.9'

any thoughts where and what to look for? :frowning:

Michael.

2

Hi Michael,

after accessing them via console i noticed that ALL services were down…

well, at least not the console login :wink:

[…]

these kdm messages might just be the result of either a stopped X server or failing network communications. Did anything turn up before that?

kernel: svc: 10.2.8.92, port=984: unknown version (43690890 for prog 100003, nfsd)

that one looks strange - I’ll take it that 10.2.8.92 is a known NFS client, that has not been tampered with? Then it points at garbled network traffic (from portmapper’s POV), which could be the consequence of some general low-level faile, i.e. corruption somewhere at kernel level.

Sep 16 12:38:30 linux-8 syslog-ng[8700]: Termination requested via signal, terminating;

was this initiated by “shutdown”, as a consequence of your actions? Else I’d look for more details, i.e. checking when the services started shutting down and if it may have been some “shutdown” call somewhere (manually, or power monitoring, or system management services, …).

If it looks like an issued shutdown command, check last logins and the bash histories of those users for traces of manual actions.

Regards,
Jens

3

thank you for your quick response,

i discovered that this host was perfoming a vulnerability scanner which i wasnt aware of.
also i noticed these 2 specific systems had vnc open and kde env , from the xinetd logs vnc was also targeted…

is anyone aware if vnc has a vulnerability like this?

Michael

4

also noticed that vulnerabily scan was running on other systems as well,

also noticed that these 2 systems had kdm instead of gdm,

anyone had issues when using kdm instead of gdm?

i will setup a test server and try to abuse vnc service with kdm…

Michael.

5

[QUOTE=maikcat;29643]also noticed that vulnerabily scan was running on other systems as well,

also noticed that these 2 systems had kdm instead of gdm,

anyone had issues when using kdm instead of gdm?

i will setup a test server and try to abuse vnc service with kdm…

Michael.[/QUOTE]

Hi Michael,
Now I come across the same problem as you did, there are many similar logs printed :
Nov 26 09:26:58 billdb2a kdm_config[8305]: Multiple occurrences of key ‘UseTheme’ in section [X--Greeter] of /usr/share/kde4/config/kdm/kdmrc
Nov 26 09:26:59 billdb2a kdm: 10.70.209.40:1[8335]: Received unknown or unexpected command -2 from greeter
Nov 26 09:26:59 billdb2a kdm: 10.70.209.40:1[8335]: Abnormal termination of greeter for display 10.70.209.40:1, code 1, signal 0
Nov 26 09:26:59 billdb2a kernel: [469170.655111] svc: 10.20.40.206, port=340: unknown version (104740642 for prog 100003, nfsd)
Nov 26 09:27:00 billdb2a kdm_config[8840]: Multiple occurrences of key ‘UseTheme’ in section [X--Greeter] of /usr/share/kde4/config/kdm/kdmrc
Nov 26 09:27:01 billdb2a kdm: 10.70.209.40:2[8845]: Received unknown or unexpected command -2 from greeter
Nov 26 09:27:01 billdb2a kdm: 10.70.209.40:2[8845]: Abnormal termination of greeter for display 10.70.209.40:2, code 1, signal 0
Nov 26 09:27:03 billdb2a kernel: Kernel logging (proc) stopped.
Nov 26 09:27:03 billdb2a kernel: Kernel log daemon terminating.
Nov 26 09:27:03 billdb2a syslog-ng[8093]: Termination requested via signal, terminating;
Nov 26 09:27:03 billdb2a syslog-ng[8093]: syslog-ng shutting down; version=‘2.0.9’

many network related services stopped at that time, I cannot find reason.

you said you had discovered the host was perfoming a vulnerability scanner. do you know which the tool which made vulnerability scan and
it will be better if you can remember how to reproduce,thank you.

6

Good Morning sir,

Nessus was used in my case (possibly the paid version),
after opening a SR to novell the response was that nessus is commercial software and they cannot buy and perform the same test
in their labs.

Michael.