OS : SLES 10.3 X86-64, Linux virtual machine. Running on VMware esxi 4.1 hyper-v
Issue: Unable to execute commands in the shell & unable to login on to the server sometimes.
In our dc one of the production server running SLES 10.3 X86-64 (virtual server), unable to execute commands in the shell, & sometimes unable to login into the server. Through this server we use to login to other servers in the dc. Earlier we faced the issue on this system, upon reboot of the system the problem will get fix. Novell support person analyzed the issue & stated that the / partition has less space, so it is not able to execute commands. We increased the space of /partition with rebooting of the server. Now we are facing the issue again & / is having enough space.
This issue is repeating on the server & we are unable to find the cause of the issue.
I suggest you get these running on the CONSOLE so you have the output during the performance issues. These will show I/O and memory issues.
Examine /var/messages for log entries around the time in question
Examine dmesg output
Implement SNMP on the server and add it to your trending system so you can see when and why it goes wrong and catch it in the act.
Also, this is the sort of thing that happens with an infested server. Precursory check for rootkit: Use rpm -V coreutils and rpm -V coreutils if ANY output is returned, these binaries have been modified, you can post output here. Or use a proper virus scanner ( like ESET ) to look for modified binaries. See if your server IP is listed as a spam source.
So what is the disk usage (df -h)?
[/color]
If tight on space, you can then hunt down the hog with
du -hx --max-depth=1
start at the root of the mount points to check, and work your way (cd)
through the big directories/folders to find the suspects. Those
parameters won’t include any mount points.