Customer started noticing the following error:
Jul 22 13:45:30 node1 kernel: [23580.074503] EXT3-fs error (device dm-3): ext3_lookup: deleted inode referenced: 1352479
and then on 20th Aug, it was noticed 32 errors in b/w 08:47:11 and 08:47:14
Aug 20 08:47:11 node1 kernel: [2510147.739895] EXT3-fs error (device dm-3): ext3_lookup: deleted inode referenced: 1352539
.
.
.
Aug 20 08:47:14 node1 kernel: [2510150.315486] EXT3-fs error (device dm-3): ext3_lookup: deleted inode referenced: 1352539
They also noticed that Application(SAP) using this file system became unfunctional, then a reboot and running the fsck.ext3 was issued to repair the file system.
Now question here is why it got corrupted ? and what should one do to avoid such errors in future.
ext3_lookup: deleted inode referenced:
Now question here is why it got corrupted ? and what should one do to avoid such errors in future.
in addition to file system implementation bugs (we all know these don’t exist ), this type of error is typical for system crashes or defective disk hardware.
Have a look at the server syslog, searching for errors indicating problems when accessing disks (i.e. “kernel.*scsi:” messages). Also, on more complicated setups like SANs, have a look at the storage back-end system and the connecting network.
Generally, with stable storage environments these errors are pretty rare. Nevertheless it is a good idea to have a file system check every year or two - typically requiring a reboot, though, to make sure the file systems aren’t accessed and are unmounted at the time of checking.
If this re-occurs in rather short time frames, i.e. weeks, I’d suspect the storage hardware to be less reliable than required.
), this type of error is typical for system crashes or defective disk hardware.