IO error at virtual disk

Its a VM(SLES 12 SP2) running on ESX 6.5. ESX is not showing any error, while at OS we are getting following errors:

kernel: [787030.198763] end_request: I/O error, dev sda, sector 3759139416
kernel: [1530802.448197] end_request: I/O error, dev sda, sector 3754016448
kernel: [1912330.306074] end_request: I/O error, dev sda, sector 3719306720
kernel: [1954566.577314] end_request: I/O error, dev sda, sector 3722956392

The disk(sda) is a virtual disk.

Please help, what and where should I check ?

Hi sharfuddin,

kernel: [787030.198763] end_request: I/O error, dev sda, sector 3759139416
Please help, what and where should I check ?

are those reported sector numbers within the virtual disk’s size? If not, those errors could be the result of a on-disk corruption (invalid sector numbers/offsets stored in whatever you put on that disk).

Everything else depends on how that virtual disk is provisioned… is it from some VMFS pool? SAN? If it’s some back-end storage system, does it report any errors?

Regards,
J

[QUOTE=jmozdzen;54722]Hi sharfuddin,

kernel: [787030.198763] end_request: I/O error, dev sda, sector 3759139416
Please help, what and where should I check ?

are those reported sector numbers within the virtual disk’s size? If not, those errors could be the result of a on-disk corruption (invalid sector numbers/offsets stored in whatever you put on that disk).

Everything else depends on how that virtual disk is provisioned… is it from some VMFS pool? SAN? If it’s some back-end storage system, does it report any errors?

Regards,
J[/QUOTE]

are those reported sector numbers within the virtual disk’s size?

How can I find this… via “fdisk -l” ? Actually don’t have access to the system right now.

is it from some VMFS pool?
Yes.

SAN?
No, locally attached disk on ESX host

Hi sharfuddin,

[QUOTE=sharfuddin;54724]>are those reported sector numbers within the virtual disk’s size?

How can I find this… via “fdisk -l” ? Actually don’t have access to the system right now. [/QUOTE]

yes, can be a good information source. Have a look at the start and end sector information for your partitions.

[QUOTE=sharfuddin;54724]>SAN?
No, locally attached disk on ESX host[/QUOTE]

You wrote that ESX had nothing in the logs - did you also look at ESX-system-related logs or just at those directly related to your virtual machine? If the latter, a look at the system logs might reveal information about intermittent disk errors reported by the system. Also, while I do not trust SMART further than I can throw our servers, maybe some SMART pre-failure error indicators may have increased. For below to work, you’ll of course need to know which disks are actually used - or if it’s hardware RAID, will have to take a totally different approach. Speaking of (hardware or software) RAID: If it’s in use, do you give your arrays a check run at least once a week? The older the disks, the more important this can get to detect silently introduced consistency errors (this is, of course, completely unrelated to ESX, but a generally applicable advice).

[CODE]# esxcli storage core device list

esxcli storage core device smart get -d device[/CODE]

Regards,
J

sharfuddin wrote:
[color=blue]

Its a VM(SLES 12 SP2) running on ESX 6.5. ESX is not showing any
error, while at OS we are getting following errors:

kernel: [787030.198763] end_request: I/O error, dev sda, sector
3759139416
kernel: [1530802.448197] end_request: I/O error, dev sda, sector
3754016448
kernel: [1912330.306074] end_request: I/O error, dev sda, sector
3719306720
kernel: [1954566.577314] end_request: I/O error, dev sda, sector
3722956392

The disk(sda) is a virtual disk.

Please help, what and where should I check ?[/color]

If SLES is reporting these errors, I would do a filesystem check from
within SLES and see what it reports.


Kevin Boyle - Knowledge Partner
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below this post.
Thank you.