I am using NVMe devices on SLES15 sp1 and sp2 both. These NVMe devices are exposed through SAN. Because they exported from SAN, these devices can go readonly or back to readwrite due to some changes by host or internally. I am trying to find out a way in sales host if the device is readonly without doing any IOs to it. The small dd command that I did, succeeded but IO errors appeared in dmesg. At a later point when device turned rw, again I couldn’t find if the device is rw now. dd command succeeded and no bad messages in dmesg.
In summary, I would like to understand if there is a programmatic way to find out if device is read only. I do see a file called Â/sys/block/nvmeXnY/ro but that always remained 0 even when device is readonly. Moreover, when I try “blockdev Ârereadpt /dev/nvmeXnY; blockdev -report /dev/nvmeXnY”, the output is always RW.
Hi
Are there nvme* kernel modules in use lsmod | grep nvme? If so, then can look at the module options (parm) via the /sbin/modinfo <module> and systool -vm <module> to see the parm value in use.
For example;
/sbin/modinfo nvme-core | grep parm:
parm: fail_request:charp
parm: multipath:turn on native support for multiple controllers per subsystem (bool)
parm: admin_timeout:timeout in seconds for admin commands (uint)
parm: io_timeout:timeout in seconds for I/O (uint)
parm: shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm: max_retries:max number of retries a command may have (byte)
parm: default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
parm: force_apst:allow APST for newly enumerated devices even if quirked off (bool)
parm: streams:turn on support for Streams write directives (bool)
systool -vm nvme_core
Module = "nvme_core"
....
Parameters:
admin_timeout = "60"
default_ps_max_latency_us= "0"
force_apst = "N"
io_timeout = "30"
max_retries = "5"
multipath = "Y"
shutdown_timeout = "5"
streams = "N"
Thanks for the response. I couldn’t find systool utility for my sles15sp1. But in spite of that, I don’t find any parameter indicating me the state of the particular device/namespace. …
e.g. in case of SCSI, I could do "blockdev --rereadpt ; blockdev --report " to get the latest state of the device, but the same isn’t working for NVMe.
Following is what I find from modinfo -
/sbin/modinfo nvme-core | grep parm:
parm: fail_request:charp
parm: multipath:turn on native support for multiple controllers per subsystem (bool)
parm: admin_timeout:timeout in seconds for admin commands (uint)
parm: io_timeout:timeout in seconds for I/O (uint)
parm: shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm: max_retries:max number of retries a command may have (byte)
parm: default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
parm: force_apst:allow APST for newly enumerated devices even if quirked off (bool)
parm: streams:turn on support for Streams write directives (bool)
Hi
The cnf command helps 
cnf systool
Program 'systool' is present in package 'sysfsutils', which is installed on your system.
Absolute path to 'systool' is '/usr/bin/systool'. Please check your $PATH variable to see whether it contains the mentioned path.
zypper if sysfsutils
Information for package sysfsutils:
-----------------------------------
Repository : SLE-Module-Basesystem15-SP1-Updates
Name : sysfsutils
Version : 2.1.0-3.3.1
zypper in sysfsutils
So are the nvme modules in use at all lsmod | grep nvme?
Below are the outputs requested. But still my query is same, how does this output help in determining device level state ?
$ lsmod | grep nvme
nvme 36864 4
nvme_core 102400 7 nvme
$ /sbin/modinfo nvme-core | grep parm:
parm: fail_request:charp
parm: multipath:turn on native support for multiple controllers per subsystem (bool)
parm: admin_timeout:timeout in seconds for admin commands (uint)
parm: io_timeout:timeout in seconds for I/O (uint)
parm: shutdown_timeout:timeout in seconds for controller shutdown (byte)
parm: max_retries:max number of retries a command may have (byte)
parm: default_ps_max_latency_us:max power saving latency for new devices; use PM QOS to change per device (ulong)
parm: force_apst:allow APST for newly enumerated devices even if quirked off (bool)
parm: streams:turn on support for Streams write directives (bool)
systool -vm nvme_core
Module = “nvme_core”
Attributes:
coresize = “102400”
initsize = “0”
initstate = “live”
refcnt = “7”
srcversion = “7EBDDE9C727C600341E6CA1”
supported = “Yes”
taint = “”
uevent =
version = “1.0”
Parameters:
admin_timeout = “60”
default_ps_max_latency_us= “100000”
force_apst = “N”
io_timeout = “30”
max_retries = “5”
multipath = “N”
shutdown_timeout = “5”
streams = “N”
Sections:
.bss = “0xffffffffc01b3340”
.data…read_mostly = “0xffffffffc01b2d98”
.data = “0xffffffffc01b1120”
.exit.text = “0xffffffffc01a9e41”
.gnu.linkonce.this_module= “0xffffffffc01b3000”
.init.text = “0xffffffffc018b000”
.note.gnu.build-id = “0xffffffffc01aa000”
.orc_unwind = “0xffffffffc01adf73”
.orc_unwind_ip = “0xffffffffc01ac4a7”
.parainstructions = “0xffffffffc01b0cd0”
.ref.data = “0xffffffffc01b2e20”
.rodata.str1.1 = “0xffffffffc01ab515”
.rodata.str1.8 = “0xffffffffc01aaab8”
.rodata = “0xffffffffc01aa340”
.smp_locks = “0xffffffffc01abfe0”
.strtab = “0xffffffffc01906c8”
.symtab = “0xffffffffc018c000”
.text = “0xffffffffc01a0000”
__bpf_raw_tp_map = “0xffffffffc01b2da0”
__bug_table = “0xffffffffc01b2bc8”
__jump_table = “0xffffffffc01b1000”
__kcrctab_gpl = “0xffffffffc01aa290”
__ksymtab_gpl = “0xffffffffc01aa030”
__ksymtab_strings = “0xffffffffc01ac034”
__mcount_loc = “0xffffffffc01b07a8”
__param = “0xffffffffc01ac2e0”
__tracepoints_ptrs = “0xffffffffc01ac448”
__tracepoints = “0xffffffffc01b2f40”
__tracepoints_strings= “0xffffffffc01ac460”
__verbose = “0xffffffffc01b2c80”
_ftrace_events = “0xffffffffc01b2e00”
Hi
There are web (AWS) reports with nvme-core timing out, fixes include increasing the max_retries (10) and io_timeout (255) parameters via grub.
What model NVME devices? I see intel has over provisioning issues;
https://www.intel.com/content/www/us/en/support/articles/000022714/memory-and-storage.html
The smartctl command can give insight to the condition of the device as well;
For example;
smartctl -a /dev/nvme0
....
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 1%
....
Have you looked through the journalctl output for errors?
Actually in my case the SSDs are not locked to read-only mode. The IOs go through fine when the NVMe device is made read-write and no errors are seen in /var/log/messages too, but when device is read-only, IOs still pass (echo $? reports 0), but IO failure errors are reported in /var/log/messages file. My question is - is there any attribute on device which indicate the right or current state ?
Hi
Depending on the device, smartctl is the only one. Depending on the device manufacturer there maybe some tools available to run on linux. Power/cabling ok?