Problems with Shutdown/Boot in a multipathed Environment

[kaiwindheuser]

Hello!

We use SLES 11 SP2
[SIZE=2]Linux 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012 (3829766) x86_64 x86_64 x86_64 GNU/Linux[/SIZE]

on an Intel Modular Server with Dual Storage Controller Setup.

Content of /etc/multipath.conf:
[SIZE=1]blacklist {
devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]"
devnode "^(hd|xvd)[a-z][[0-9]]”
devnode “^cciss!c[0-9]d[0-9][p[0-9]]”
}

devices {
device {
vendor “Intel”
product “Multi-Flex”
path_grouping_policy “group_by_prio”
getuid_callout “/lib/udev/scsi_id -g -u /dev/%n”
prio “alua”
path_checker tur
path_selector “round-robin 0”
hardware_handler “1 alua”
failback immediate
rr_weight uniform
rr_min_io 100
no_path_retry queue
features “1 queue_if_no_path”
}
}
[/SIZE]

Output of “multipath -ll”
[SIZE=1]222d60001557a1912 dm-1 Intel,Multi-Flex
size=550G features=‘1 queue_if_no_path’ hwhandler=‘1 alua’ wp=rw
|-± policy=‘round-robin 0’ prio=130 status=active
| - 0:0:0:1 sdb 8:16 active ready running -± policy=‘round-robin 0’ prio=1 status=enabled
- 0:0:1:1 sdd 8:48 active ready running 2228900015565f670 dm-0 Intel,Multi-Flex size=40G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=130 status=active | - 0:0:0:0 sda 8:0 active ready running
-+- policy='round-robin 0' prio=1 status=enabled - 0:0:1:0 sdc 8:32 active ready running[/SIZE]

The redundant Storage connection just works fine. We can remove one of the controllers and the Server switches
to the other path.
But when we shutdown the server, SuSE Linux is not able to do a clean unmount.
The following message is printed to the console:
[SIZE=4][COLOR="#FF0000"]
“Not shutting down MD Raid - reboot/halt scripts do this.” missing
Removing multipath targets: May 21 08:53:52 | 22289xxxxx_part2: map in use
May 21 08:53:52 | failed to remove multipath map 22289xxxxx
[/COLOR][/SIZE]

As a consequence it happened one time, that on bootup, fsck (File-System-Check) found
an orphaned inode and remove it.

How can we configure SLES 11 SP2 that it does a clean shutdown? Or is this normal behaviour?

Thanks for your reply in advance!

[amo_vzug]

Hi

It’s possible that mdadmd should shutdown the md-raid-devices.

Try enabling it:

[CODE]

Just check, if it’s already enabled within the runlevel 3 and 5

$ chkconfig --list mdadmd
mdadmd 0:off 1:off 2:off 3:off 4:off 5:off 6:off[/CODE]

If all runlevels are set to “off”, then activate runlevel 3 and 5 with:

$ chkconfig mdadmd on

Perhaps this helps.
Kind regards,
Tom

[kaiwindheuser]

Thanks for your reply.
I tried it.

<HOSTNAME>:~ # chkconfig --list mdadmd
mdadmd                    0:off  1:off  2:off  3:on   4:off  5:on   6:off

Sadly, it didn’t solve the problem.
(rebootet several times, the unmount was never clean during the shutdown process)

[amo_vzug]

Are you using software-raid (dm)?

The message your receive is from the init-script ‘/etc/init.d/boot.md’. And it seems, that this script is enabled → that’s the reason, why if you reboot or shutdown your system, this script (and also this warning) is executed / shown.
The only thing what this script does - if runlevel 0 (shutdown) or runlevel 6 (reboot) are called (stop-section) - is send this message to the console:

...
...
    stop)
        echo -n "Not shutting down MD RAID - reboot/halt scripts do this."
...
...

So, if your not using software-raid, then you can disable the ‘boot.md’-script with

$ chkconfig boot.md off

And then you can also deactivate the other script (because it didn’t solved your problem):

$ chkconfig mdadmd off

Hope, this helps.
Regards,
Tom

[kaiwindheuser]

Thanks for you reply again, Tom.

I did both two steps.

The result is the following at shutdown:

[FONT=Courier New]
Removing multipath targets: | 22289xxxxx_part2: map in use
failed to remove multipath map 22289xxxxx
[/FONT]

Note: The Line [SIZE=1]“Not shutting down MD Raid - reboot/halt scripts do this.” missing[/SIZE] is not being displayed anymore

We have three partitions on our server:

Volume1
22289xxxxx_part1 ===> Swap-File
22289xxxxx_part2 ===> Root

Volume2
222d6xxxxx_part1 ===> Files

Only the Root-Partition has the unmount problems.

Maybe it is an issue regarding wait/delay time. Meaning that the shutdown/unmount
process would need more time than it’s given by the system to orderly shut down?

[amo_vzug]

You can try adding an “lsof /dev/mapper/xxx >/tmp/lsof 2>&1” (instead of ‘xxx’ put your root-partition) in the ‘/etc/init.d/boot.multipath’-script (before the line ‘$PROGRAM -F’). Then you can see in the file ‘/tmp/lsof’, which files still are open or in use.
Another hint is to look to the order of the “K”-Scripts in /etc/init.d/boot.d.

What’s your output of ‘chkconfig --list -A | grep -i multi’? Are you using the root-partition within LVM? Or as a normal primary-partition?

Can you then also try to remark the line ‘features “1 queue_if_no_path”’ in your multipath.conf?

From “man multipath.conf”:

…
The usage of queue_if_no_path option can lead to D state processes being hung and not killable in situations where all the paths to the LUN go offline. It is advisable to use the no_path_retry option instead.
…

[kaiwindheuser]

@amo_vzug;4787

You can try adding an “lsof /dev/mapper/xxx >/tmp/lsof 2>&1” (instead of ‘xxx’ put your root-partition) in the ‘/etc/init.d/boot.multipath’-script (before the line ‘$PROGRAM -F’). Then you can see in the file ‘/tmp/lsof’, which files still are open or in use.

The result is the following:

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
init         1 root  cwd    DIR  253,4     4096       2 /
init         1 root  rtd    DIR  253,4     4096       2 /
init         1 root  txt    REG  253,4    40784  376901 /sbin/init
init         1 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
init         1 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
init         1 root  mem    REG  253,4   244800 1704002 /lib64/libsepol.so.1
init         1 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
init         1 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
kthreadd     2 root  cwd    DIR  253,4     4096       2 /
kthreadd     2 root  rtd    DIR  253,4     4096       2 /
ksoftirqd    3 root  cwd    DIR  253,4     4096       2 /
ksoftirqd    3 root  rtd    DIR  253,4     4096       2 /
kworker/0    4 root  cwd    DIR  253,4     4096       2 /
kworker/0    4 root  rtd    DIR  253,4     4096       2 /
migration    6 root  cwd    DIR  253,4     4096       2 /
migration    6 root  rtd    DIR  253,4     4096       2 /
watchdog/    7 root  cwd    DIR  253,4     4096       2 /
watchdog/    7 root  rtd    DIR  253,4     4096       2 /
migration    8 root  cwd    DIR  253,4     4096       2 /
migration    8 root  rtd    DIR  253,4     4096       2 /
ksoftirqd   10 root  cwd    DIR  253,4     4096       2 /
ksoftirqd   10 root  rtd    DIR  253,4     4096       2 /
watchdog/   12 root  cwd    DIR  253,4     4096       2 /
watchdog/   12 root  rtd    DIR  253,4     4096       2 /
migration   13 root  cwd    DIR  253,4     4096       2 /
migration   13 root  rtd    DIR  253,4     4096       2 /
kworker/2   14 root  cwd    DIR  253,4     4096       2 /
kworker/2   14 root  rtd    DIR  253,4     4096       2 /
ksoftirqd   15 root  cwd    DIR  253,4     4096       2 /
ksoftirqd   15 root  rtd    DIR  253,4     4096       2 /
watchdog/   16 root  cwd    DIR  253,4     4096       2 /
watchdog/   16 root  rtd    DIR  253,4     4096       2 /
migration   17 root  cwd    DIR  253,4     4096       2 /
migration   17 root  rtd    DIR  253,4     4096       2 /
kworker/3   18 root  cwd    DIR  253,4     4096       2 /
kworker/3   18 root  rtd    DIR  253,4     4096       2 /
ksoftirqd   19 root  cwd    DIR  253,4     4096       2 /
ksoftirqd   19 root  rtd    DIR  253,4     4096       2 /
watchdog/   20 root  cwd    DIR  253,4     4096       2 /
watchdog/   20 root  rtd    DIR  253,4     4096       2 /
migration   21 root  cwd    DIR  253,4     4096       2 /
migration   21 root  rtd    DIR  253,4     4096       2 /
kworker/4   22 root  cwd    DIR  253,4     4096       2 /
kworker/4   22 root  rtd    DIR  253,4     4096       2 /
ksoftirqd   23 root  cwd    DIR  253,4     4096       2 /
ksoftirqd   23 root  rtd    DIR  253,4     4096       2 /
watchdog/   24 root  cwd    DIR  253,4     4096       2 /
watchdog/   24 root  rtd    DIR  253,4     4096       2 /
migration   25 root  cwd    DIR  253,4     4096       2 /
migration   25 root  rtd    DIR  253,4     4096       2 /
kworker/5   26 root  cwd    DIR  253,4     4096       2 /
kworker/5   26 root  rtd    DIR  253,4     4096       2 /
ksoftirqd   27 root  cwd    DIR  253,4     4096       2 /
ksoftirqd   27 root  rtd    DIR  253,4     4096       2 /
watchdog/   28 root  cwd    DIR  253,4     4096       2 /
watchdog/   28 root  rtd    DIR  253,4     4096       2 /
migration   29 root  cwd    DIR  253,4     4096       2 /
migration   29 root  rtd    DIR  253,4     4096       2 /
kworker/6   30 root  cwd    DIR  253,4     4096       2 /
kworker/6   30 root  rtd    DIR  253,4     4096       2 /
ksoftirqd   31 root  cwd    DIR  253,4     4096       2 /
ksoftirqd   31 root  rtd    DIR  253,4     4096       2 /
watchdog/   32 root  cwd    DIR  253,4     4096       2 /
watchdog/   32 root  rtd    DIR  253,4     4096       2 /
cpuset      37 root  cwd    DIR  253,4     4096       2 /
cpuset      37 root  rtd    DIR  253,4     4096       2 /
khelper     38 root  cwd    DIR  253,4     4096       2 /
khelper     38 root  rtd    DIR  253,4     4096       2 /
netns       39 root  cwd    DIR  253,4     4096       2 /
netns       39 root  rtd    DIR  253,4     4096       2 /
sync_supe   40 root  cwd    DIR  253,4     4096       2 /
sync_supe   40 root  rtd    DIR  253,4     4096       2 /
bdi-defau   41 root  cwd    DIR  253,4     4096       2 /
bdi-defau   41 root  rtd    DIR  253,4     4096       2 /
kintegrit   42 root  cwd    DIR  253,4     4096       2 /
kintegrit   42 root  rtd    DIR  253,4     4096       2 /
kblockd     43 root  cwd    DIR  253,4     4096       2 /
kblockd     43 root  rtd    DIR  253,4     4096       2 /
md          44 root  cwd    DIR  253,4     4096       2 /
md          44 root  rtd    DIR  253,4     4096       2 /
khungtask   45 root  cwd    DIR  253,4     4096       2 /
khungtask   45 root  rtd    DIR  253,4     4096       2 /
kworker/1   46 root  cwd    DIR  253,4     4096       2 /
kworker/1   46 root  rtd    DIR  253,4     4096       2 /
kswapd0     47 root  cwd    DIR  253,4     4096       2 /
kswapd0     47 root  rtd    DIR  253,4     4096       2 /
ksmd        48 root  cwd    DIR  253,4     4096       2 /
ksmd        48 root  rtd    DIR  253,4     4096       2 /
khugepage   49 root  cwd    DIR  253,4     4096       2 /
khugepage   49 root  rtd    DIR  253,4     4096       2 /
fsnotify_   50 root  cwd    DIR  253,4     4096       2 /
fsnotify_   50 root  rtd    DIR  253,4     4096       2 /
crypto      51 root  cwd    DIR  253,4     4096       2 /
crypto      51 root  rtd    DIR  253,4     4096       2 /
kthrotld    53 root  cwd    DIR  253,4     4096       2 /
kthrotld    53 root  rtd    DIR  253,4     4096       2 /
kpsmoused   54 root  cwd    DIR  253,4     4096       2 /
kpsmoused   54 root  rtd    DIR  253,4     4096       2 /
kworker/5   55 root  cwd    DIR  253,4     4096       2 /
kworker/5   55 root  rtd    DIR  253,4     4096       2 /
mpt_poll_  100 root  cwd    DIR  253,4     4096       2 /
mpt_poll_  100 root  rtd    DIR  253,4     4096       2 /
mpt/0      101 root  cwd    DIR  253,4     4096       2 /
mpt/0      101 root  rtd    DIR  253,4     4096       2 /
kworker/2  102 root  cwd    DIR  253,4     4096       2 /
kworker/2  102 root  rtd    DIR  253,4     4096       2 /
kworker/3  103 root  cwd    DIR  253,4     4096       2 /
kworker/3  103 root  rtd    DIR  253,4     4096       2 /
kworker/4  104 root  cwd    DIR  253,4     4096       2 /
kworker/4  104 root  rtd    DIR  253,4     4096       2 /
kworker/6  105 root  cwd    DIR  253,4     4096       2 /
kworker/6  105 root  rtd    DIR  253,4     4096       2 /
kworker/7  106 root  cwd    DIR  253,4     4096       2 /
kworker/7  106 root  rtd    DIR  253,4     4096       2 /
scsi_eh_0  107 root  cwd    DIR  253,4     4096       2 /
scsi_eh_0  107 root  rtd    DIR  253,4     4096       2 /
kworker/u  108 root  cwd    DIR  253,4     4096       2 /
kworker/u  108 root  rtd    DIR  253,4     4096       2 /
kmpathd    116 root  cwd    DIR  253,4     4096       2 /
kmpathd    116 root  rtd    DIR  253,4     4096       2 /
kmpath_ha  117 root  cwd    DIR  253,4     4096       2 /
kmpath_ha  117 root  rtd    DIR  253,4     4096       2 /
kmpath_rd  150 root  cwd    DIR  253,4     4096       2 /
kmpath_rd  150 root  rtd    DIR  253,4     4096       2 /
kworker/u  291 root  cwd    DIR  253,4     4096       2 /
kworker/u  291 root  rtd    DIR  253,4     4096       2 /
khubd      367 root  cwd    DIR  253,4     4096       2 /
khubd      367 root  rtd    DIR  253,4     4096       2 /
kdmflush   490 root  cwd    DIR  253,4     4096       2 /
kdmflush   490 root  rtd    DIR  253,4     4096       2 /
kdmflush   493 root  cwd    DIR  253,4     4096       2 /
kdmflush   493 root  rtd    DIR  253,4     4096       2 /
kdmflush   506 root  cwd    DIR  253,4     4096       2 /
kdmflush   506 root  rtd    DIR  253,4     4096       2 /
kdmflush   507 root  cwd    DIR  253,4     4096       2 /
kdmflush   507 root  rtd    DIR  253,4     4096       2 /
kdmflush   510 root  cwd    DIR  253,4     4096       2 /
kdmflush   510 root  rtd    DIR  253,4     4096       2 /
kjournald  659 root  cwd    DIR  253,4     4096       2 /
kjournald  659 root  rtd    DIR  253,4     4096       2 /
flush-253  672 root  cwd    DIR  253,4     4096       2 /
flush-253  672 root  rtd    DIR  253,4     4096       2 /
udevd      743 root  cwd    DIR  253,4     4096       2 /
udevd      743 root  rtd    DIR  253,4     4096       2 /
udevd      743 root  txt    REG  253,4   126304  377462 /sbin/udevd
udevd      743 root  mem    REG  253,4    61622 1703961 /lib64/libnss_files-2.11.3.so
udevd      743 root  mem    REG  253,4    52424 1703965 /lib64/libnss_nis-2.11.3.so
udevd      743 root  mem    REG  253,4   108248 1703955 /lib64/libnsl-2.11.3.so
udevd      743 root  mem    REG  253,4    38676 1703957 /lib64/libnss_compat-2.11.3.so
udevd      743 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
udevd      743 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
udevd      743 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
udevd      743 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
edac-poll  905 root  cwd    DIR  253,4     4096       2 /
edac-poll  905 root  rtd    DIR  253,4     4096       2 /
kauditd   1163 root  cwd    DIR  253,4     4096       2 /
kauditd   1163 root  rtd    DIR  253,4     4096       2 /
kworker/0 1521 root  cwd    DIR  253,4     4096       2 /
kworker/0 1521 root  rtd    DIR  253,4     4096       2 /
kworker/1 2071 root  cwd    DIR  253,4     4096       2 /
kworker/1 2071 root  rtd    DIR  253,4     4096       2 /
udevd     4505 root  cwd    DIR  253,4     4096       2 /
udevd     4505 root  rtd    DIR  253,4     4096       2 /
udevd     4505 root  txt    REG  253,4   126304  377462 /sbin/udevd
udevd     4505 root  mem    REG  253,4    61622 1703961 /lib64/libnss_files-2.11.3.so
udevd     4505 root  mem    REG  253,4    52424 1703965 /lib64/libnss_nis-2.11.3.so
udevd     4505 root  mem    REG  253,4   108248 1703955 /lib64/libnsl-2.11.3.so
udevd     4505 root  mem    REG  253,4    38676 1703957 /lib64/libnss_compat-2.11.3.so
udevd     4505 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
udevd     4505 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
udevd     4505 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
udevd     4505 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
udevd     4506 root  cwd    DIR  253,4     4096       2 /
udevd     4506 root  rtd    DIR  253,4     4096       2 /
udevd     4506 root  txt    REG  253,4   126304  377462 /sbin/udevd
udevd     4506 root  mem    REG  253,4    61622 1703961 /lib64/libnss_files-2.11.3.so
udevd     4506 root  mem    REG  253,4    52424 1703965 /lib64/libnss_nis-2.11.3.so
udevd     4506 root  mem    REG  253,4   108248 1703955 /lib64/libnsl-2.11.3.so
udevd     4506 root  mem    REG  253,4    38676 1703957 /lib64/libnss_compat-2.11.3.so
udevd     4506 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
udevd     4506 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
udevd     4506 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
udevd     4506 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
pulseaudi 4810  gdm  cwd    DIR  253,4     4096       2 /
pulseaudi 4810  gdm  rtd    DIR  253,4     4096       2 /
pulseaudi 4810  gdm  txt    REG  253,4    75984  287361 /usr/bin/pulseaudio
pulseaudi 4810  gdm  mem    REG  253,4    14536   49767 /usr/lib64/pulse-0.9.23/modules/module-cork-music-on-phone.so
pulseaudi 4810  gdm  mem    REG  253,4    10456   49793 /usr/lib64/pulse-0.9.23/modules/module-position-event-sounds.so
pulseaudi 4810  gdm  mem    REG  253,4    18856   49766 /usr/lib64/pulse-0.9.23/modules/module-console-kit.so
pulseaudi 4810  gdm  mem    REG  253,4    22880   49804 /usr/lib64/pulse-0.9.23/modules/module-suspend-on-idle.so
pulseaudi 4810  gdm  mem    REG  253,4    14616   49781 /usr/lib64/pulse-0.9.23/modules/module-intended-roles.so
pulseaudi 4810  gdm  mem    REG  253,4    18944   49789 /usr/lib64/pulse-0.9.23/modules/module-null-sink.so
pulseaudi 4810  gdm  mem    REG  253,4    10488   49759 /usr/lib64/pulse-0.9.23/modules/module-always-sink.so
pulseaudi 4810  gdm  mem    REG  253,4    14544   49795 /usr/lib64/pulse-0.9.23/modules/module-rescue-streams.so
pulseaudi 4810  gdm  mem    REG  253,4    14576   49768 /usr/lib64/pulse-0.9.23/modules/module-default-device-restore.so
pulseaudi 4810  gdm  mem    REG  253,4    14672   49788 /usr/lib64/pulse-0.9.23/modules/module-native-protocol-unix.so
pulseaudi 4810  gdm  mem    REG  253,4    52792   49751 /usr/lib64/pulse-0.9.23/modules/libprotocol-esound.so
pulseaudi 4810  gdm  mem    REG  253,4    14656   49776 /usr/lib64/pulse-0.9.23/modules/module-esound-protocol-unix.so
pulseaudi 4810  gdm  mem    REG  253,4    56056 1706420 /lib64/libudev.so.0.5.1
pulseaudi 4810  gdm  mem    REG  253,4    27280   49807 /usr/lib64/pulse-0.9.23/modules/module-udev-detect.so
pulseaudi 4810  gdm  mem    REG  253,4    19008   49760 /usr/lib64/pulse-0.9.23/modules/module-augment-properties.so
pulseaudi 4810  gdm  mem    REG  253,4    14632   49761 /usr/lib64/pulse-0.9.23/modules/module-card-restore.so
pulseaudi 4810  gdm  mem    REG  253,4   127112   49753 /usr/lib64/pulse-0.9.23/modules/libprotocol-native.so
pulseaudi 4810  gdm  mem    REG  253,4    35616   49803 /usr/lib64/pulse-0.9.23/modules/module-stream-restore.so
pulseaudi 4810  gdm  mem    REG  253,4    23024   49771 /usr/lib64/pulse-0.9.23/modules/module-device-restore.so
pulseaudi 4810  gdm  mem    REG  253,4   198280 1664437 /usr/lib64/libvorbis.so.0.4.0
pulseaudi 4810  gdm  mem    REG  253,4    22816 1664255 /usr/lib64/libogg.so.0.5.3
pulseaudi 4810  gdm  mem    REG  253,4  1947904 1664439 /usr/lib64/libvorbisenc.so.2.0.3
pulseaudi 4810  gdm  mem    REG  253,4   312376 1664731 /usr/lib64/libFLAC.so.8.2.0
pulseaudi 4810  gdm  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
pulseaudi 4810  gdm  mem    REG  253,4   135690 1703970 /lib64/libpthread-2.11.3.so
pulseaudi 4810  gdm  mem    REG  253,4   541821 1703952 /lib64/libm-2.11.3.so
pulseaudi 4810  gdm  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
pulseaudi 4810  gdm  mem    REG  253,4    47241 1703974 /lib64/librt-2.11.3.so
pulseaudi 4810  gdm  mem    REG  253,4    22952 1664331 /usr/lib64/libgdbm.so.3.0.0
pulseaudi 4810  gdm  mem    REG  253,4   253488 1706423 /lib64/libdbus-1.so.3.4.0
pulseaudi 4810  gdm  mem    REG  253,4   385176 1664921 /usr/lib64/libsndfile.so.1.0.20
pulseaudi 4810  gdm  mem    REG  253,4    81768 1664237 /usr/lib64/libspeexdsp.so.1.5.0
pulseaudi 4810  gdm  mem    REG  253,4  1480912 1664239 /usr/lib64/libsamplerate.so.0.1.4
pulseaudi 4810  gdm  mem    REG  253,4    39600 1664286 /usr/lib64/libltdl.so.7.2.0
pulseaudi 4810  gdm  mem    REG  253,4    42016 1703984 /lib64/libwrap.so.0.7.6
pulseaudi 4810  gdm  mem    REG  253,4    14592 1664176 /usr/lib64/libXau.so.6.0.0
pulseaudi 4810  gdm  mem    REG  253,4   113520 1664410 /usr/lib64/libxcb.so.1.0.0
pulseaudi 4810  gdm  mem    REG  253,4    14472  377531 /usr/lib64/libxcb-atom.so.0.0.0
pulseaudi 4810  gdm  mem    REG  253,4     6224 1664400 /usr/lib64/libxcb-xlib.so.0.0.0
pulseaudi 4810  gdm  mem    REG  253,4  1298176 1665721 /usr/lib64/libX11.so.6.2.0
pulseaudi 4810  gdm  mem    REG  253,4    73624 1666222 /usr/lib64/libXext.so.6.4.0
pulseaudi 4810  gdm  mem    REG  253,4    27224  377513 /usr/lib64/libXtst.so.6.1.0
pulseaudi 4810  gdm  mem    REG  253,4   102544 1664178 /usr/lib64/libICE.so.6.3.0
pulseaudi 4810  gdm  mem    REG  253,4    19152 1704053 /lib64/libuuid.so.1.3.0
pulseaudi 4810  gdm  mem    REG  253,4    35136 1664841 /usr/lib64/libSM.so.6.0.0
pulseaudi 4810  gdm  mem    REG  253,4     6112 1665719 /usr/lib64/libX11-xcb.so.1.0.0
pulseaudi 4810  gdm  mem    REG  253,4   325928  377771 /usr/lib64/libpulsecommon-0.9.23.so
pulseaudi 4810  gdm  mem    REG  253,4   275896  377770 /usr/lib64/libpulse.so.0.12.4
pulseaudi 4810  gdm  mem    REG  253,4   531864  378054 /usr/lib64/libpulsecore-0.9.23.so
pulseaudi 4810  gdm  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
pulseaudi 4810  gdm  mem    REG  253,4   123193 1600576 /usr/share/locale/de/LC_MESSAGES/libc.mo
pulseaudi 4810  gdm  mem    REG  253,4   256484 1771347 /usr/lib/locale/de_DE.utf8/LC_CTYPE
pulseaudi 4810  gdm  mem    REG  253,4  1163682 1771451 /usr/lib/locale/de_DE.utf8/LC_COLLATE
pulseaudi 4810  gdm  mem    REG  253,4    38658 1601843 /usr/share/locale/de/LC_MESSAGES/pulseaudio.mo
pulseaudi 4810  gdm  mem    REG  253,4       54 1778992 /usr/lib/locale/de_DE.utf8/LC_NUMERIC
pulseaudi 4810  gdm  mem    REG  253,4     2354 1771453 /usr/lib/locale/de_DE.utf8/LC_TIME
pulseaudi 4810  gdm  mem    REG  253,4      294 1786318 /usr/lib/locale/de_DE.utf8/LC_MONETARY
pulseaudi 4810  gdm  mem    REG  253,4       54 1778947 /usr/lib/locale/de_DE.utf8/LC_MESSAGES/SYS_LC_MESSAGES
pulseaudi 4810  gdm  mem    REG  253,4       34 1786329 /usr/lib/locale/de_DE.utf8/LC_PAPER
pulseaudi 4810  gdm  mem    REG  253,4       83 1779070 /usr/lib/locale/de_DE.utf8/LC_NAME
pulseaudi 4810  gdm  mem    REG  253,4      155 1770366 /usr/lib/locale/de_DE.utf8/LC_ADDRESS
pulseaudi 4810  gdm  mem    REG  253,4       56 1778940 /usr/lib/locale/de_DE.utf8/LC_TELEPHONE
pulseaudi 4810  gdm  mem    REG  253,4       23 1788036 /usr/lib/locale/de_DE.utf8/LC_MEASUREMENT
pulseaudi 4810  gdm  mem    REG  253,4      375 1771452 /usr/lib/locale/de_DE.utf8/LC_IDENTIFICATION
pulseaudi 4810  gdm  mem    REG  253,4    26050 1786333 /usr/lib64/gconv/gconv-modules.cache
pulseaudi 4810  gdm   10u   REG  253,4     3898  385662 /var/lib/gdm/.pulse/2cbcb5ea89b045bfd1660cb04fb22f8f-device-volumes.x86_64-suse-linux-gnu.gdbm
pulseaudi 4810  gdm   11u   REG  253,4     3072  385663 /var/lib/gdm/.pulse/2cbcb5ea89b045bfd1660cb04fb22f8f-stream-volumes.x86_64-suse-linux-gnu.gdbm
pulseaudi 4810  gdm   12u   REG  253,4     3072  385664 /var/lib/gdm/.pulse/2cbcb5ea89b045bfd1660cb04fb22f8f-card-database.x86_64-suse-linux-gnu.gdbm
sshd      4838 root  cwd    DIR  253,4     4096       2 /
sshd      4838 root  rtd    DIR  253,4     4096       2 /
sshd      4838 root  txt    REG  253,4   454768 1666561 /usr/sbin/sshd
sshd      4838 root  mem    REG  253,4   217016  369452 /var/run/nscd/group
sshd      4838 root  mem    REG  253,4     6104 1722560 /lib64/security/pam_deny.so
sshd      4838 root  mem    REG  253,4     6192 1722597 /lib64/security/pam_warn.so
sshd      4838 root  mem    REG  253,4    10456 1722595 /lib64/security/pam_umask.so
sshd      4838 root  mem    REG  253,4    18832 1722571 /lib64/security/pam_limits.so
sshd      4838 root  mem    REG  253,4    10392 1722574 /lib64/security/pam_loginuid.so
sshd      4838 root  mem    REG  253,4    39496 1664784 /usr/lib64/libcrack.so.2.8.0
sshd      4838 root  mem    REG  253,4    23024 1722734 /lib64/security/pam_pwcheck.so
sshd      4838 root  mem    REG  253,4    22896 1703988 /lib64/libxcrypt.so.2.0.0
sshd      4838 root  mem    REG  253,4    52288 1722735 /lib64/security/pam_unix2.so
sshd      4838 root  mem    REG  253,4    14552 1722562 /lib64/security/pam_env.so
sshd      4838 root  mem    REG  253,4    10360 1722579 /lib64/security/pam_nologin.so

sshd      4838 root  mem    REG  253,4    61622 1703961 /lib64/libnss_files-2.11.3.so
sshd      4838 root  mem    REG  253,4   135690 1703970 /lib64/libpthread-2.11.3.so
sshd      4838 root  mem    REG  253,4    10336 1704032 /lib64/libkeyutils-1.2.so
sshd      4838 root  mem    REG  253,4    35800 1665941 /usr/lib64/libkrb5support.so.0.1
sshd      4838 root  mem    REG  253,4    22944  378061 /usr/lib64/libscconf.so.2.0.0
sshd      4838 root  mem    REG  253,4    39600 1664286 /usr/lib64/libltdl.so.7.2.0
sshd      4838 root  mem    REG  253,4    39824 1664253 /usr/lib64/libopenct.so.1.0.0
sshd      4838 root  mem    REG  253,4   343040 1666427 /usr/lib64/libssl.so.0.9.8
sshd      4838 root  mem    REG  253,4   110440 1666050 /usr/lib64/libsasl2.so.2.0.22
sshd      4838 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
sshd      4838 root  mem    REG  253,4    14720 1704064 /lib64/libcom_err.so.2.1
sshd      4838 root  mem    REG  253,4   154776 1665929 /usr/lib64/libk5crypto.so.3.1
sshd      4838 root  mem    REG  253,4   664216 1665939 /usr/lib64/libkrb5.so.3.3
sshd      4838 root  mem    REG  253,4   185904 1665925 /usr/lib64/libgssapi_krb5.so.2.2
sshd      4838 root  mem    REG  253,4    98115 1703972 /lib64/libresolv-2.11.3.so
sshd      4838 root  mem    REG  253,4   661736  378057 /usr/lib64/libopensc.so.2.0.0
sshd      4838 root  mem    REG  253,4    57699 1703948 /lib64/libcrypt-2.11.3.so
sshd      4838 root  mem    REG  253,4   108248 1703955 /lib64/libnsl-2.11.3.so
sshd      4838 root  mem    REG  253,4    88704 1703982 /lib64/libz.so.1.2.3
sshd      4838 root  mem    REG  253,4    14360 1703978 /lib64/libutil-2.11.3.so
sshd      4838 root  mem    REG  253,4    60584 1666335 /usr/lib64/liblber-2.4.so.2.7.1
sshd      4838 root  mem    REG  253,4   298088 1666337 /usr/lib64/libldap-2.4.so.2.7.1
sshd      4838 root  mem    REG  253,4  1621368 1664713 /usr/lib64/libcrypto.so.0.9.8
sshd      4838 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
sshd      4838 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
sshd      4838 root  mem    REG  253,4    56048 1706410 /lib64/libpam.so.0.83.1
sshd      4838 root  mem    REG  253,4   109128 1704036 /lib64/libaudit.so.0.0.0
sshd      4838 root  mem    REG  253,4    42016 1703984 /lib64/libwrap.so.0.7.6
sshd      4838 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
sshd      4838 root  DEL    REG  253,4           369453 /var/run/nscd/dbyEJuqe
sshd      4838 root  mem    REG  253,4   217016  369451 /var/run/nscd/passwd
bash      4861 root  cwd    DIR  253,4     4096  860161 /tmp
bash      4861 root  rtd    DIR  253,4     4096       2 /
bash      4861 root  txt    REG  253,4   584016 1482762 /bin/bash
bash      4861 root  mem    REG  253,4   293936 1704013 /lib64/libncurses.so.5.6
bash      4861 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
bash      4861 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
bash      4861 root  mem    REG  253,4   263568 1704043 /lib64/libreadline.so.5.2
bash      4861 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
bash      4861 root  mem    REG  253,4   256484 1771347 /usr/lib/locale/de_DE.utf8/LC_CTYPE
bash      4861 root  mem    REG  253,4       54 1778992 /usr/lib/locale/de_DE.utf8/LC_NUMERIC
bash      4861 root  mem    REG  253,4     2354 1771453 /usr/lib/locale/de_DE.utf8/LC_TIME
bash      4861 root  mem    REG  253,4  1163682 1771451 /usr/lib/locale/de_DE.utf8/LC_COLLATE
bash      4861 root  mem    REG  253,4      294 1786318 /usr/lib/locale/de_DE.utf8/LC_MONETARY
bash      4861 root  mem    REG  253,4       54 1778947 /usr/lib/locale/de_DE.utf8/LC_MESSAGES/SYS_LC_MESSAGES
bash      4861 root  mem    REG  253,4       34 1786329 /usr/lib/locale/de_DE.utf8/LC_PAPER
bash      4861 root  mem    REG  253,4       83 1779070 /usr/lib/locale/de_DE.utf8/LC_NAME
bash      4861 root  mem    REG  253,4      155 1770366 /usr/lib/locale/de_DE.utf8/LC_ADDRESS
bash      4861 root  mem    REG  253,4       56 1778940 /usr/lib/locale/de_DE.utf8/LC_TELEPHONE
bash      4861 root  mem    REG  253,4       23 1788036 /usr/lib/locale/de_DE.utf8/LC_MEASUREMENT
bash      4861 root  mem    REG  253,4    26050 1786333 /usr/lib64/gconv/gconv-modules.cache
bash      4861 root  mem    REG  253,4      375 1771452 /usr/lib/locale/de_DE.utf8/LC_IDENTIFICATION
bash      4861 root  mem    REG  253,4   217016  369451 /var/run/nscd/passwd
kworker/u 5107 root  cwd    DIR  253,4     4096       2 /
kworker/u 5107 root  rtd    DIR  253,4     4096       2 /
flush-253 5126 root  cwd    DIR  253,4     4096       2 /
flush-253 5126 root  rtd    DIR  253,4     4096       2 /
rc        5127 root  cwd    DIR  253,4     4096       2 /
rc        5127 root  rtd    DIR  253,4     4096       2 /
rc        5127 root  txt    REG  253,4   584016 1482762 /bin/bash
rc        5127 root  mem    REG  253,4   293936 1704013 /lib64/libncurses.so.5.6
rc        5127 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
rc        5127 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
rc        5127 root  mem    REG  253,4   263568 1704043 /lib64/libreadline.so.5.2
rc        5127 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
rc        5127 root  255r   REG  253,4    10161 1180634 /etc/init.d/rc
kworker/1 6969 root  cwd    DIR  253,4     4096       2 /
kworker/1 6969 root  rtd    DIR  253,4     4096       2 /
splash    7333 root  cwd    DIR  253,4     4096       2 /
splash    7333 root  rtd    DIR  253,4     4096       2 /
splash    7333 root  txt    REG  253,4   421680  378132 /sbin/splash
splash    7333 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
splash    7333 root  mem    REG  253,4   541821 1703952 /lib64/libm-2.11.3.so
splash    7333 root  mem    REG  253,4    88704 1703982 /lib64/libz.so.1.2.3
splash    7333 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
S01reboot 7350 root  cwd    DIR  253,4     4096       2 /
S01reboot 7350 root  rtd    DIR  253,4     4096       2 /
S01reboot 7350 root  txt    REG  253,4   584016 1482762 /bin/bash
S01reboot 7350 root  mem    REG  253,4   293936 1704013 /lib64/libncurses.so.5.6
S01reboot 7350 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
S01reboot 7350 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
S01reboot 7350 root  mem    REG  253,4   263568 1704043 /lib64/libreadline.so.5.2
S01reboot 7350 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
S01reboot 7350 root  255r   REG  253,4     6255 1180631 /etc/init.d/halt
K07boot.m 7541 root  cwd    DIR  253,4     4096       2 /
K07boot.m 7541 root  rtd    DIR  253,4     4096       2 /
K07boot.m 7541 root  txt    REG  253,4   584016 1482762 /bin/bash
K07boot.m 7541 root  mem    REG  253,4   293936 1704013 /lib64/libncurses.so.5.6
K07boot.m 7541 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
K07boot.m 7541 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
K07boot.m 7541 root  mem    REG  253,4   263568 1704043 /lib64/libreadline.so.5.2
K07boot.m 7541 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
K07boot.m 7541 root  255r   REG  253,4     4320 1182081 /etc/init.d/boot.multipath
lsof      7548 root  cwd    DIR  253,4     4096       2 /
lsof      7548 root  rtd    DIR  253,4     4096       2 /
lsof      7548 root  txt    REG  253,4   127416 1664421 /usr/bin/lsof
lsof      7548 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
lsof      7548 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
lsof      7548 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
lsof      7548 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so
lsof      7548 root    1w   REG  253,4        0  860263 /tmp/lsof
lsof      7548 root    2w   REG  253,4        0  860263 /tmp/lsof
lsof      7549 root  cwd    DIR  253,4     4096       2 /
lsof      7549 root  rtd    DIR  253,4     4096       2 /
lsof      7549 root  txt    REG  253,4   127416 1664421 /usr/bin/lsof
lsof      7549 root  mem    REG  253,4    19149 1703950 /lib64/libdl-2.11.3.so
lsof      7549 root  mem    REG  253,4  1754140 1703944 /lib64/libc-2.11.3.so
lsof      7549 root  mem    REG  253,4   118080 1704003 /lib64/libselinux.so.1
lsof      7549 root  mem    REG  253,4   151051 1706614 /lib64/ld-2.11.3.so

@amo_vzug;4787

Another hint is to look to the order of the “K”-Scripts in /etc/init.d/boot.d.

<HOSTNAME>:/etc/init.d/boot.d # ls -al
total 8
drwxr-xr-x  2 root root 4096 May 22 13:33 .
drwxr-xr-x 11 root root 4096 May 23 09:42 ..
lrwxrwxrwx  1 root root   15 May 15 12:26 K01boot.cleanup -> ../boot.cleanup
lrwxrwxrwx  1 root root   18 May 15 12:29 K01boot.compliance -> ../boot.compliance
lrwxrwxrwx  1 root root   13 May 15 12:28 K01boot.cycle -> ../boot.cycle
lrwxrwxrwx  1 root root   15 May 15 12:26 K01boot.debugfs -> ../boot.debugfs
lrwxrwxrwx  1 root root   12 May 15 12:28 K01boot.fuse -> ../boot.fuse
lrwxrwxrwx  1 root root   16 May 15 12:26 K01boot.ipconfig -> ../boot.ipconfig
lrwxrwxrwx  1 root root   13 May 15 12:32 K01boot.kdump -> ../boot.kdump
lrwxrwxrwx  1 root root   12 May 15 12:26 K01boot.klog -> ../boot.klog
lrwxrwxrwx  1 root root   16 May 15 12:26 K01boot.ldconfig -> ../boot.ldconfig
lrwxrwxrwx  1 root root   19 May 15 12:28 K01boot.lvm_monitor -> ../boot.lvm_monitor
lrwxrwxrwx  1 root root   12 May 15 12:26 K01boot.proc -> ../boot.proc
lrwxrwxrwx  1 root root   13 May 15 12:27 K01boot.quota -> ../boot.quota
lrwxrwxrwx  1 root root   12 May 15 12:25 K01boot.scpm -> ../boot.scpm
lrwxrwxrwx  1 root root   14 May 15 12:25 K01boot.sysctl -> ../boot.sysctl
lrwxrwxrwx  1 root root   18 May 15 12:26 K01boot.udev_retry -> ../boot.udev_retry
lrwxrwxrwx  1 root root   13 May 15 12:26 K02boot.clock -> ../boot.clock
lrwxrwxrwx  1 root root   19 May 15 12:28 K02boot.loadmodules -> ../boot.loadmodules
lrwxrwxrwx  1 root root   16 May 15 12:30 K02boot.localnet -> ../boot.localnet
lrwxrwxrwx  1 root root   12 May 15 12:26 K02boot.swap -> ../boot.swap
lrwxrwxrwx  1 root root   15 May 15 12:28 K04boot.localfs -> ../boot.localfs
lrwxrwxrwx  1 root root   16 May 15 12:28 K06boot.rootfsck -> ../boot.rootfsck
lrwxrwxrwx  1 root root   17 May 15 13:18 K07boot.multipath -> ../boot.multipath
lrwxrwxrwx  1 root root   21 May 15 12:28 K08boot.device-mapper -> ../boot.device-mapper
lrwxrwxrwx  1 root root   12 May 15 12:28 K09boot.udev -> ../boot.udev
lrwxrwxrwx  1 root root   12 May 15 12:26 S01boot.udev -> ../boot.udev
lrwxrwxrwx  1 root root   16 May 15 12:26 S02boot.rootfsck -> ../boot.rootfsck
lrwxrwxrwx  1 root root   13 May 15 12:26 S03boot.clock -> ../boot.clock
lrwxrwxrwx  1 root root   21 May 15 12:27 S04boot.device-mapper -> ../boot.device-mapper
lrwxrwxrwx  1 root root   19 May 15 12:26 S04boot.loadmodules -> ../boot.loadmodules
lrwxrwxrwx  1 root root   17 May 15 13:18 S06boot.multipath -> ../boot.multipath
lrwxrwxrwx  1 root root   15 May 15 12:28 S10boot.localfs -> ../boot.localfs
lrwxrwxrwx  1 root root   18 May 15 12:29 S12boot.compliance -> ../boot.compliance
lrwxrwxrwx  1 root root   13 May 15 12:29 S13boot.cycle -> ../boot.cycle
lrwxrwxrwx  1 root root   12 May 15 12:29 S13boot.fuse -> ../boot.fuse
lrwxrwxrwx  1 root root   12 May 15 12:29 S13boot.klog -> ../boot.klog
lrwxrwxrwx  1 root root   19 May 15 12:29 S13boot.lvm_monitor -> ../boot.lvm_monitor
lrwxrwxrwx  1 root root   12 May 15 12:29 S13boot.proc -> ../boot.proc
lrwxrwxrwx  1 root root   12 May 15 12:29 S13boot.scpm -> ../boot.scpm
lrwxrwxrwx  1 root root   12 May 15 12:29 S13boot.swap -> ../boot.swap
lrwxrwxrwx  1 root root   18 May 15 12:29 S13boot.udev_retry -> ../boot.udev_retry
lrwxrwxrwx  1 root root   15 May 15 12:29 S14boot.debugfs -> ../boot.debugfs
lrwxrwxrwx  1 root root   16 May 15 12:29 S14boot.ldconfig -> ../boot.ldconfig
lrwxrwxrwx  1 root root   13 May 15 12:29 S14boot.quota -> ../boot.quota
lrwxrwxrwx  1 root root   14 May 15 12:29 S14boot.sysctl -> ../boot.sysctl
lrwxrwxrwx  1 root root   15 May 15 12:29 S15boot.cleanup -> ../boot.cleanup
lrwxrwxrwx  1 root root   16 May 15 12:29 S15boot.ipconfig -> ../boot.ipconfig
lrwxrwxrwx  1 root root   16 May 15 12:29 S16boot.localnet -> ../boot.localnet
lrwxrwxrwx  1 root root   13 May 15 12:32 S17boot.kdump -> ../boot.kdump

@amo_vzug;4787

What’s your output of ‘chkconfig --list -A | grep -i multi’? Are you using the root-partition within LVM? Or as a normal primary-partition?

The Output is:

<HOSTNAME>:/etc/init.d/boot.d # chkconfig --list -A | grep -i multi
boot.multipath            0:off  1:off  2:off  3:off  4:off  5:off  6:off  B:on
multipathd                0:off  1:off  2:off  3:on   4:off  5:on   6:off

All Partitions are created with YaST as normal Primary Partitions with EXT3 Filesystem.
No LVM is used.

[QUOTE=amo_vzug;4787]
Can you then also try to remark the line ‘features “1 queue_if_no_path”’ in your multipath.conf?
From “man multipath.conf”:[/QUOTE]
Would you suggest using the option:

"no_path_retry <VALUE>" 

instead of queueing the I/O load???

Thanks again!

[amo_vzug]

I was also able to reproduce this behavior (SAN-Boot with root-fs within lvm). But I did not have special fsck’s after a shutdown/reboot. Because of this, my filesystems looks consistent. So for me, the “map in use”-error is poor, but it seems harmless.