How to fix hosed system after kernel update 3.0.76-0.11

SUSE Linux Enterprise Desktop SLED Install-Boot
1

While doing the rounds and patching a bunch of SLES server for ShellShock, I then proceeded to patch my trusty SLED 11 SP3 workstation. While at it, I noticed the security patch listed in YOU for kernel 3.0.76-0.11 and installed it as well. It completed OK, but after a reboot I have some weirdness like I have never seen before.

Basically the system only mounts /boot, /home and SWAP.

After rebooting into failsafe mode with runlevel 1, and “confirm” I was able to find the first sign on the problem.

Activating swap-devices in /etc/fstab... rm: cannot remove /etc/mtab : No space left on device can't create lock file /etc/mtab~833: No space left on device (use -n flag to override)

Then after bootup script boot.localfs starts, it displays similar errors for mounting all the fstab entries, with the same error:

can't create lock file /etc/mtab~833: No space left on device (use -n flag to override)

except that the number above “833” increments. See screen shot - file sys mount errors.
(The screen shot was difficult to catch as it scrolled by, so I filmed it with my ipad, then extracted a frame).

Once the system completes boot, I can actually log in as root.
lsblk lists all my partitions correctly, but with only /boot, /home and swap mounted.
Even though / isn’t mounted I can navigate it and /home with with cd, ls etc. !? I didn’t think this would be possible!

mount - shows nothing is mounted, so contradicts lsblk!

mount -a

mount: /dev/sdc1 already mounted or /boot busy mount: /dev/md0 already mounted or /home busy mount: devpts already mounted or /dev/pts busy

If it matters, / is btrfs and is mounted using UUID in fstab. I’m just about to reboot with a rescue disk and set this to good old fashion /dev/sdc2 (instead of UUID) to see if it helps…

I’ve tried to fix this for a few hours already, so I hope someone may have seen this before, and can offer some insight into what the kernel (and/or shellshock) update might have done.

Cheers, Gordon

2

A bit closer to understanding what is going on…

Booted off SLED DVD.
Manually mounted /boot (/dev/sdc1 ext3), / (/dev/sdc2 btrfs) to /mnt
and I found that I could touch/edit a file on the /mnt/boot partition but not the btrfs one. So therein lies the core issue, the / partition is read only.
Yet in contradiction mount lists the mounted /dev/sdc2 as RW!?

BTRFS has helped me out many times with its excellent rollback capabilities, but this time, I can’t see a way out yet. Any BTRFS experts out there that can provide some pointers?

3

No btrfs snap shot disk space issues:

btrfs filesystem show

Label: none uuid: 8c84ee47-e350-45da-8819-a21f1429f3e4
Total devices 1 FS bytes used 18.26GiB
devid 1 size 25.00GiB used 25.00GiB path /dev/sdc2

Btrfs v0.20+

4

Your post makes it sound like you’ve updated to kernel 3.0.76-0.11, but that’s the version SLED 11 SP 3 shipped with and there have been about eight kernel updates since then. So I assume you’re actually updating from 3.0.76-0.11, not to it. If you’ve really updated to 3.0.76-0.11 that would be…strange.

I’m afraid I’ve no idea how your machine has ended up in this state or how you recover from it. In your position I’d figure out how to copy the contents of /home somewhere else (unless your backup of everything you care about is sufficiently recent) then re-install, probably with the default filesystem rather than btrfs.

When that stuff you filmed with your iPad is scrolling up the screen, try pressing scroll lock and then pressing shift and page up/down.

Since you appear to be a fan of btrfs be sure to check out SLED 12 when it gets released. (Scheduled for Q4.)

5

You must be right. At first look at my issue because I selected a kernel update, I was checking for some sort of split kernel issue, and saw that everything was the same in /boot and assumed it was the updated version, when now it seems that the kernel update didn’t take hold.

Too right, enough hrs have been spent trying to get around this to no avail. /home is already on a separate RAID 1 partion, running XFS, plus have rsnapshot backups. I like SLED 11, but some of it’s packages were showing their age. Can’t wait for SLED 12 so I might actually give OpenSuse 13.1 a whirl for something different. By all reports it seems quite decent.

–
Gordon