Booting SLES from hard drive/SSD on RPi-3

I’ve been testing SLES on RPi for 6-8 weeks. It’s my first experience with SUSE and it has been a smooth one. Thanks for this implementation … it gives me the ARM8 environment I’d been wanting to learn, and it’s a very high quality implementation.

One question, though, about the ability to boot from or redirect booting to a USB-attached hard or SSD drive on a Pi-3. Under Raspbian, editing cmdline.txt on mmcblk0 to set root to /dev/sda1 redirects the booting process to the hard drive. Or by preparing the Pi-3 with a special command one time, you can then boot directly from the hard drive (no µSD at all) or with a special BOOTCODE.BIN as the only file on the µSD, it will redirect booting to the hard drive.

I don’t see any option in the /BOOT directory to redirect the boot device, and the only reference to mmcblk is in start.elf (binary, not text).

Is there a way to direct the boot process to go to a USB-attached drive, or to redirect the boot to a hard drive with a modified bootstrap program on a µSD? While the booting starts a little slower over the USB rather than on the µSD, the performance is much better once you’re up and running. This is coming from Firefox/Raspbian/SSD/Pi-3, and I’d like to do the same with SLES on my other Pi-3.

Hi
Based on this ML thread today it needs the usb driver in initrd (which
I’m guessing it’s not for SLES…)
https://lists.opensuse.org/opensuse-arm/2017-01/msg00060.html


Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE Leap 42.1|GNOME 3.16.2|4.1.36-44-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

Thank, yet again!, Malcom! I’ll update my SUSE installation on the microSD card, move it to a SD-USB adapter, and give it a try before installing on a hard drive. I’ll post the results.

David

Well, the straightforward approach of installing SLES on a USB thumbdrive (following http://tinyurl.com/slespi) then using the Raspbian MSD installation process (https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md) to copy the updated, working version of SLES from a µSD to the USB drive didn’t work. After installing SLES on the USB drive, when I insert it into the USB socket with SLES having booted off the µSD card, SLES remounts /dev/sda3 as the root, in place of /dev/mmcblk0p3. Makes it hard to rsync the working partition over the USB partition. [ :slight_smile: ] I’ve never seen that before and haven’t figured out how to remount mmcblk0p3 as the / partition. I’ll post again if I make any progress.

If anyone else is trying this. please note that I’d already prepared my Pi-3 to boot off the USB drive (have booted Raspbian on it) with the one-time fuse-blowing step. And note that not all USB drives will work: the Raspbian MSD reference above lists some that are known to work and some that are known not to work.

In particular, though, note that USB 3.0 devices seem especially tricky. Neither my USB 3.0 µSD-USB adapter nor a Verbatim 3.0 USB thumbdrive were recognized by my Pi-3 and so couldn’t be used for USB booting.

More later (I hope).

Hi
The USB 3.0 thing not only affects RPI’s… I have a few systems here
that want the USB 3.0 device in a USB 2.0 port to boot the installer…

Feed it through a powered USB 2.0 hub to start with? Also try dd of the
partitions rather than rsync.

I have an old 8GB IDE SSD, need to hook that up and see if can get
something running…

What boot options are you using and your method to get there (u-boot
or grub?).


Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE Leap 42.1|GNOME 3.16.2|4.1.36-44-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

Thanks for the reminder about using a USB 2 hub … I had done that when I was first setting up USB booting with Raspbian – that would have been very helpful to have remembered last night! :slight_smile:

I’m using grub to boot. I noticed that when trying to boot off the USB drive (no microSD installed), the USB drive seems to have some activity, but nothing comes up on the monitor. So it might be a bootloader issue, and it might be better with u-boot. But I was unaware that I had options. (Still learning SUSE) I’ll work through that option when I get back to it later today.

Good idea about trying dd rather than rsync. I’ll give that a shot, too. I’m hopeful that eventually one of these approaches will work out, and until then I’ll just work through the alternatives.

No ntfsprogs on SUSE? I couldn’t create the file system partitions with fdisk/parted + mkfs because there’s no mkfs.ntfs and no ntfsprogs available through zypper/yast. Hence my approach of doing the SUSE install on the USB drive – to have it create the partitions. I probably could have used rsync or dd to copy from the working microSD to the USB drive if I could have created the partitions directly. I’ll check Raspbian to see if it has mkfs.ntfs, and if so, that might be the more direct route.

Any idea how/why the system, after having booted off the microSD and having mounted /dev/mmcblk0p3 as the “/” partition, remounts “/” as being on /dev/sda3 when I insert the USB drive on which I did a fresh SUSE install (even though it seems to still be listing the /dev/mmcblk0p3 contents)? I’ve not seen that behavior before and have no idea how it was done. I’m sure I’m misinterpreting what I’m seeing. I’ll poke around with it a bit more today to see if I can make sense of it.

More later. Good luck on your setup.

Hi
On SLES ‘EFI’ is vfat, ‘BOOT’ is ext3 and ‘ROOT’ btrfs. But with dd it won’t matter…

Check /etc/fstab entries, they need to be changed for the drive (or did they change to sda3).

To clean up devices for prepping with dd I use wipefs -a /dev/device_partition_number, then once all the partitions are done, also the disk wipefs -a /dev/device.

Malcolm, just following up with progress to date.

I ended up stuck where Freigeist is in your link above, but I got there by a different route. I get it to boot, sorta, off the USB drive, but it hangs looking for device “dev-sda3”: note the “-” in there. I’m sure that’s a constructed reference, but I can’t find where it’s constructed in any of the boot config files. So I think I need to wait and watch as Alex and Freigeist work through the problem – way out of my depth here.

I took a different route to get to the same point. I didn’t recompile everything. If it ever works, I’ll document in detail, but briefly:
o installed ntfs-3g and tools and btrfs on Raspbian
o partitioned a new USB thumb drive to mimic the microSD partitioning that was created by the SLES install process
o mounted the sequence of partitions on the new drive as suggested in the Raspbian MSD install process (mkdirs to create /boot and /boot/efi)
o mounted the sequence of partitions on the microSD drive (installed on a USB 2.0 adapter)
o did rsyncs to copy the drive contents from the microSD to the USB thumb drive
o edited the thumb drive fstab to replace the “dev-disk-by” references with “/dev/sda”
o removed the Raspbian microSD drive and the SLES microSD drive (on USB adapter) and rebooted, with only the USB SLES thumb drive installed
o that got some boot activity going, but the various config files in /boot and /etc/default and /etc/sysconfig/bootloader needed to have the “dev-disk-by” references changed to “/dev/sda”
o so I rebooted in SLES (using the SLES microSD in the microSD slot) and edited those references to “dev-disk-by” that I could find; pulled the microSD and rebooted from SLES USB
o that gets a boot that fails but then fails over to something vaguely like the splash screen, from which you can select the image you want to boot
o and that boot then hangs looking for “dev-sda3”. So it gets as far as the systemctl loading process but hangs.

There’s got to be an easier and more systematic way to edit those “dev-disk-by” references, but I did it manually – they’d be reverted by any subsequent zypper or yast updates to the kernel. But I was just trying to verify that it would work at all before I delved into the grub install process. Pretty ugly and not very reproducible to this point.

I’ll go work on something else for a while and hope that Alex gets back to look at this again. Doesn’t look like it’s something I can get working yet.

Hi
Did you add to the grub command line root=/dev/sda3?

Yes, all four menu entries [“Linux 4.4.38-93-default” and recovery mode and “Linux 4.4.21-90-default” and recovery mode] in BOOT/grub2/grub.cfg have these within the {}:

    set root='hd0,msdos2'
    if [ x$feature_platform_search_hint = xy ]; then
      search --no-floppy --set=root --hint='hd0,msdos2'  /dev/sda3
    else
      search --no-floppy --set=root /dev/sda3
    fi
    echo    'Loading Linux 4.4.38-93-default ...'
    linux   /Image-4.4.38-93-default root=/dev/sda3 disk=/dev/sda resume=/dev/sda4 loglevel=3 plymouth.enable=0  console=ttyS0,115200n8 console=tty quiet ${extra_cmdline}

(BOOT is /dev/sda2, which comes up as /boot when the system is running).

Malcolm, quick update: there’s a new post by Freigeist in the thread to which you posted the link above in which he explains how he got USB booting to work. I edited the grub template as he suggested (and commented out another uuid reference I found there) and re-generated the boot files and installed them. But while I got a really clean boot (that process cleaned up all the error messages I’d gotten before), I ended up at the same hung condition, waiting for “dev-sda3”. Freigeist reported that that’s what his latest post fixed. I’m guessing that I’ve left a trail of erroneous edits as I’ve tried to get SUSE booting off USB, so I will give it another try after generating a clean install on a microSD card to get started.

Anyway, it looks like there’s hope that this does, in fact, work.

David

Malcolm: Well, I’m still getting hung at the same point (searching for “dev-sda3”) at boot time. I have a feeling that I’m missing a step in the grub regeneration/install process. The boot process itself seems pretty normal with the USB drive there (except that it does note that it can’t find mmc). But it hangs when systemd (I think it is) starts wanting to mount drives. Odd because I’ve verified that fstab correctly points to /dev/sdax. So I think there’s some step I’m missing that sets parameters for systemd. I’m using instructions to regenerate and install the grub boot files from here:
https://forums.opensuse.org/content.php/128-Re-install-Grub2-from-DVD-Rescue
Is there a better (or more complete) set of instructions? Essentially two commands in that reference.
grub2-mkconfig -o /boot/grub2/grub.cfg
Seems to work fine, but gives a warning:
/run/lvm/lvmetad.socket: connect failed: No such file or directory
WARNING: Failed to connect to lvmetad. Falling back to internal scanning.
Found SUSE Linux Enterprise Server 12 (aarch64) on /dev/mmcblk0p3
done
and
grub2-install /dev/sda
gives me an error message and then says there were no errors:
Installing for arm64-efi platform.
requested operation failed: status=8000000000000002
Installation finished. No error reported.
No idea where that “requested operation” message is coming from.

Anyway, I also noticed that the grub.cfg file that results from all this still references UUID – can’t find where that’s being generated. So I’m missing some step in here, probably in a
config file associated with the grub build/install process. Any suggestions where to look?

Here’s the process I followed:
1. Install a fresh copy of SLES on a microSD drive (32GB in this case)
2. Boot a Raspberry Pi-3 with that SLES microSD drive and login
3. Connect to the Internet (hardwired or WiFi) and update software; reboot
4. Clone the SLES microSD partition structure on a USB hard/thumb drive (8GB)
5. Copy the microSD files to the USB hard/thumb drive, by partition
6. Fetch the Raspbian USB-booting bootcode.bin & start.elf and copy them to the USB drive efi partition
7. chroot to the target-drive mount point so subsequent work will be done on the USB cloned drive
8. Delete ssh host keys so SLES will regenerate them on next boot
9. Edit the USB-drive grub configuration template to use the USB drive (/dev/sda) designations rather than uuid
10. Create and then install the grub bootloader system with that new configuration
11. Edit etc/fstab on the new USB / partition so that it references the /dev/sda partitions
12. Verify that there are no references to “uuid” or “UUID” or “mmcblk0” on the USB /boot partion (grep -r)
Edit files manually if necessary to replace them with references to /dev/sda.
12. Exit from “chroot”, poweroff, remove microSD drive, powerup with USB drive still installed and watch it boot.

Hi David
OK, looks like enough details for me to try and duplicate, give me a day or so to get some free time to look at it.

Malcolm, re-reading Freigeist, I think he started with a Tumbleweed image. I started with the SLES image, which got me exactly the same systemd loop looking for dev-sda1 that he got initially. I’m going back to try the Tumbleweed Raspberry Pi XFCE 2017-01-13 image. My first attempt, trying to modify Tumbleweed on a USB thumb drive from the SLES microSD environment failed, because of partition structure differences between the two, I think. I’ll try installing a Tumbleweed on microSD and then install and modify on USB. More later.

[QUOTE=hdtodd;36212]
Is there a way to direct the boot process to go to a USB-attached drive, or to redirect the boot to a hard drive with a modified bootstrap program on a microSD? [/QUOTE]
I feel a bit like Scott trying to get back from the South Pole: the expedition was unsuccessful, but I feel I should leave notes in case others follow this so they might learn from my failures. Thanks to Malcolm Lewis for his help: I wouldn’t have made it this far without his suggestions.

And for a related thread, on a problem that arose when trying to set up WiFi in Tumbleweed but ended up in a similar place, see https://forums.opensuse.org/showthread.php/522941-How-to-resolve-quot-A-start-job-is-running-for-dev-disk-by-quot-problems-Raspberry-Pi-openSUSE

Key references were:
[LIST=1]
[]Installing Raspbian on a bootable USB drive: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md
[
]Conversation between Freigeist and Alex Graf indicating that, ultimately, it was possible to get Tumbleweed to boot from USB (but providing insufficient details about how to actually do it): https://lists.opensuse.org/opensuse-arm/2017-01/msg00060.html
[]Installing WAN on RPi under openSUSE: https://lists.opensuse.org/opensuse-arm/2016-11/msg00018.html
[
]Installing grub: https://forums.opensuse.org/content.php/128-Re-install-Grub2-from-DVD-Rescue
[/LIST]

I worked on a Raspberry Pi-3 that I had prepared to boot off USB (first reference above). I had developed a Raspbian USB drive and confirmed that that Pi-3 would boot Raspbian off the USB drive (the preparation part is a one-time process). Subsequently, I did my work in SLES or openSUSE on that Pi-3, booting off a microSD card and working on either a USB thumb drive (Verbatim, Lexar) or on another microSD inserted into a USB adapter (so that I could confirm that that drive would boot when inserted into the microSD slot of the Pi-3).

Briefly, I was initially working in SLES-12.2 and hoped to get it booting off USB. When that was unsuccessful, I also tried LEAP 42.2 and various stable releases and snapshots of Tumbleweed, : 2017-01-13, 2017-01-28, 2017-01-31, 2017-0203. I used two techniques to create and edit my USB boot drive:
[LIST=1]
[]boot up some version of SUSE/openSUSE from the microSD in the microSD slot, and either install onto a USB drive (or microSD inserted into a USB adapter) from the .raw.xz file with the usual xzcat/dd process and then edit the files in the installed partitions on that USB drive; or
[
]boot from the microSD slot, then create the USB boot drive and partitions with parted, then populate the EFI and ROOT partitions with rsync from the microSD partitions, then edit the USB partitions to configure for USB booting.
[/LIST]
In both cases, subsequent work with grub2-mkconfig, grub2-install, mkinitrd, and dracut were done after isolating the USB drive from the microSD drive (as best possible). The USB root drive was mounted as /mnt/tgt, for example, and the USB efi drive (first partition, FAT32) was mounted as /mnt/tgt/boot/efi, mimicing the microSD mount structure. Except for the /dev, /sys, and /proc mountpoints (which may be a critical issue!), the two environments were separated for editing and grub/initrd creation by:

cd /mnt/tgt mount --bind /dev dev mount --bind /sys sys mount --bind /proc proc chroot /mnt/tgt <work on USB> exit
The configuration-generating programs wouldn’t work without /proc (or /dev, as I recall). But I think that is the step in the process that caused the ultimate boot loop failure.

Over the many attempts to get this to work, I worked at various times with the grub configuration files (/boot/grub2/, /etc/default/, /usr/share/grub2/*), with /etc/fstab, and with the mkinitrd and dracut shell scripts.

I routinely worked the USB drive configuration to the point where, when booting off the USB drive (no microSD installed), the Pi-3 would bring up the SLES/openSUSE splash screen (default or recover boot option), then initiate the boot, then hang (systemd, I believe) in a wait loop with the message:

A start job is running for dev-disk-by\\x2did-mmc\\x2dSL32G_0x28b56c90\\x2dpart2.device

where the part after x2did varied over the many attempts I made to get this to work. The hang was caused by some component that insisted upon referencing the root and swap partitions “by-id”, with references in grub to UUID; I was unable to coerce it into using “/dev/sda3” or “LABEL=ROOT”, for example.

I think the problem is that grub2-mkconfig (and mkinitrd/dracut) reference the /proc/cmdline from the running OS and use it to see the configuration for grub for the USB drive. I could find no way to avoid that. BUT, I could also find no way to get the hanging component (systemd?) to use “/dev/sda3” (for example) or “LABEL=ROOT” to identify the root device, for which it was stuck in a “no limit” loop.

I believe there is some development work going on in this area within the openSUSE group. It seems increasingly that the Raspberry Pi folks are preparing to use systems that boot off the network or off USB drives, and the RPi-3 is a pretty capable little computer when you put an SSD drive onto the USB port. But the characteristics of the boot problems (and particularly the related WAN-install problem, which once worked but failed on newer Tumbleweed releases) changed over the last three weeks, suggesting that there have been changes in the code in this area (grub/mkinitrd configuration).

So I’m guessing that this will get fixed in some future release, but for now it’s a non-functioning moving target.

If anyone sees mistakes I’ve made in my approach, please let me know and I’ll give it another try. But for now, I’ll just wait for it to work in a future release.

David Todd

Hi David,

I don’t believe it is so much about failures but more about sharing experiences so that others can progress beyond where we have been.

I hope others will jump in with their experiences. That’s how we make progress.

While this forum is a great place to exchange ideas, there are other vehicles that may provide greater visibility. Have you considered promoting your post to an Article or creating a Blog which could generate additional interest and may elicit feedback from others who may not participate in the forums?

You have provided a lot of valuable information. Thank you and please continue to contribute!

[QUOTE=KBOYLE;36556]Hi David,

I don’t believe it is so much about failures but more about sharing experiences so that others can progress beyond where we have been.

I hope others will jump in with their experiences. That’s how we make progress.

While this forum is a great place to exchange ideas, there are other vehicles that may provide greater visibility. Have you considered promoting your post to an Article or creating a Blog which could generate additional interest and may elicit feedback from others who may not participate in the forums?

You have provided a lot of valuable information. Thank you and please continue to contribute![/QUOTE]

Thanks for the kind comments and encouragement, Kevin! I’m still figuring my way around SUSE/SLES/openSUSE-LEAP-Tumbleweed distinctions, and I surely don’t know the best place to post queries or solutions. I’ll look into the Article vs Blog business. Given the volume of postings I’ve generated around these two issues, a blog might make more sense – there’s still a lot of give-and-take in the discussion as we try to figure out how to make it work. “Article” sounds more like a more-or-less finished approach to solving a problem, and I’m surely not there yet!

I had gotten pretty discouraged after spending a good deal of time on one problem, then switching to another and getting to the same dead end (!), but exchanges and guidance from Forum participants has been encouraging again. So I’ll see if I can make some more progress.

Thanks again for the note and encouragement!

David Todd
(old techie / SUSE newbie)

[QUOTE=hdtodd;36560]Thanks for the kind comments and encouragement, Kevin![/QUOTE]They are well deserved!

[QUOTE]I’m still figuring my way around SUSE/SLES/openSUSE-LEAP-Tumbleweed distinctions[/QUOTE]I was in much the same place myself not too long ago. You should think of this as a journey that offers new experiences and opportunities allong the way.

The forums are still the best medium for exploring ideas and resolving issues. They are the most interactive and the most informal. Blogs and Articles are less interactive but do allow others to provide feedback. A Blog or series of blogs is a good way to generate interest in a topic and to push out information if you have lots to offer.

That is also my assessment. Often, for those who are interested, solutions can be found by combing through the forums looking for them. An Article, on the other hand, consolidates all the pertinent information in one place and makes it available to a larger audience. I’m sure it’s only a matter of time before you are there.

Often real progress can only be achieved after the occasional setback. This is a Community of people helping other people. While the Knowledge Partners, like Malcolm and myself, each have our own areas of expertise and we provide general support in the various forums, we always keep our eyes open looking for others who have something to offer. It’s the contributions from other community members like yourself that makes these forums something special.

Your contributions are greatly appreciated!

Yes, there is. How-To posted here:https://forums.opensuse.org/showthread.php/523341-Create-a-USB-Bootable-aarch64-openSUSE-Raspberry-Pi-3?p=2813799#post2813799
Thanks to Malcolm Lewis and Alex Graf for working through to the solution.

Thanks for the encouragement, Kevin. Great support from the community. Solution posed here: https://forums.opensuse.org/showthread.php/523341-Create-a-USB-Bootable-aarch64-openSUSE-Raspberry-Pi-3?p=2813799#post2813799