LVM migration and boot issues

[peterb]

A bit of an obscure one...

basically the setup is a Redhat 9 installation upgraded to Fedora core 6. As an upgrade it retained the hard disk fixed partitions, which ere organised so that /boot was on one samall partition, / was on another larger partition (20GB) and /home was on another larger partition.

After some perusing and research, I decided to migrate to LVM (which is used by FC6 in a new installation anyway) as I think the flexibility it offers oytweighs the added underlying complexity.

So having created the VG, I have three LVs which are earmarked for /home /var and eventually root.

The first stage of copying /home to the relevant LV went uneventfully without taking the system down, and I subsequently migrated /var by going to single user mode.

Now I (naivley) thought that migrating the remainder of the root filesystem would be a simple case of booting to the rescue disK (so that I am not using the system I am migrating) copying the remaining directories across, creating the mount points for /boot /var /home /sys and /proc , altering /etcfstab to mount the new lv, and finally changing grub.conf to boot off the new filesystem. (/boot has to remain on its own dedicated partition as LVM is only enabled during the boot process)

However, on rebooting, the system loaded the initial kernel (in ranmdisk), but then panicked with the error message "cannot find /dev/root" and sure enough there isn't a /dev/root on the new or old file systems.

A look at another system that had FC6 installed from scratch shows that there IS a file called root in /dev. It is a block device, and my first thought was to make it using mknod although figuring the major and minor numbers might be a bit tricky.... but of course /dev is created from scratch by the udev process at each and every boot, so there is no need to explicitly create it.

There is a surprising lack of detailed knowledge on google (at least I haven't found it)about the nitty gritty of the boot process, but my guess is that /dev/root is a block device that does something with the main kernel image and as that is missing, the 'main' kernel cannot run from the hard disk. (almost as if it is an alias)

So the question is... what bit that should create the /dev/root block device is missing, and from where? (one lead from a Carla Schroeder website indicated that there is a rule set in /etc that might control this, but I haven't had a chance to investigate this yet)

[b0redom]

You went from RH9 -> FC6? Seriously I would consider a clean install unless there's some really good reason you can't do it - especially if you're messing about with filesystems.

[ikonia]

ok - the problem you've got on this is......

1.) you've migrated your root file system to lvm - so therefore your lvm devices have got to be established before you actually boot your system.
2.) the rule set your probably looking at is the udev rule set, which in this case probably isn't the problem.
The problem is more likley to be your volume groups and volumes can't bee seen at udev node creation time so you're not getting a block file created by udev.

Tell me a little more about your volume groups, are they just disks or raid devices in a volume group.
Do you have any unusual controllers that may be problematic for your initrd ?

[peterb]

Thank you...

There is nothing fancy about the LVM set up - the spare space on the drive (hda) was partitioned into 5 small partitions and 4 of these grouped to make one VG called main. This was divided into 3 LVs called root, var and home. I copied the existing /var and /homme directories into the respective LVs and mounted them at the appropriate points, editing the fstab entries accordingly

/dev/main/home /home ext3 defaults 1 2) and the same for /var

This worked Ok as you would expect as the sdystem was still booting off the fixed partition system

I then booted the system using the rescue disk to give a ram based ssystem with the main system image at /mnt/sysimage. I dis a vgscan to detect the vg and created a mounty point /new on the sysimage. I mounted /dev/main/root at /new, and then did cp -a /new on all the system directories, making sure I created the appropriate mountpoints for /boot /sys and /proc. Finally I altered the /etc/fstab file on the system on the lv to include a line like the one above, but using /dev/main/root to be mounted at /

Finally I altered the grub.conf file so that the line pointing to the kernel read /dev/main/root instead of /dev/hda3 label=/ (or something like that - I haven't got the system in front of me at the moment) These were taken from system that had FC installed from scratch.

b0redom - you are quite right! And in fact that is what the Fedora project recommends. I had previously upgraded a RH9 system to FC4 without problem, and then from FC4 to 6 again without problem. However the jump to FC6 from RH9 was not easy! The basics were fine, but the X windows system was a nightmare and basically wouldn't run. Eventually I used yum to erase all the X system and then to re-install it, but it probably took more effort to do that than install from scratch. The reason why I didn't was that the system had grown a bit since I had it, and the thought of setting all that up from scratch was a bit daunting! (SSH keys, apache virtual hosts etc) That said, all the server functions upgraded without problems - it was only the x server that was tricky (and I learned a lot fixing it!)

[ikonia]

First thing to do is check for typos, this is probably wrong but in your two posts you've mentioned configs for /dev/main/root - which is correct and /dev/root which is in correct.

Use the livecd to double check these configs for you menu.lst and fstab.

If you're getting to the point where the kernel actually boots then your initrd is fine for the disk controller/lvm modules. You're not doing anything with /boot on a seperate partition as I recall there are rules for having /boot in lvm control in the same way there is for raid 0 and 5.

[peterb]

Checked for typos - none found (apart from the one in the previous post!)

I get the following output on booting:

root (hd0,0) (which is correct)
Filesystem type is ext2fs, partition type 0x83
...... rest of initial boot sequence all OK

Uncompressing Linux... OK, booting the kernel.
ACPI: Invlaid PBLK length [5] (This is OK - it happens with normal boot - not set up ACPI)

Red Hat nash version 5.1.19.0.3 starting
mount: could not find filesysten '/dev/root'
setuproot: error moving /dev failed: No such file or directory
setuproot: error mounting /proc failed: No such file or directory
setuproot: error mounting /sys failed: No such file or directory
switchroot: mount failed: No such file or directory
Kernel panic - not syncing; Attempted to kill init!

And that is it. I then repeated this having cleared /dev to see if it would repopulate, but it looks as if udev isn't running. BUT udev can't run until the kernel is loaded (or can it) in which case some entries in /dev must have to be made previously - (there is a list in /etc/makedev that shows that - but it doesn't include /dev/root and anyway, that can't run until the filesystem is mounted... and /dev/root is a block device anyway) Maybe I should manually create /dev/root using makenod - but what should I use for the major and minor numbers? (time to reach for The Linux Kernel book)

I wish I understood it more!!!

(btw, I think boot has to be in a physical partition, it cannot be in an lv because lvm hasn't started at that point)

A bit later... tried creating a block device with major and minor numbers as 253 and 0 (which I think is the same as on the clean install system - but it made no difference...)

[ikonia]

yes, your correct on a few things here, boot can only be on a physical partition hence why I said to check that.

And yes - certain nodes /dev/console /dev/null are the bare minimum to get a system to boot. However /dev/root looks to be required for your situation. so create it with

mknod /dev/root b 9 0

See what that progresses with.

[ikonia]

ok - update on this

/dev/root - is really just a "link" to the real root device for boot time

eg: if you have your root file system disk as /dev/sda1 then /dev/root should have the same major / minor numbers as that.

That is to get the system to boot as lvm is not running at that point.

Ignore the major / minor numbers on in my earlier example, This will be system specific

For example.

the root file system on my system is on a meta device

Filesystem Size Used Avail Use% Mounted on
/dev/md0 38G 1.7G 35G 5% /

which looks like this

brw-r----- 1 root disk 9, 0 Jun 27 16:55 /dev/md0

Therefore my root "device" could in theory be a symlink to /dev/md0 - or in my case another block device to access it

brw------- 1 root root 9, 0 Jun 27 16:55 /dev/root

As you can see on another system

/dev/md1 35G 1.5G 31G 5% /

so on this system /dev/md1 is my root fs.

brw-r----- 1 root disk 9, 1 2007-06-26 18:12 /dev/md1

so.....

brw------- 1 root root 9, 1 2007-06-26 18:12 /dev/root

Does that make sense ?

[peterb]

yes, that does make sense - thank you, and doing ls -l /dev/hda1 brings up

brw-r----- 1 root disk 3, 1 Jul 11 21:28 /dev/hda1


but where do I create it? If it is /dev. then it won't be read until the file system is mounted, and that can't happen until the system is booting, which it isn't - so does that mean that I should remake initrd? I will try it later this evening though!

Sorry for late reply - I have been out all day - and thanks for the help so far.

EDIT

Just tried it - but again no difference - suppose because the root fs isn't being mounted - the more I think about it, the more I think the problem/solution lies in initrd - as that has to bring up the fs so that control can be transferred to the physical system - but I'm still hazy on that - more reading required!

EDIT

Didn't get a chance to read up last night, but thinking about it, I am pretty sure the answer lies in the initrd file, and I have just been looking at the mkinitrd man pages at http://www.netadmintools.com/html/mkinitrd.man.html - I haven''t got the system in front of me. The initrd contains the drivers for ide devices etc so it would seem reasonable that it would contain the LVM drivers too, but as my initrd was generated before I installed lvm on the system - it won't contain them! In fact the mkinitrd options allow LVM to be specifically excluded, so I guess that if the appropriate modules for LVM are in the /etc/build directory, they would be included by default - and as LVM is running - they must be there. So back to the man page and this weekend I'll see if I can generate a new initrd file.

[ikonia]

dev isn't mounted as such and the static nodes of say /dev/console /dev/root etc will be readable.

The initrd for your disks/lvm is a likley candidate for not being loaded. However thats hard to tell.

I'll actually play with this later on today and walk through the boot process to get this sorted.

[peterb]

Thank you - I appreciate the time you are spending on this.

EDIT

Making some progress...

I have remade the initrd image, using the new fstab file as a source for the file system, and the system now brings up the lvs and recognises the volume group and logical volumes. However it then goes on to give the following error :

mkrootdev: expected fs options
mount: missing mount point
setuproot: moving/dev failed: bo such file or device

The remaining errors are the same as the earlier post.

So I deduce that it is now recognising the LVM and the lvm file systems, but not creating a mount point in /dev. However I think I need to see what mkrootdev should do.

A bit later (after a google search - google IS my friend)...

mkrootdev is a command in the nash scripting language. #man nash brings up the details, but basically

mkrootdev path
Makes path a block inode for the device which should be mounted
as root. To determine this device nash uses the device suggested
by the root= kernel command line argument (if root=LABEL is used
devices are probed to find one with that label). If no root=
argument is available, /proc/sys/kernel/real-root-dev provides
the device number.

So I'm not quite sure where to go from here - but it seems so close to the solution... I need to find the nash script...

[peterb]

OK - cracked it

If you look back at the previous post, I had made a new initrd which recognised the logical volumes.

I did that by mounting the root lvm on the old file system at /new and I then altered /new/etc/fstab (that is, the fstab that would be mounted when the new root fs came up) so that the root fs line became

/dev/main/root / ext3 defaults 1 1

and commented out

label=/ / ext3 defaults 1 1

then ran (from /sbin) mkinitrd /boot/grub --fstab=/new/etc/fstab (etc, etc etc)

to create the new initrd in /boot/grub and I edited the relevant parameter line in grub too. (I - fortunately) backed up the old initrd first)

This didn't boot, as I said in the last post, but at least I could reboot the system as was by using the old initrd and old grub option line.

So it was back to the books on boot sequences, (not a lot of help - one was down in the kernel processes which gave a couple of pointers as to where the process was failing - the other up at too high a level) and then I had a look at the mkinitrd script, and while I am no expert at reading comp[licated scripts, it did give me an idea.

The solution was to boot the system using the old root fs, then mount the new fs at the root mount point (so instead of mounting it as /new, it was mounted at / ) and the running mkinitrd using the default settings. And it works!

I'm not entirely sure why, but my guess is that the first attempt at running mkinitrd detected the requirment for the lvm drivers, but was using sysinfo from the old fs and it ended up with a mish mash of conflicting info. Running it against the target fs that was mounted resolved those conflicts, although the system did hiccup a bit at first reboot and insisted on a manual run of fsck on the / fs and resolved a few errors. I wish I could claim that it was purely the result of logical fault finding and deductuion, rather than an educated fuess - but hey ho!

So problem solved - thank you Ikonia for your input - two heads are much better than one - and it goes to show (again) that it is only when things don't work as expected that you discover a bit more about how they work!

(and I now know why the system initially comes up with the nash message at the very start of the boot sequence)

[Agent]

Feel free to use the "Thank you" button on the posts which have helped you

[peterb]

Oh yes! (But there are so many (posts) if I clicked all of them, Ikonia might get big-headed !)!

[ikonia]

Peter,

Good job on walking this through, I think I may have missed the part about you replacing your initrd in the way that you did. I tried a good few builds at the weekend and although I could quite easily break LVM at boot time, I couldn't get into the situation you where in, but your explaination makes it a little clearer. Interesting jam you got your self into and nice to understand how it all came about for future reference, although I can't see this situation appearing all too often

Its nice to see interesting issue like this and more so to see them resolved in this manner with out random "try rebooting" or insults being thrown.

Nice job Peter.

[peterb]

Thank you - and particularly for the time and effort you put in to this. There is a bit of tidying up to do - while I was trying this, I think there were a couple of system updates applied, and as /var and /home are on separate mountpoints, the updates as they affect those fs are out of step with the new root fs, so this evening I will copy the old root fs to the new, which should sort it out.

Without wishing to provoke anyone into a Wndows/Linux flame war, it is interesting to speculate whether solving this problem would have been any easier with Windows than with Linux. I think not (unless qualified as an MCSE). Although it wasn't straightforward diagnosing the problem, the open spource nature made it much easier.