Suddenly I receive an error with the bellow dump when the SLES 12.4 started. Then, the system becomes unavailable.
When I repair the root filesystem [on a guest VM otherwise it isn’t possible because the root filesystem is mounted] then the root partition is marked as full even an empty space is shown in [FONT=Courier New][COLOR="#008000"]df[/COLOR][/FONT].
What happened, how can I solve it? The very strange thing is that when I restore a 3 weeks old backup [where all was OK ] into a new VM I receive the same error messages.
This is the output from [FONT=Courier New][SIZE=2][COLOR="#008000"]btrfs check --repair /dev/xvdc2[/COLOR][/SIZE][/FONT]
[FONT=Courier New][SIZE=2]enabling repair mode
Checking filesystem on /dev/xvdc2
UUID: c88dbf5b-3513-4966-b3d6-5bb6c9b7717e
checking extents
Fixed 0 roots.
checking free space cache
cache and super generation don’t match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 9380937728 bytes used err is 0
total csum bytes: 8061100
total tree bytes: 230670336
total fs tree bytes: 181600256
total extent tree bytes: 34013184
btree space waste bytes: 41120097
file data blocks allocated: 10417725440
referenced 8217731072[/SIZE][/FONT]
Then I get this:
[FONT=Courier New][SIZE=2]Filesystem 1MB-blocks Used Available Use% Mounted on
/dev/xvda2 11249MB 9918MB 0MB 100% /[/SIZE][/FONT]
@df: You are right, but this is not the problem. @volume full: This is not the problem and does not solve my issue.
I may have been not accurate enough: The root filesystem is totally damaged. When I start the server the 1st time [from a 3 weeks old backup] so I get after a while the above error messages.
When I try to start the server then the lot of error messages appear and the server becomes unavailable. I found also this post where the same error messages are posted.
Please note that the system is referring with the message ‘[FONT=Courier New][SIZE=2][COLOR="#FF0000"]kernel BUG at …/fs/btrfs/relocation.c:1449![/COLOR][/SIZE][/FONT]’ to a bug in the kernel in the 1st line of the dump!
Again: The server is idle, this means, apart from the normal services, nothing special is running. Suddenly, the error messages in the 1st post appear in the console and then the VM is broken!
It looks to me like the kernel got a very serious bug in one of the previous updates.
[QUOTE=AAEBHolding;57678]@df: You are right, but this is not the problem. @volume full: This is not the problem and does not solve my issue.
I may have been not accurate enough: The root filesystem is totally damaged. When I start the server the 1st time [from a 3 weeks old backup] so I get after a while the above error messages.
When I try to start the server then the lot of error messages appear and the server becomes unavailable. I found also this post where the same error messages are posted.
Please note that the system is referring with the message ‘[FONT=Courier New][SIZE=2][COLOR="#FF0000"]kernel BUG at …/fs/btrfs/relocation.c:1449![/COLOR][/SIZE][/FONT]’ to a bug in the kernel in the 1st line of the dump!
Again: The server is idle, this means, apart from the normal services, nothing special is running. Suddenly, the error messages in the 1st post appear in the console and then the VM is broken!
It looks to me like the kernel got a very serious bug in one of the previous updates.[/QUOTE]
Hi
At GRUB, can you select advanced boot options and boot to an earlier snapshot?
[QUOTE=malcolmlewis;57684]Hi
At GRUB, can you select advanced boot options and boot to an earlier snapshot?[/QUOTE]
It is a VM running under XenServer. I have only snapshots available which are provided by XenServer - the SLES snapshots are not available.
I think I should provide this information because it may be the real reason for this issue; the root partition was running out of space so I did this in order to increase the root partition:
[LIST=1]
[]Resized in XenCenter the root partition.
[]Detached the SLES 12.4 VM.
[]Attached the root partition to another SLES 12.4 VM.
[]Resized the root partition in the other VM with yast/Partition Manager.
[]Detached from the helper VM the root partition.
[]Attached the resized root partition to the origin VM.
[*]
[/LIST]
All steps like increasing, detaching and attaching have been performed when the VMs have been down.
Since then even an old backup crashes after few seconds when the VM started.
Is it possible there is a more deeper information stored in the disk and when even I restore to a snapshot where the disk hasn’t been resized so it doesn’t match and the problems starts?
Hi
It does make it clearer Are you in a position to raise a SR (Support
Request)?
–
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
Tumbleweed 20190512 | GNOME Shell 3.32.1 | 5.0.13-1-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!
[QUOTE=malcolmlewis;57691]Hi
It does make it clearer Are you in a position to raise a SR (Support
Request)?
[/QUOTE]
Not really, I never raised a SR. I am alone with in business and I am maintaining my servers and the XenServer on my own. It is now working like this more or less perfectly since 4 years but with this issue I am totally overstrained.
Hi
So if you boot the system to runlevel 1 (at grub and 1 to the options), can you mount the / partition and look at the logs to see what’s failing? Or boot the system in rescue mode.
[QUOTE=malcolmlewis;57716]Hi
So if you boot the system to runlevel 1 (at grub and 1 to the options), can you mount the / partition and look at the logs to see what’s failing? Or boot the system in rescue mode.[/QUOTE]
Which logs should I check? I am in the rescue mode and mounted the root partition as [SIZE=2][FONT=Courier New][COLOR="#008000"]/mnt[/COLOR][/FONT][/SIZE].
Can I try to fix it somehow? Running [FONT=Courier New][SIZE=2][COLOR="#008000"]btrfs check --repair /dev/xvda2[/COLOR][/SIZE][/FONT] doesn’t solve the problem. As I wrote, when I then boot regularly the / partion is out of space.
[QUOTE=AAEBHolding;57720]Which logs should I check? I am in the rescue mode and mounted the root partition as [SIZE=2][FONT=Courier New][COLOR="#008000"]/mnt[/COLOR][/FONT][/SIZE].
Can I try to fix it somehow? Running [FONT=Courier New][SIZE=2][COLOR="#008000"]btrfs check --repair /dev/xvda2[/COLOR][/SIZE][/FONT] doesn’t solve the problem. As I wrote, when I then boot regularly the / partion is out of space.[/QUOTE]
Hi
Check down in /var/log for big files, check those for clues (maybe even copy them off to an external drive), especially messages log.
Maybe it’s coredumping?
Run;
coredumpctl list
If there are old logs you think can be deleted, remove those and see how it goes getting some disk space.
[QUOTE=malcolmlewis;57722]Hi
Check down in /var/log for big files, check those for clues (maybe even copy them off to an external drive), especially messages log.
Maybe it’s coredumping?
If there are old logs you think can be deleted, remove those and see how it goes getting some disk space.[/QUOTE]
I summarize what I did:
[LIST=1]
[]Start the VM in the rescue mode.
[][FONT=Courier New][SIZE=2][COLOR="#0000FF"]mount /dev/xvda2 /mnt[/COLOR][/SIZE][/FONT]
[][FONT=Courier New][SIZE=2][COLOR="#0000FF"]btrfs balance start -v --full-balance /mnt[/COLOR][/SIZE][/FONT]
[]
[/LIST]
When I got [FONT=Courier New][SIZE=2][COLOR="#FF0000"]Done, had to relocate 12 out of 12 chunks[/COLOR][/SIZE][/FONT] then it worked (1).
When I got the message
[FONT=Courier New][SIZE=2][COLOR="#FF0000"]ERROR: error during balancing ‘/mnt’ - No space left on device
There may be more info in syslog - try dmesg | tail[/COLOR][/SIZE][/FONT]
I had to proceed with (2).
And here, I had two different situations:
One VM has been completely recovered and works well even before the VM didn’t start and crashed while booting - with other words the VM became void.
The other VM was more persistent and really tiresome. I had to find a XenServer snapshot from the VM where the VM did at least start properly even it crashed within few seconds.
Then, to check if it really works I was running some heavy disk access routines where before the VM crashed within seconds. Now, it runs without any problem.
I hope, it will remain like this.
I summarize what I did:
[LIST=1]
[]Start the VM in the rescue mode.
[][FONT=Courier New][SIZE=2][COLOR="#0000FF"]mount /dev/xvda2 /mnt[/COLOR][/SIZE][/FONT]
[][FONT=Courier New][SIZE=2][COLOR="#0000FF"]btrfs balance start -v --full-balance /mnt[/COLOR][/SIZE][/FONT]
[]
[/LIST]
When I got [FONT=Courier New][SIZE=2][COLOR="#FF0000"]Done, had to relocate 12 out of 12 chunks[/COLOR][/SIZE][/FONT] then it worked (1).
When I got the message
[FONT=Courier New][SIZE=2][COLOR="#FF0000"]ERROR: error during balancing ‘/mnt’ - No space left on device
There may be more info in syslog - try dmesg | tail[/COLOR][/SIZE][/FONT]
I had to proceed with (2).
And here, I had two different situations:
One VM has been completely recovered and works well even before the VM didn’t start and crashed while booting - with other words the VM became void.
The other VM was more persistent and really tiresome. I had to find a XenServer snapshot from the VM where the VM did at least start properly even it crashed within few seconds.
Then, to check if it really works I was running some heavy disk access routines where before the VM crashed within seconds. Now, it runs without any problem.
I hope, it will remain like this.[/QUOTE]
Hi
Thanks for the feedback and good work
I’ve experienced this exact same error at relocation.c:1449 running “4.12.14-lp150.12.58-default #1 openSUSE Leap 15.0”. The call to btrfs_insert_root() within create_reloc_root() returns an error code which causes the subsequent BUG_ON() assertion to fail.
I’m hopeful that the following commit in the kernel will fix the underlying issue.
OMG, after the last updates SUSE offered and I ran it started again. But now I cannot fix it. One VM which had the problem wasn’t updated and it doesn’t have any problem.
Again the same. There is no way I fix it with restoring a snapshot. This VM is apparently broken.
Are you in a position to raise a SR (Support
Are you in a position to raise a SR (Support