dmesg WARNING btrfs_free_reserved_data_space_noquota

jimsmithson · July 12, 2016, 6:25pm

Noticed multiple messages like this in dmesg output:

[FONT=Courier New][SIZE=1][ 1889.128759] ------------[ cut here ]------------
[ 1889.128800] WARNING: CPU: 10 PID: 1510 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[ 1889.128802] Modules linked in: af_packet iscsi_ibft iscsi_boot_sysfs vmw_vsock_vmci_transport vsock xfs libcrc32c coretemp crct10dif_pclmul crc32_pclmul dm_mod crc32c_intel aesni_intel aes_x86_64 ppdev vmxnet3 parport_pc vmw_balloon lrw parport i2c_piix4 gf128mul shpchp glue_helper mptctl pcspkr serio_raw ablk_helper vmw_vmci battery cryptd processor button ac btrfs sr_mod cdrom ata_generic xor raid6_pq sd_mod ata_piix vmwgfx ahci libahci ttm mptspi scsi_transport_spi drm vmw_pvscsi mptscsih libata mptbase floppy sg scsi_mod autofs4
[ 1889.128831] Supported: Yes
[ 1889.128835] CPU: 10 PID: 1510 Comm: kworker/u50:1 Tainted: G        W         3.12.59-60.45-default #1
[ 1889.128837] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[ 1889.128844] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-1)
[ 1889.128846]  0000000000000000 ffffffff81520a30 0000000000000000 ffffffffa029384f
[ 1889.128851]  ffffffff81058f72 ffff880c39fb4800 ffff880c39f8fc00 0000000000001000
[ 1889.128854]  0000000000001000 ffff880b2a51082c ffffffffa01fe2e8 ffff880b2a5109b8
[ 1889.128857] Call Trace:
[ 1889.128870]  [<ffffffff8100474d>] dump_trace+0x7d/0x2d0
[ 1889.128875]  [<ffffffff81004a34>] show_stack_log_lvl+0x94/0x170
[ 1889.128879]  [<ffffffff81005ce1>] show_stack+0x21/0x50
[ 1889.128885]  [<ffffffff81520a30>] dump_stack+0x5d/0x78
[ 1889.128892]  [<ffffffff81058f72>] warn_slowpath_common+0x82/0xc0
[ 1889.128905]  [<ffffffffa01fe2e8>] btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]
[ 1889.128942]  [<ffffffffa0219c5f>] btrfs_clear_bit_hook+0x22f/0x2f0 [btrfs]
[ 1889.128992]  [<ffffffffa023209c>] clear_state_bit+0x5c/0x1e0 [btrfs]
[ 1889.129055]  [<ffffffffa0232a96>] __clear_extent_bit+0x2b6/0x400 [btrfs]
[ 1889.129116]  [<ffffffffa0233abf>] extent_clear_unlock_delalloc+0x5f/0x250 [btrfs]
[ 1889.129177]  [<ffffffffa021c62c>] cow_file_range+0x29c/0x480 [btrfs]
[ 1889.129223]  [<ffffffffa021d6fa>] run_delalloc_range+0x35a/0x390 [btrfs]
[ 1889.129271]  [<ffffffffa023437e>] writepage_delalloc.isra.39+0xfe/0x160 [btrfs]
[ 1889.129332]  [<ffffffffa0235041>] __extent_writepage+0xc1/0x2e0 [btrfs]
[ 1889.129392]  [<ffffffffa023550f>] extent_write_cache_pages.isra.33.constprop.55+0x2af/0x360 [btrfs]
[ 1889.129454]  [<ffffffffa023741d>] extent_writepages+0x4d/0x60 [btrfs]
[ 1889.129506]  [<ffffffff811d0389>] __writeback_single_inode+0x39/0x2b0
[ 1889.129510]  [<ffffffff811d10e4>] writeback_sb_inodes+0x1a4/0x400
[ 1889.129514]  [<ffffffff811d13d6>] __writeback_inodes_wb+0x96/0xc0
[ 1889.129518]  [<ffffffff811d1683>] wb_writeback+0x283/0x330
[ 1889.129522]  [<ffffffff811d2f72>] bdi_writeback_workfn+0x2c2/0x480
[ 1889.129530]  [<ffffffff81073f31>] process_one_work+0x171/0x460
[ 1889.129535]  [<ffffffff81074bea>] worker_thread+0x11a/0x3c0
[ 1889.129539]  [<ffffffff8107b564>] kthread+0xb4/0xc0
[ 1889.129547]  [<ffffffff8152edd8>] ret_from_fork+0x58/0x90
[ 1889.129550] ---[ end trace f527d775878825ad ]---
[/SIZE][/FONT]
# dmesg|grep WARNING|wc -l
10
This is after doing the Yast Online Update last night (7/11/16). Was getting same warnings (but a different CPU) before YOL.
# uname -a
Linux SR0005IB02 3.12.59-60.45-default #1 SMP Sat Jun 25 06:19:03 UTC 2016 (396c69d) x86_64 x86_64 x86_64 GNU/Linux

VMware 5.5 Virtual Machine Version 8. Non-SSD SAN. root file system is BTRFS, data filesystem is XFS.
22 CPUs, 48GB RAM

This system is scheduled to go live soon but I will not do it until I’m sure that I’m not going to lose the root file system unexpectedly.
TIA for any help!

malcolmlewis · July 12, 2016, 7:19pm

Hi
Not running out of space with snapshots?

btrfs fi usage /
snapper list

What services are running on this system, not using a custom directory on / that may be included in snapper snapshots?

PS, if you use code tags around terminal output, makes life easier

malcolmlewis · July 12, 2016, 7:49pm

Hi
I also see this recent post, so wonder if it’s present in the 3.12.x kernel;
http://www.spinics.net/lists/linux-btrfs/msg56951.html

Are you in a position to raise an SR? If so, think that may be the best course of action if the current filesystem is not full or needs checking (See thread I linked to).

jimsmithson · July 13, 2016, 7:33pm

[QUOTE=malcolmlewis;33449]Hi
I also see this recent post, so wonder if it’s present in the 3.12.x kernel;
http://www.spinics.net/lists/linux-btrfs/msg56951.html

Are you in a position to raise an SR? If so, think that may be the best course of action if the current filesystem is not full or needs checking (See thread I linked to).[/QUOTE]
No. I have only patch update support.

There is also chatter on other boards about this issue being a kernel bug. I donÂt see that it has been resolved.
http://www.spinics.net/lists/linux-btrfs/msg55329.html
https://lime-technology.com/forum/index.php?topic=47744.0

Thanks you for your other reply. Let me try to answer your questions.

[CODE]SR0005IB02:~ # btrfs fi usage /
Overall:
Device size: 298.54GiB
Device allocated: 115.16GiB
Device unallocated: 183.37GiB
Device missing: 0.00B
Used: 31.72GiB
Free (estimated): 266.00GiB (min: 174.31GiB)
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 304.00MiB (used: 0.00B)

Data,single: Size:112.60GiB, Used:29.97GiB
/dev/sdb2 112.60GiB

Metadata,DUP: Size:1.25GiB, Used:895.20MiB
/dev/sdb2 2.50GiB

System,DUP: Size:32.00MiB, Used:16.00KiB
/dev/sdb2 64.00MiB

I removed the postgres, mysql and mariadb subvolumes from /etc/fstab ( not sure if just commenting out of /etc/fstab is correct)
/home was also made a part of the root file system instead of being separate.
I recreated /dd1 filesystem after giving it a separate VMware Paravirtual SCSI controller. Root is using a LSI Logic SCSI Controller.

[CODE]SR0005IB02:~ # diff /etc/fstab.orig /etc/fstab
12,13c12,13
< UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0
< UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0

#UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0
#UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0
15c15
< UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/pgsql btrfs subvol=@/var/lib/pgsql 0 0

#UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/pgsql btrfs subvol=@/var/lib/pgsql 0 0
21,22c21
< UUID=8c8fd1d8-a0c8-408a-8193-62a9b95fc8f9 /home xfs defaults 1 2
< UUID=a908851f-a203-415b-8df8-438de37eb1b6 /dd1 xfs defaults 1 2

UUID=3163e6e6-86cd-4304-8750-5a9c13f534eb /dd1 xfs defaults 1 2[/CODE]

I booted from the Rescue Disk and tried a fsck. There were no errors found.

malcolmlewis · July 13, 2016, 9:19pm

Hi
Well the database ones should be excluded from snapshots, hence they are added as subvolumes, no cow etc.

So, maybe I should clarify with respect to services, is this system running apache, database etc (@22 cpus and 48GB of ram it’s going to be busy?)? Anything like that should be running on I’m guessing, a xfs partition rather than btrfs.

I normally run home on / as well as on a default install is will be added as a subvolume for exclusion.

If it needs to be like it is currently configured, let me see if can as my SUSE contacts on what could be done.

jimsmithson · July 14, 2016, 12:07am

[QUOTE=malcolmlewis;33474]Hi
Well the database ones should be excluded from snapshots, hence they are added as subvolumes, no cow etc.

So, maybe I should clarify with respect to services, is this system running apache, database etc (@22 cpus and 48GB of ram it’s going to be busy?)? Anything like that should be running on I’m guessing, a xfs partition rather than btrfs.

I normally run home on / as well as on a default install is will be added as a subvolume for exclusion.

If it needs to be like it is currently configured, let me see if can as my SUSE contacts on what could be done.[/QUOTE]

The postgres DB had been moved to the XFS file system as that is where most of the I/O will be done.
We are running apache2 and logging is done to root (btrfs) file system.
But right now, the system is pre-production and not under any load at all.
Good news is that I have not seen any dmesg WARNINGs since after the reboot from the fsck 5 hours ago.
Thanks again for your help!

malcolmlewis · July 14, 2016, 1:48am

On Wed 13 Jul 2016 09:14:01 PM CDT, jimsmithson wrote:

malcolmlewis;33474 Wrote:[color=blue]

Hi
Well the database ones should be excluded from snapshots, hence they
are added as subvolumes, no cow etc.

So, maybe I should clarify with respect to services, is this system
running apache, database etc (@22 cpus and 48GB of ram it’s going to
be busy?)? Anything like that should be running on I’m guessing, a xfs
partition rather than btrfs.

I normally run home on / as well as on a default install is will be
added as a subvolume for exclusion.

If it needs to be like it is currently configured, let me see if can
as my SUSE contacts on what could be done.[/color]

The postgres DB had been moved to the XFS file system as that is where
most of the I/O will be done.
We are running apache2 and logging is done to root (btrfs) file system.
But right now, the system is pre-production and not under any load at
all.
Good news is that I have not seen any dmesg WARNINGs since after the
reboot from the fsck 5 hours ago.
Thanks again for your help!

Hi
So maybe that cleaned things up?

Maybe (I guess it would create more work) consider lvm? I tend to
keep / to 40-50GB and then configure snapper (/etc/snapper/configs/root)
to keep snapshots at a reasonable level.

For example here is my SuMA vm;

NAME                     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                        8:0    0   350G  0 disk
ââsda1                     8:1    0   101M  0 part /boot
ââsda2                     8:2    0 349.9G  0 part
ââSumaVG-Suma--LVSwap  254:0    0     4G  0 lvm  [SWAP]
ââSumaVG-Suma--LVRoot  254:1    0    50G  0 lvm  /
ââSumaVG-Suma--LVPgsql 254:2    0    50G  0 lvm  /var/lib/pgsql
ââSumaVG-Suma--LVSwalk 254:3    0 245.9G  0 lvm  /var/spacewalk

I’ve never run btrfs on such a large / seems like a lot of overhead?

Anyway, lets see how it goes and if you run into more (or the
same) issues, just pop back here and can see what we can do

–
Cheers Malcolm Â°Â¿Â° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
openSUSE Leap 42.1|GNOME 3.16.2|4.1.26-21-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks!

jimsmithson · July 14, 2016, 3:43pm

[QUOTE=malcolmlewis;33480]Hi
So maybe that cleaned things up?

Maybe (I guess it would create more work) consider lvm? I tend to
keep / to 40-50GB and then configure snapper (/etc/snapper/configs/root)
to keep snapshots at a reasonable level.

For example here is my SuMA vm;

NAME                     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                        8:0    0   350G  0 disk
ââsda1                     8:1    0   101M  0 part /boot
ââsda2                     8:2    0 349.9G  0 part
ââSumaVG-Suma--LVSwap  254:0    0     4G  0 lvm  [SWAP]
ââSumaVG-Suma--LVRoot  254:1    0    50G  0 lvm  /
ââSumaVG-Suma--LVPgsql 254:2    0    50G  0 lvm  /var/lib/pgsql
ââSumaVG-Suma--LVSwalk 254:3    0 245.9G  0 lvm  /var/spacewalk

I’ve never run btrfs on such a large / seems like a lot of overhead?

Anyway, lets see how it goes and if you run into more (or the
same) issues, just pop back here and can see what we can do

–
Cheers Malcolm Â°Â¿Â° LFCS, SUSE Knowledge Partner (Linux Counter #276890)
openSUSE Leap 42.1|GNOME 3.16.2|4.1.26-21-default
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below… Thanks![/QUOTE]

I keep / on most of my smaller VMs at 30GB but there is a lot of logging and mail on this one.
Not resolved. Here is what I see this morning. The system was idle overnight.

# dmesg|grep WARN
[38431.125793] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38496.208094] WARNING: CPU: 20 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38531.251353] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38651.403847] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38651.404607] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38651.405329] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()

Is there any way to convert the btrfs root to ext3 without a reinstall?

malcolmlewis · July 14, 2016, 4:23pm

[QUOTE=jimsmithson;33482]I keep / on most of my smaller VMs at 30GB but there is a lot of logging and mail on this one.
Not resolved. Here is what I see this morning. The system was idle overnight.

# dmesg|grep WARN
[38431.125793] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38496.208094] WARNING: CPU: 20 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38531.251353] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38651.403847] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38651.404607] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()
[38651.405329] WARNING: CPU: 21 PID: 121 at ../fs/btrfs/extent-tree.c:3731 btrfs_free_reserved_data_space_noquota+0xe8/0x100 [btrfs]()

Is there any way to convert the btrfs root to ext3 without a reinstall?[/QUOTE]
Hi
No to convert filesystems would require a re-install

Is it possible to move the mail/logging to a xfs partition(s)?

I’ve asked my SUSE contacts if some additional help is available to help resolve (eg bug report).

malcolmlewis · July 18, 2016, 3:59pm

Hi
Seems it’s been reported https://bugzilla.suse.com/show_bug.cgi?id=985562 (hopefully you can see the bug?).

jimsmithson · July 19, 2016, 2:21am

[QUOTE=malcolmlewis;33518]Hi
Seems it’s been reported https://bugzilla.suse.com/show_bug.cgi?id=985562 (hopefully you can see the bug?).[/QUOTE]

Thanks but I get: You are not authorized to access bug #985562.

I’m giving up on btrfs. I’ve been working on trying to salvage the many man hours I have into this migration without having to do a complete reinstall.
I will start another thread on the issues I’ve run into so far.

ab1 · July 19, 2016, 4:31am

Before you throw away a working system, unless you have evidence to the
contrary, the open bug describes this as a cosmetic error, a warning that
should not happen, which already has a patch test file (PTF) created. If
you’d like to open an SR, you should get it refunded to you because this
is an unresolved bug. Otherwise, because it is a cosmetic bug, you should
be safe to ignore it until it is resolved. If you have concerns about the
validity of this, perhaps contact a sales representative.

BtrFS is pretty awesome, and it’s being used on thousands of systems as
the default filesystem for root (and other non-transactional-data-heavy
loads). The benefits you get from it, such as snapshots, are pretty
significant, so I would definitely not rebuild without having a problem
that at least indicates a problem with the filesystem that is not
cosmetic, but that’s just me.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…

HvdHeuvel · July 19, 2016, 2:47pm

Hi jimsmithson

It is unfortunate you ran into this problem, but as was mentioned before, the messages is what it is, just a message. Kernel developer on the bug have confirmed there is not a single indication of any file system corruption, and a proposed solution is currently awaiting testing/confirmation from another customer that reported the same. Once they can confirm the solution, this should become available through an online update fairly soon thereafter.
So, when you are in the ability to log an SR, I could also also ask the SUSE L3 team to built a PTF for you too. Hope you did not take any too drastic measures.

best regards
Hans

jimsmithson · July 21, 2016, 8:24pm

[QUOTE=ab;33538]Before you throw away a working system, unless you have evidence to the
contrary, the open bug describes this as a cosmetic error, a warning that
should not happen, which already has a patch test file (PTF) created. If
you’d like to open an SR, you should get it refunded to you because this
is an unresolved bug. Otherwise, because it is a cosmetic bug, you should
be safe to ignore it until it is resolved. If you have concerns about the
validity of this, perhaps contact a sales representative.

BtrFS is pretty awesome, and it’s being used on thousands of systems as
the default filesystem for root (and other non-transactional-data-heavy
loads). The benefits you get from it, such as snapshots, are pretty
significant, so I would definitely not rebuild without having a problem
that at least indicates a problem with the filesystem that is not
cosmetic, but that’s just me.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…[/QUOTE]

Really? Putting WARNING into the logs for a “cosmetic problem” is like yelling FIRE in a movie theater when there is no danger.

ab1 · July 21, 2016, 9:09pm

On 07/21/2016 11:34 AM, jimsmithson wrote:[color=blue]

Really? Putting WARNING into the logs for a “cosmetic problem” is like
yelling FIRE in a movie theater when there is no danger.[/color]

I do not think you understood what I wrote. I did not state that the
warning is being written because of a cosmetic bug, but rather that
printing warning as it does IS the cosmetic bug. The warning, per the bug
at least, should not be printed, but it is (the bug), which is unfortunate
and still merely cosmetic.

Just because somebody (incorrectly) yells “Fire” in a theater does not
mean that there is one, or that the movie will be bad.

It’s your call at the end of the day; use your time however you see best.

–
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below…

jimsmithson · July 25, 2016, 5:32pm

I converted from btrfs to ext4 over the weekend. Not too hard to do. I ran some performance measurements of my application and it runs just as fast on ext4 as it did on btrfs.

Topic		Replies	Views
BTRFS errors possibly patched by LTSS kernel patch SLES Updates	2	243	April 27, 2017
Server crashes with a long BTRFS error list SLES Configure-Administer	14	297	June 22, 2019
I found error messages in the message. I would like help. Support	2	1055	September 26, 2020
plz help find out the reason behind the server reboot. SLES Hardware	1	418	May 30, 2014
BTRFS: superblock checksum mismatch SLES for Raspberry Pi	2	1398	July 19, 2017

dmesg WARNING btrfs_free_reserved_data_space_noquota

[CODE]SR0005IB02:~ # diff /etc/fstab.orig /etc/fstab 12,13c12,13 < UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0 < UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0

Related topics

[CODE]SR0005IB02:~ # diff /etc/fstab.orig /etc/fstab
12,13c12,13
< UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mariadb btrfs subvol=@/var/lib/mariadb 0 0
< UUID=775fb57e-247f-4e11-a85a-688de19d8621 /var/lib/mysql btrfs subvol=@/var/lib/mysql 0 0