disk out of space errors causing lots of tests to fail

[lrumancik]

Upon updating my image to xfstests-amd64-202410151341, I am experiencing a huge uptick in failures due to disk out-of-space errors:

cat 525.out.bad
QA output created by 525
Silence is golden
/root/xfstests/tests/xfs/525: line 33: /xt-vdd/a: No space left on device

The other failing tests are also experiencing this issue.

Before I was on a custom build image, so I am not sure when this started occuring with regards to the varying official image releases.

xfs/v4: 3459 tests, 650 failures, 1502 skipped, 17352 seconds
Failures: generic/054 generic/055 generic/260 generic/363 generic/365
generic/743 generic/754 xfs/014 xfs/049 xfs/073 xfs/077 xfs/078
xfs/080 xfs/084 xfs/107 xfs/119 xfs/148 xfs/149 xfs/164 xfs/165
xfs/206 xfs/216 xfs/217 xfs/222 xfs/250 xfs/252 xfs/253 xfs/259
xfs/260 xfs/263 xfs/284 xfs/289 xfs/290 xfs/292 xfs/294 xfs/299
xfs/304 xfs/305 xfs/512 xfs/513 xfs/514 xfs/515 xfs/522 xfs/523
xfs/524 xfs/525 xfs/526 xfs/557 xfs/602 xfs/605 xfs/606 xfs/613
xfs/614 xfs/629 xfs/630 xfs/631 xfs/632 xfs/789 xfs/803 xfs/804
xfs/806
Flaky: generic/001: 75% (3/4) generic/002: 50% (2/4)
generic/004: 75% (3/4) generic/005: 75% (3/4)
generic/006: 75% (3/4) generic/007: 75% (3/4)
generic/008: 75% (3/4) generic/009: 75% (3/4)
generic/011: 75% (3/4) generic/012: 75% (3/4)
generic/013: 75% (3/4) generic/014: 75% (3/4)
generic/016: 75% (3/4) generic/020: 75% (3/4)
generic/021: 75% (3/4) generic/022: 75% (3/4)
generic/023: 75% (3/4) generic/024: 75% (3/4)
generic/025: 75% (3/4) generic/026: 75% (3/4)
generic/028: 75% (3/4) generic/035: 75% (3/4)
generic/036: 50% (2/4) generic/037: 25% (1/4)
generic/053: 25% (1/4) generic/058: 75% (3/4)
generic/060: 75% (3/4) generic/061: 75% (3/4)
generic/063: 75% (3/4) generic/066: 25% (1/4)
generic/067: 75% (3/4) generic/070: 75% (3/4)
generic/072: 75% (3/4) generic/074: 75% (3/4)
generic/075: 75% (3/4) generic/077: 25% (1/4)
generic/078: 75% (3/4) generic/080: 75% (3/4)
generic/082: 75% (3/4) generic/086: 75% (3/4)
generic/087: 50% (2/4) generic/088: 50% (2/4)
generic/089: 50% (2/4) generic/091: 50% (2/4)
generic/092: 75% (3/4) generic/093: 75% (3/4)
generic/097: 75% (3/4) generic/099: 75% (3/4)
generic/100: 67% (2/3) generic/112: 67% (2/3)
generic/113: 67% (2/3) generic/114: 67% (2/3)
generic/123: 67% (2/3) generic/124: 67% (2/3)
generic/125: 67% (2/3) generic/126: 67% (2/3)
generic/127: 67% (2/3) generic/131: 67% (2/3)
generic/133: 67% (2/3) generic/184: 67% (2/3)
generic/192: 67% (2/3) generic/193: 67% (2/3)
generic/198: 67% (2/3) generic/207: 67% (2/3)
generic/208: 67% (2/3) generic/209: 67% (2/3)
generic/210: 67% (2/3) generic/211: 67% (2/3)
generic/212: 67% (2/3) generic/214: 67% (2/3)
generic/215: 67% (2/3) generic/221: 67% (2/3)

[lrumancik]

The overlay failures look a bit different:

QA output created by 009
rm: cannot remove '/xt-vdc/ovl-upper/symlink.288': Structure needs cleaning
rm: cannot remove '/xt-vdc/ovl-upper/symlink.320': Structure needs cleaning
Silence is golden
QA output created by 010
rm: cannot remove '/xt-vdc/ovl-upper/symlink.288': Structure needs cleaning
rm: cannot remove '/xt-vdc/ovl-upper/symlink.320': Structure needs cleaning
mount: /xt-vdc/ovl-mnt: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.
Silence is golden
QA output created by 011
rm: cannot remove '/xt-vdc/ovl-upper/symlink.288': Structure needs cleaning
rm: cannot remove '/xt-vdc/ovl-upper/symlink.320': Structure needs cleaning
mount: /xt-vdc/ovl-mnt: mount point does not exist.
       dmesg(1) may have more information after failed mount system call.
getfattr: /xt-vdc/ovl-mnt/testdir: No such file or directory
Silence is golden
QA output created by 012
rm: cannot remove '/xt-vdc/ovl-upper/symlink.288': Structure needs cleaning
rm: cannot remove '/xt-vdc/ovl-upper/symlink.320': Structure needs cleaning

[tytso]

I'm not seeing this failure, so I decided to take a closer look.

I just tested using a test appliance built using the tip of xfstests-bld and it's not failing for me. Looking at line 33 of xfstests/tests/525, what I see at line 33 is "rm -rf $def_cfgfile", but I'm guessing you have a slightly different version of xfstests, and what is at line 33 corresponds to "cat > $def_cfgfile << ENDL", where def_cfg is $TEST_DIR/a.

So what I am guessing is that some previous test is leaving enough gunk left over in the test file system. So what I would suggest adding is before line 33, something like this:

df >> $seqres.full
ls -sFR $TEST_DIR >> $seqres.full

... and then see what shows up in /results/xfs/results-4k/xfs/525.full

This might give us a hint as to what test might be causing the problem?

Also, which kernel version were you testing, and what set of tests were you running? Does this show up easily with "gce-xfstests -c xfs/4k -g auto"?