[PATCH 2/2] xfstests-bld: test-apliance add castomization options

[Dmitry Monakhov]

From: root <***@deb7build.(none)>

Add a possibility to customize variables similar to kvm-xfstests/config.custom
One can add local mirrors to speedup image generation
---
kvm-xfstests/test-appliance/gen-root-image | 3 +++
kvm-xfstests/test-appliance/gen-test-image | 3 +++
2 files changed, 6 insertions(+)

diff --git a/kvm-xfstests/test-appliance/gen-root-image b/kvm-xfstests/test-appliance/gen-root-image
index 977c4c2..516b7b5 100644
--- a/kvm-xfstests/test-appliance/gen-root-image
+++ b/kvm-xfstests/test-appliance/gen-root-image
@@ -12,6 +12,9 @@ RAW_ROOT_FS=$DIR/root_fs.raw
ROOT_FS=$DIR/root_fs.img
COMPAT="-o compat=0.10"

+if test -f $DIR/config.custom ; then
+ . $DIR/config.custom
+fi

if test $(df -k /tmp | tail -1 | awk '{print $4}') -gt 350000
then
diff --git a/kvm-xfstests/test-appliance/gen-test-image b/kvm-xfstests/test-appliance/gen-test-image
index ab02592..fde1798 100755
--- a/kvm-xfstests/test-appliance/gen-test-image
+++ b/kvm-xfstests/test-appliance/gen-test-image
@@ -8,6 +8,9 @@ TEST_FS=$DIR/test_fs.img
TESTDIR=$DIR/testdir
COMPAT="-o compat=0.10"

+if test -f $DIR/config.custom ; then
+ . $DIR/config.custom
+fi

if test $(df -k /tmp | tail -1 | awk '{print $4}') -gt 350000
then

[Theodore Ts'o]

This patch is really huge, and unfortunately I've been doing some work
in parallel that hadn't yet gotten pushed to git.kernel.org, so your
patches don't apply cleanly any more.

In particular, I've already added support for config.custom (ala your
2/2 patch) to gen-image. This can be used in conjuction to a
convenience script which takes the cached packages in
var.cache.apt.archives and var.lib.apt.lists and create a local
mirror.

So now after running:

# ./gen-image --mirror http://snapshot.debian.org/archive/debian/20141013T184415Z
% ./create-local-mirror snapshot.debian.org_archive_debian_20141013T184415Z

you can now run the command:

# ./gen-image --mirror file:mirror

without needing access to the network.

Currently (1) and (2) are combined together to root_fs.img, but
it is not always good because (1) is usually very stable, but
(2) is not especially in case of xfstests development.
root_fs.img: contains debian distribution and generic xfstest startup scripts
test_fs.img: contains xfstests and xfstest-config (mounted in /tests)

Adding an extra image file makes life a bit harder for people who
aren't actually doing xfstests/kvm-xfstests development, since they
now need to download and install two files. We also have to worry
about potential incompatibilities in the future that might arise
between the generic xfstest starutp scripts (in root_fs.img) and the
xfstest-config files (in test_fs.img).

So what I suggest instead is that we have a new script, update-image,
which takes an existing root_fs.img, and installs the updated
xfstests, the configuration files, and startup files, leaving the
existing debian base image intact.

The advantage of this is that patch will be much smaller and easier
for me to review :-), and it also makes life easier for end-users who
aren't developing, where having a single image is much, much simpler.
Also, having a single image will also make it simpler to support
deploying an test image to cloud hosting provider. (One my side
project I've been working on is trying to make kvm-xfststs work on
Google Compute Engine.)

Will creating a new "update-image" script meet your needs of making it
easier to do xfstests development? (Actually, what I've done when I'm
doing xfstests development is I just copy xfstests.tar.gz to the file
that gets mounted under /vdb, run "./kvm-xfstests maint", and then
unpack the xfstests.tar.gz file. Or sometimes I'll just copy the
modified xfstests scripts /vdb and just install them so I don't have
to recompile all of the xfstests binaries.)

Cheers,

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Theodore Ts'o]

Post by Theodore Ts'o
So what I suggest instead is that we have a new script, update-image,
which takes an existing root_fs.img, and installs the updated
xfstests, the configuration files, and startup files, leaving the
existing debian base image intact.

Here's an enhancement to the gen-image script which adds a new
--update option.

- Ted

commit debd9734bc23108f37c6f62914cb5f9e16da20cc
Author: Theodore Ts'o <***@mit.edu>
Date: Wed Oct 22 20:42:32 2014 -0400

test-appliance: add --update option to the gen-image script

This allows gen-image to to update the xfstests.tar.gz tarball and the
xfstests driver scripts into an existing root_fs.img file, keeping the
original operating systems files unchanged. This takes less than 15
seconds, compared to about 90 seconds to do a full gen-image build
using a local disk debian mirror. It also allows a xfstests developer
to generate new versions of the root_fs.img more simply on a system
that does not have debootstrap installed or has access to a Debian
archive or mirror.

Signed-off-by: Theodore Ts'o <***@mit.edu>

diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
index 550d070..27f9d96 100755
--- a/kvm-xfstests/test-appliance/gen-image
+++ b/kvm-xfstests/test-appliance/gen-image
@@ -12,6 +12,7 @@ ROOT_FS=$DIR/root_fs.img
COMPAT="-o compat=0.10"
SAVE_RAW_ROOT=no
DO_GCE=no
+DO_UPDATE=no

if test -r config.custom ; then
. $(pwd)/config.custom
@@ -29,8 +30,11 @@ while [ "$1" != "" ]; do
DO_GCE=yes
SAVE_RAW_ROOT=yes;
;;
+ --update)
+ DO_UPDATE=yes
+ ;;
*)
- echo "usage: gen-image [--save_raw_root] [--do_gce] [--mirror MIRROR_LOC]"
+ echo "usage: gen-image [--save_raw_root] [--do_gce] [--update] [--mirror MIRROR_LOC]"
exit 1
;;
esac
@@ -42,6 +46,31 @@ then
RAW_ROOT_FS=/tmp/root_fs.raw.$$
fi

+update_xfstests()
+{
+ tar -C $ROOTDIR/root -xf ../../xfstests.tar.gz
+ rsync -avH files/* $ROOTDIR
+}
+
+finalize_rootfs()
+{
+ e2fsck -fyD $RAW_ROOT_FS
+ e2fsck -fy -E discard $RAW_ROOT_FS
+ qemu-img convert -f raw -O qcow2 $COMPAT -c $RAW_ROOT_FS $ROOT_FS
+}
+
+mkdir -p $ROOTDIR
+if test $DO_UPDATE = "yes" ; then
+ qemu-img convert -f qcow2 -O raw $ROOT_FS $RAW_ROOT_FS
+ mount -t ext4 -o loop $RAW_ROOT_FS $ROOTDIR
+ rm -rf $ROOTDIR/xfstests
+ update_xfstests
+ umount $ROOTDIR
+ rmdir $ROOTDIR
+ finalize_rootfs
+ exit 0
+fi
+
mkdir -p gce
if test $DO_GCE = "yes" ; then
sh get-gce-debs
@@ -51,7 +80,6 @@ else
variant=xfstests
fi

-mkdir -p $ROOTDIR
mkdir -p var.cache.apt.archives
mkdir -p var.lib.apt.lists
mkdir -p debs
@@ -69,9 +97,7 @@ mount --bind gce $ROOTDIR/gce
mkdir -p $ROOTDIR/imgdir
mount --bind $(dirname $RAW_ROOT_FS) $ROOTDIR/imgdir
debootstrap --variant=$variant $ARCH $SUITE $ROOTDIR $MIRROR $DIR/kvm-xfstest.script
-cp -r files/* rootdir
-echo "untaring xfstests"
-tar -C rootdir/root -xf ../../xfstests.tar.gz
+update_xfstests
for i in vda vdb vdc vdd vde vdf results
do
mkdir $ROOTDIR/$i
@@ -139,9 +165,7 @@ rmdir $ROOTDIR

if test $DO_GCE != "yes" ; then
tune2fs -O has_journal $RAW_ROOT_FS
- e2fsck -fyD $RAW_ROOT_FS
- e2fsck -fy -E discard $RAW_ROOT_FS
- qemu-img convert -f raw -O qcow2 $COMPAT -c $RAW_ROOT_FS $ROOT_FS
+ finalize_rootfs
fi

if test "$SAVE_RAW_ROOT" = "yes" ; then
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

[Dmitry Monakhov]

Post by Theodore Ts'o

Post by Theodore Ts'o
So what I suggest instead is that we have a new script, update-image,
which takes an existing root_fs.img, and installs the updated
xfstests, the configuration files, and startup files, leaving the
existing debian base image intact.

Here's an enhancement to the gen-image script which adds a new
--update option.

Yes. This one looke much than mine, than you.

Post by Theodore Ts'o
- Ted
commit debd9734bc23108f37c6f62914cb5f9e16da20cc
Date: Wed Oct 22 20:42:32 2014 -0400
test-appliance: add --update option to the gen-image script
This allows gen-image to to update the xfstests.tar.gz tarball and the
xfstests driver scripts into an existing root_fs.img file, keeping the
original operating systems files unchanged. This takes less than 15
seconds, compared to about 90 seconds to do a full gen-image build
using a local disk debian mirror. It also allows a xfstests developer
to generate new versions of the root_fs.img more simply on a system
that does not have debootstrap installed or has access to a Debian
archive or mirror.
diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
index 550d070..27f9d96 100755
--- a/kvm-xfstests/test-appliance/gen-image
+++ b/kvm-xfstests/test-appliance/gen-image
@@ -12,6 +12,7 @@ ROOT_FS=$DIR/root_fs.img
COMPAT="-o compat=0.10"
SAVE_RAW_ROOT=no
DO_GCE=no
+DO_UPDATE=no
if test -r config.custom ; then
. $(pwd)/config.custom
@@ -29,8 +30,11 @@ while [ "$1" != "" ]; do
DO_GCE=yes
SAVE_RAW_ROOT=yes;
;;
+ --update)
+ DO_UPDATE=yes
+ ;;
*)
- echo "usage: gen-image [--save_raw_root] [--do_gce] [--mirror MIRROR_LOC]"
+ echo "usage: gen-image [--save_raw_root] [--do_gce] [--update] [--mirror MIRROR_LOC]"
exit 1
;;
esac
@@ -42,6 +46,31 @@ then
RAW_ROOT_FS=/tmp/root_fs.raw.$$
fi
+update_xfstests()
+{
+ tar -C $ROOTDIR/root -xf ../../xfstests.tar.gz
+ rsync -avH files/* $ROOTDIR
+}
+
+finalize_rootfs()
+{
+ e2fsck -fyD $RAW_ROOT_FS
+ e2fsck -fy -E discard $RAW_ROOT_FS
+ qemu-img convert -f raw -O qcow2 $COMPAT -c $RAW_ROOT_FS $ROOT_FS
+}
+
+mkdir -p $ROOTDIR
+if test $DO_UPDATE = "yes" ; then
+ qemu-img convert -f qcow2 -O raw $ROOT_FS $RAW_ROOT_FS
+ mount -t ext4 -o loop $RAW_ROOT_FS $ROOTDIR
+ rm -rf $ROOTDIR/xfstests
+ update_xfstests
+ umount $ROOTDIR
+ rmdir $ROOTDIR
+ finalize_rootfs
+ exit 0
+fi
+
mkdir -p gce
if test $DO_GCE = "yes" ; then
sh get-gce-debs
@@ -51,7 +80,6 @@ else
variant=xfstests
fi
-mkdir -p $ROOTDIR
mkdir -p var.cache.apt.archives
mkdir -p var.lib.apt.lists
mkdir -p debs
@@ -69,9 +97,7 @@ mount --bind gce $ROOTDIR/gce
mkdir -p $ROOTDIR/imgdir
mount --bind $(dirname $RAW_ROOT_FS) $ROOTDIR/imgdir
debootstrap --variant=$variant $ARCH $SUITE $ROOTDIR $MIRROR $DIR/kvm-xfstest.script
-cp -r files/* rootdir
-echo "untaring xfstests"
-tar -C rootdir/root -xf ../../xfstests.tar.gz
+update_xfstests
for i in vda vdb vdc vdd vde vdf results
do
mkdir $ROOTDIR/$i
@@ -139,9 +165,7 @@ rmdir $ROOTDIR
if test $DO_GCE != "yes" ; then
tune2fs -O has_journal $RAW_ROOT_FS
- e2fsck -fyD $RAW_ROOT_FS
- e2fsck -fy -E discard $RAW_ROOT_FS
- qemu-img convert -f raw -O qcow2 $COMPAT -c $RAW_ROOT_FS $ROOT_FS
+ finalize_rootfs
fi
if test "$SAVE_RAW_ROOT" = "yes" ; then

[Theodore Ts'o]

Post by Dmitry Monakhov

Post by Theodore Ts'o

Post by Theodore Ts'o
So what I suggest instead is that we have a new script, update-image,
which takes an existing root_fs.img, and installs the updated
xfstests, the configuration files, and startup files, leaving the
existing debian base image intact.

Here's an enhancement to the gen-image script which adds a new
--update option.

Yes. This one looke much than mine, than you.

Great, I've pushed it out to git.kernel.org.

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html