Skip to content

Latest commit

 

History

History
486 lines (363 loc) · 22.7 KB

gce-xfstests.md

File metadata and controls

486 lines (363 loc) · 22.7 KB
Raw

Running xfstests on Google Compute Engine

Before running on Google Compute Engine for the first time, you will need to do a number of setup tasks:

  • Get a Google Compute Engine account
  • Install the gce-xfstests script
  • Install the software needed by gce-xfstests
  • Get access to the File system test appliance
  • Run "gce-xfstests setup"

Get a Google Compute Engine account

If you don't have a GCE account, you can go to https://cloud.google.com and sign up for a free trial. This will get you $300 worth of credit which you can use over a 12-month period (as of this writing). Given that a full test for ext4 costs around $1.50, and a smoke test costs pennies, that should be enough for plenty of testing. :-)

Setting up a GCE project

Although you can use a pre-existing project, it is a good idea to set up a new GCE project for gce-xfstests. To set up a GCE project, go to the GCE Projects page, pick a project name and then click on the blue "Create Project" button at the top of the page. The GCE projects namespace is a global one, so you will need to pick something like unique, such as "yourName-xfstests" or "yourUserName-xfstests". After you create it, you will need to enable billing for your newly created project.

Next, go to GCE Instances page in order to enable the GCE API for your project. You can optionally try creating a VM instances via the web interface, or follow the quickstart tutorial if you like, although this won't be necessary, since the gce-xfstests command line interface will take care of starting and stopping instances for you automatically. If you do start up some test instances yourself, please make a point of going to the GCE Instances page when you are done to make sure you have shut down any test VM's so that you don't have unexpected charges to your account.

Setting up a GCS bucket

The gce-xfstests system needs a Google Cloud Storage (GCS) bucket to send kernel images to be tested and to save the results from the test runs. If you are already using GCS you can use a pre-existing bucket, but it is strongly advisable that you use a dedicated bucket for this purpose. Detailed instructions for creating a new bucket can be found in the GCS Quickstart.

Setting up a Sendgrid account

Since virtual machines in GCE aren't allowed to send connect to the normal outgoing mail ports (in order prevent abuse by spammers), in order to send e-mail we have to use a cloud mail service. Using a cloud mail service is optional --- you can wait for the test to complete and then use the gce-xfstests ls-results and get-results command to fetch the test results --- but it's very handy to have the test reports show up in your inbox once they are finished.

The gce-xfstests system uses sendgrid, so if you would like to get e-mailed reports, you will need to sign up for a free Sendgrid account. The Sendgrid free account provides 100 e-mails per day and it's unlikely that you will be running more than 100 test runs per day.

To start, visit the Sendgrid website and click on the "Start for Free" button. It may take a day or two for sendgrid to decide you are a not a robot spammer, so please start the process right away while you familiarize yourself with the rest of gce-xfstests.

NOTE: Starting from April 6 2020, Sendgrid requires new users to verify their Sender Identities before using the service. You could check this guide for more details.

Once you have set up a Sendgrid account, get a new API key by going to the url https://app.sendgrid.com/settings/api_keys and click on the blue "Create API Key" button. Pick a name such as "gce-xfstests" and enter it into "API Key Name". Then click on the Mail Send's "Full Access" bubble and then click on the blue "Create & View" button. Copy the API key that was generated and use it to set the GCE_SG_API configuration variable in gce-xfstests's config file.

Then go to Sendgrid's Tracking Settings web page at https://app.sendgrid.com/settings/tracking and make sure all of the Tracking settings are set to inactive. If one of the trackers, such as Click Tracking, are enabled, click on the down arrow in the Options column, and then click on "Off" to disable the tracker. This is important, because the reports are sent as plain ASCII text, and the way sendgrid tries to translate the text report into HTML results in something that looks really mangled if you are using a mail client that tries to display the HTML version of an e-mail message.

Get and install the gce-xfstests script

The gce-xfstests and its associated helper scripts are part of the xfstests-bld git repository. If you have not fetched it, you will need to do so now:

    git clone https://git.kernel.org/pub/scm/fs/ext2/xfstests-bld.git fstests

The gce-xfstests driver script needs to be customized so it can find the "real" gce-xfstests script, which is located in fstests/run-fstests/gce-xfstests. To do this:

    cd fstests
    make

And then copy this script to a directory in your PATH. For example, if ~/bin is in your shell's search path:

    cp gce-xfstests ~/bin/gce-xfstests

Install software required by gce-xfstests

  1. Install the Google Cloud SDK. Instructions for can be found at: https://cloud.google.com/sdk/docs/quickstart-linux

  2. Install the following packages (debian package names used):

    % sudo apt-get install jq xz-utils dnsutils python3-crcmod

Configure gce-xfstests

You will need to set up the following configuration parameters in ~/.config/gce-xfstests:

  • GS_BUCKET
    • The name of the Google Storage bucket which should be used by gce-xfstests. Your Google Compute Engine account must have access to read and write files in this bucket.
  • GCE_PROJECT
    • The name of Google Compute Engine project which should be used to create and run GCE instances and disks.
  • GCE_ZONE
    • The name of the Google Compute Engine zone which should be used by xfstests.
  • GCE_KERNEL
    • The pathname to kernel that should be used for gce-xfstests by default.

If you have a sendgrid account, you can set the following configuration parameters in order to have reports e-mailed to you:

  • GCE_SG_API
    • The Sendgrid API used to send the test report
  • GCE_REPORT_EMAIL
    • The email addressed for which test results should be sent.
  • GCE_REPORT_SENDER
    • The email used as the sender for the test report. This defaults to the GCE_REPORT_EMAIL configuration parameter. If the domain used by GCE_REPORT_EMAIL has restrictive SPF settings, and you don't have control over the domain used by GCE_REPORT_EMAIL, you may need to choose a different sender address.

Other optional parameters include:

  • GCE_FIREWALL_RULES
    • List of firewall rules to add to the GCP project if not already present. By default a rule "allow-http" is created which makes the gce-xfstests web interface accessible to anyone over the Internet. It may be useful to override this if you want to implement more restrictive firewall rules or disable access to the web interface entirely. Note that existing firewall rules associated with the GCP project will not be removed, and by default there is a default-allow-ssh rule which allows SSH access.
  • GCE_USER
    • Optional identifier for all test instance names. By default, if this is unset, test instances will be named "xfstests-USER-DATECODE". (USER will be the evaluation of $USER) This option can be set to the empty string, i.e. GCE_USER= or GCE_USER="" to disable having "$USER-" in instance names, and simply have them named "xfstests-DATECODE". This doesn't affect the instances managed by LTM server.
  • GCE_UPLOAD_SUMMARY
    • If set to a non-empty string value, test appliances will upload a summary.*.txt file in addition to the regular results tarball. This summary file will be a copy of the summary file normally found at the root directory of the results tarball.
  • BUCKET_SUBDIR
    • Optional parameter to specify the subdirectory to be used to upload results instead of the default BUCKET_ROOT/results/ directory. e.g. BUCKET_SUBDIR="4.13-rc5" or BUCKET_SUBDIR="my_subdir"
  • GCE_MIN_SCR_SIZE
    • Optional value to use as a minimum scratch disk size. Must be a number between 0 and 250. If specified, the scratch disk created by any test appliances will have this value as a minimum size in GB. This is useful for particularly IO-bound tests (e.g. generic/027), which will run faster with a larger disk size. This is because GCE assigns IOPS per GB, so a larger scratch disk will have more IOPS available to it.
  • GCE_LTM_KEEP_DEAD_VM
    • Optional string. If specified as a non-empty string, the LTM instance will preserve VMs that are presumed to have wedged/timed out rather than deleting the VM.
  • GIT_REPO
    • Optional git repo url. If specified, all kernel building requests will use this repo be default. It can be overridden by command line argument --repo.

An example ~/.config/gce-xfstests might look like this:

    GS_BUCKET=tytso-xfstests
    GCE_PROJECT=tytso-linux
    GCE_ZONE=us-central1-c
    GCE_KERNEL=/build/ext4-64/arch/x86/boot/bzImage

Add yourself to the gce-xfstests group

By default gce-xfstests uses the pre-built image which is made available via the xfstests-cloud project. In order gain access to this image, you will need add the google account used for your GCE project to the gce-xfstests Google Group. To do this, go the gce-xfstests Google Groups page and click on the blue "Join group" button. This group is an announcement-only so it will not have a large number of posts.

The pre-built image will receive periodic updates, and while we try to keep backwards compatibility, it may be that some new images may require updating your local copy of the xfstests-bld git repository to get the latest version of the gce-xfstests script.

If you don't want to use the pre-built image, please see the section Creating a new GCE test appliance image below for instructions to build your own image from source.

Run "gce-xfstests setup"

The command gce-xfstests setup will set some GCE settings for gce-xfstests, but more importantly, it will sanity check the configuration parameters for gce-xfstests. If there are any problems or potential problems, it will report them so you can fix them.

Running gce-xfstests

Once you have completed all of the set up tasks listed above, you can now start using gce-xfstests. The GCE_KERNEL configuration parameter should be set to the location of your build directory or the kernel that you want to boot. So for example, you could set it to /build/ext4, or /build/ext4/arch/x86/boot/bzImage. If gce-xfstests is run from the top-level of a kernel build or source tree where there is a built kernel, gce-xfstests will use it. Otherwise, it will use the kernel specified by the GCE_KERNEL variable.

The design of gce-xfstests allows you to to apply a patch to your kernel, build it, and then run gce-xfstests smoke, which will test the kernel without needing to install it first; gce-xfstests will upload it to Google Cloud Storage, and then the test appliance VM will kexec into that kernel. This speeds up your edit, compile, debug cycle, so you can improve your development velocity.

Running gce-xfstests is much like kvm-xfstests:

gce-xfstests [-c <cfg>] [-g <group>]|[<tests>] ...

As with kvm-xfstests, you can also use gce-xfstests smoke and gce-xfstests full, to run the a quick smoke test and the full file system regression test. The command gce-xfstests help will provide a quick summary of how tests can be run.

The gce-xfstests command also has a few other commands, some of which are described below:

gce-xfstests ssh INSTANCE

Remotely login as root to a test instances. This is a convenience shorthand for: gcloud compute --project GCE_PROJECT ssh root@INSTANCE --zone GCE_ZONE.

gce-xfstests console INSTANCE

Fetch the serial console from a test instance. This is a convenience shorthand for: gcloud compute --project GCE_PROJECT get-serial-port-output INSTANCE --zone GCE_ZONE.

gce-xfstests console [--port N] INSTANCE

Connect to serial port N on a test instance. Port 1 is the serial console; the magic sysrq key can be accessed via the Enter key, followed by the tilde ('~') key, followed by the 'B' key. Ports 2, 3, and 4 will connect to a serial port with a shell running on it. In the future serial port #4 may be repurposed to allow a remote gdb connection to kgdb, if the kernel under test is built with with kgdb support.

gce-xfstests ls-instances [-l ]

List the current test instances. With the -l option, it will list the current status of each instance.

This command can be abbreviated as gce-xfstests ls.

The ls-gce option is a convenience command for gcloud compute --project GCE_PROJECT instances list --regexp ^xfstests.*

gce-xfstests rm-instances INSTANCE

Shut down the instance. If test kernel has hung, it may be useful to use gce-xfstests console to fetch the console, and then use gce-xfstests rm and examine the results disk before deleting it.

This command can be abbreviated as gce-xfstests rm

gce-xfstests abort-instances INSTANCE

this command functions much as the gce-xfstests rm-instances command, except it makes sure the results disk will be deleted.

This command can be abbreviated as gce-xfstests abort

NOTE: This command will delete the cache pd for KCS.

gce-xfstests ls-disks

List the GCE disks. This is a convenience command for gcloud compute --project "$GCE_PROJECT" disks list --regexp ^xfstests.*

ALIAS: gce-xfstests ls-disk

gce-xfstests rm-disks DISK

Delete a specified GCE disk. This is a convenience command for gcloud compute --project "$GCE_PROJECT" disks delete DISK

ALIAS: gce-xfstests rm-disk

gce-xfstests ls-results|ls-gce

List the available results tarballs stored in the Google Cloud Storage bucket. This is a convenience command for gsutil ls gs://GS_BUCKET/results.* (ls-results) or gsutil ls gs://GS_BUCKET (ls-gcs).

gce-xfstests rm-results RESULT_FILE

Delete a specified result tarball. This is a convenience command for gsutil rm gs://GS_BUCKET/RESULT_FILE.

gce-xfstests get-results [--unpack | --summary | --failures ] RESULT_FILE

Fetch the run-tests.log file from the RESULT_FILE stored in the Google Cloud Storage bucket. The --summary or --failures option will cause the log file to be piped into the get-results script to summarize the log file using the -s or -F option, respectively. The --failures or -F option results in a more succinct summary than the --summary or -s option.

The --unpack option will cause the complete results directory to be unpacked into a directory in /tmp instead.

Managing multiple tests with LTM server

Gce-xfstests provides a service called Lightweight Test Manager (LTM) to launch and manage multiple tests automatically. It can parallelize tests, monitor test progress, aggregate test results and do more. To launch the LTM server, use the following command:

    gce-xfstests launch-ltm

It launches a VM named "xfstests-ltm" in your GCE project. Then you can run tests through LTM server by adding ltm to the regular command:

  	gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ...

The LTM server attempts to split your config into smaller tests, one cfg for each. For example, it splits the config full into -c ext4/data_journal full, -c ext4/encrypt full, -c ext4/ext3 full... For more details about how LTM splits the config, check parser.go.

Then the LTM server will launch these tests in parallel and monitor the status of each test VM. If a single test makes no progress in an hour, it will kill that test VM early. After all tests finish, the LTM server aggregates the test results into one tarball and upload it to the GCS bucket.

NOTE: Some command line arguments takes no affect with LTM., including --instance-name, --gce-zone, --hooks and more.

When LTM server is running, the following command queries for LTM running status, and prints a json response with active sharders, watchers and bisectors info.

  	gce-xfstests ltm-info

Building kernels remotely with KCS server

Gce-xfstests also provides a way to build kernel images remotely on the Kernel Compile Server (KCS). To build a kernel and run tests on it, you can specify the git repo for the kernel source code with --repo and a single revision (SHA-1 hash, tag name or branch name) with --commit:

    gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] --commit <rev>

The --repo <url> specifies the Git repository that should used to find the specified commit (or branch, or tag). If --repo is not specified, the repository that will be used is the value of GIT_REPO in ~/.config/gce-xfstests, or Linus Torvalds' linux top-level git repository if GIT_REPO is not configured in the user's config file.

The LTM server will launch the KCS server in a separate VM to build the kernel and use the built image to run xfstests. KCS builds your kernel using a kernel config installed via gce-xfstests install-kconfig feature. Options can be passed to install-kconfig via --kconfig-opts <opts>. Alternatively, you can specify a custom kernel config using --config <filepath>.

Under rare conditions you might want to build the kernel image only. First make sure the KCS server is running with the command:

    gce-xfstests launch-kcs

Then use the following command to build the kernel:

    gce-xfstests kcs --repo <url> --commit <rev>

After the compilation finishes, you will find the built kernel under GCS path BUCKET_ROOT/kernels/.

The KCS server uses a cache pd to store local repositories, cached compilations and build logs. The cache pd is auto-generated when launching KCS for the first time and gets reused later. While the LTM server keeps running unless you kill it explicitly, the KCS server shuts down itself automatically after being idle for more than one hour.

Running gce-xfstests test spinner

With LTM and KCS, gce-xfstests supports a test spinner that watches a git repo and run tests on newly pushed code automatically. You can initiate a new watcher with command:

    gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] --watch <branch>

LTM server will check for new commit on branch periodically, build kernels and launch tests when new code are pushed to this branch. If you've set up the email service, a new email is sent to you every time a new round of tests finishes.

You can have multiple watchers running at the same time, even on the same branch. To terminate a watcher, find the watcher's testID with command gce-xfstests ltm-info and run command:

    gce-xfstests ltm --unwatch <testID>

Searching for buggy commits with git bisect

Sometimes you might apply some patches only to find some xfstests tests fails, and you want to find the commit that introduces the bug. Gce-xfstests provides an automated way to perform git-bisect on a git tree. To use git bisect, you need to provide the tests that you expect to fail, a bad_rev commit that is known to have the bug, and at least one good_rev commit that is known to be before the bug was introduced:

    gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] \
    --bisect-bad <bad_rev> --bisect-good <good_rev> [--bisect-good <good_rev1> ...]

The KCS server will use a binary search approach to find which commit between two endpoints that introduced a bug causing any of the tests to fail. This process involves multiple rounds of building kernels and running xfstests. If the server encounters a kernel build error or any test error (e.g. crashed test VMs), the corresponding commit is skipped.

After git bisect finishes, you will receive an email containing the bisect log report. Test results are also uploaded to the GCS bucket.

Creating a new GCE test appliance image

By default gce-xfstests uses the prebuilt image which is made available via the xfstests-cloud project. However, if you want to build your own image, you must first build the xfstests tarball as described in the instructions for building xfstests. Then run the command gce-xfstests create-image. This will create a new GCE image with a name such as "xfstests-201608132226" where 201608132226 indicates when the image was created (in this case, August 13, 2016 at 22:26).

As with kvm-xfstests, if you want to include any additional Debian packages, place them in the directory test-appliance/debs. See the documentation for building kvm-xfstests appliances for more information. Note that gce-xfstests requires packages for the amd64 architecture; packages for other architectures will not be installed.

This image will be created as part of an image family called xfstests. By default, when you start a test using gce-xfstests, the most recently created image in the xfstests image family will be used.

In order to use the xfstests image family created in your GCE project (instead of the xfstests-cloud project), add the following line to your ~/.config/gce-xfstests configuration file after the GCE_PROJECT variable is defined:

    GCE_IMAGE_PROJECT="$GCE_PROJECT"

Normally, the most recently created image in the xfstests image family will be used by default. You can however override this and use a specific image by setting ROOT_FS in your ~/.config/gce-xfstests configuration file, or by using the -I option to gce-xfstests. (For example: ROOT_FS=xfstests-201608130052, or "gce-xfstests -I xfstests-201608130052 smoke".) You can also use the --image-project command line option to override the GCE_IMAGE_PROJECT setting in your configuration file.

Visualizing test results using gce-xfstests dashboard

Test results uploaded to GCS bucket can be visualized in a web based GUI. To setup the web dashboard, ensure that you have run gce-xfstests setup command. If you are upgrading from older gce-xfstests version, please re-run gce-xfstests setup command to setup the dashboard.

Run gce-xfstests launch-dashboard command to launch the test dashboard on Google cloud run. This may take a while. Once this command completes, it will print a URL where the dashboard is live.

You can optionally pass --local flag to above command to run dashboard locally.