Ktest

ktest.pl is a perl script in the kernel source tree that reads a simple config file to perform kernel testing. If you have a target machine that can be remotely rebooted and have its console viewed by a remote server, then you can use ktest.pl to perform automated tests. The tests that you can perform are:

* git bisecting * config bisecting * randconfig * any .config boot/test * patch checking

The tests allow you to perform build, boot or tests on each kernel that is tested. Although ktest is written in perl, you do not need to know anything about perl to use ktest. ktest.pl takes a single config file for its input and will run the tests according to that config file. ktest.pl has been included in the mainline Linux kernel under the directory tools/testing/ktest. There exists a sample config that covers all the configurations that can be specified to use ktest.

= Sub Types of tests =

Each test that can be executed has three types of testing: build, boot and test. Some of these sub types may perform differently with different test types.

Build testing
Build testing will just build the kernel. This does not requier having another machine to do remote testing, although some of the options may need to tell ktest that it can reboot this non-existent machine but if all you do is build the kernel, it will never get to the point that it will require rebooting it.

Boot testing
If you can supply ktest.pl with a way to remotely boot the target, as well as read its console via stdio, then you can perform boot testing. This does currently require ssh (actually scp) connectivity to the target (not necessarily with the kernel that was built) to be able to copy the boot image to the kernel (if one is needed) and perhaps the modules that that kernel uses, again if one is needed.

After a successful build, ktest will copy the necessary files over to the target and reboot the target to that kernel. It will watch the console for a specified string (default "login:") that will tell ktest.pl that the boot succeeded. There's configurable timeouts that ktest.pl uses to determine if the boot locked up or not.

Test testing
The config option "TEST" may point to any bash string (which could also call a script) that will be executed after the build and boot has succeeded. The TEST bash line could ssh to the target to execute some test (perhaps even LTP). The one requirement is that on success the bash command returns 0 to let ktest.pl know if the test succeeded or not.

= Test types =

Simple type
The simple test type usually just takes a standard config and builds, boots and tests depending on what the sub type states.

Git Bisect type
The git bisect type will perform a git bisect on the source tree (which ktest assumes is a git repository). Depending on the test type, it will test build, boot or some other test. If a boot or test fails on build (or test fails on boot) it will execute git bisect skip as ktest.pl assumes the current git commit has corrupted the environment, and the git bisect skip should hopefully take the git repo to something that can successfully get past the problem areas.

The bisect test type also supports checking the initial good and bad commits before it wastes time doing a bisect in those cases you are not sure the good and bad points or perhaps the test you set up is correct. When BISECT_CHECK is set, the good commit is checked out and ktest.pl makes sure that the test succeeds, and then the bad commit is checked out to make sure it fails the test.

In those strange cases that something just started working when it was broken before, you can enable BISECT_REVERSE, where a ktest.pl will execute git bisect good on failure, and git bisect bad when the test succeeds. This will find that commit that fixed the bug.

Patch check type
For those maintainers that take patches from others, or simply want to run that last set of patches that have been committed into git through some verification tests before sending the results upstream, the patch check can help. It will checkout the git repo of one commit and incrementally checkout each commit leading up to a final commit to check and for each commit it will execute the PATCHCHECK_TYPE subtype of test. This test modifies what the build does (for all test subtypes). If warning appears in a file that the commit touches, then it will fail the build. As there is no good reason that new code should add new warnings to the kernel.

randconfig type
The randconfig test type will execute randconfig to create the config file for the build. You can also specify a MIN_CONFIG that will hold the default configs that should be set for the config (both enabled and disabled configs, so make sure you remove anything that you do not want to force in that file). This is a test that is useful to use the ITERATE option of ktest.pl to iterate a single test type several times.

config bisect
On those off occasions that you find that you have two configs for the same version of the kernel, yet one works and the other does not and you have no idea which config is the problem config. ktest.pl supplies a config bisect that will try various configs doing a semi bisect to find the culprit. It is a semi config as the config dependencies prevents a true bisect to be done.

make_min_config
When a user of your code has a bug it is common to ask the user for their config file. The problem is that their config file may not boot on your machine. The solution is to add a minimum configuration that you know is needed to boot your box. But if your minimum configuration is not the true minimum, it is likely that one of your configurations might override the config that causes the bug. Having a true minimum config limits this collision, and if the minimum still hides the bug, the bug is a machine specific bug and not a configuration issue.

Using make_min_config, you can bring down a config that was made by localmodconfig from 1200 configurations to 148 configurations. Even better, ktest.pl will keep track of the individiual configurations that were needed otherwise the box did not boot. That is, just the ones that failed to boot and not set because another config was set. This config ended up being 13 configs, and just adding these configs to an allnoconfig allowed it to boot.