[Ksummit-2012-discuss] [ATTEND] writeback and kernel testing

[Jason Wessel]

On 06/19/2012 03:59 PM, Grant Likely wrote:
> On Tue, Jun 19, 2012 at 2:55 PM, Jason Wessel
> <jason.wessel at windriver.com> wrote:
>> On 06/19/2012 03:38 PM, Grant Likely wrote:
>>> On Sat, 16 Jun 2012 07:07:25 -0700, Guenter Roeck <linux at roeck-us.net> wrote:
>>>> On Sat, Jun 16, 2012 at 08:44:18PM +0800, Fengguang Wu wrote:
>>>>
>>>> I would love to be able to expand that to runtime tests, specifically for
>>>> hardware monitoring functionality, but that is a bit more than I can afford
>>>> and/or maintain on my own.
>>> Actually, this should be doable *without* any special hardware or
>>> setup.  QEMU has models for all the major architectures.  It would be
>>> useful to have a tool in the kernel tree that builds and boots a set
>>> of kernel configs on QEMU, assuming that the user has the relevant
>>> cross compilers installed and that there is are a set of rootfs images
>>> to use for each architecture.
>>>
>>> If done right, a maintainer (or submitter) can fire off a command to
>>> sanity-test their kernel tree before pushing it out to linux-next.
>>> I've been thinking about this for a while now, but haven't had the
>>> time to hack on it.
>> Not that ktest.pl needs more plugs, but it might be something we could
>> use for this.  For any of the pieces that I submit to the kernel they
>> are all run time boot tested with kvm or qemu with pre-canned very
>> small rootfs's that were built 3 years ago.  I have a set of
>> defconfigs and test scripts which return a 1 or a 0 if things work or
>> not.
>>
>> It is particularly useful to have the ability for runtime tests that
>> produce a 1 or 0 in that you can feed it to "git bisect run" after
>> setting the initial good and bad (ktest.pl does something similar but
>> requires a some setup).  Ultimately I was thinking more along the
>> lines of a make target where you can do "make runtime-tests
>> TARGET=qemu" where you have all the plumbing you need to run some tests.
>>
>> In terms of providing a root file system it could be done in a similar
>> manner to the "firmware git" so we don't have to bloat the kernel
>> tree.
> buildroot is looking pretty nice these days.  It could also be as
> simple as a set of buildroot configs to (re)create some basic
> busybox-based rootfs images.

Sure.  I bet there are quite a few different things we could leverage to this end.

I would absolutely consider just checking in the raw binaries for both qemu and the target fs's, and then also check in the recipes that were used to build them should developers want to reproduce things, or augment the tests.  Quite a few of the tests I run don't even need a root file system because you just use the kernel's own internal tests, so long as they are enabled.

If I was going to choose a root file system, I would probably go with one of the Yocto Project's tiny rootfs.  In part I would choose the Yocto Project because it is aligned with the Linux Foundation already.  Of course I am bias because I am also a contributor to the project :-)  With in the Yocto Project, we already do some amount of automated testing on kernel / rootfs combinations including every time we hit an -rc1 we try everything out again.  In a nutshell, the Yocto Project already contains all the pieces we would need such as the simulators, rootfs, cross compilers etc...

Adding a top level make target to the kernel tree along with pre-canned binaries would be aimed at simply making something very easy for a one time use where you can know in 60 seconds or less you have some kind of basic kernel runtime sanity.  Certainly this does not catch everything, but it really has caught lots of problems with my patches over the years.

Cheers,
Jason.