[Accessibility] Accessibility testing in LSB

[Mark Doffman]

Hello everyone,

I apologize for giving you all so little time to mull this over, but 
here is a short report detailing the status of Accessibility testing in LSB.

Brief overview of the LSB test suite
------------------------------------

The LSB test suite is split up into two sections the Application Test 
Kit and the Distribution Test Kit. The purpose of the Application Test 
Kit is to check that applications are compatible with LSB standards and 
to help application developers write applications that will work on LSB 
certified distributions. The purpose of the Distribution Test Kit is to 
check LSB compatibility of distribution libraries, aiding 
interoperability and checking that LSB certified applications will run.

ATK tests in LSB
-----------------

There are some ATK tests that form part of the t2c-desktop suite in LSB. 
Test reports indicate this suite performed 222 tests and has 100% 
coverage of the ATK interface. The tests seem comprehensive, there are a 
large number of requirements that have been translated from upstream 
documentation, and they all seem to be tested.

The issue with these tests is that a large number of the requirements 
are functional, and depend on the implementation of ATK, rather than 
just the interface. The test suite provides its own 'Dummy' ATK 
implementation, so I'm not sure of its usefulness in checking many of 
the requirements.

In addition to this I'm not sure that testing the ATK interface is that 
useful to LSB. Its not an interface likely to be used directly by 
application programmers. The only access likely is by ATs and they will 
usually only do this indirectly through AT-SPI client libraries.

AT-SPI testing in LSB
---------------------

There are some incomplete AT-SPI tests checked in to the LSB repository 
under atk-tests. These are the IBM prototype tests that George Kraft has 
pointed us to. As George mentioned the tests have little coverage; 
essentially just the Component interface.

I think these tests are more useful to LSB than the ATK tests, as the 
cspi and pyatspi libraries are more likely to be accessed directly. LSB 
should have tests to check the compatibility of AT-SPI client libraries.

As far as I can tell, the purpose of the shallow tests is to check 
interface compatibility. As AT-SPI and ATK are interfaces, that may have 
multiple implementations, shallow tests should be developed for the cspi 
library. This will ensure API compatibility across LSB certified 
distributions. Deeper, 'Medium', quality tests could then be created 
using the pyatspi bindings. These tests would be similar to the IBM 
prototype tests - end-to-end tests checking a particular implementation 
of AT-SPI.


TODO list
---------

1) (Optional) Modify the t2c-desktop tests so that they check the GAIL 
implementation.

2) (Optional) Write shallow tests for the cspi library to check API 
compatibility.

3) Create AT-SPI requirements from upstream documentation.

This is likely to be a very involved process, translating the IDL 
documentation into test requirements. I would imagine that there would 
be some additional requirements on the interfaces that could be taken 
from ATK documentation. These requirements would also include the 
implementations responsibility to send events under certain conditions.

Ideally these requirements would be implementation agnostic, but this 
may only come about through some trial and error.

4) Write end-to-end tests for GTK and pyatspi.

The initial tests would probably be created using pyatspi and GTK. Test 
applications would have to be written in GTK that would enable all of 
the requirements to be properly tested.

5) Integrate the tests with LSB

Stew Benedict has attempted to package the IBM prototype tests for LSB 
and found that they contained many non-LSB dependencies and Gnome 
infrastructure. We must assume the GTK-pyatspi tests would be similarly 
difficult to package and may take some work to move into the LSB test 
suite.

Application Tool Kit Accessibility tests
----------------------------------------

Accerciser has some "AT-SPI Validator" tests that look at the 
accessibility status of applications. I haven't had any word yet from 
LSB guys on whether these sorts of tests would be considered for inclusion.

Thanks

Mark


-- 
Mark Doffman, Codethink Ltd. -  http://codethink.co.uk