[Openais] RE: [Saftest-ais-devel] Re: [Saftest-devel] [ANNOUNCE] saftest_AIS-B.01.01_0.1.0 released!

Yu, Ling L ling.l.yu at intel.com
Thu Feb 3 18:39:16 PST 2005


Steve
Thanks for your good comments; it's very helpful for us. Pls see my
comments as below.

Thanks
- Ling

-----Original Message-----
From: saftest-ais-devel-admin at lists.sourceforge.net
[mailto:saftest-ais-devel-admin at lists.sourceforge.net] On Behalf Of
Steven Dake
Sent: Friday, February 04, 2005 5:00 AM
To: Xiong, Crystal; openais at lists.osdl.org
Cc: saftest-devel at lists.sourceforge.net;
saftest-hpi-devel at lists.sourceforge.net;
saftest-ais-devel at lists.sourceforge.net
Subject: [Saftest-ais-devel] Re: [Saftest-devel] [ANNOUNCE]
saftest_AIS-B.01.01_0.1.0 released!

Crystal

Thanks for the early work on a B implementation of the test suite.  As
you may know openais is quickly moving to the B implementation in the
next few weeks and an early test suite will really help us out.  If you
need order of priority, we are doing conversions something like
CLM (done)
EVT (done)
CKPT (to be done)
AMF (may be next release)

I have a few comments on this test package.

I fired up the clm tests against our first ported service (the cluster
membership API B.01.01).  The test suite runs well after I figured out a
few things.

First of all, PLEASE integrate the timeout code into the run_tests.sh
script.  It took me a few hours to determine there was indeed a need for
an external script which I didn't have and had a hard time finding (but
eventualy found).  Then after I found the script, the calling parameters
didn't match the script I have so I had to figure out how to hack
run_tests.sh and the timeout script together.  External dependencies in
test suites on scripts with changing interfaces is a bad bad idea.
[Yu, Ling L]  Yes, "timeout" are from netatalk-1.6.4-2.1, Thank for
reminder. We will try to find a resolution about this issue. 
I also found a serious defect:
it appears test cases fail for the test cases that use the exec call. 
An example of this is saClmDispatch/8-1.c

int main(int argc, char *argv[])
{
        char same_case[]="./1-2.test";

        execl(same_case, NULL);
        printf("        Can't find case %s\n", same_case);
        return SAF_TEST_UNKNOWN;
}

Unfortunately we are not in the correct directory but instead in the top
level directory, which makes these test cases fail (execl can't find the
application).  This results in about 62 "Unknown reason" failures which
is over half the test cases.

In a perfect world the tests should be runnable from any directory
either top level or in the directory themselves.  This would be the
ideal scenario for developers (so individual test cases can be
debugged).
[Yu, Ling L]  Yes, this is a defect; we can refine the run_tests.sh to
fix it.

I really dislike having to say pass fail for the manual tests in
report.sh.  This is terribly annoying and skews the results of my test
cases because I either always answer pass or fail.  A command line
option to report or not manual tests would solve the problem.  Another
option should be added as well, which is "NOT TESTED".
[Yu, Ling L]  I think add a command option is a good resolution. If
"report.sh -m", all manual test cases will be calculated together. By
default, only automatic test cases are considered. 
Once saftest has these few issues fixed up, I'll post some results on
the B-01-01 implementation of CLM that has been developed in the openais
project.

Thanks for the hard work

Regards
-steve

On Thu, 2005-02-03 at 03:13, Xiong, Crystal wrote:
> The first release of AIS B.01.01 tests is available at
http://sourceforge.net/projects/saftest. This release includes the tests
for AIS-B.01.01 membership service.
> 
> >From this release, we will separate the releases for HPI, AIS and
A.01.01, B.01.01. The new release is named as
saftest_<SPEC-NAME>_<VERSION>, this release is named as
saftest_AIS-B.01.01_0.1.0
> The existed releases for SAF A.01.01 will remain the same. Bug fix
version for A.01.01 will be named as saftest_<VERSION>, such as
saftest_1.0.1.
> 
> The major changes in this release are:
>      - Create AIS B.01.01 clm test cases (Thanks YuLing and YuPing)
>      - Support manual cases in test framework. (Thanks YuLing and
XiaoWei)
>      - Update test framework scripts to support different releases we
will make. (Thanks YuLing).
>      - Create scripts under utilities to facilate our work (Thanks
XiaoWei)
>      - Update test framework document to v0.9 (Thanks YuLing)
> 
> Test case contribution and test execution are welcome!
> 
> The README in the release package describes how to use the framework,
how to contribute more tests to the framework in a brief. For more
details, please refer to Framework_Specification.htm document under doc
directory in the release package.
> 
> The goal of this project is to create conformance test suite for SAF
spec, including AIS A.01.01, AIS B.01.01, HPI A.01.01 and HPI B.01.01
spec. 
> 
> Any more comments, please discuss in the following mailing list:
> saftest-devel at lists.sourceforge.net (For the whole project)
saftest-hpi-devel at lists.sourceforge.net (For HPI part)
saftest-ais-devel at lists.sourceforge.net (For AIS part)
> 
> The webpage of this project can be found here:
http://saftest.sourceforge.net/
> 
> Thanks,
> Crystal    
> 
> * This is my personal opinion *
> 
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive
Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Saftest-devel mailing list
> Saftest-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/saftest-devel



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Saftest-ais-devel mailing list
Saftest-ais-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/saftest-ais-devel




More information about the Openais mailing list