Test Harness 2 - Advanced Test Harness Features

skinapi

链接请见：
http://blogs.msdn.com/steverowe/archive/2006/05/06/591764.aspx

In a past post I talked about the basic functionality of a test harness.  That is, it should be something that provides a reusable framework for an application, for running and reporting the results of test cases.  There is much more that a test harness can do, however.  It can provide mechanisms for lightweight test development, model based testing, and scripting.

Basic test harnesses like the Shell98 I spoke of last time or cppunit are what I would call heavyweight harnesses.  By that I mean that the testing code is statically bound to the harness at compile time.  In the most basic form the harness comes in the form of source code that is compiled along with the test code.  A slightly more advanced model involves a separate library that is statically linked with the test code to form a standalone executable.  This is fine but it means longer compile times, larger binaries, and potentially less flexibility.

There is a better way to do this.  The test harness and test cases can be separated.  The harness is compiled into an executable and the test cases are loaded by it dynamically.  I call this a lightweight harness.  As the harness no longer knows what test cases it will be tasked with executing, this requires that the tests are discoverable in some manner by the test harness.  The test cases are usually collected in dlls, jar files, or assemblies which are loaded by the harness.  The harness uses reflection (C# or Java) or a custom interface to discover which tests are in the test file.  This system is much more complex than the static binding but it offers several advantages.  It separates the test cases from the harness, allowing them to be varied independently.  It decreases compile times and reduces binary size.  It can also allow a single instance of a test harness to run many different types of tests.  With a static model, this scenario would require running many different executables.  Nunit is an example of a lightweight test harness.

Another feature that advanced test harnesses will have is support for model based testing.  Model based testing is a method of testing where test cases are generated automatically by the test framework based on a well-defined finite state machine.  A good description can be found on Nihit Kaul’s blog.  A test harness which supports model based testing will provide mechanisms for defining states and state transitions (actions) which it will use to generate and execute test cases.  The harness will also need to support a mechanism for verifying that the system is still correct after each transition.  Setting up model based testing usually requires a lot of work up front.  They payoff can be quite high though.

In a simple test harness, a test case is not provided any context in which to run.  Imagine the test cases as functions which take no parameters.  They will run exactly the same each time they are executed.  An advanced test harness will provide a mechanism to modify the test cases via parameters or scripting.  The simple method is to allow parameters to be passed to the test cases via a configuration file.  This is useful when you want several tests which vary only on a single parameter.  I wished for this feature when I was testing DVD playback.  In order to test the performance of our DVD navigator on different discs, I needed test cases to play different chapters.  I was forced to create a test case per chapter I wanted to play.  It would have been much simpler to write one test case and then allow a parameter which supplied the chapter to play.  Some harnesses might even provide a full scripting model where you could programmatically call test cases and provide parameters.  It is easy to envision embedding VB, Python, or even Lisp into a test harness and using that to control the test case execution.  The advantages to both of these methods are the ability to easily vary the test coverage without being required to program.  This makes test case creation accessible to non-programmers and saves programmers a lot of time over compilation.

Very advanced test harnesses provide mechanisms to execute the tests on a pool of machines.  We might call these test systems rather than test harnesses.  I’ll discuss these in another post.

carol2000

yeah~It's time for our tester to coding！

brilliantking

Advanced Test Harness Features
高级 Test Harness

1)In a past post I talked about the basic functionality of a test harness.  That is, it should be something that provides a reusable framework for an application, for running and reporting the results of test cases.  There is much more that a test harness can do, however.  It can provide mechanisms for lightweight test development, model based testing, and scripting.
在以前的的一个帖子里我探讨了 test harness 的基本作用。这就是， test harness 应该为应用程序提供一个可以重复使用的框架，用来运行测试用例以及报告测试结果。 test harness 还可以做更多的工作。它可以提供一些手段以便于简化建立测试的过程，实现基于模块的测试以及进行测试前的准备工作。

2)Basic test harnesses like the Shell98 I spoke of last time or cppunit are what I would call heavyweight harnesses.  By that I mean that the testing code is statically bound to the harness at compile time.  In the most basic form the harness comes in the form of source code that is compiled along with the test code.  A slightly more advanced model involves a separate library that is statically linked with the test code to form a standalone executable.  This is fine but it means longer compile times, larger binaries, and potentially less flexibility.
上回我说过的Shell98或是cppUnit这样的初级的 test harness ，我想把它们叫做重量级的 harness 。我的意思是这些 test harness 的测试代码在编译时是静态地绑定在里面的。大多数的初级的 test harness 都是采用源代码与测试代码一共编译的方式。稍微高级一些的方式是用一个与源代码分开的的运行库，静态链接上测试代码，然后编译成独立的可执行文件。这样做是不错的，但是这样也意味着需要更长的编译时间，文件的长度会更大，而且暗中也降低了灵活性。

3)There is a better way to do this.  The test harness and test cases can be separated.  The harness is compiled into an executable and the test cases are loaded by it dynamically.  I call this a lightweight harness.  As the harness no longer knows what test cases it will be tasked with executing, this requires that the tests are discoverable in some manner by the test harness.  The test cases are usually collected in dlls, jar files, or assemblies which are loaded by the harness.  The harness uses reflection (C# or Java) or a custom interface to discover which tests are in the test file.  This system is much more complex than the static binding but it offers several advantages.  It separates the test cases from the harness, allowing them to be varied independently.  It decreases compile times and reduces binary size.  It can also allow a single instance of a test harness to run many different types of tests.  With a static model, this scenario would require running many different executables.  Nunit is an example of a lightweight test harness .
要让 test harness 变得轻便，还有更好的方式。 test harness 与测试用例可以完全分隔开。harness 首先编译成可执行文件然后动态地加载测试用例。我把这种方式叫做轻量级的 harness 。这种方式下， harness 事先不再知道要执行哪些测试用例，这就要求 test harness 以某种方式来获取测试用例。通常，测试用例都集中存放于由harness所加载的 dll ， jar 文件或用例包中. test harness 使用响应机制(在C#或java中)或某种自定义的接口来探知测试文件中有哪些用例。这种方式比那种静态绑定方式要复杂很多，但是这种方式有几大优势。首先，此方式将测试用例与 test harness 分离开来从而允许独立更换用例。第二，此方式缩短了编译时间以及编译文件的长度。此外，此方式还允许 test harness 的一个单独实例来运行不同类型的测试用例。而在那种静态模式中，这种情形可能会需要运行很多不同的 test harness 可执行文件。Nunit就是轻量级 test harness 的一个例子。

4)Another feature that advanced test harnesses will have is support for model based testing.  Model based testing is a method of testing where test cases are generated automatically by the test framework based on a well-defined finite state machine.  A good description can be found on Nihit Kaul’s blog.  A test harness which supports model based testing will provide mechanisms for defining states and state transitions (actions) which it will use to generate and execute test cases.  The harness will also need to support a mechanism for verifying that the system is still correct after each transition.  Setting up model based testing usually requires a lot of work up front.  They payoff can be quite high though.
高级的 test harness 还有一个特性就是支持基于模块的测试。基于模块的测试机制是这样的一种测试方法。其中有一个以精确定义的自动状态机为基础的测试框架，用来自动地产生测试用例。关于这种测试机制，Nihit Kaul 的博客里有很好的介绍。支持基于模块测试的 test harness 需要提供用于定义状态及状态转换（动作）以便产生和执行测试用例的一些机制。这种 test harness 还需要能够确认在每一次状态转换后系统是否仍正常运行。建立一个基于模块的测试系统通过需要大量的前期工作，其资金投入非常大。

5)In a simple test harness, a test case is not provided any context in which to run.  Imagine the test cases as functions which take no parameters.  They will run exactly the same each time they are executed.  An advanced test harness will provide a mechanism to modify the test cases via parameters or scripting.  The simple method is to allow parameters to be passed to the test cases via a configuration file.  This is useful when you want several tests which vary only on a single parameter.  I wished for this feature when I was testing DVD playback.  In order to test the performance of our DVD navigator on different discs, I needed test cases to play different chapters.  I was forced to create a test case per chapter I wanted to play.  It would have been much simpler to write one test case and then allow a parameter which supplied the chapter to play.  Some harnesses might even provide a full scripting model where you could programmatically call test cases and provide parameters.  It is easy to envision embedding VB, Python, or even Lisp into a test harness and using that to control the test case execution.  The advantages to both of these methods are the ability to easily vary the test coverage without being required to program.  This makes test case creation accessible to non-programmers and saves programmers a lot of time over compilation.
初级的 test harness 没有为测试用例提供运行时的环境，而只是将用例作为没有输入参数的函数看待。所以用例每次执行的情况都相同。高级的 test harness 则会有一套方法通过调整参数或是输入测试脚本来修改测试用例。一个简单的方法是用配置文件向测试用例传递参数。我测试DVD的播放功能时就很期待这个特性。为了测试我们的DVD机播放不同的光盘时的性能，我需要有播放不同片断的测试用例。我不得不为我想播放的每一个片断写一个测试用例。要是能只写一个用例而允许有一个代表播放片断的参数，那就容易多了。有一些 test harness 甚至会给你提供全脚本模块，你可以编写程序来调用测试用例同时给出参数。可以方便地向 test harness 中添加嵌入式的VB,Python, 甚至Lisp程序片断用以控制测试用例的执行情况。 这两种方式的优点就是可以在不需编程的情况下很容易地修改测试的覆盖度。这使得非编程人员也可以创建测试用例，并且可以节省编程人员在编译程序花费的大量时间。

6)Very advanced test harnesses provide mechanisms to execute the tests on a pool of machines.  We might call these test systems rather than test harnesses.  I’ll discuss these in another post.
更加高级的 test harness 提供的方式可以允许在机器群组中执行测试用例。我们应该把它们叫做测试系统而非 test harness了。关于这类 test harness，我将在另外一个贴子里讨论。  

[ 本帖最后由 brilliantking 于 2006-5-10 17:59 编辑 ]

李逍遥

顶啊!