Test Harness 1 - Test Harness Basics

skinapi · 发表于 2006-4-29 23:20:39

链接请见：
http://blogs.msdn.com/steverowe/archive/2006/04/27/585758.aspx

A short while back one of my readers asked what a test harness was.  I will answer that in a pair of posts.  This first post will describe the basics of a test harness.  The second post will talk about more advanced features that a test harness might have.

When you are writing test cases , whether they are unit tests, tests for test-driven development, regression tests, or any other kind, there are some functions you will find yourself doing each time.  These include the basics of an application, a mechanism for launching tests, and a way to report results.  You could write these functions each time you start testing a new feature or you could write them once and leverage them every time.  A test harness, at its most simple, is just that—a system to handle the elements you’ll repeat each time you write a test.  Think of it as scaffolding upon which you will build your tests.

A basic test harness will contain at least the following elements:  application basics, test launching, and result reporting.  It may also include a graphical user interface, logging, and test case scripting.  It should be noted that harnesses will generally be written for one language or runtime (C/C++, Java, .Net, etc.).  It is hard to write a good harness which will work across runtimes.

To run a test, the first thing you need is an application.  Each operating system has a different way to write an application.  For example, on windows if you want any GUI, you need things like a message pump and a message loop.  This handles the interaction with the operating system.  The harness will include code to start the program, open any required files, select which cases are run, etc.

The next thing you need is a way to actually launch the tests.  Most of the time a test case is just a function or method.  This function does all of the actual work of the test case.  It calls the API in question, verifies the results, and informs the framework whether the case passed or failed.  The test harness will provide a standardized way for test cases to advertise themselves to the system and an interface by which they will be called.  In the most simple system, a C/C++ harness might advertise its cases by adding function pointers to a table.  The harness may provide some extra services to the test cases by allowing them to be called in a specified order or to be called on independent threads.

The third basic pillar of a test harness is a way to inform the user of which cases pass and which fail.  They provide a way for test cases to output messages and to report pass/fail results.  In the most basic harness, this could be just a console window in which test cases will print their own messages.  Better than that is a system which automatically displays each test case name and its results.  Usually there is a summary at the end of how many cases passed or failed.  This could be textual or even a big red or green bar.  More advanced systems have built-in logging systems.  They provide a standardized way for the test cases to output trace messages informing the user of each important call as it is being made.  The harness may simply log to a text file but it may also provide a parsable format like XML or even interact directly with a database for result retention.

At Microsoft, many groups write their own test harnesses which are specialized to support the unique needs of the organization.  For example, my team uses a harness called Shell98 which has, among other things, support for device enumeration.  Good examples of a freely available test harnesses are the xUnit series of tests like cppUnit, nUnit, and jUnit.  These are designed for unit testing and are not very feature-rich.  I’ve used cppUnit which is very basic and nUnit which is pretty slick.  The xUnit harnesses do not do any logging and you do not have control over the order in which tests are run.  They are intended for a user to run and visually inspect the results.  The harness I use allows for scripting and outputs its results to a database.

brilliantking · 发表于 2006-5-9 13:48:36

Test Harness Basics
Test Harness 基础篇

A short while back one of my readers asked what a test harness was.  I will answer that in a pair of posts.  This first post will describe the basics of a test harness.  The second post will talk about more advanced features that a test harness might have.
　　前不久，有位网友问我什么叫test harness。我想用两个贴子来回答这个问题。第一个贴子我想叙述一下test harness包含的基本的东西。第二个贴子会探讨test harness所包含的更深层次的东西。

When you are writing test cases , whether they are unit tests, tests for test-driven development, regression tests, or any other kind, there are some functions you will find yourself doing each time.  These include the basics of an application, a mechanism for launching tests, and a way to report results.  You could write these functions each time you start testing a new feature or you could write them once and leverage them every time.  A test harness, at its most simple, is just that—a system to handle the elements you’ll repeat each time you write a test.  Think of it as scaffolding upon which you will build your tests.
　　当你在书写测试用例时，无论是单元测试用例还是测试驱动式开发中的测试用例，还是回归测试用例或是其他类型的用例，你会发现有一些事情你每次都要做。这些事情包括应用程序的基础部件，启动测试的一套程序以及报告测试结果的方法。每次你要测试程序的一个新的方面的时候，你都要去写这些东西，或者你一次把它们写好，然后每次都修改它们。简而言之，所谓 test harness 就是指管理每次写用例时需重复进行的事情的一套系统。你可以把它当作构建测试时用的脚手架。

A basic test harness will contain at least the following elements:  application basics, test launching, and result reporting.  It may also include a graphical user interface, logging, and test case scripting.  It should be noted that harnesses will generally be written for one language or runtime (C/C++, Java, .Net, etc.).  It is hard to write a good harness which will work across runtimes.
　　基本的 test harness 至少要包括下面几个要素：应用程序基本部件，测试启动机制和结果报告机制。同时也可以包含有一套图形用户界面，日志系统与用例书写系统。应该注意的是 test harness 通常只是为某一种语言或程序运行平台而构建的。很难去构建一个跨平台运作的 test harness。

To run a test, the first thing you need is an application.  Each operating system has a different way to write an application.  For example, on windows if you want any GUI, you need things like a message pump and a message loop.  This handles the interaction with the operating system.  The harness will include code to start the program, open any required files, select which cases are run, etc.
　　要进行一次测试，你需要的第一件东西就是一个应用程序。每个操作系统都会有各自不同的方法去构建应用程序。比如，在 windows 平台上，如果你想要 GUI ，你需要的就是一个类似消息泵和消息环的部件。　这个部件专门处理应用程序与操作系统的交互。Test harness 要包含用于启动程序，打开所需文件，以及选择用例等操作的代码。
The next thing you need is a way to actually launch the tests.  Most of the time a test case is just a function or method.  This function does all of the actual work of the test case.  It calls the API in question, verifies the results, and informs the framework whether the case passed or failed.  The test harness will provide a standardized way for test cases to advertise themselves to the system and an interface by which they will be called.  In the most simple system, a C/C++ harness might advertise its cases by adding function pointers to a table.  The harness may provide some extra services to the test cases by allowing them to be called in a specified order or to be called on independent threads.
　　Test harness 里的第二件东西是实际运行测试用例的一套途径。大多数情况下，一个测试用例仅仅可看作是一个功能或是方法。这个功能要执行测试用例的所有实际的工作。该功能调用现在仍有争议的所谓API，核对测试结果，以及向程序所在的运行框架报告该用例是通过还是未通过。Test harness 会提供一种标准的途径，以便向系统提交测试用例，还要提供系统用来调用测试用例的接口。在最简单的系统中，一套 C 或 C＋＋的 harness通过向一个表格添加功能指针的方法来向系统提交用例。Test harness 还要提供一些额外的服务，以允许系统依次调用测试用例，或是使用独立的线程来调用。

The third basic pillar of a test harness is a way to inform the user of which cases pass and which fail.  They provide a way for test cases to output messages and to report pass/fail results.  In the most basic harness, this could be just a console window in which test cases will print their own messages.  Better than that is a system which automatically displays each test case name and its results.  Usually there is a summary at the end of how many cases passed or failed.  This could be textual or even a big red or green bar.  More advanced systems have built-in logging systems.  They provide a standardized way for the test cases to output trace messages informing the user of each important call as it is being made.  The harness may simply log to a text file but it may also provide a parsable format like XML or even interact directly with a database for result retention.
　　Test harness 的第三个基本的组成部分就是用以通知用户哪些用例通过哪些没通过测试的一个途径。这个途径使得测试用例可以对外发出测试状况消息并报告通过/失败结果。在最基本的 test harness 中，这种机制可能只是一个控制台窗口，测试用例会将测试状况发送其上。更好的测试结果报告机制则是可自动显示测试用例名称及各自测试结果的一个系统。通过在末尾会有一个关于有多少用例通过或失败的总结，可以是文本方式或者红色或绿色的长条。更高级的机制则会内置日志系统。这种机制提供一种标准化的方法，测试用例输出各种跟踪信息，用来向用户报告正在进行的每一个重要的调用。Test harness 可能会简单地记录到文本文件中，但也可以提供像XML一样的可解释格式文件，或者更进一步地直接与保存测试结果的数据库直接交互。

At Microsoft, many groups write their own test harnesses which are specialized to support the unique needs of the organization.  For example, my team uses a harness called Shell98 which has, among other things, support for device enumeration.  Good examples of a freely available test harnesses are the xUnit series of tests like cppUnit, nUnit, and jUnit.  These are designed for unit testing and are not very feature-rich.  I’ve used cppUnit which is very basic and nUnit which is pretty slick.  The xUnit harnesses do not do any logging and you do not have control over the order in which tests are run.  They are intended for a user to run and visually inspect the results.  The harness I use allows for scripting and outputs its results to a database.
　　在微软公司里，很多小组都有自己的 test harness ，都是为支持组织里特定的需要而专门开发的。举个例子，我的小组使用一种叫 Shell98 的 test harness ，与其他的相比，它支持设备列举。可以免费获取的 test harness的一些好例子是关于测试的 xUnit 系列，像 cppUnit ， nUnit 以及 jUnit 。这些软件是用单元测试而设计的，功能不是很强大。我用过 cppUnit 用起来很简单，还有 nUnit 用起来很顺手。xUnit 这类 test harness 不会对测试过程做任何记录，你也不能控制测试用例执行的顺序。这些 test harness 的设计初衷是让用户运行用例并且用可视化的方式检查得到的结果。我用的 test harness 可以为测试用例写脚本并把测试结果输出到数据库。