Simple performance testing part I

skinapi

原文链接请见：
http://www.testingreflections.com/node/view/3837

I’m going to describe a quick and dirty approach to performance investigation. To no means it is a replacement for full-scale performance testing, but it could give a good understand of the system under test and even detect the major issues so that they could be fixed/removed before starting actual performance testing.
Please read detailed description of each approach below. Summary of approach is simple: record how much CPU time each separate component use and compare the sum of it with real time used. It will provide a useful assumption about your bottlenecks: CPU (per component) or hardware (network, I/O).


Some assumptions/context
This approach suppose there are no asynchronous executions, no programmed delays, etc.
It is useless if your application is not CPU-intensive, such as static content serving web server, or a file system.
I’m talking about things that Scott Barber refer as performance investigation . Even more those are only the very first steps I believe the investigator should undertake. This is not about doing performance testing with objective to see to that the customer’s SLA’s are matched.

Detailed description

I always start performance testing with single-thread case. I will describe what I do next when need to switch to multi-thread issues in next blog. Now, if unit under test is batch process this is the only case to test. But even if this is multi-user application I want first to make sure there are no obviously slow operations; lack of table indexes, etc.
But first of all I only take hosts with single CPU unit (and prefer windows, you will see later why). I run each separate component (tier) on separate host. I start the long-running batch or script emulating queries in a loop without think times. This is fast enough and I could stop it within few minutes and look at results. So what I measure: CPU time used on each host. Now it’s time for a simple calculations. Suppose we only have 3 tiers: DB, WEB server and client (my script working in a loop). Suppose DB time is 20 seconds, WEB: 40 seconds; client: 10 seconds and overall execution time – real time is 1 minute 30 seconds. 20+40+10=70 seconds. Anyone see the 20 seconds lack? This is time spend on hardware operations other than CPU: typically network and I/O (I never experienced other cases, but would be glad to hear about other cases). If this time is big compared to CPU time (in example above it is rather small) I would inspect I/O statistics, although I typically suspect too much (trivial) SQL statements, i.e. selecting rows one-by-one instead of retrieving single large result-set.
If hardware time is small, you should inspect CPU time usage distribution, taking into account CPU speed of each host. Now do you see the distribution is as expected, or your WEB server takes 50% of the time, but the content is mostly static? Or your DB takes most of the time?

Extensions
If you test end-user applications and automated script creation is somehow complicated, you could use this approach even for manual interaction with UI. The only issue is that you will not be able to see the real time, so will not be able to learn hardware time. This is because hardware time will be added with your type-in time. Well, you could escape it if application support short-cuts. For example press and hold ctrl+r if this is refresh shortcut, your application will be going to database to reload data frequently.
I love to use “2 seconds” rule. If any single user transaction uses 2 or more seconds of server CPU time (either DB or web) this mean it is critical transaction. But only if it is supposed as typically frequently used transaction.

Argues ? Need a proof of the approach?
I typically encounter initial disagreement with my approach. However I have convinced most of the people who I was able to speak in voice to. I tried to address most of the argues, still if you don’t feel OK about this approach – feel free to post your argues, I would love to discuss them.

What’s next?
As I described above, this is the first step. The next step that I do and still without requiring detailed performance requirements are multi-threading issue hunting. Hope to post about it soon.