A small intro …
Now that I am almost finished with the “stress test” task I was talking about in my previous post, I have several thoughts and experience to share concerning on the subject. I am also planing to write about Java web application profiling on a following post as it somehow relates with the results of a “stress test” task.
The tool I have used to carry on stress test tasks is JMeter (the latest version available at the time of this writing) thus, I will write about JMeter. However, I am interested in any feedback (experience) concerning other tools (or JMeter).
State clearly your objectives …
It is important that you state your objectives clearly as the overall methodology of the stress tests will greatly depend on these objectives.
Some classical examples follow:
- Give a precise estimate of the maximum load that a given system may serve (peak): this is usually done in order to help plan the future infrastructure of a live system.
- Find precisely the bottlenecks of a live system during a peak: this is usually done as a preliminary task to profiling and performance tuning tasks.
- Find precisely the origin of eventual leaks (memory, connection to resources, various resources) during a long run: this is also usually done as a preliminary task to profiling and tuning tasks.
- Prove that the system you have implemented can hold a theoretical load: usually this was a client’s requirement expressed during the very early stages of a project (for example in the call for tender)
- Any combination of the aforementioned objectives …
These various different objectives lead to different types of scenarios. To my opinion a good methodology is always to try and implement scenarios that are as close as possible to real and typical use cases of the system you are willing to test. However, in some cases (bullets 2 and 3 above) you may need to write artificial scenarios that will help you identify precisely a functionality of your system that has performance problems.
The following paragraph is about writing “real case” scenarios and test plans covering the aforementioned objectives.
Write good quality scenarios and test plans …
First a difference must be made between “scenarios” on one hand and “test plans” on the other:
A scenario is (or at least should be) an actual use case of your application carried out by a single user. In JMeter terms, a scenarios is a combination of “samplers” and “controllers” that will be executed by a single “thread” of a “thread group”.
A test plan is the “way” a given scenario will be executed in order to achieve a given objective (as the ones described in the previous paragraph). In JMeter terms, the “way” the scenario will be executed mainly means playing with the following variables on the thread group: the number of threads, the ramp up time and the number of loops executed by a thread.
It is very important to understand the exact meaning of these 3 parameters:
- The “number of threads” in a thread group is the actual number of threads spawned by JMeter, each one of them used to execute the scenario. In other words, this variable is the number of users executing a “real life” use case on your system. This number is not the number of concurrent / parallel users executing a “real life” use case on your system: the concurrency of the users depends on both the duration of your scenario and the ramp up time configured on the thread group.
- The “ramp up time” in a thread group is the actual time taken by JMeter to spawn all the threads. If the ramp up time is small compared to the number of threads and the mean duration of a scenario then the number of concurrent threads accessing your system will be high and vice versa. A rough estimation of the throughput (number of requests per second) during the ramp up period of your test plan is: number of threads / ramp up time (in seconds).
- The “number of loops” in a thread group is the actual number of times that the scenario will be executed by each thread.
Now let’s go back to the implementation of “real case” scenarios using JMeter. I recommend this interesting article on the subject sent to me by a colleague (thanks Petros ;-) ). Some very good methodological hints are given concerning the writing of scenarios in the first paragraphs. Basically, I can give 3 main hints on the subject that are easy to follow and implement with JMeter:
- Keep scenarios simple:
Each scenario should correspond to one use case. This makes things much more simple and logical particularly when it comes to interpreting the results of the stress tests.
- Use “recording” techniques to generate your scenario from a “real” usage of the application:
JMeter comes with a proxy component, which when started, will record all the HTTP Requests and Response cycles originating from a web browser configured to access your system through this proxy. There are well-known problems with the usage of this proxy when dealing with HTTPS: often, a simple solution is to do all the recording in HTTP and turn the protocol to HTTPS in your scenario afterwards (this supposes that you can make your system run under HTTP for the time of the recording).
- Don’t forget to record the “think time” of the users:
The “think time” of a user is the elapsed time between 2 user actions. During this time, the user may be thinking what to do next, answering an urgent call on the phone, talking with a friend … this must be part of the scenario. Fortunately, JMeter allows to record these “think times” and translate them into “Gaussian Waits” inside your scenario (see the article mentioned above for hints on how to do it). In any case, you should always have “waits” in your scenarios simulating in the most realistic manner these “think times” of the real users.
- Read the JMeter User’s Manual particularly the “Component Reference” in order to find all possibilities provided by the tool. For example:
You can use an external csv file containing (username, password) couples in order to have each thread login into your system with different credentials.
You can use regular expressions to parse HTTP Responses and extract data necessary to chain your samplers
Once you have your scenario ready, you must configure your test plan in order to meet your objectives. The tuning of the main parameters of your test plan (number of threads, ramp up and number of loops) is often a “try and error” procedure. However, we can give the 3 following hints:
- You should try to have a constant throughput during a run:
It is often very difficult to “control” the throughput particularly during the ramp up period
- If your objective is to simulate a “peak”:
You should have a “high” number of threads and a “low” ramp up time and number of loops
- If your objective is to simulate a “long run”:
You should have a “medium” number of threads, a “higher” ramp up time and a “high” number of loops
Note: The terms “high”, “higher”, “medium” and “low” are voluntary qualitative in the 3 bullets above as they depend on the system you are testing.
To be continued …
This post is already too long: seems I have to much to say on the subject ;-) Never mind, I will carry on in a following post tomorrow covering the remaining subjects: running the test plans,recording the meaningful measures, interpreting the results, monitoring the systems …