Hello Readers!! Welcome back to Part 3 of our Load Testing Best Practices series, which focuses on writing good load testing scripts.

As I’m sure you would agree to, load testing is an imperative part of every development effort. One of the key components to your load testing efforts is a good load testing script that simulates real user behavior on your web site in the most realistic and accurate way possible.

Before you take a deep dive into the scripting, there are few things you need to plan ahead of time.

  • The first one obviously is the scenario outline. Many clients seek the help of their analytics to identify their most visited pages.
  • Secondly, identify a reasonable range of pause times between steps.
  • As the next step, identify if your script will involve randomization or parameterization. If so, are you planning on using a datafile or an array of random values in the script?

If you have the answers for all of the above then you are half way through already as what’s remaining is packaging this into a script which is the objective of this post.  This post is a step by step tutorial on how to write a good load testing script using the Selenium API and the BrowserMob platform.  However, some of the tips and information can be used to help with script creation on other platforms as well. We will be focusing on how to organize your scripts, how and why to perform content validation, how to build robust and reusable scripts with the help of optimized selenium locators amongst other tips and guidelines on the best practices. I’m confident that at the end of this article you will be able to create your own load testing scripts in no time.

If you haven’t done so already, you are encouraged to install the following to assist in building the load testing scripts.

What makes a good load testing script?

    1. Detailed Comments and Descriptions

Although the presence or absence of these is in no way going to affect the performance of your load test, it is always a good practice to include comments wherever possible as they make a good reference for later use, irrespective of the scripting language under use. It is easy to add comments while scripting with the IDE. Right click on the IDE and choose ‘Insert New Comment’.

Tip: All comments added in the IDE are automatically converted to “Step Descriptions” when the script is uploaded to BrowserMob. i.e the comment in the picture above would be converted to a BrowserMob step description like below:

browserMob.beginStep("Open CNN Home Page");

Giving each step/page a brief description rather than just “Step 1”, “Step 2” etc also helps find out easily how each step/page is performing during the load testing.

    1. Content validation combined with proper use of ‘waitForPageToLoad’ and ‘waitForNetworkTrafficToStop’

“Content validation” is an essential part of every load testing script. Content validation is the only means to confirm the right page got downloaded or the right action was performed during a load test and this can be accomplished by searching for an appropriate piece of content in every step that confirms the validity of the page being downloaded. Selenium has various useful commands that can be used for content validation such as verifyTextPresent, VerifyElementPresent, assertTextPresent, assertElementPresent, waitForTextPresent, waitForElementPresent etc.

Tip: While recording actions with IDE, it is easy to add content validation at the appropriate places. Highlight the appropriate piece of content in each page load, right click and choose the appropriate content validation command as in the screenshot below:

Consider the snippets below:

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1a]

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForTextPresent("Latest news");
browserMob.endStep();

[Example: 1b]

Which one of the above is the right one?

The first one will fail immediately because the page needs to load completely or at least partially until the content “Latest news” becomes visible. The second one may not fail but is recommended only if you intend to measure the “Perceived Load Time” rather than the complete “Page Load Time”. What is the difference between the two? In the second snippet above (script 1b), the script is going to wait only until the content “Latest news” becomes visible before moving on to the next step. To a user that intends to load the CNN Home page and click on the first article under “Latest news”, this perceived load time is all they are going to care about. Whereas, the complete “Page Load Time” is measured only if the script waits for the page to load completely before moving on to the next step and this can be accomplished by using selenium API’s waitForPageToLoad and/or BrowserMob API’s waitForNetworkTrafficToStop.

Modifying the second snippet from above:

var timeout = 60000;
selenium.setTimeout(timeout);

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1c]

When is BrowserMob API’s waitForNetworkTrafficToStop helpful?

Consider the scenario wherein the user loads the site united.com and tries to find the airport code using the site’s AutoSuggest feature.

Let’s consider the snippet below that tries to simulate the above scenario:

browserMob.beginStep("Step 1: United Airlines Home");
selenium.open("http://www.united.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent(“Travel information”);
browserMob.endStep();

browserMob.beginStep("Step 2: Choose Flights using AutoSuggest");
selenium.click("id=shop_from0_temp");
selenium.typeKeys("id=shop_from0_temp", "SAN ");
selenium.mouseOver("css=div#autoSuggestDiv > div");
selenium.clickAt("css=div#autoSuggestDiv > div", "");
………
browserMob.endStep();

[Example: 1d]

Running a validation for the script above threw the error “Element css=div#autoSuggestDiv > div not found”. What was the reason? Look at the waterfall chart for united.com Home page below:

united 1a1 In the Driver’s Seat: Load Testing Best Practices – Scripting united 2a1 In the Driver’s Seat: Load Testing Best Practices – Scripting united 3a1 In the Driver’s Seat: Load Testing Best Practices – Scripting united 4a In the Driver’s Seat: Load Testing Best Practices – Scripting

The ‘waitForPageToLoad’ at Step 1 captured the traffic only until the document was complete. But as the waterfall chart above shows, several images, JavaScript files and iframes loaded asynchronously after the document was complete, especially the JavaScript file ‘airportSuggestion.js’ which was required for the autosuggestion at Step 2. Since the script simulated entering of the airport code ‘SAN’ even before the airportSuggestion.js was loaded, the autosuggest did not work as expected, resulting in a script failure.

Here is the modified script that works as expected:

browserMob.beginStep("Step 1: United Airlines Home ");
selenium.open("http://www.united.com/");
browserMob.waitForNetworkTrafficToStop(2000,timeout);
selenium.verifyTextPresent("Travel information");
browserMob.endStep();

browserMob.beginStep("Step 2: Choose Flights using AutoSuggest ");
selenium.click("id=shop_from0_temp");
selenium.typeKeys("id=shop_from0_temp", "SAN ");
// wait for the network traffic to stop and the autosuggest to populate
browserMob.waitForNetworkTrafficToStop(2000,timeout);
selenium.mouseOver("css=div#autoSuggestDiv > div");
selenium.clickAt("css=div#autoSuggestDiv > div", "");
browserMob.endStep();

[Example: 1e]

I’d like to highlight one more important aspect of the ‘waitForPageToLoad/waitFor[Text/Element]Present’ and ‘waitForPageToLoad/verify[Text/Element]Present’ combination. While both work fine it is recommended to use the combination waitForPageToLoad/verify[Text/Element]Present for load testing. Why?

Consider the snippets below:

var timeout = 60000;
selenium.setTimeout(timeout);

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

[Example: 1f]

var timeout = 60000;
selenium.setTimeout(timeout);

browserMob.beginStep("Open CNN Home Page");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.waitForTextPresent("Latest news");
browserMob.endStep();

[Example: 1g]

Both would exhibit the same behavior when the page loads successfully. However, in the instance of an error, say an ‘Internal Server’ error or a ‘Connection Reset’ error, the script would declare an error immediately after the error page loads with script 1f while it’ll wait for 60 more seconds to declare the same error with script 1g as the ‘waitForTextPresent(“Latest news”) is going to wait for 60 more seconds, which is the timeout we’ve set in the script, to declare the same content error, irrespective of the page’s status code. Besides,  script 1f will help simulate more transactions in the same amount of time than script 1g, especially in the event of errors.

Setting Proper Timeouts

The default selenium timeout is 30 seconds (30000 milliseconds) and the default TimeUnit in use by BrowserMob is milliseconds. It must be remembered that most of the ‘waitFor’ commands like the ‘waitForElementpresent’, ‘waitForVisible’, ‘waitFortextPresent’ etc wait for however long this default selenium timeout is set to. Timeouts can be set the following way in the script:

var timeout = 60000;
selenium.setTimeout(timeout);

It is recommended to set the timeout value before the actual transaction begins in the script. Studies show that the ideal page load time is 5 seconds. Any amount of load time greater than this is considered unacceptable for real users. However, 5 seconds is too small of a timeout threshold for load testing as load times are expected to rise with increase in user load. Hence, 60 seconds to 120 seconds would be considered ideal. The upper limit on the timeout threshold set by BrowserMob is 120 seconds.

Occasionally, you may have the necessity to set a different timeout value for different page loads which can be accomplished by the solution proposed in this article. The article on waitForCondition may also make an interesting read to learn more about advanced handling of timeouts.

Optimization of Locators

Selenium provides several ways of locating elements on pages. It may be worthwhile to read through this before proceeding further: http://release.seleniumhq.org/selenium-remote-control/0.9.2/doc/dotnet/Selenium.html

The most commonly used locator is the xpath. FireFox plugins like xpath finder, firebug may be used to identify the correct xpath for the appropriate elements. Below are a couple of useful articles on this topic:

Using Firebug for Load Testing.

Test Your Selenium xPath Easily with Firebug

Optimization of the locators used is extremely important for script robustness and re-usability. In the screenshot below, the IDE has captured the xpath ‘link=Arrested GI had gunpowder, ammo’ when clicking on the first article under “Latest News”. Obviously, the script will work only until this article is available on the Home Page. The xpath shown by firefbug for the same is ‘/html/body/div[5]/div/div[4]/div[3]/div/div[3]/div/ul[2]/li/a’ . Again, this might work temporarily but the moment the site is updated and the DOM changes, the script could break because of this complex xpath as the hierarchy of the elements could have been altered. ‘//ul[@class='cnn_bulletbin']/li[1]/a’ would be a much more optimized and a less brittle xpath for this example. Learn more about locators and pattern matching here.

Pause Times

Inserting pause times in between steps is essential in order to simulate real user actions more closely. A range of pause times is recommended over a fixed time as different users browse through a web site at different paces. Both BrowserMob and Selenium API’s pause method can be used to insert think/pause times in between steps.

Randomization

In the context of load testing, I want to emphasize the extreme importance of having large datasets available for testing. Many important bugs simply do not surface unless you deal with very large entities such thousands of users, thousands of mail server mailboxes etc. For scenarios involving product checkout, browsing through product pages etc, consider randomizing the product being purchased or browsed through, from a list of several products in order to simulate real user behavior more realistically as not all users are going to be purchasing the same product at the same time. Randomization and parameterization can be achieved in a few ways which are explained in depth here and here.

Use of quickStop() and stopExpected()

The BrowserMob API has two methods that come in handy when you want to pause a test that has exceptionally lengthy scripts running, such as loops that repeat certain actions several times over the course of the script execution.

By default, when a test is paused, BrowserMob will wait for the actively running scripts to finish executing and complete the current transaction. Long running scripts with loops or scripts testing a streaming video will not be interrupted until they complete. So in order to abort such transactions immediately on pausing, it is important to include the method browserMob.quickStop() at the top of the script.

However, forcefully aborting the scripts could result in an increased error rate, skewing the results. This can be avoided by using quickStop() in conjunction with stopExpected(). This instructs BrowserMob that the script is expecting to be stopped at any time and if that happens, the sudden stop should not be considered an error.

Here is a complete script, incorporating all of the above:

browserMob.quickStop();
browserMob.stopExpected();

var selenium = browserMob.openBrowser();
var c = browserMob.getActiveHttpClient();

//  Set the timeout threshold
//  Set a different timeout threshold for local validation as it has a limit on the
//  transaction length, capped at 2 minutes.
if (browserMob.isValidation()){
 	var timeout = 30000;
}
else
{
 	var timeout = 120000;
}

selenium.setTimeout(timeout);

//*************** Randomize think time *********************//
function randomXToY(minVal,maxVal)
{
 	var randVal = minVal+(Math.random()*(maxVal-minVal));
 	return Math.floor(randVal);
}

function sleep(x,y)
{
 	if(!browserMob.isValidation()){
  		browserMob.pause(randomXToY(x,y)*1000);
 }
 	else{
  		browserMob.pause(randomXToY(1,2)*1000);
  }
}

//********** Randomly pick an articleId, SearchString combination *************//
function randOption()
{
	var list = "1,PERMALINK;2,Related Links;3,Related Articles;4,Most Popular";
	var myArray = list.split(";");
	//pick a random combination from the array every time
	var position = Math.floor(Math.random()*myArray.length);
	var myArray1 = myArray[position];
	return myArray1;
}

var selected = randOption().split(",");
var id = selected[0];
var string = selected[1];

// Logging the variables being set provides a good means of troubleshooting
browserMob.log(id);
browserMob.log(string);

// Begin Transaction
transaction = browserMob.beginTransaction();

browserMob.beginStep("Open CNN Home");
selenium.open("http://www.cnn.com/");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent("Latest news");
browserMob.endStep();

sleep(2,5);

browserMob.beginStep("Click on a random article");
// Randomly clicking on an article with every script run
selenium.click("//ul[@class='cnn_bulletbin']/li["+id+"]/a");
selenium.waitForPageToLoad(timeout);
selenium.verifyTextPresent(string);
browserMob.endStep();

browserMob.endTransaction();

As web sites get complex, the scenarios for the load testing tend to get complex and so does the scripting and other load testing requirements such as configuration and set up. Our team of trained load testing experts is waiting to assist you with all your complex load testing requirements.

Hope you enjoyed reading this article as much as I enjoyed writing it. Coming up next in this Load Testing Best Practices blog series is “Test Execution” so be sure to stay tuned.

Happy Testing!!