In the Driver’s Seat: Load Testing Best Practices – Planning
This is the first part of a five-part blog series that will cover all the aspects of load testing, including planning, configuring, scripting, executing and analyzing. This first article focuses on the key areas and questions to consider when planning to run a test.
I generally see customers greatly OVER-estimating their expected capacity and many are often shocked when they get the first set of test results back from us. Usually, there is a lot of work to do in order to get the website or application tuned up and ready for a full production release.
Load testing helps you scientifically determine how many customers the website or application will support – BEFORE you potentially find out the hard way (i.e. when users actually come to your site and you have to scramble to make last minute capacity improvements). In the case where your site generates online revenue, it will also help you to quantify your investment and give you specific metrics on your overall cost per transaction in terms of required infrastructure. For example, if the site is able to support 50,000 transactions under peak conditions, you can take this number and divide it by your monthly infrastructure costs (say $2,000) to determine your infrastructure’s cost per transaction (e.g. 4 cents).
The key here is to find out this information BEFORE it causes irreparable harm to your business, brand and reputation. The only reliable way of doing this is to plan a load test.
When to test?
The ideal time to run your first load test is after User Acceptance Testing (UAT) is complete, but prior to moving the application into production. Load testing should be part of your project plan and it’s important to allow for sufficient time in your plan to complete performance and load testing. Based on my experience, you need to allow for an absolute minimum of 2-3 weeks for appropriate load test planning, scripting and execution. Many customers try to compress this time down when things are running behind, but there are no tricks to accelerating this process (and generally this results in a painful experience for everyone involved and severe sleep deprivation!). The process of load testing and performance analysis is an incremental process that requires careful review and dedicated time from the development team to address any problems or recommendations from the load tests.
How many tests should I run?
Based on our work with hundreds of customers over the years, we have found that three is the magic number.
- The first test provides you a benchmark and a baseline snapshot of the current performance of the site. There are always a number of recommendations for improvement from the first test and you need time to let your team react to the results and implement the necessary changes.
- The second test is designed to validate all the changes made after the first test and identify any remaining bottlenecks. Additional tweaks to the infrastructure or application itself are then made in preparation for the third test.
- The third test is then designed to be the final validation point prior to go-live and ideally will uncover the peak performance point of the application.
Of course, your mileage may vary here. Some customers are able to complete this process with just two tests, but generally we have found it’s best to start with three tests and add additional ones as needed.
Real Browsers or Virtual Users?
With the rapid adoption and investment in Cloud Computing, it’s now possible to cost effectively launch thousands of real browsers an hour to test a website. Neustar offers this service in both a self-service platform as well as a fully managed engagement. This has added a lot of realism to the process of load testing, which until recently has been limited to the testing using a “virtual user” or “virtual browser” that simply mimics the http request/response sequence issued by the browser.
Testing with real browsers provides unprecedented realism in driving user load to the site and generating a realistic load profile against the application. This is especially true with the wide prevalence of AJAX and client side plug-ins such as Flash/Flex. We have recently partnered with Gorilla Logic to make the process of load testing of Flex applications as easy as possible. Now you can quickly create automated Flex test scripts and launch thousands of real browsers to stress test your Flash/Flex applications!
Testing with real browsers also greatly speeds up the time to develop test scripts. The scripts also interact with the site from a functional standpoint by referring directly to the appropriate DOM elements that comprise the page rather than relying on mimicking of the underlying http request/response sequence. This helps to improve script re-use and provides an additional assurance that the test scripts will be accurately performing the actions of a real user when visiting the site. Virtual user testing still has its place for very high volume testing and testing of web service transactions or very simple websites, but in general load testing with real browsers is highly recommended for most sites.
How much load should I test with?
500. Just kidding.
The answer comes down to whether you need to execute a “stress test” or a “load test.” Let me explain the difference between these two terms and how they impact your overall test plan.
A “stress test” is designed to take the existing application infrastructure and test it to the breaking point where one or more bottlenecks are exposed. This involves sizing the test based on the amount of expected traffic the total infrastructure is expected to handle. A good general rule of thumb here is approximately 250 concurrent users per each CPU core available to process front-end web server requests. Please note that this is a very rudimentary metric and varies widely based on the type of web server, network infrastructure and most importantly the application. Performance benchmarking can help you better determine exactly what this number is, but 250 is generally a good number to start with. For example, if you had a new application with 2 front-end web servers installed on a quad-core machine, a reasonable test would be to run with 2000 concurrent users (250 users x 2 machines x 4 cores).
A “load test” is designed to test the site to a certain volume of page views or transactions, in order to validate that the site has sufficient capacity to support expected user traffic. This type of data can be obtained from web analytics data such as Omniture or Google Analytics. Generally, the target volume is based on the peak hour of historical usage for the application.
For example, say you find that in the peak hour there are 50,000 pages being loaded and you expect a 20% increase in overall web traffic in the next several months. A good test approach would be to target a total load of 75,000 pages views so you can determine the ability of the application to serve up to and above the target page view count.
Concurrent User Calculator
When planning out your load test, it’s helpful to use a calculator to determine the total number of concurrent users necessary to reach your target load level. To illustrate and help with this process, I’ve created a simple concurrent user calculator. Simply plug in the desired number of page views and the rate (e.g. 100,000 page views per hour), the session time and the desired test time (we generally recommend 60 minutes) and the calculator will determine approximately the number of users required to generate the associated number of page views. The calculator can be easily extended to estimate the number of transactions expected to complete during the test.
Before going too far explaining the calculator though, let me first explain a common misconception between a “concurrent user” and a “unique user” or “user”. We have lots of customers that come to us and say that they need to test their site with 50,000 “users.” Most of the time, what they are trying to do is simulate the business requirement of 50,000 unique users coming to the site and performing a specific transaction. This is very different from the idea of a “concurrent user.” A concurrent user is designed to run through a transaction from start to finish and then immediately start again, repeating the same transaction. This user will do this repeatedly, until the end of the test. A “unique user,” on the other hand is simply a single execution of a concurrent user or the completion of one transaction (execution of the test script from start to finish). Depending on a number of factors, concurrent users are able to generate the traffic equivalent of thousands of unique users over the course of a standard test. The calculator determines the number of concurrent users required to generate the expected amount of load. Now with definitions out of the way, let’s look at a specific of example of how to effectively use the calculator to plan your load test.
Say you have a 10-step check out process and you want to ensure the site can handle 1,000 checkouts in one hour. The total number of page views associated with this process is 10 steps x 1,000 transactions = 10,000 page views. Let’s also assume that we expect each page load to take 5 seconds and the user to wait 5 seconds on each page. This would give us a total session time of 10 steps x 10 seconds = 100 seconds. So plugging in 10,000 pages per hour and 100 seconds for the session time, gives us a concurrent user count of approximately 278. In this way, we can calculate how many users are needed to complete the key business transactions for the website. It’s important to note here that these are just estimations and highly dependent on the actual page load times encountered during the test. As page load times start to degrade, the overall throughput the site in terms of page views also degrades. Adding additional users at this point becomes overkill and will not result in an increase in the number of total pages loaded. This is why sizing of the load test is critical to make sure you have enough traffic at the ready to validate your objectives for the site.
Another aspect of load test planning that is often overlooked is the creation of appropriate test account data. This often takes the form of creating individual user ID’s to log in to the site or a list of products to select from, etc. Creation of a realistic and accurate data set is important to get the most out of the load testing process. Many times, errors encountered during the load test are the result of an incomplete or inaccurate test data creation process. Depending on the scope of testing, the data creation can be time consuming so it’s important to identify early on what data needs to be created and the process of creating it. Typically, data creation scripts will be needed to create the necessary data sets. Alternatively, custom user test scripts can be written as part of the load testing engagement to create the necessary data sets.
Server and Network Monitoring
Finally, you want to make sure you have a plan and resources in place to monitor the network devices, web server, and application and database servers that support the application. Load testing will give you all the key front-end metrics, and to get the most out of your test you’ll want to cross correlate this information closely with the performance counters from all the devices in the application delivery chain.
I hope this article has given you a good sense of the importance of load testing, as well as the key steps and questions to consider when planning to run a load test on your website or application. Next up in this series is “Configuration” – so be sure to check back to read more, as this blog series will be taking you through the full load testing process.