How to write a load test script – Part 2

In the last post, we discussed some strategies for deciding which user actions should go into your load test. It’s all pretty simple, include those actions that a typical user will take when using your app. But how easy is this to do in a very realistic manner. Let’s say users will visit one page twice as often as another, or the application is a game and the user will take different actions based on whether he won or lost. How can we do this in FriendRunner?

To get to the answer to these questions, you first need to understand a small secret about FriendRunner. Load test scripts are created in Selenium. However, that isn’t where the process of test script generation ends. The Selenium script is simply an easy way to communicate intentions about what the test should do, but there is no way to define all of the richness that a test should be able to hold. But that’s okay, the Selenium script isn’t even the thing that’s executed during a Friendrunner test. When you submit your Selenium script, FriendRunner will load it back into Selenium, and convert it into Java code. When you run your test through FriendRunner, it is this code that is actually controlling what your virtual users are doing.

This is great because it means that the virtual users are being controlled programmatically. If we need them to do some fancy things, all we’ll need to do is hand tweak the program. Let’s look at different things we can do:

Going to a page only some of the time

If there’s a page that should be a part of a load test, but users will only go to it on occasion, then we’ll need to take advantage of FriendRunner’s ability to use randomness in load test scripts. At any point in the script, instead of saying that the virtual user should go to some page, we can say that the user goes to some page X% of the time. Then, when we get to that point in the script we generate a random number between 1 and 100. If this number is less than X, then we’ll go to that page. This behavior can be used in any situation when some amount of randomness will help to make the test script more realistic.

User action determined by what’s on the page

Often, it’s a bit tricky to write a static serial test script because the application may respond to user requests in an unpredictable way.  For instance, if the application is a game, the response to a user’s move may depend on whether that user has won or lost the game. The action that the user takes next on a winning page may not even make sense if the page is a losing page. This means that the page must be “read” in order to determine the next action to take. FriendRunner is able to search a page for a piece of text, and control the next user action beased on whether that text is or isn’t there.

Two users who perform different actions

Each virtual user in FriendRunner has a user id, just like in Facebook. If we’d like 1% of the users to perform advanced functionality, and the rest of the users to perform normal functions, we can do that easily be taking advantage of the user ids.

Warning - Don’t get carried away

When writing your load test script, it’s often useful to take a break every half hour , take a deep breath, and say to yourself “I’m writing a load test script.” It’s often very easy to fall into the trap of trying to throw every user action and function into your test script, but this must be avoided. You aren’t writing a functional test, it’s a load test, and you should treat it that way. In our experience, even very large Facebook applications can be load tested with a handful of user actions.

The reason that you want to keep your load tests simple is because timing is so important to interpreting the results of the test. If you’d like to know the point at which your test, and your application began to go south, the best place to look is the transaction response graph. This graph shows you the average time it took to run through one cycle of the test script. If the test script is very long or complex, this graph will be less predictable and harder to interpret.

How to write a load test script – Part 1

One of the hardest parts of running a load test for your Facebook application is simply figuring out what to test. Since we’ve been through this a few times, we’ve come up with some ideas and tips for  how to do this properly and efficiently so that you can build a load test script that effectively tests your app.

An interesting observation we made was how closely the process of creating a test script is to Richard Feynman’s famous algorithm for solving every problem that you could possibly imagine. So let’s look at the steps you’ll need to take:

Building your FriendRunner test script

Step 1: Write down all of the things your user can do in the app

This is the tedius part of the exercise. Make a list of all of the features available to a user while he uses your application. Sometimes it’s useful to clump several discrete user actions into a single logical action. For instance, a user may browse to an account information page in your app, modify his information, and then save that change. When deciding what gets into a load test, it’s often better to think of user actions at a more macro level.

Step 2: Think really hard about what the user typically does when using your app

Does you application have a  page where you discuss your Privacy Policy? If not, you probably should, but that isn’t the issue. The issue is that very few of your users are ever going to this page. Ones that do will go once, and never return. Serving this page will probably incur very little load on your server that fulfills the request. Ergo, looking at the Privacy Policy should not be a part of a load test. Seems kind of obvious when you see it written down, but we’ve seen many things like this in proposed scripts.

What you should be putting into your script are the “normal” things that your user does when he interacts with your application. You can figure out what this is by observing how users interact with similar services (e.g. if you’re already running a non-Facebook version of your web application) or with the liberal use of some common sense.

Step 3: Write down the list of things to do for your load test script

Here’s the easy part. Once you decide on what a normal user does when he interacts with your app, put the steps together in a reasonable order. If several steps have some type of dependency on each other, take this into account. It’s also important to consider whether one user’s actions will have consequences for another user.

Special handling of FriendRunner load test scripts

In our next post, we’ll discuss some cool FriendRunner features that will allow your test scripts to more closely approximate the real world. Things such as going to some pages for some users but not others, or changing the user behavior based on information that’s displayed on the pages shown.

Contest

Identify the company that’s represented on the t-shirt that Feynman wears in the above picture, and name one thing that he did for that company. First person to answer correctly will be named Alpha Geek, and have their name posted on this page. Send your responses to contest@friendrunner.com.

Winner

Dr. Saul Kravitz was the first one to identify Thinking Machines as the company represented on Richard Feynman’s shirt. Feynman worked there for several years doing everything from painting the walls to developing deep insights into parallel computation.

For more info about Feynman’s relationship with Thinking Machines, see http://www.longnow.org/essays/richard-feynman-connection-machine/ 

Facebook Graph API on Android

Last week at Google I/O, Google’s developer conference, an important announcement about Facebook was made that’s gone fairly unnoticed (well, All Facebook did notice it). Android now has its very own API to take advantage of the Facebook Graph API, and is callable directly from any Android application. This is very similar to what’s happening with Facebook Connect for iPhone, but technically more sophisticated. The Android library supports the new Graph API, as well as support for authentication through OAuth 2.0. All the things the cool kids are using to build their Facebook apps.

Of course you’re wondering “How does FriendRunner fit into all of this?”. Is there really a need to load test an application that’s running on a mobile device? The answer to that is no! It’s a mobile device – that doesn’t even make any sense. Well…., unless there is also a backend server you run that the Android app talks to. If that’s the case, and you want to test the scalability of your backend server, then using FriendRunner makes loads of sense.

The way we’d imagine this to work is the following: We’d take your Facebook-enabled Android app and instead of loading it onto an Android device, we’d run it with an Android simulator. In fact, we’d run a whole lot of simulators in parallel. They’d all interact with your with your backend server giving it quite a workout, and allowing you to determine whether it would survive in the wild. The FriendRunner goodness comes into play because the Android app also needs to interact with a Facebook server, but typical Facebook load testing rules will apply, which are Don’t Do This. FriendRunner alleviates this problem by allowing the Android app to interact with it instead of a real Facebook server, and so allows a full load test to be run.

So do you think a tool like this would be of value? Does this address issues that mobile developers sometimes find themselves faced with, and would it solve their problems. Leave us a note and let us know what you think.

No more 24 hour rule!

One of the more interesting announcements coming out of f8 was the relaxation on the prohibition of cacheing data for more than 24 hours. Now I’ve always thought that this prohibition came from the privacy world – don’t save people’s information because it isn’t yours. Turns out, it isn’t a privacy thing at all – it’s really a user experience issue. The problem with holding on to people’s data is what happens if it changes? If you store the data and rely on it being current, things will fall apart when the user changes it several weeks from when you stored it. However, if you can only hold onto it for 24 hours, the chances of something going wrong go down dramatically.

These concerns go away now because of the new service that allows developers to register with Facebook to be notified when a user’s information changes. This will allow any cached data to be updated without any concerns of getting out of whack. This can dramatically simplify writing applications, and it was one new feature that seems to be universally embraced by developers.

And how does this affect FriendRunner? The FriendRunner profiler features that allow developers to see what APIs get called during the execution of their application become more important. There is now no excuse at all to call the same API with the same data, since the result may be cached, FriendRunner can help you find these types of problems, and identify places where you can optimize your code.

The Facebook Graph API

The new Facebook Graph API announced at f8 represents a revolution in the way that Facebook applications will be developed from here on in. You want to get a list of a users’ friends? No longer will you need to weed through the PHP documentation to try to figure out which API to call, and which parameters to pass. Now, all you need to do is make a simple RESTful HTTP call, and get a nicely formatted JSON response in return. How do you format the REST call? Pretty simple, I’m guessing that after you do 2 or 3 of these it will become second nature, and you won’t even need to look at the documentation.

So how will this affect FriendRunner? My initial thoughts is that it will affect FriendRunner in a positive way. All of the original API will still be in force, so nothing there will change. What will change it that FriendRunner will need to support all of the REST API calls at https://graph.facebook.com. But that won’t be so hard because it’s simply getting at the same data that we already provide, just in a different way. But even if it is hard, that’s okay – we subscribe to the “Why does Rice play Texas” philosophy.

Since the RESTful API is language agnostic, the FriendRunner support can be written any way we want, and not necessarily in PHP. This makes us very happy.

Stay tuned to get more information about when we’ll have the new API coded up and ready. In the meantime, all of the investment you’ve made in wriing Facebook apps the old way is secure, and your apps will continue to work. And FriendRunner will continue to be available for testing them.

FriendRunner @ f8

We’re here in San Francisco at the f8 conference awaiting all of the cool new Facebook things that are announced today. We’ve already experienced the Presence technology that looks pretty cool.

We hope to update later today some of the announcements, and how FriendRunner will fit in.

Redirecting requests to Facebook during testing

In the interest of bringing you the best product possible, we sometimes need to delve into some hairy technical issues. With that in mind, we apologize for the hairy technical issue we’re about to discuss. If you gut it out to the end, there’s a question to answer because we value your opinion.

When load test a Facebook application with FriendRunner, we need to support everything that can happen while running an app. Almost all apps make API calls into the Facebook server to get information about a user and his friends, publish info to a user’s stream, etc. During testing, these calls cannot be sent to Facebook: they must be sent to the FriendRunner server that’s acting as the Facebook server. This is a bit tricky since the application will make these calls to the facebook.com domain, and since we don’t want to force developers to modify their applications, we need to play games with DNS to redirect the calls to us.

Historically, we’ve asked developers to simply modify the /etc/hosts files on their servers running the app to be tested, and point several DNS names (e.g. www.facebook.com, api.facebook.com, etc.) to an IP address we control. No one has ever had a problem with this, it’s temporary, and aside from a short inconvenience it works great.

Today during a technical discussion, we came up with a much better solution to this redirection problem. Instead of changing the HOSTS file, you would modify /etc/resolv.conf  to point to a DNS server that we control. This DNS server would map the facebook.com domains to our servers, and do nothing else. You’d select our DNS server as primary, and whatever you use now as secondary. Of course this change would only happen on the test server, so your employees could still get to their Facebook accounts and not start a revolt.

The reason for this change is that it would make the FriendRunner infrastructure easier to manage and more robust. We thought about whether the change would cause any performance/security/management problems, and frankly we can’t find any downside to it. However, before we make a change like this, we wanted to ask you, our customers and users, whether you’d have a problem with using our DNS server for testing instead of just updating your HOSTS file.

So what do you think? Is there any problem in doing this?

Load testing Facebook applications with random user databases

When we run load tests with FriendRunner, we’re doing it outside the realm of Facebook, so our users are not only virtual, they’re also fictional. They’re cooked up by a program we wrote to populate the user database we use when running tests.

We randomly threw together early versions of this database  since we were mostly concerned with sheer numbers of users. It wasn’t uncommon to have a 17 year old guy with 3 university degrees and a 58 year old wife. It was pretty random. But we found that some applications we’re testing query this kind of information, and they actually act on it enough to make a difference during a load test. This can cause problems, so we’ve had to go back and make the random generation process a little less random. And it’s made us feel, well, a little…

Omnipotent

That’s right – there’s a feeling of power associated with deciding the social rules of your little virtual and fictious world. For us, deciding on rules was easy – support what Facebook supports, and have fictional people reflect everyday society. Gay marriage? There’s no debate here, Facebook allows this and we do too. You won’t find many married same-sex couples in our database, but you’ll find some. Fourteen year old kids? In our world they can have girlfriends, but they can’t get married. Married people in the database live in the same city, are roughly the same age, and both partners have their Relationship status set to Married. In contrast to the real world, our fictional people don’t fight over things like this.

We do this so that any applications we test can query this information and act on it as realistically as possible. Which doesn’t mean there aren’t anomalies in the database. It’s possible to generate a user named Ahmed Goldstein who was born in Mississippi, and happens to be Catholic. But it’s hard to imagine how this would affect a load test.

Our improved algorithms for generating virtual users make it much easier for us to create custom user databases for you when necessary. If your app targets seniors, we’ll make all of your test users seniors. Writing an app for online dating? We can ensure that all of the users during the test are single.

Let us know how a custom user database could help you test your Facebook application.

FriendRunner – Now more scalable!

As software developers, we’re constantly reminded to “eat our own dog food”.  Of course, this means that we need to force ourselves to actually use the products we build (in the same way that our customers will) so we can see how difficult they are to use, and that hopefully we’ll improve them for the better. Some people call this “drinking your own champagne”, but they’re just delusional.

With this in mind, we’re happy to report that FriendRunner is now more scalable than ever. Our own testing showed that at higher levels of virtual users, one of our internal components that processes Profiler data would get swamped and could start to fail. That’s right, we humbly and ironically report that the tool you use to test the scalability of your Facebook apps was itself susceptible to scalability problems. We’ve spent the last few days rearchitecting this component so that we can run several copies of it on concurrent machines which can be duplicated as much as the load requires, just like the rest of the FriendRunner infrastructure. The end result is a much more robust and scalable tool that handles large loads easily.

Okay, but why blog about it? Well, we have several reasons:

• We come from a computer security background, so we believe that it’s important to be open and honest about the capabilities of our products. If that includes limitations and problems, so be it.
• To remind ourselves about the importance of testing software products
• Fixing problems like this means that we’re constantly trying to make FriendRunner a better tool
• We found this cool dog food picture, and needed a reason to share it.

So how about your products? Are they really as scalable as you think? Do they have the virtual cojones to stand up to load testing from FriendRunner? Let us know when you’re ready to try it out.

Meet us at f8 2010

We’ll be in San Francisco on April 21 and 22 for f8 2010, Facebook’s premier developer conference. If you’d like to learn more about FriendRunner, see a demo, or you’d simply like to talk about testing Facebook applications, send us a note at f8@friendrunner.com.

We’re looking forward to meeting you there.