FriendRunner adds Advanced Monitoring

One of the first things taught to medical students is Primum non nocere, which roughly translates into First, do no harm. This is understood to mean that it’s better to do nothing than to treat a patient in a way that you do more damage than good, or put another way: Don’t let the cure be worse than the illness.

So what does this have to do with FriendRunner? Lots – in fact it was the inspiration for our latest feature that you don’t even get to see since it works behind the scenes, but will be thankful it’s there. To understand why, you need to think about the two jobs that FriendRunner performs:

1. Calls the test application in the same way that Facebook would so the application can experience the load. While doing this, FriendRunner monitors the time it takes the application to respond to determine its health. (Steps 2 and 5 in the diagram below)
2. Allows the application to call the FriendRunner ”proxy” Facebook server to service any Facebook API calls that the application needs to make. (Steps 3 and 4 in the diagram below)

That’s really all that FriendRunner does. These two functions act together to form a system that can put enough load on the test application so we can learn something meaningful, and hopefully make the application more scalable and better.

But what about FriendRunner? What happens when it gets large loads put on it? Once we start running with a few thousand virtual users, will the proxy Facebook server be able to satisfy API requests fast enough? And what if it can’t? Well, that’s pretty easy to answer: a poorly performing API will result in poor performance at the test application level. In other words, FriendRunner (the tool used to measure an app’s performance) will be the reason that the app being tested exhibits poor performance.

This is the stuff that keeps us up at night. However, in reality, things really aren’t so bad. FriendRunner’s server that replies to API calls is not a single server, but a collection of state-less scalable servers deployed onto a computing infrastructure with almost infinite expandability. It’s very unlikely that it would ever be the cause of a problem and make an application under test appear to be slowing down. Yet, we still think about these things to ensure that our customers get our best service.

Our initial attempts to allay these concerns were to simply monitor the servers that the FriendRunner software ran on. We’d monitor the health of the computers, and watch the memory and CPU usage. This works pretty well at the macro level, but still doesn’t really say what happens at the individual call level. The server’s memory may be doing just fine, but we still may be affecting the performance of the application under test. Truthfully, we’ve never seen any evidence of this, but it still weighs heavy on us. Remember: First, do no harm. We take our customers very seriously, and don’t want to be the cause of misinformation.

Which brings us to our latest feature, Advanced Monitoring. We’ve now instrumented our Facebook API server so that we can measure the performance of the FriendRunner server at the API call level to prove that the system acts consistently. And that’s a very important point to this whole process – we aren’t interested in measuring our performance because we want to make it faster. No, we measure to prove consistency – something that’s very important when measuring the performance of another system. Our calls should return in about the same amount of time regardless of whether there are ten or ten thousand users running in the system. That’s a difficult standard to live up to, and not only do we do it, but Advanced Monitoring can now prove that this is what happens.

Unfortunately, despite this very sophisticated system working very well, there’s really nothing that you, the user, can see. But we’ll sleep much better at night knowing that FriendRunner is not doing any harm. And you … well, you should be happy that FriendRunner will always provide you with a large and consistent load so that testing and tuning your application is as simple as possible.

How to write a load test script – Part 2

In the last post, we discussed some strategies for deciding which user actions should go into your load test. It’s all pretty simple, include those actions that a typical user will take when using your app. But how easy is this to do in a very realistic manner. Let’s say users will visit one page twice as often as another, or the application is a game and the user will take different actions based on whether he won or lost. How can we do this in FriendRunner?

To get to the answer to these questions, you first need to understand a small secret about FriendRunner. Load test scripts are created in Selenium. However, that isn’t where the process of test script generation ends. The Selenium script is simply an easy way to communicate intentions about what the test should do, but there is no way to define all of the richness that a test should be able to hold. But that’s okay, the Selenium script isn’t even the thing that’s executed during a Friendrunner test. When you submit your Selenium script, FriendRunner will load it back into Selenium, and convert it into Java code. When you run your test through FriendRunner, it is this code that is actually controlling what your virtual users are doing.

This is great because it means that the virtual users are being controlled programmatically. If we need them to do some fancy things, all we’ll need to do is hand tweak the program. Let’s look at different things we can do:

Going to a page only some of the time

If there’s a page that should be a part of a load test, but users will only go to it on occasion, then we’ll need to take advantage of FriendRunner’s ability to use randomness in load test scripts. At any point in the script, instead of saying that the virtual user should go to some page, we can say that the user goes to some page X% of the time. Then, when we get to that point in the script we generate a random number between 1 and 100. If this number is less than X, then we’ll go to that page. This behavior can be used in any situation when some amount of randomness will help to make the test script more realistic.

User action determined by what’s on the page

Often, it’s a bit tricky to write a static serial test script because the application may respond to user requests in an unpredictable way.  For instance, if the application is a game, the response to a user’s move may depend on whether that user has won or lost the game. The action that the user takes next on a winning page may not even make sense if the page is a losing page. This means that the page must be “read” in order to determine the next action to take. FriendRunner is able to search a page for a piece of text, and control the next user action beased on whether that text is or isn’t there.

Two users who perform different actions

Each virtual user in FriendRunner has a user id, just like in Facebook. If we’d like 1% of the users to perform advanced functionality, and the rest of the users to perform normal functions, we can do that easily be taking advantage of the user ids.

Warning - Don’t get carried away

When writing your load test script, it’s often useful to take a break every half hour , take a deep breath, and say to yourself “I’m writing a load test script.” It’s often very easy to fall into the trap of trying to throw every user action and function into your test script, but this must be avoided. You aren’t writing a functional test, it’s a load test, and you should treat it that way. In our experience, even very large Facebook applications can be load tested with a handful of user actions.

The reason that you want to keep your load tests simple is because timing is so important to interpreting the results of the test. If you’d like to know the point at which your test, and your application began to go south, the best place to look is the transaction response graph. This graph shows you the average time it took to run through one cycle of the test script. If the test script is very long or complex, this graph will be less predictable and harder to interpret.

How to write a load test script – Part 1

One of the hardest parts of running a load test for your Facebook application is simply figuring out what to test. Since we’ve been through this a few times, we’ve come up with some ideas and tips for  how to do this properly and efficiently so that you can build a load test script that effectively tests your app.

An interesting observation we made was how closely the process of creating a test script is to Richard Feynman’s famous algorithm for solving every problem that you could possibly imagine. So let’s look at the steps you’ll need to take:

Building your FriendRunner test script

Step 1: Write down all of the things your user can do in the app

This is the tedius part of the exercise. Make a list of all of the features available to a user while he uses your application. Sometimes it’s useful to clump several discrete user actions into a single logical action. For instance, a user may browse to an account information page in your app, modify his information, and then save that change. When deciding what gets into a load test, it’s often better to think of user actions at a more macro level.

Step 2: Think really hard about what the user typically does when using your app

Does you application have a  page where you discuss your Privacy Policy? If not, you probably should, but that isn’t the issue. The issue is that very few of your users are ever going to this page. Ones that do will go once, and never return. Serving this page will probably incur very little load on your server that fulfills the request. Ergo, looking at the Privacy Policy should not be a part of a load test. Seems kind of obvious when you see it written down, but we’ve seen many things like this in proposed scripts.

What you should be putting into your script are the “normal” things that your user does when he interacts with your application. You can figure out what this is by observing how users interact with similar services (e.g. if you’re already running a non-Facebook version of your web application) or with the liberal use of some common sense.

Step 3: Write down the list of things to do for your load test script

Here’s the easy part. Once you decide on what a normal user does when he interacts with your app, put the steps together in a reasonable order. If several steps have some type of dependency on each other, take this into account. It’s also important to consider whether one user’s actions will have consequences for another user.

Special handling of FriendRunner load test scripts

In our next post, we’ll discuss some cool FriendRunner features that will allow your test scripts to more closely approximate the real world. Things such as going to some pages for some users but not others, or changing the user behavior based on information that’s displayed on the pages shown.

Contest

Identify the company that’s represented on the t-shirt that Feynman wears in the above picture, and name one thing that he did for that company. First person to answer correctly will be named Alpha Geek, and have their name posted on this page. Send your responses to contest@friendrunner.com.

Winner

Dr. Saul Kravitz was the first one to identify Thinking Machines as the company represented on Richard Feynman’s shirt. Feynman worked there for several years doing everything from painting the walls to developing deep insights into parallel computation.

For more info about Feynman’s relationship with Thinking Machines, see http://www.longnow.org/essays/richard-feynman-connection-machine/ 

Facebook Graph API on Android

Last week at Google I/O, Google’s developer conference, an important announcement about Facebook was made that’s gone fairly unnoticed (well, All Facebook did notice it). Android now has its very own API to take advantage of the Facebook Graph API, and is callable directly from any Android application. This is very similar to what’s happening with Facebook Connect for iPhone, but technically more sophisticated. The Android library supports the new Graph API, as well as support for authentication through OAuth 2.0. All the things the cool kids are using to build their Facebook apps.

Of course you’re wondering “How does FriendRunner fit into all of this?”. Is there really a need to load test an application that’s running on a mobile device? The answer to that is no! It’s a mobile device – that doesn’t even make any sense. Well…., unless there is also a backend server you run that the Android app talks to. If that’s the case, and you want to test the scalability of your backend server, then using FriendRunner makes loads of sense.

The way we’d imagine this to work is the following: We’d take your Facebook-enabled Android app and instead of loading it onto an Android device, we’d run it with an Android simulator. In fact, we’d run a whole lot of simulators in parallel. They’d all interact with your with your backend server giving it quite a workout, and allowing you to determine whether it would survive in the wild. The FriendRunner goodness comes into play because the Android app also needs to interact with a Facebook server, but typical Facebook load testing rules will apply, which are Don’t Do This. FriendRunner alleviates this problem by allowing the Android app to interact with it instead of a real Facebook server, and so allows a full load test to be run.

So do you think a tool like this would be of value? Does this address issues that mobile developers sometimes find themselves faced with, and would it solve their problems. Leave us a note and let us know what you think.

Redirecting requests to Facebook during testing

In the interest of bringing you the best product possible, we sometimes need to delve into some hairy technical issues. With that in mind, we apologize for the hairy technical issue we’re about to discuss. If you gut it out to the end, there’s a question to answer because we value your opinion.

When load test a Facebook application with FriendRunner, we need to support everything that can happen while running an app. Almost all apps make API calls into the Facebook server to get information about a user and his friends, publish info to a user’s stream, etc. During testing, these calls cannot be sent to Facebook: they must be sent to the FriendRunner server that’s acting as the Facebook server. This is a bit tricky since the application will make these calls to the facebook.com domain, and since we don’t want to force developers to modify their applications, we need to play games with DNS to redirect the calls to us.

Historically, we’ve asked developers to simply modify the /etc/hosts files on their servers running the app to be tested, and point several DNS names (e.g. www.facebook.com, api.facebook.com, etc.) to an IP address we control. No one has ever had a problem with this, it’s temporary, and aside from a short inconvenience it works great.

Today during a technical discussion, we came up with a much better solution to this redirection problem. Instead of changing the HOSTS file, you would modify /etc/resolv.conf  to point to a DNS server that we control. This DNS server would map the facebook.com domains to our servers, and do nothing else. You’d select our DNS server as primary, and whatever you use now as secondary. Of course this change would only happen on the test server, so your employees could still get to their Facebook accounts and not start a revolt.

The reason for this change is that it would make the FriendRunner infrastructure easier to manage and more robust. We thought about whether the change would cause any performance/security/management problems, and frankly we can’t find any downside to it. However, before we make a change like this, we wanted to ask you, our customers and users, whether you’d have a problem with using our DNS server for testing instead of just updating your HOSTS file.

So what do you think? Is there any problem in doing this?

Load testing Facebook applications with random user databases

When we run load tests with FriendRunner, we’re doing it outside the realm of Facebook, so our users are not only virtual, they’re also fictional. They’re cooked up by a program we wrote to populate the user database we use when running tests.

We randomly threw together early versions of this database  since we were mostly concerned with sheer numbers of users. It wasn’t uncommon to have a 17 year old guy with 3 university degrees and a 58 year old wife. It was pretty random. But we found that some applications we’re testing query this kind of information, and they actually act on it enough to make a difference during a load test. This can cause problems, so we’ve had to go back and make the random generation process a little less random. And it’s made us feel, well, a little…

Omnipotent

That’s right – there’s a feeling of power associated with deciding the social rules of your little virtual and fictious world. For us, deciding on rules was easy – support what Facebook supports, and have fictional people reflect everyday society. Gay marriage? There’s no debate here, Facebook allows this and we do too. You won’t find many married same-sex couples in our database, but you’ll find some. Fourteen year old kids? In our world they can have girlfriends, but they can’t get married. Married people in the database live in the same city, are roughly the same age, and both partners have their Relationship status set to Married. In contrast to the real world, our fictional people don’t fight over things like this.

We do this so that any applications we test can query this information and act on it as realistically as possible. Which doesn’t mean there aren’t anomalies in the database. It’s possible to generate a user named Ahmed Goldstein who was born in Mississippi, and happens to be Catholic. But it’s hard to imagine how this would affect a load test.

Our improved algorithms for generating virtual users make it much easier for us to create custom user databases for you when necessary. If your app targets seniors, we’ll make all of your test users seniors. Writing an app for online dating? We can ensure that all of the users during the test are single.

Let us know how a custom user database could help you test your Facebook application.