Random header image... Refresh for more!

Scrambled Keys

F5.

F5.

F5 F5 F5.

WTF?

Why isn’t this running?

F5 F5.

Grrr…

Run->Debug.

(wait)

(wait)

F10.

F10.

Why is that making the File menu highlight?

F10.

F11.

WTF?!  It restarted?

Tip of the day:  If you install the C/C++ Developer Tools in Eclipse, it’ll come with a set of Visual Studio key bindings.

December 7, 2010   1 Comment

Pizza Pizza

Visual Studio never fails to come up with amusing ways to go horribly wrong.

And yet somehow, “Append” is immune.

May 18, 2010   No Comments

SWA in WPF and Silverlight

A couple of weeks ago, I wrote about using the System.Windows.Automation libraries in the .Net Framework to write UI Automation, primarily for use in writing automated test cases.  Well, that’s only half the story.  In order to truly unlock the power of SWA and the UIAutomation library, you’ll want to add SWA instrumentation to your application.  It’s not required for using SWA, but believe me, it will make your testers happier.

Microsoft has provided a way to add enhanced support for UIA into Win32 and Windows Forms apps, but WPF and Silverlight have been built from the ground up to have full and simple support for the UI Automation properies.  In this post, I’ll be exploring how to instrument a Silverlight application, largely because I can give a link to my finished sample app for you to explore.  Working with WPF should be mostly the same. 1

Let’s start by looking at what you get from Silverlight, straight out of the box.  We’ll begin with a simple text box and button, the sort of thing you’ll find in pretty much any UI program ever written.

<StackPanel x:Name="LayoutRoot" Orientation="Horizontal" Width="250" Height="32">
    <TextBox x:Name="TextEntryBox" Width="150"/>
    <Button x:Name="SubmitButton" Content="Submit"/>
</StackPanel>

That produces a super exciting display that looks like this:

SWASilverlightApp1

I has mad UX skillz.  Fear them.

If you open good old trusty UISpy and examine the control tree of your XAML masterpiece, you’ll find something that looks a bit like this:

SWASilverlightApp-UISpy1

An “edit” control and a “button” control in a Silverlight Control?  That looks an awful lot like what we just wrote up in XAML, doesn’t it?  That’s because it is.  Yep.  You write it and presto, it’s seen by SWA.  And if you look at the automation properties panel in UISpy, you’ll see that the edit control has an AutomationId with the value of “TextEntryBox” and that the button control has an AutomationId of “”SubmitButton”.  Those are also what we put in the x:Name attributes back in the XAML.  In other words, for Silverlight and WPF apps, most of your controls will automatically be exposed to UIAutomation using the programmatic names you’ve already given them in your code.  You get that for free.

But wait!  There’s more!

Do you see all those other properties in UISpy?  Well, if you don’t, here’s a screenshot for your enlightenment:

SWASilverlightApp-UISpy2

Well, you have direct access to set some of them straight from XAML.  Okay, so it’s not completely free, but it’s still really really easy.  They’re exposed through the AutomationProperties attribute.  As an example, let’s set the “Name” property of the text box in the sample.  If you look at the UISpy control tree, you’ll see that it’s called “”, instead of anything useful.  The automatic population of the “Name” property usually uses the content of a control.  In the case of the button, it has a text label of “Submit”, which is where it gets the name from.  But the text box doesn’t have any content, so it gets left out.  Let’s give it a name before it gets depressed.

<TextBox x:Name="TextEntryBox" Width="150" AutomationProperties.Name="Search Query"/>

All it needs is that “AutomationProperties.Name” attribute set and there you have it.  our edit box is now showing up with a name in UISpy and everyone’s happy.

SWASilverlightApp-UISpy3

Most of the time, I’ll stick to “Name” and “AutomationId” for helping me find elements for my automated tests.

For a good number of decently simple applications, that’s all you’ll need to know.  For that reason, most tutorials I’ve seen on the subject stop here and pretend that this much information is actually adequate for using SWA in the real world.  I, however, have actually done some work in the real world and know that this alone can be painful to work with.  So, I’m going to keep rambling for a bit.

Consider, for a moment, that you’ve assembled some combination of controls that is useful to you, and that you’d like to reuse that combination of controls and not have to copy/paste/rename.  Now, I understand that many of you will never attempt something as wild and complex as this, and that the thought of it alone might drive you to madness, but please bear with me, as it’s required for my next demonstration.  For the sake of conversation, let’s say that this useful combination of controls is a text box and a button to go with it.

So, you have a user control with a box and a button.  I’ve given it the bleedingly obvious name of “BoxAndButton”.  This code should look familar to you:

<UserControl x:Class="SWASilverlightApp.BoxAndButton"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Width="250" Height="32">
    <StackPanel x:Name="LayoutRoot" Orientation="Horizontal">
        <TextBox x:Name="TextEntryBox" Width="150" AutomationProperties.Name="Search Query"/>
        <Button x:Name="SubmitButton" Content="Submit"/>
    </StackPanel>
</UserControl>

Okay, now that we have the control, let’s use it somewhere.  In fact, it’s a reusable control, so let’s use it twice, just because we can!

<StackPanel x:Name="LayoutRoot">
    <MathPirate:BoxAndButton x:Name="SearchDatabase"/>
    <MathPirate:BoxAndButton x:Name="SearchWeb"/>
</StackPanel>

(Please note the “MathPirate” namespace that you’ll have to add to your XAML file, if you’re playing the home game.)

Now we run our app and see our two glorious text boxes and buttons.

SWASilverlightApp2

Now let’s go into UISpy and see what Silverlight or WPF has given us for free.

SWASilverlightApp-UISpy4

Now, this is the point in the application development cycle where you’ve made your UI tester cry.  (Or, if they’re anything like me, you’ve made them cry and shoot you with a fully automatic Nerf gun.) While all of the elements are there, everything is jumbled together.  Despite naming one of the controls as “SearchDatabase” and the other as “SearchWeb” and using a hierarchy of controls, SWA has decided that you really want a flat list of useless.  Anyone trying to automate against this isn’t going to know which button is which, and will therefore have to rely on ordinal positions of controls and a whole lot of luck.  If, for some reason, you decide that Web should come first and you go and swap the boxes, you’ll decimate every single test written against this app.  Database queries will go to the Web and Web queries will end up at the Database, and a quantum singularity will form that will destroy the entire universe.  So, let’s fix things, shall we?

I know!  I know!  AutomationProperties!

<StackPanel x:Name="LayoutRoot">
    <MathPirate:BoxAndButton x:Name="SearchDatabase" AutomationProperties.Name="Database"/>
    <MathPirate:BoxAndButton x:Name="SearchWeb" AutomationProperties.Name="Web"/>
</StackPanel>

Give it a run, and…

SWASilverlightApp-UISpy4

FAIL.

So, what gives?  You’re using a nicely hierarchical control tree, you’re giving things nice names and identifiers, why aren’t you getting anything useful from SWA?  The reason:  UIAutomation is primarily designed to expose interactive controls and elements, and your user control isn’t an interactive element in itself, only the text box and button inside it are.  It’s wrapped in a StackPanel, but things like StackPanels, Grids and Canvases in Silverlight and WPF are considered to be strictly for graphical layout and say nothing about the interaction with the system.  As a result, they don’t show up in the control tree. 2  And the BoxAndButton control itself doesn’t show up because you haven’t told SWA that it’s interesting yet.  Let’s do something about that.

The method that WPF and Silverlight use to expose these AutomationProperties is called an “AutomationPeer”.  AutomationPeer is an abstract class that has a bunch of methods for you to implement, like GetNameCore() and GetAutomationIdCore().  The idea is that these Get*Core are supposed to return the value of one of the Automation Properties for UIA to expose.  This gives you a lot of control over what gets seen by UI Automation clients or tools like UISpy.  If you like that level of control, then feel free to inherit directly from AutomationPeer and implement all thirty or so Automation Properties.  The rest of us, who don’t like tedium, will simply use the implementation provided by FrameworkElementAutomationPeer.

Of course, having an AutomationPeer isn’t going to help you much, unless you tell SWA to use it.  You do that by overriding the method OnCreateAutomationPeer() on your control class.  The method is defined on the UIElement base class, so every element in Silverlight already has it, you’ve just probably never noticed it before.  OnCreateAutomationPeer() just needs to return the appropriate AutomationPeer object that exposes your object and the Automation Properties and Control Patterns you want the world to know about.

Okay, that sounded scary and confusing, but it’s not really that bad when you see it in action.  So, let’s get to the code!  Open up the .cs file for your control.

First, add a using:

using System.Windows.Automation.Peers;

Then, override OnCreateAutomationPeer:

protected override AutomationPeer OnCreateAutomationPeer()
{
    return new FrameworkElementAutomationPeer(this);
}

And now let’s run our pretty application once more and see what UISpy has for us this time.

SWASilverlightApp-UISpy5

Boom-diggity, that’s what I’m talking about.

Now, you have a nice control tree, with your web and database search controls uniquely identified.  While your efforts may not have made the testers completely happy, at the very least, they should have stopped shooting you with the Nerf guns for a while.

But wait!  There’s more!

If you were paying attention, you may have noticed the mention of Control Patterns that I ever-so-slyly snuck in a few paragraphs back.  Control Patterns were the prime focus of the earlier article on using SWA and UIAutomation, but I haven’t really said anything about them here at all.  Why is that?  Because on the implementation side, you typically don’t have to care about them.  You’re using a text box or a button or a scroll bar or whatever that already implements all of the control patterns that it needs, for the most part.  That means you typically don’t have to do a damned thing to get the ValuePattern or InvokePattern to work, while the testers all have to suffer with that idiotic rehash of QueryInterface and COM.

Before you start gloating about that, you need to realize that I’m a tester and now I’m going to make you suffer, too.

All along, we’ve been using a Box and Button as two separate elements.  We’ve put them in a single control, though.  Why not make that single control appear as a single control to UIAutomation?  In other words, why don’t we make our custom control look like it is a text box and a button, rather than something that just contains a text box and button?3

So, somehow, we now have to implement the ValuePattern and the InvokePattern on our BoxAndButton control.  The way to do that, as mentioned before, is through the AutomationPeer.  First, we’ll have to create our own subclass of AutomationPeer.  Again, since I’m not crazy, I’m going to use FrameworkElementAutomationPeer as a base, but you don’t have to.

public class BoxAndButtonAutomationPeer : FrameworkElementAutomationPeer
{
    public BoxAndButtonAutomationPeer(BoxAndButton owner) : base(owner) { }
}

Be sure to change your OnCreateAutomationPeer in BoxAndButton to return one of these instead.  If you run this now, you should see everything act exactly the same as it was before you added this new class.

If you go inside the your new AutomationPeer class and type “public override “, you’ll conjure the magic Intellisense window that’ll show you the list of all those Get*Core() and other Automation Property methods I talked about earlier.

AutomationPeerMethods

You can implement any of them if you want to, but the implementation in FrameworkElementAutomationPeer is usually good enough.  Except for the method GetPattern…

GetPattern is the method that SWA calls on your AutomationPeer when someone using SWA calls “GetCurrentPattern” on an automation element representing your control.  You’ll need to implement it and return an object that implements whatever ControlPatterns you wish to support.  In our case, we want to implement ValueProvider for the Text Box and InvokeProvider for the Button.  The only parameter to GetPattern is a PatternInterface enum containing a value for each of the existing ControlPatterns, so a simple implementation is to switch on patternInterface and return the appropriate object.

public override object GetPattern(PatternInterface patternInterface)
{
    switch(patternInterface)
    {
        case PatternInterface.Value:
        return Owner;

        case PatternInterface.Invoke:
        return Owner;
    }

    return base.GetPattern(patternInterface);
}

If you run it right now and look at it with UISpy, you’ll see that your BoxAndButton control now is apparently supporting the Invoke Pattern and the Value Pattern, just like we want it to do.  Trouble is…  It doesn’t actually support anything.  You may have noticed that I was simply returning “Owner” from GetPattern and found that odd.  Owner is the UIElement we passed into the constructor for FrameworkElementAutomationPeer.  In other words, Owner is the BoxAndButton control itself.  But, we haven’t actually done anything to BoxAndButton to let it know that it’s supposed to support InvokePattern.  Let’s take care of that now.

Each of the control patterns is exposed through an interface. 4  For the Invoke Pattern, it’s IInvokeProvider.  For the Value Pattern, you’ll want to use IValueProvider.  These and other Provider interfaces for for the other patterns live in the System.Windows.Automation.Providers namespace, so be sure to add that.

IInvokePattern is an interface with a single method, called Invoke.  It takes no parameters and returns nothing.  The function is supposed to cause the control to perform some action.  In our case, we’re going to want it to act like the button was clicked.  Like so:

#region IInvokeProvider Members

void IInvokeProvider.Invoke()
{
    AutomationPeer peer = FrameworkElementAutomationPeer.CreatePeerForElement(SubmitButton);
    IInvokeProvider invokeProvider = (IInvokeProvider)peer.GetPattern(PatternInterface.Invoke);
    invokeProvider.Invoke();
}

#endregion

Okay, that was freaking ugly.  You see, Silverlight and WPF don’t really have a way to easily simulate a button click on the Button class.  So, this method is diving into the world of UIAutomation itself and using the InvokePattern that’s already available on the Button.  Basically, it’s a passthrough.  Making this function suck less is left as an exercise to the reader.

IValueProvider is somewhat more straightforward and sane.  It has three members.  IsReadOnly, a boolean property that returns true/false based on whether or not the control is read only.  Value, a string getter property that returns the value of the control.  And SetValue, a function that takes a string and sets the value of the control with it, because apparently the designers hadn’t heard that properties like “Value” can have both getters AND setters.

#region IValueProvider Members

bool IValueProvider.IsReadOnly
{
    get { return TextEntryBox.IsReadOnly; }
}

void IValueProvider.SetValue(string value)
{
    TextEntryBox.Text = value;
}

string IValueProvider.Value
{
    get { return TextEntryBox.Text; }
}

#endregion

 

 Conveniently, all three of those members on IValueProvider map directly to things we can easily do with a text box already.

Now, if you go into UISpy and play around, you’ll see that not only does the BoxAndButton control say that it supports the Value and Invoke patterns, but that you can now actually use them.  You can easily see the Value Pattern in action.  Simply put some text in the box, then look in UISpy, and you’ll see that the custom BoxAndButton now can see the text value.  To see that the InvokePattern is wired up properly, you’ll have to hook up an event handler on the Click event of the button.

Of course, you don’t have to do exactly what I did in my example.  You can use whatever you need and whatever you have available in the I*Provider implementations.  Do it the way that makes sense for you.   You don’t even have to implement these interfaces on the control class directly.  I did it because it was convenient in this case, but it might not make sense for you.  You can use any class you want, it’s entirely up to your design.  Just change what you return from GetPattern.  However, if you put the interfaces on the control class, like I’m doing, I would recommend explicitly implementing these interfaces, since it’s unlikely you’ll want to expose these methods to anyone using the class normally.

There’s just one little annoying thing left about the control.  If you look in UISpy, you’ll see that our BoxAndButton control now supports the Invoke and Value patterns, but we still have the Text Box and Button as children, and those also have the Value and Invoke patterns on them.  They’re not needed anymore.  Let’s axe them, shall we?

If you’ll look at the list of functions available to override on our AutomationPeer, you’ll see one called GetChildrenCore.  You can override that function to tell UIA what your children should be.

protected override List GetChildrenCore()
{
    return new List<AutomationPeer>();
}

Obviously, it’s more useful to tell the system that you have children that it wouldn’t ordinarily know about (Like in the case of an owner drawn control), than it is to disown the children that you do have.  So, when using this function in a normal situation, you’re probably going to add something to the list that’s being returned.

Now, if you run the app and look at it in UISpy, you’ll see that the children of the BoxAndButton control have been written out of the will.  You’ll also likely see that, in fact, UISpy is having trouble finding our control.

Element : Element details not available.
Name : TreeValidationException
Message : UI Automation tree navigation is broken. The parent of one of the descendants exist but the descendant is not the child of the parent
Stack Trace : at UISpy.Base.Tree.BuildTree(AutomationElement element)
    at UISpy.Base.Tree.BuildTreeAndRelatedTrees(AutomationElement element)

Well now…  That sucks.  See, what’s happening is that UISpy is finding the real Text Box and real Button controls that still exist and trying to select them, but it’s unable to find their parents in the tree.  In other words, we’ve tampered with the natural order of things and are making UISpy go all loopy.  Good luck with that, I’m gonna end on this high note.

You can find my sample application here (A version that’s not hOrking UISpy):  http://mathpirate.net/log/UploadedFiles/SWASilverlightApp1/SWASilverlightAppTestPage.html

You can get the source code out of SVN here:  http://www.mathpirate.net/svn/Projects/SWAExample/SWASilverlightApp/

  1. I haven’t done anything with SWA and Windows Forms or Win32 yet, so I don’t know what that entails, but from a quick glance at the docs, it’s a bit more complicated. []
  2. And honestly, you don’t want them to.  It would be insane to try to navigate through all the different panels that appear everywhere.  One of the Silverlight 2 Betas had all of the panels and things exposed and it was frightening. []
  3. Why don’t we?  Because it’s actually useless to do so.  A text box and a button are perfetly good controls to contain.  However, treat this as an exercise in exposing a completely custom owner drawn control to System.Windows.Automation. []
  4. If you’ll remember from the previous article, a large serving of wrath was thrown toward the designers of SWA for not using interfaces so finding out that they do, in fact, use interfaces on objects internally was a bit of a relief, but also upsetting.  I still can’t understand why the person responsible for the design on this side didn’t smack the designers of the consuming side around until they used interfaces, too.  And if it’s the same designer, then they need to smack themselves around until they make up their mind. []

October 10, 2009   No Comments

Being Mean To The Average

I hate the average.

Specifically, I hate the use of the average as the predominant, sometimes only bit of information given when talking about software performance test results.  I will grant that it’s easy to understand, it’s easy to calculate, and I know that it’s one of the few mathematical concepts that can be passed along to management without much explanation.  However, it’s often misleading, presents an incomplete picture of performance, and is sometimes just plain wrong.  What’s worse is that most people don’t understand just how dangerously inaccurate it can be, and so they happily report the number and don’t understand when other things go wrong. 

Calculating the average is simple.  You take the sum of a set of numbers and divide it by the number of numbers in that set.  But what exactly does that give you?  Most people will say that you get a number in the middle of the original set or something like that.   That’s where the faith in the number begins and, more importantly, where the mistakes begin.  The average will not necessarily be a number in the middle of your original set and it won’t necessarily be anywhere near any of the numbers in your original set.  In fact, even if the average turns out to be dead-center in the middle of your data, it doesn’t tell you anything about what that data looks like.

Consider, for a moment, the annual income of someone we’ll call Mr. Lucky.  Mr. Lucky’s salary starts at $50,000.  Every year, Mr. Lucky gets a $2500 raise.  So, for five years, here’s his income:  $50000, $52500, $55000, $57500, $60000.  Over that period, his average annual income is $55000.   Great, smack in the middle.  Now, in the sixth year, Mr. Lucky wins a $300 million lottery jackpot.  What’s his average income over all six years?  Over $50 million a year.  However, it’s clearly wrong to try to claim that Mr. Lucky made $50 million dollars a year over six years, because once you look at the data, it is obvious that the $300 million is skewing the average well away from what he was actually making at the time.

Let’s take another example, one closer to home.  Every month, you get an electric bill.  Since heating and cooling are often the largest chunks of power consumption, the bill will have the average temperature for the month, in order to help you make sense of the fluctuating charges.   This year, you get a bill for $200.  Shocked, you pull out last year’s bill to compare, and discover that you paid only $50 then.  Last year, according to the bill, the average temperature was 54.3 degrees, and this year it was 53.8 degrees.    The average temperature was roughly the same, so your heating/cooling shouldn’t have changed that much.  You didn’t buy a new TV or an electric car, you turn off the lights when you’re not in the room, your shut down the computer at night, you’ve got CFL bulbs everywhere, and as far as you know, the neighbors aren’t tapped into your breaker box to power the grow op in their basement.  So…  What happened?  Let’s take a closer look at that weather…

Daily Temperature Last Year:

ElectricityBill-LastYear

Daily Temperature This Year:

ElectricityBill-ThisYear

Once you look at the actual daily temperature, it becomes clear what happened to your power bill.  Last year, the temperature was fairly constant, but this year, there were wild temperature swings.  You had your AC cranking full blast for the first part of the month, then you kept nudging up the thermostat during the end of the month.  However, since the temperature extremes offset one another, the average temperature makes it seem like both months had the same weather.

That’s the key:  Once you look at the data, it’s often clear that the average doesn’t tell the whole story, or worse, tells the wrong story.  And the problem is that people typically don’t look at the data.

Let me pull this back to the world of software performance testing and tell a story about a product I’ve worked on and how I first came to realize that typical performance testing was dead wrong.  The product was a keyword processor at the heart of a larger system.  Our customer had demanded a 1.5 second response time from our overall system, and that requirement got chipped down and used up by other components on its way to the keyword processor I was involved with.  By the time it got to us, we only had a response time cap of 250 ms in which to return our data, otherwise the larger system would give up on us and continue on its way.  So, great, I thought.  I’ll just load up the system with an ever increasing number of concurrent requests and find out how many requests we can process at the same time before our average hits 250 ms.  I did that and came up with 20 requests at once.

So we set up enough machines to handle the anticipated load with a maximum of 15 requests per box at one time, so we’d have some room to grow before we needed to add capacity.  All was well.

Until, that is, the day we launched and our keyword processor kept getting dumped on the floor by the system above us.

Something was obviously wrong.  We knew what our performance was.  At 20 concurrent requests, we had a 250 ms response time, at 15 requests, we has seen an average 200 ms response time.  We’re fast enough, and we’d proved that we were fast enough.  Statistics said so!

That right there was the problem.  We trusted the wrong information.  Sure, the average response time was 200 ms at the load we were seeing, but that said absolutely nothing about the something like 30% of the requests that were hitting the 250 ms timeout.  We frantically reran the performance tests.  The results were stunning.  While the average response time did not hit 250 ms until we reached the 20 concurrent request level, we saw a significant (and SLA violating) number of requests that took more than 250 ms by the time we reached the 10 concurrent request level.

People aren’t very happy when you tell them that a cluster size has to double…

At the time, I thought I might have just made a rookie mistake.  It was the first major system I’d done the performance testing for, and I’d had no training or mentoring.  I did what I thought was right and ended up getting burned.  Surely, I thought, everyone else knows what they’re doing.  Real performance testers using real performance testing tools will get it right.  Trouble is, I’ve since discovered that’s not the case.  Everyone else makes these mistakes and they don’t even realize that they’re making any mistakes.  And performance testing tools actively encourage testers to make these mistakes by not giving the tester the information that they really need and, in some cases, giving testers information that is just plain invalid.1

So…  What do you do about it?  For starters, don’t use the average in isolation.  Pull in some other measurement.  I like using the 95th percentile for a second opinion.2  The 95th percentile means that 95% of all requests take less than that amount of time.  That’s really more what you care about, anyway.  You’re probably not really concerned with the 5% that lie beyond that point, since they’re usually outliers or abberations in your performance anyway.  This will get rid of things like Mr. Lucky’s lottery winnings.   Additionally, you’re probably not really concerned with where the average response time lies.  People often use the results of performance testing to feed into capacity planning or SLAs.  When there are dollars on the line tied to the Five Nines, why do you care about a number that you think of as the middle of the road?  You care about the worst case, not the average case.  If we’d used the 95th or 99th percentile in our inital performance tests of that keyword processor, we would not have had the problem that we did.

But even so, the 95th percentile has its own set of issues and should also not be used in isolation.  It, too, does not tell you the complete story, and can easily hide important trends of the data.

There I go again with “the data”.  You have to look at the data.  In order to get the full picture of your system performance, you actually have to look at the full picture of your system performance.  You can’t go to single aggregate numbers here or there and call it good.

Of course, that leads to the obvious problem.  When you run a performance test, you’ll often end up running thousands upon thousands of iterations, collecting thousands upon thousands of data points.  You cannot wrap your head around that kind of data in the same way that you can see Mr. Lucky’s income or the daily chart of the temperature.

Well, not without help…

 Whenever I do performance testing now, I rely on a tool that I built that will produce several graphs from the results of a performance test run.  The simplest graph to produce is a histogram of response times.

NormalHistogram

A histogram will show you the distribution of the response times for the performance test.  At a glance, you can see the where the fastest and slowest responses lie, and get a sense for how the service behaves.  Are the response times consistent, with most of them around 150 ms, or are they spread out between 100 and 200 ms?  A histogram can also show you when things are acting strangely.  One piece of software I tested had most of the response times centered below 400 ms, but there was a secondary bump up between 500-900 ms.  That secondary bump indicated something was strange, perhaps there was a code path that took three times as long to execute that only some inputs would trigger, or there were random slowdowns due to garbage collection or page swapping or network hiccups.

AbnormalHistogram

As you can see, there are a sizeable number of responses in that bump, enough to make you want to investigate the cause.  This potential problem would have been completely invisible if you were only concerned with the average, and the full extent would not have been known if you’d been looking at the 95th percentile.3  However, it’s plainly visible that something strange is going on when you see the graph like that.

While helpful, simple histograms like this are not enough.  In particular, they fall down if anything changes during the test run.  If something like network latency slows down your test for a brief period of time, that will be invisible in this graph.  If you’re changing the number of users or the number of concurrent requests, then the times from all of them get squeezed together, rendering the graph invalid.  What you’re missing here is the time dimension.

One way to bring in the time dimension is to animate the histogram.  You produce multiple histograms, each representing a slice of time in your test.  That way, you can watch the behavior of your service change over the length of the performance test.  The problem I have with an animated histogram is that you can’t just glance at the information.  You have to watch the whole thing, which can be time consuming for a long running performance test.

Instead of animation, I prefer to visualize the time in this way:

 NormalDensityGraph

Going up the side, you have divisions for buckets of response times.  Going across, you have time slices.4   Essentially, this is what you’d see if you stacked a bunch of histograms together side by side, then looked at them from the top.  It’s basically like a heat map or a graph of the density of the response times.  The red zones have the most responses, while the green areas have the least.5  In the example above, you can see that there are a lot of responses in the 100 and 150 ms buckets and then it quickly trails off to zero.  There’s a lot of noise up to about 600 ms, and sporadic outliers above that.  The performance remains fairly stable throughout the test run.  All in all, this is a fairly standard graph for a well behaved piece of software.6

These density graphs aren’t terribly interesting when things are well behaved, though.  Here’s another graph I saw:

AbnormalDensityGraph1

First, there’s bands of slowness between about 210 and 280 and 380-450.  These bands would appear in a histogram like the secondary hill shown above.  But what a histogram isn’t going to show you is the apparent pattern in the 380-450 band.  It appears that there’s groups of slow responses in a slice of time, then none for the next couple of slices, then another group of slow responses, then none, and so on.  Seeing this kind of behavior can help you find the problem faster.  In this case, the slow responses may be caused by something else running on the box that’s scheduled to run at a regular interval, like an anti-virus scanner or a file indexer, or they can be caused by something like the garbage collector waking up to do a sweep on a somewhat regular basis.

Another benefit of a density graph is that they’re still useful, even if you change the parameters of the test during the test run.  For instance, a common practice in performance testing is to increase the load on a system during the run, in order to see how performance changes.

AbnormalDensityGraph2

In this example, the number of concurrent users was steadily increased over the run of the test.  In the first part of the test, you can see that increasing the number of users will directly influence the response time.  It also increases the variation of those response times:  For the first part, all of the responses came in within 100 ms of one another, but pretty quickly, they’re spread over a 300-400 ms range.  And then, during the final third of the test, everything went all kerflooey.  I don’t know what happened, exactly, but I know that something bad happened there.

I think this graph is one of my favorites:

AbnormalDensityGraph3

As you can see, this graph is distinctly trimodal.  There’s a steep, but well defined band shooting off the top, then a wide and expanding band in the middle, followed by a sharply narrow band with a shallow slope.  I like this graph because it doesn’t actually show anything that’s wrong.  What it illustrates is the huge impact that your test inputs can have on the results.  This test was run against a keyword index system.  The input I used was a bunch of random words or phrases.  Different words caused the keyword index system to do different things.  The shallow band at the bottom was created by the keywords for which the index system found no results.  When it didn’t find anything, the system simply returned immediately, making it very fast.  The middle band was filled with keywords that found a single result.  The top band was the set of keywords that found multiple results, which required extra processing to combine the two results into one before returning.

 

Performance tests are the same as any other test, your goal is to find problems with the software.  You’re not going to find them if you’re only looking at the average.  So, the next time you’re involved with a performance test, remember: Look deeper.  There are more problems to be found under the surface.

  1. I’ve seen several cases of odd numbers coming out of VS Perf Tests, but the one I’m specifically thinking of here is the fact that VS will still report an average response time, even when you’re ramping up the number of users.  The performance of your system when you have a single user is vastly different than when you have 100 users, so the single “Average Request Time” number that it will report is just plain useless. []
  2. You can get the 95th percentile in Visual Studio, if you tweak a setting.  Set “Timing Details Storage” to “Statistics Only” or “All Individual Details” and it’ll start being recorded. []
  3. Likely, the 95th percentile would lie in the middle of the bump, leading you to believe that the system performance is slow overall, not that there’s an anomaly. []
  4. Time isn’t marked in these examples because the tool I use to generate them will usually generate other graphs, as well, making it possible to correlate the time on that graph with the time on this graph.  For these examples, it’s not really necessary to know how much time each column is or how many requests are represented by each block. []
  5. And the white zones are for loading and unloading only.  There is no parking in the white zones. []
  6. For comparison, this graph is from the same test run that produced the “Average Response Time: 149 ms” histogram that I showed earlier. []

September 13, 2009   No Comments