Random header image... Refresh for more!

SWA in WPF and Silverlight

A couple of weeks ago, I wrote about using the System.Windows.Automation libraries in the .Net Framework to write UI Automation, primarily for use in writing automated test cases.  Well, that’s only half the story.  In order to truly unlock the power of SWA and the UIAutomation library, you’ll want to add SWA instrumentation to your application.  It’s not required for using SWA, but believe me, it will make your testers happier.

Microsoft has provided a way to add enhanced support for UIA into Win32 and Windows Forms apps, but WPF and Silverlight have been built from the ground up to have full and simple support for the UI Automation properies.  In this post, I’ll be exploring how to instrument a Silverlight application, largely because I can give a link to my finished sample app for you to explore.  Working with WPF should be mostly the same. 1

Let’s start by looking at what you get from Silverlight, straight out of the box.  We’ll begin with a simple text box and button, the sort of thing you’ll find in pretty much any UI program ever written.

<StackPanel x:Name="LayoutRoot" Orientation="Horizontal" Width="250" Height="32">
    <TextBox x:Name="TextEntryBox" Width="150"/>
    <Button x:Name="SubmitButton" Content="Submit"/>
</StackPanel>

That produces a super exciting display that looks like this:

SWASilverlightApp1

I has mad UX skillz.  Fear them.

If you open good old trusty UISpy and examine the control tree of your XAML masterpiece, you’ll find something that looks a bit like this:

SWASilverlightApp-UISpy1

An “edit” control and a “button” control in a Silverlight Control?  That looks an awful lot like what we just wrote up in XAML, doesn’t it?  That’s because it is.  Yep.  You write it and presto, it’s seen by SWA.  And if you look at the automation properties panel in UISpy, you’ll see that the edit control has an AutomationId with the value of “TextEntryBox” and that the button control has an AutomationId of “”SubmitButton”.  Those are also what we put in the x:Name attributes back in the XAML.  In other words, for Silverlight and WPF apps, most of your controls will automatically be exposed to UIAutomation using the programmatic names you’ve already given them in your code.  You get that for free.

But wait!  There’s more!

Do you see all those other properties in UISpy?  Well, if you don’t, here’s a screenshot for your enlightenment:

SWASilverlightApp-UISpy2

Well, you have direct access to set some of them straight from XAML.  Okay, so it’s not completely free, but it’s still really really easy.  They’re exposed through the AutomationProperties attribute.  As an example, let’s set the “Name” property of the text box in the sample.  If you look at the UISpy control tree, you’ll see that it’s called “”, instead of anything useful.  The automatic population of the “Name” property usually uses the content of a control.  In the case of the button, it has a text label of “Submit”, which is where it gets the name from.  But the text box doesn’t have any content, so it gets left out.  Let’s give it a name before it gets depressed.

<TextBox x:Name="TextEntryBox" Width="150" AutomationProperties.Name="Search Query"/>

All it needs is that “AutomationProperties.Name” attribute set and there you have it.  our edit box is now showing up with a name in UISpy and everyone’s happy.

SWASilverlightApp-UISpy3

Most of the time, I’ll stick to “Name” and “AutomationId” for helping me find elements for my automated tests.

For a good number of decently simple applications, that’s all you’ll need to know.  For that reason, most tutorials I’ve seen on the subject stop here and pretend that this much information is actually adequate for using SWA in the real world.  I, however, have actually done some work in the real world and know that this alone can be painful to work with.  So, I’m going to keep rambling for a bit.

Consider, for a moment, that you’ve assembled some combination of controls that is useful to you, and that you’d like to reuse that combination of controls and not have to copy/paste/rename.  Now, I understand that many of you will never attempt something as wild and complex as this, and that the thought of it alone might drive you to madness, but please bear with me, as it’s required for my next demonstration.  For the sake of conversation, let’s say that this useful combination of controls is a text box and a button to go with it.

So, you have a user control with a box and a button.  I’ve given it the bleedingly obvious name of “BoxAndButton”.  This code should look familar to you:

<UserControl x:Class="SWASilverlightApp.BoxAndButton"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Width="250" Height="32">
    <StackPanel x:Name="LayoutRoot" Orientation="Horizontal">
        <TextBox x:Name="TextEntryBox" Width="150" AutomationProperties.Name="Search Query"/>
        <Button x:Name="SubmitButton" Content="Submit"/>
    </StackPanel>
</UserControl>

Okay, now that we have the control, let’s use it somewhere.  In fact, it’s a reusable control, so let’s use it twice, just because we can!

<StackPanel x:Name="LayoutRoot">
    <MathPirate:BoxAndButton x:Name="SearchDatabase"/>
    <MathPirate:BoxAndButton x:Name="SearchWeb"/>
</StackPanel>

(Please note the “MathPirate” namespace that you’ll have to add to your XAML file, if you’re playing the home game.)

Now we run our app and see our two glorious text boxes and buttons.

SWASilverlightApp2

Now let’s go into UISpy and see what Silverlight or WPF has given us for free.

SWASilverlightApp-UISpy4

Now, this is the point in the application development cycle where you’ve made your UI tester cry.  (Or, if they’re anything like me, you’ve made them cry and shoot you with a fully automatic Nerf gun.) While all of the elements are there, everything is jumbled together.  Despite naming one of the controls as “SearchDatabase” and the other as “SearchWeb” and using a hierarchy of controls, SWA has decided that you really want a flat list of useless.  Anyone trying to automate against this isn’t going to know which button is which, and will therefore have to rely on ordinal positions of controls and a whole lot of luck.  If, for some reason, you decide that Web should come first and you go and swap the boxes, you’ll decimate every single test written against this app.  Database queries will go to the Web and Web queries will end up at the Database, and a quantum singularity will form that will destroy the entire universe.  So, let’s fix things, shall we?

I know!  I know!  AutomationProperties!

<StackPanel x:Name="LayoutRoot">
    <MathPirate:BoxAndButton x:Name="SearchDatabase" AutomationProperties.Name="Database"/>
    <MathPirate:BoxAndButton x:Name="SearchWeb" AutomationProperties.Name="Web"/>
</StackPanel>

Give it a run, and…

SWASilverlightApp-UISpy4

FAIL.

So, what gives?  You’re using a nicely hierarchical control tree, you’re giving things nice names and identifiers, why aren’t you getting anything useful from SWA?  The reason:  UIAutomation is primarily designed to expose interactive controls and elements, and your user control isn’t an interactive element in itself, only the text box and button inside it are.  It’s wrapped in a StackPanel, but things like StackPanels, Grids and Canvases in Silverlight and WPF are considered to be strictly for graphical layout and say nothing about the interaction with the system.  As a result, they don’t show up in the control tree. 2  And the BoxAndButton control itself doesn’t show up because you haven’t told SWA that it’s interesting yet.  Let’s do something about that.

The method that WPF and Silverlight use to expose these AutomationProperties is called an “AutomationPeer”.  AutomationPeer is an abstract class that has a bunch of methods for you to implement, like GetNameCore() and GetAutomationIdCore().  The idea is that these Get*Core are supposed to return the value of one of the Automation Properties for UIA to expose.  This gives you a lot of control over what gets seen by UI Automation clients or tools like UISpy.  If you like that level of control, then feel free to inherit directly from AutomationPeer and implement all thirty or so Automation Properties.  The rest of us, who don’t like tedium, will simply use the implementation provided by FrameworkElementAutomationPeer.

Of course, having an AutomationPeer isn’t going to help you much, unless you tell SWA to use it.  You do that by overriding the method OnCreateAutomationPeer() on your control class.  The method is defined on the UIElement base class, so every element in Silverlight already has it, you’ve just probably never noticed it before.  OnCreateAutomationPeer() just needs to return the appropriate AutomationPeer object that exposes your object and the Automation Properties and Control Patterns you want the world to know about.

Okay, that sounded scary and confusing, but it’s not really that bad when you see it in action.  So, let’s get to the code!  Open up the .cs file for your control.

First, add a using:

using System.Windows.Automation.Peers;

Then, override OnCreateAutomationPeer:

protected override AutomationPeer OnCreateAutomationPeer()
{
    return new FrameworkElementAutomationPeer(this);
}

And now let’s run our pretty application once more and see what UISpy has for us this time.

SWASilverlightApp-UISpy5

Boom-diggity, that’s what I’m talking about.

Now, you have a nice control tree, with your web and database search controls uniquely identified.  While your efforts may not have made the testers completely happy, at the very least, they should have stopped shooting you with the Nerf guns for a while.

But wait!  There’s more!

If you were paying attention, you may have noticed the mention of Control Patterns that I ever-so-slyly snuck in a few paragraphs back.  Control Patterns were the prime focus of the earlier article on using SWA and UIAutomation, but I haven’t really said anything about them here at all.  Why is that?  Because on the implementation side, you typically don’t have to care about them.  You’re using a text box or a button or a scroll bar or whatever that already implements all of the control patterns that it needs, for the most part.  That means you typically don’t have to do a damned thing to get the ValuePattern or InvokePattern to work, while the testers all have to suffer with that idiotic rehash of QueryInterface and COM.

Before you start gloating about that, you need to realize that I’m a tester and now I’m going to make you suffer, too.

All along, we’ve been using a Box and Button as two separate elements.  We’ve put them in a single control, though.  Why not make that single control appear as a single control to UIAutomation?  In other words, why don’t we make our custom control look like it is a text box and a button, rather than something that just contains a text box and button?3

So, somehow, we now have to implement the ValuePattern and the InvokePattern on our BoxAndButton control.  The way to do that, as mentioned before, is through the AutomationPeer.  First, we’ll have to create our own subclass of AutomationPeer.  Again, since I’m not crazy, I’m going to use FrameworkElementAutomationPeer as a base, but you don’t have to.

public class BoxAndButtonAutomationPeer : FrameworkElementAutomationPeer
{
    public BoxAndButtonAutomationPeer(BoxAndButton owner) : base(owner) { }
}

Be sure to change your OnCreateAutomationPeer in BoxAndButton to return one of these instead.  If you run this now, you should see everything act exactly the same as it was before you added this new class.

If you go inside the your new AutomationPeer class and type “public override “, you’ll conjure the magic Intellisense window that’ll show you the list of all those Get*Core() and other Automation Property methods I talked about earlier.

AutomationPeerMethods

You can implement any of them if you want to, but the implementation in FrameworkElementAutomationPeer is usually good enough.  Except for the method GetPattern…

GetPattern is the method that SWA calls on your AutomationPeer when someone using SWA calls “GetCurrentPattern” on an automation element representing your control.  You’ll need to implement it and return an object that implements whatever ControlPatterns you wish to support.  In our case, we want to implement ValueProvider for the Text Box and InvokeProvider for the Button.  The only parameter to GetPattern is a PatternInterface enum containing a value for each of the existing ControlPatterns, so a simple implementation is to switch on patternInterface and return the appropriate object.

public override object GetPattern(PatternInterface patternInterface)
{
    switch(patternInterface)
    {
        case PatternInterface.Value:
        return Owner;

        case PatternInterface.Invoke:
        return Owner;
    }

    return base.GetPattern(patternInterface);
}

If you run it right now and look at it with UISpy, you’ll see that your BoxAndButton control now is apparently supporting the Invoke Pattern and the Value Pattern, just like we want it to do.  Trouble is…  It doesn’t actually support anything.  You may have noticed that I was simply returning “Owner” from GetPattern and found that odd.  Owner is the UIElement we passed into the constructor for FrameworkElementAutomationPeer.  In other words, Owner is the BoxAndButton control itself.  But, we haven’t actually done anything to BoxAndButton to let it know that it’s supposed to support InvokePattern.  Let’s take care of that now.

Each of the control patterns is exposed through an interface. 4  For the Invoke Pattern, it’s IInvokeProvider.  For the Value Pattern, you’ll want to use IValueProvider.  These and other Provider interfaces for for the other patterns live in the System.Windows.Automation.Providers namespace, so be sure to add that.

IInvokePattern is an interface with a single method, called Invoke.  It takes no parameters and returns nothing.  The function is supposed to cause the control to perform some action.  In our case, we’re going to want it to act like the button was clicked.  Like so:

#region IInvokeProvider Members

void IInvokeProvider.Invoke()
{
    AutomationPeer peer = FrameworkElementAutomationPeer.CreatePeerForElement(SubmitButton);
    IInvokeProvider invokeProvider = (IInvokeProvider)peer.GetPattern(PatternInterface.Invoke);
    invokeProvider.Invoke();
}

#endregion

Okay, that was freaking ugly.  You see, Silverlight and WPF don’t really have a way to easily simulate a button click on the Button class.  So, this method is diving into the world of UIAutomation itself and using the InvokePattern that’s already available on the Button.  Basically, it’s a passthrough.  Making this function suck less is left as an exercise to the reader.

IValueProvider is somewhat more straightforward and sane.  It has three members.  IsReadOnly, a boolean property that returns true/false based on whether or not the control is read only.  Value, a string getter property that returns the value of the control.  And SetValue, a function that takes a string and sets the value of the control with it, because apparently the designers hadn’t heard that properties like “Value” can have both getters AND setters.

#region IValueProvider Members

bool IValueProvider.IsReadOnly
{
    get { return TextEntryBox.IsReadOnly; }
}

void IValueProvider.SetValue(string value)
{
    TextEntryBox.Text = value;
}

string IValueProvider.Value
{
    get { return TextEntryBox.Text; }
}

#endregion

 

 Conveniently, all three of those members on IValueProvider map directly to things we can easily do with a text box already.

Now, if you go into UISpy and play around, you’ll see that not only does the BoxAndButton control say that it supports the Value and Invoke patterns, but that you can now actually use them.  You can easily see the Value Pattern in action.  Simply put some text in the box, then look in UISpy, and you’ll see that the custom BoxAndButton now can see the text value.  To see that the InvokePattern is wired up properly, you’ll have to hook up an event handler on the Click event of the button.

Of course, you don’t have to do exactly what I did in my example.  You can use whatever you need and whatever you have available in the I*Provider implementations.  Do it the way that makes sense for you.   You don’t even have to implement these interfaces on the control class directly.  I did it because it was convenient in this case, but it might not make sense for you.  You can use any class you want, it’s entirely up to your design.  Just change what you return from GetPattern.  However, if you put the interfaces on the control class, like I’m doing, I would recommend explicitly implementing these interfaces, since it’s unlikely you’ll want to expose these methods to anyone using the class normally.

There’s just one little annoying thing left about the control.  If you look in UISpy, you’ll see that our BoxAndButton control now supports the Invoke and Value patterns, but we still have the Text Box and Button as children, and those also have the Value and Invoke patterns on them.  They’re not needed anymore.  Let’s axe them, shall we?

If you’ll look at the list of functions available to override on our AutomationPeer, you’ll see one called GetChildrenCore.  You can override that function to tell UIA what your children should be.

protected override List GetChildrenCore()
{
    return new List<AutomationPeer>();
}

Obviously, it’s more useful to tell the system that you have children that it wouldn’t ordinarily know about (Like in the case of an owner drawn control), than it is to disown the children that you do have.  So, when using this function in a normal situation, you’re probably going to add something to the list that’s being returned.

Now, if you run the app and look at it in UISpy, you’ll see that the children of the BoxAndButton control have been written out of the will.  You’ll also likely see that, in fact, UISpy is having trouble finding our control.

Element : Element details not available.
Name : TreeValidationException
Message : UI Automation tree navigation is broken. The parent of one of the descendants exist but the descendant is not the child of the parent
Stack Trace : at UISpy.Base.Tree.BuildTree(AutomationElement element)
    at UISpy.Base.Tree.BuildTreeAndRelatedTrees(AutomationElement element)

Well now…  That sucks.  See, what’s happening is that UISpy is finding the real Text Box and real Button controls that still exist and trying to select them, but it’s unable to find their parents in the tree.  In other words, we’ve tampered with the natural order of things and are making UISpy go all loopy.  Good luck with that, I’m gonna end on this high note.

You can find my sample application here (A version that’s not hOrking UISpy):  http://mathpirate.net/log/UploadedFiles/SWASilverlightApp1/SWASilverlightAppTestPage.html

You can get the source code out of SVN here:  http://www.mathpirate.net/svn/Projects/SWAExample/SWASilverlightApp/

  1. I haven’t done anything with SWA and Windows Forms or Win32 yet, so I don’t know what that entails, but from a quick glance at the docs, it’s a bit more complicated. []
  2. And honestly, you don’t want them to.  It would be insane to try to navigate through all the different panels that appear everywhere.  One of the Silverlight 2 Betas had all of the panels and things exposed and it was frightening. []
  3. Why don’t we?  Because it’s actually useless to do so.  A text box and a button are perfetly good controls to contain.  However, treat this as an exercise in exposing a completely custom owner drawn control to System.Windows.Automation. []
  4. If you’ll remember from the previous article, a large serving of wrath was thrown toward the designers of SWA for not using interfaces so finding out that they do, in fact, use interfaces on objects internally was a bit of a relief, but also upsetting.  I still can’t understand why the person responsible for the design on this side didn’t smack the designers of the consuming side around until they used interfaces, too.  And if it’s the same designer, then they need to smack themselves around until they make up their mind. []

October 10, 2009   No Comments

Oh, Thanks, WordPress!

It looks like WordPress is deciding that the Silverlight Video Player is obviously a Flash object, because it’s using an object tag. So, it’s being helpful and converting it to a Flash object for me.

Thank you very much!

However, it looks like there’s trouble with that video player even when I’m not getting second-guessed about its type.  Something seems to make it not want to appear when there’s more than one on the page, or in certain cases when there’s more than one on the page.  Suck.  Unfortunately, I’m building a robot that can play Atari this weekend, I’m not debugging Silverlight Video Player Embedding, so I can’t spend time on that right now to sort out these issues.  No more embedded videos, I guess, except for the YouTubery ones.  Sorry.

September 6, 2009   No Comments

Sweet. Silverlight Works With Real Videos.

It looks like videos from my video camera will work in a Silverlight player after all. The screen capture videos don’t work, but I think they’re using some crazy codec. Hopefully these will work for people and you won’t have to download the videos to watch them.

Robot Demonstration

Things Going Wrong

September 6, 2009   No Comments