Microsoft automation framework

Microsoft automation framework DEFAULT

UI Automation

Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop. It enables assistive technology products, such as screen readers, to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI.

Where Applicable

By using UI Automation and following accessible design practices, developers can make applications running on Windows more accessible to many people with vision, hearing, or motion disabilities. Also, UI Automation is specifically designed to provide robust functionality for automated testing scenarios.

Developer Audience

UI Automation is designed for experienced C/C++ developers. In general, developers need a moderate level of understanding about Component Object Model (COM) objects and interfaces, Unicode, and Windows API programming.

For information about UI Automation for managed code, please see Accessibility in the .NET Framework Developer's Guide section of MSDN.

Run-Time Requirements

UI Automation is supported on the following operating systems: Windows XP, Windows Server 2003, Windows Server 2003 R2, Windows Vista, Windows 7, Windows 10, Windows Server 2008, Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2, Windows Server 2016, and Windows Server 2019.


Windows XP and Windows Server 2003 also require Microsoft .NET Framework 3.0.

Support for Down-level Operating Systems

The Platform Update for Windows Vista is a set of run-time libraries that enables developers to target applications to both Windows 7 and down-level operating systems. The Platform Update for Windows Server 2008 is a set of run-time libraries that enables developers to target applications to both Windows Server 2008 R2 and previous versions of Windows Server. The Platform Update for Windows Vista and the Platform Update for Windows Server 2008 will be available to all Windows Vista and Windows Server 2008 customers through Windows Update. Third-party applications that require Platform Update for Windows Vista or Platform Update for Windows Server 2008 can have Windows Update detect whether it is installed; if it is not, Windows Update will download and install it in the background.

The Platform Update for Windows Vista and the Platform Update for Windows Server 2008 both support the entire Windows Automation API 3.0 feature set on the following operating systems.

  • Windows XP (English)
    Windows XP Home SP3 x86
    Windows XP Professional SP3 x86
  • Windows Server 2003 (English)
    Windows Server 2003 SP2 (x86 and x64)
  • Windows Vista (English)
    Starter SP2 (x86 and x64)
    Home Premium SP2 (x86 and x64)
    Business SP2 (x86 and x64)
    Enterprise SP2 (x86 and x64)
    Ultimate SP2 (x86 and x64)
  • Windows Server 2008 (English)
    Windows Server 2008 SP2 (x86 and x64)

For more information about both updates, see Platform Update for Windows Vista.

In this section

Information in this topic applies to desktop and web applications.

Microsoft .NET Framework v.3.0 and later includes a special UI Automation Framework. One of its purposes is to assist in application testing by providing program access to user interface elements. TestComplete includes a special plugin, UI Automation Support, that provides support for the UI Automation technology. For more information on the plugin, see Requirements.

If the plugin is installed, TestComplete uses the UI Automation providers to obtain information on UI elements of the application under test. UI Automation Framework includes special providers that expose properties of buttons, check boxes, menus and menu items, data grids, list boxes and other controls. For more information, please see Windows SDK documentation.

Currently, the plugin only supports properties exposed by standard providers. That is, if your application implements a custom provider exposing specific properties, they will not be available to TestComplete.

Note:The UI Automation Support plugin only exposes properties of those windows whose class names are specified by the UI Automation project properties. So, in order to work with windows via UI Automation, you may need to use these properties to select the desired window classes.

Which properties are retrieved by TestComplete depends on the control type. All retrieved properties are displayed in the Object Browser’s Properties page:

Object Browser displays properties exposed by the UI Automation providers

You can then use these properties for testing.

If a property name coincides with the name of a property, method or action that is provided by TestComplete, this automation’s property is moved to the NativeUIAObject namespace.

If TestComplete retrieves the property, then it uses the method to address the object (this method is added by the UI Automation Support plugin for the TestComplete and objects).

Note that the name must be a valid identifier, that is, it must start with a letter and contain no spaces or other disallowed characters. If the name contains these type of characters, TestComplete replaces them with underscores. However, in some cases even replacements cannot make the name a valid script identifier.

So, if TestComplete is unable to retrieve the property or if it cannot make the name a valid identifier, TestComplete addresses the object in the usual way, that is, by class name, caption and index (that is, ). Other properties that were exposed by UI Automation providers are still displayed in the Object Browser.

See Also

Using Microsoft UI Automation Technology
UIAObject Method

  1. Younger full episode
  2. What is mugetsu
  3. Fivem bcso pack
  4. Pennsylvania trapping regulations

Using UI Automation for Automated Testing

This overview describes how Microsoft UI Automation can be useful as a framework for programmatic access in automated testing scenarios.

UI Automation provides a unified object model that enables all user interface (UI) frameworks to expose complex and rich functionality in an accessible and easily automated manner.

UI Automation was developed as a successor to Microsoft Active Accessibility. Active Accessibility is an existing framework designed to provide a solution for making controls and applications accessible. Active Accessibility was not designed with test automation in mind even though it evolved into that role due to the very similar requirements of accessibility and automation. UI Automation, in addition to providing more refined solutions for accessibility, is also specifically designed to provide robust functionality for automated testing. For example, Active Accessibility relies on a single interface to both expose information about the UI and collect the information needed by AT products; UI Automation separates the two models.

Both a provider and client are required to implement UI Automation for it to be useful as an automated test tool. UI Automation providers are applications such as Microsoft Word, Excel, and other third-party applications or controls based on the Microsoft Windows operating system. UI Automation clients include automated test scripts and assistive technology applications.


The intent of this overview is to showcase the new and improved automated testing capabilities of UI Automation. This overview is not intended to provide information on accessibility features and will not address accessibility other than where necessary.

UI Automation in a Provider

For a UI to be automated, a developer of an application or control must look at what actions an end-user can perform on the UI object using standard keyboard and mouse interaction.

Once these key actions have been identified, the corresponding UI Automation control patterns (that is, the control patterns that mirror the functionality and behavior of the UI element) should be implemented on the control. For example, user interaction with a combo box control (such as the run dialog) typically involves expanding and collapsing the combo box to hide or display a list of items, selecting an item from that list, or adding a new value via keyboard input.


With other accessibility models, developers must gather information directly from individual buttons, menus, or other controls. Unfortunately, every control type comes in dozens of minor variations. In other words, even though ten variations of a pushbutton may all work the same way and perform the same function, they must all be treated as unique controls. There is no way to know that these controls are functionally equivalent. Control patterns were developed to represent these common control behaviors. For more information, see UI Automation Control Patterns Overview.

Implementing UI Automation

As mentioned earlier, without the unified model provided by UI Automation, test tools and developers are required to know framework-specific information in order to expose properties and behaviors of controls in that framework. Since there can be several different UI frameworks present at any single time within Windows operating systems, including Win32, Windows Forms, and Windows Presentation Foundation (WPF), it can be a daunting task to test multiple applications with controls that seem similar. For example, the following table outlines the framework-specific property names required to retrieve the name (or text) associated with a button control and shows the single equivalent UI Automation property.

UI Automation Control TypeUI FrameworkFramework Specific PropertyUI Automation Property
ButtonWindows Presentation FoundationContentNameProperty

UI Automation providers are responsible for mapping the framework-specific properties of their controls to the equivalent UI Automation properties.

Information on implementing UI Automation in a provider can be found at UI Automation Providers for Managed Code. Information on implementing control patterns is available at UI Automation Control Patterns and UI Automation Text Pattern.

UI Automation in a Client

The goal of many automated test tools and scenarios is the consistent and repeatable manipulation of the user interface. This can involve unit testing specific controls through to the recording and playback of test scripts that iterate through a series of generic actions on a group of controls.

A complication that arises from automated applications is the difficulty synchronizing a test with a dynamic target. For example, a list box control, such as one contained in the Windows Task Manager, that displays a list of currently running applications. Since the items in the list box are dynamically updated outside the control of the test application, attempting to repeat the selection of a specific item in the list box with any consistency is impossible. Similar issues can also arise when attempting to repeat simple focus changes in a UI that is outside the control of the test application.

Programmatic Access

Programmatic access provides the ability to imitate, through code, any interaction and experience exposed by traditional mouse and keyboard input. UI Automation enables programmatic access through five components:

Key Properties for Test Automation

The ability to uniquely identify and subsequently locate any control within the UI provides the basis for automated test applications to operate on that UI. There are several Microsoft UI Automation properties used by clients and providers that assist in this.


Uniquely identifies an automation element from its siblings. AutomationIdProperty is not localized, unlike a property such as NameProperty that is typically localized if a product gets shipped in multiple languages. See Use the AutomationID Property.


AutomationIdProperty does not guarantee a unique identity throughout the automation tree. For example, an application may contain a menu control with multiple top-level menu items that, in turn, have multiple child menu items. These secondary menu items may be identified by a generic scheme such as "Item1, Item 2, Item3, etc.", allowing duplicate identifiers for children across top-level menu items.


Identifies the type of control represented by an automation element. Significant information can be inferred from knowledge of the control type. See UI Automation Control Types Overview.


This is a text string that identifies or explains a control. NameProperty should be used with caution since it can be localized. See UI Automation Properties Overview.

Implementing UI Automation in a Test Application

Add the UI Automation References.The UI Automation dll's necessary for UI Automation clients are listed here.

- UIAutomationClient.dll provides access to the UI Automation client-side APIs.
- UIAutomationClientSideProvider.dll provides the ability to automate Win32 controls. See UI Automation Support for Standard Controls.
- UIAutomationTypes.dll provides access to the specific types defined in UI Automation.
Add the System.Windows.Automation namespace.This namespace contains everything UI Automation clients need to use the capabilities of UI Automation except text handling.
Add the System.Windows.Automation.Text namespace.This namespace contains everything a UI Automation clients need to use the capabilities of UI Automation text handling.
Find controls of interest.Automated test scripts locate UI Automation elements that represent controls of interest within the automation tree.

There are multiple ways to obtain UI Automation elements with code.

- Query the UI using a Condition statement. This is typically where the language-neutral AutomationIdProperty is used. Note: An AutomationIdProperty can be obtained using a tool such as Inspect.exe that is able to itemize the UI Automation properties of a control.

- Use the TreeWalker class to traverse the entire UI Automation tree or a subset thereof.
- Track focus.
- Use the hWnd of the control.
- Use screen location, such as the location of the mouse cursor.

See Obtaining UI Automation Elements
Obtain Control Patterns.Control patterns expose common behaviors for functionally similar controls.

After locating the controls that require testing, automated test scripts obtain the control patterns of interest from those UI Automation elements. For example, the InvokePattern control pattern for typical button functionality or the WindowPattern control pattern for window functionality.

See UI Automation Control Patterns Overview.
Automate the UI.Automated test scripts can now control any UI of interest from a UI framework using the information and functionality exposed by the UI Automation control patterns.

Related Tools and Technologies

There are a number of related tools and technologies that support automated testing with UI Automation.

  • Inspect.exe is a graphical user interface (GUI) application that can be used to gather UI Automation information for both provider and client development and debugging. Inspect.exe is included in the Windows SDK.

  • MSAABridge exposes UI Automation information to Active Accessibility clients. The primary goal of bridging UI Automation to Active Accessibility is to allow existing Active Accessibility clients the ability to interact with any framework that has implemented UI Automation.


For security information, see UI Automation Security Overview.

See also

How to use Microsoft Power Automate - Tutorial for Beginners

Microsoft UI Automation

Microsoft UI Automation (UIA) is an application programming interface (API) that allows one to access, identify, and manipulate the user interface (UI) elements of another application.[1][2]

UIA is targeted at providing UI accessibility and it is a successor to Microsoft Active Accessibility. It also facilitates GUI test automation, and it is the engine upon which many test automation tools are based. RPA tools also use it to automate applications in business processes.

UIA's property providers support both Win32 and .NET programs.

The latest specification of UIA is found as part of the Microsoft UI Automation Community Promise Specification. Microsoft claims that portability to platforms other than Microsoft Windows was one of its design goals. It has since been ported to Mono.[3]


In 2005, Microsoft released UIA as a successor to MSAA framework.

Managed UI Automation API was released as a part of .NET Framework 3.0. The native UI Automation API (provider) is included as part of the Windows Vista and Windows Server 2008 SDK and is also distributed with the .NET Framework.

UIA is available out of the box in Windows 7 as a part of Windows Automation API 3.0 and as a separate download for Windows XP, Windows Vista, and Windows Server 2003 and 2008.[4]

Motivation and goals[edit]

As a successor to MSAA, UIA aims to address the following goals:

  • Enable efficient client performance without forcing clients to hook into a target application’s process.
  • Expose more information about the UI.
  • Co-exist with and use MSAA, but do not inherit problems that exist in MSAA.
  • Provide an alternative to MSAA that is simple to implement.

Technical overview[edit]


At client side, UIA provides a .NET interface in assembly and a COM interface implemented directly in .

At server side, is injected into all or selected processes on the current desktop to perform data retrieval on behalf of a client. The DLL can also load UIA plugins (called providers) into its host process to extract data using different techniques.

UIA has four main provider and client components, as shown in the following table.

Component Description
UIAutomationCore (UIAutomationCore.dll and dependents) The underlying code (sometimes called the UIA core) that handles communication between providers and clients. UI Automation Core also offers the provider and client API interfaces for unmanaged applications and clients; unmanaged applications (either clients or providers) do not require the managed assemblies listed below.
Managed Provider API (UIAutomationProvider.dll and dependents)

A set of interface definitions and functions that are implemented by managed UIA provider applications. Providers are objects that provide information about UI elements and respond to programmatic input.

Managed Client API (UIAutomationClient.dll and dependents) A set of interface definitions and functions for managed UIA client applications.
UIAutomationClientsideProviders.dll A set of UIA provider implementations for legacy Win32 controls and MSAA applications. This client-side provider is available to managed client applications by default.


UIA exposes every piece of the UI to client applications as an Automation Element. Elements are contained in a tree structure, with the desktop as the root element.

Automation Element objects expose common properties of the UI elements they represent. One of these properties is the control type, which defines its basic appearance and functionality as a single recognizable entity (e.g., a button or check box).

In addition, elements expose control patterns that provide properties specific to their control types. Control patterns also expose methods that enable clients to get further information about the element and to provide input.

Clients can filter the raw view of the tree as a control view or a content view. Applications can also create custom views.


Within the UIA tree there is a root element that represents the current desktop and whose child elements represent application windows. Each of these child elements may contain elements representing pieces of UI such as menus, buttons, toolbars, and list boxes. These elements, in turn, can contain other elements, such as list items.

The UIA tree is not a fixed structure and is seldom seen in its totality because it might contain thousands of elements. Parts of the tree are built as they are needed, and the tree can undergo changes as elements are added, moved, or removed.

Control types[edit]

UIA control types are well-known identifiers that can be used to indicate what kind of control a particular element represents, such as a combo box or a button.

Having a well-known identifier allows assistive technology (AT) devices to more easily determine what types of controls are available in the user interface (UI) and how to interact with the controls. A human-readable representation of the UIA control type information is available as a property, which can be customizable by control or application developers.

Control patterns[edit]

Control patterns provide a way to categorize and expose a control's functionality independent of the control type or the appearance of the control.

UIA uses control patterns to represent common control behaviors. For example, the control pattern is used for controls that can be invoked (such as buttons) and the control pattern is used for controls that are scrollable viewports (such as list boxes, list views, or combo boxes). Because each control pattern represents a separate functionality, they can be combined to describe the full set of functionality supported by a particular control.


UIA providers expose properties on UIA elements and the control patterns. These properties enable UIA client applications to discover information about pieces of the user interface (UI), especially controls, including both static and dynamic data.


UIA event notification is a key feature for assistive technologies (AT) such as screen readers and screen magnifiers. These UIA clients track events that are raised by UIA providers that occur within the UIA, and use the information to notify end users.

Efficiency is improved by allowing provider applications to raise events selectively, depending on whether any clients are subscribed to those events, or not at all, if no clients are listening for any events.


UIA exposes the textual content, including format and style attributes, of text controls in UIA-supported platforms. These controls include, but are not limited to, the Microsoft .NET Framework and as well as their Win32 equivalents.

Exposing the textual content of a control is accomplished through the use of the control pattern, which represents the contents of a text container as a text stream. In turn, requires the support of the class to expose format and style attributes. supports by representing a contiguous text span in a text container with the and endpoints. Multiple or disjoint text spans can be represented by more than one objects. supports functionality such as clone, selection, comparison, retrieval and traversal.

UI Automation for automated testing[edit]

UIA can also be useful as a framework for programmatic access in automated testing scenarios. In addition to providing more refined solutions for accessibility, it is also specifically designed to provide robust functionality for automated testing.

Programmatic access provides the ability to imitate, through code, any interaction and experience exposed by traditional user interactions. UIA enables programmatic access through five components:

  • The UIA tree facilitates navigation through the logical structure of the UI for accessibility and automation.
  • UI Automation Elements are individual components in the UI.
  • UI Automation Properties provide specific information about UI elements or the Control Pattern.
  • UI Automation Control Patterns define a particular aspect of a control's functionality or feature; they can consist of property, method, event, and structure information.
  • UI Automation Events provide a trigger to respond to changes and notifications in UIA information.


UIA was initially available on Windows Vista and Windows Server 2008, and it was also made available to Windows XP and Windows Server 2003 as part of .NET Framework 3.0. It has been integrated with all subsequent Windows versions, up to and including Windows 7.[5]

Besides Windows platforms, the Olive project (which is a set of add-on libraries for the Mono core aiming for the .NET Framework support) includes a subset of WPF ( and ) and UI Automation.[6]

Novell's Mono Accessibility project is an implementation of the UIA Provider and Client specifications targeted for the Mono framework. Additionally, the project provides a bridge to the Accessibility Toolkit (ATK) for Linux assistive technologies (ATs). Novell is also working on a bridge for UIA-based ATs to interact with applications that implement ATK.[7]

Related technology and interoperability[edit]

  • Microsoft Active Accessibility (MSAA): UIA is the successor to MSAA. However, since there are still MSAA based applications in existence, bridges are used to allow communication between UIA and MSAA applications. So information can be shared between the two APIs, an MSAA-to-UIA Proxy and UIA-to-MSAA Bridge were developed. The former is a component that consumes MSAA information and makes it available through the UIA client API. The latter enables client applications using MSAA access applications that implement UIA.[8]
  • Accessible Rich Internet Applications(ARIA): The UIA and properties can provide access to the ARIA attribute values corresponding to an HTML element (which can be exposed as an automation element by web browsers). General mapping from ARIA attributes to UIA is also available.[3]
  • Windows Automation API: Starting with Windows 7, Microsoft is packaging its accessibility technologies under a framework called Windows Automation API. Both MSAA and UIA will be part of this framework. For older versions of Windows see KB971513.[9]
  • Mono Accessibility Project: On November 7, 2007, Microsoft and Novell Inc., after completion of a year of their interoperability agreement, announced that they would be extending their agreement to include accessibility.[10][11] Specifically, it was announced that Novell would develop an open source adapter allowing the UIA framework to work with existing Linux accessibility projects such as the Linux Accessibility Toolkit (ATK), which ships with SUSELinux Enterprise Desktop, Red HatEnterprise Linux and Ubuntu Linux. This would eventually make UIA cross-platform.


  1. ^Darryl K. Taft: Microsoft Promotes Cross-Platform Accessibility Tech, EWeek (2005-11-28), accessed 2007-02-07.
  2. ^Microsoft: Microsoft's New Accessibility Model To Be Offered as Cross-Platform Solution for Industry, accessed 2007-02-06.
  3. ^ abMicrosoft Developer Network: UI Automation Specification and Community Promise
  4. ^Description of the Windows Automation API
  5. ^Microsoft: UI Automation Overview, accessed 2007-02-07.
  6. ^Mono: Olive.
  7. ^Miguel de Icaza and Philippe Cohen: Mono, Mainsoft and Cross-Platform Enterprise DevelopmentArchived 2008-06-17 at the Wayback Machine, Enterprise Open Source Magazine (2007-01-14), accessed 2007-02-07.
  8. ^Microsoft Developer Network (MSDN): Microsoft, UI Automation and Microsoft Active Accessibility, accessed 2007-02-07.
  9. ^KB971513: Windows Automation API download
  10. ^Microsoft: Microsoft and Novell Celebrate Year of Interoperability, Expand Collaboration Agreement.
  11. ^Mono Accessibility Project Homepage.


External links[edit]


Framework microsoft automation

Microsoft UI Automation

Microsoft UI Automation (UI Automation)is a framework that enables you to access, identify, and manipulate UI elements of any application by providing programmatic access to these user interface elements. When testing against Windows-based applications that have implemented UI Automation provider interfaces, you can use UI Automation to improve the object recognition for the controls in these applications. In this Help, we will refer to such controls as UI Automation controls.

Note:Silk Test Workbench supports testing Windows-based applications that have implemented UI Automation on machines with Microsoft Windows 8 or later.

UI Automation provides fallback support for applications that are based on the following technologies:
  • Win32
  • WPF
  • WinForms
  • Oracle JavaFX
  • QT
  • PowerBuilder
  • Delphi
  • Microsoft Office

For example, if you cannot record a test against your application because Silk Test Workbench cannot recognize the objects in the application or because Silk Test Workbench recognizes all objects in the application as Control, you could try to enable the UI Automation support.

To enable the UI Automation support during recording, stop recording, enable the option Microsoft UI Automation, and resume recording. For additional information, see Setting UI Automation Options.

Note: The UI Automation support overrides the standard technology-domain-specific support. When you are finished interacting with the controls that require UI Automation support, disable the UI Automation support again to resume working with standard controls.

Note: If you are testing against a Java FX application, you do not have to enable the UI Automation support, as Silk Test Workbench enables this out-of-the-box for Java FX applications.

To ensure that the methods supported for a UI Automation control cover the corresponding controls in all supported technologies, the Silk Test Workbench API supports only a subset of the methods and properties available for these controls. To call additional methods and properties that are not available in the Silk Test Workbench API for a control, use dynamic invoke.

How to use the NEW Power Automate app in Microsoft Teams

UI Automation Overview

Microsoft UI Automation is the new accessibility framework for Microsoft Windows, available on all operating systems that support Windows Presentation Foundation (WPF).

UI Automation provides programmatic access to most user interface (UI) elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI.


UI Automation does not enable communication between processes started by different users through the Run as command.

UI Automation client applications can be written with the assurance that they will work on multiple frameworks. The UI Automation core masks any differences in the frameworks that underlie various pieces of UI. For example, the property of a WPF button, the property of a Win32 button, and the property of an HTML image are all mapped to a single property, Name, in the UI Automation view.

UI Automation provides full functionality on supported Windows operating systems running the .NET Framework (see .NET Framework system requirements or versions of .NET Core starting with .NET Core 3.0.

UI Automation providers offer some support for Microsoft Active Accessibility client applications through a built-in bridging service.

Providers and Clients

UI Automation has four main components, as shown in the following table.

Provider API (UIAutomationProvider.dll and UIAutomationTypes.dll)A set of interface definitions that are implemented by UI Automation providers, objects that provide information about UI elements and respond to programmatic input.
Client API (UIAutomationClient.dll and UIAutomationTypes.dll)A set of types for managed code that enables UI Automation client applications to obtain information about the UI and to send input to controls.
UiAutomationCore.dllThe underlying code (sometimes called the UI Automation core) that handles communication between providers and clients.
UIAutomationClientsideProviders.dllA set of UI Automation providers for standard legacy controls. (WPF controls have native support for UI Automation.) This support is automatically available to client applications.

From the software developer's perspective, there are two ways of using UI Automation: to create support for custom controls (using the provider API), and creating applications that use the UI Automation core to communicate with UI elements (using the client API). Depending on your focus, you should refer to different parts of the documentation. You can learn more about the concepts and gain practical how-to knowledge in the following sections.

The following table lists UI Automation namespaces, the DLLs that contain them, and the audience that uses them.

UI Automation Model

UI Automation exposes every piece of the UI to client applications as an AutomationElement. Elements are contained in a tree structure, with the desktop as the root element. Clients can filter the raw view of the tree as a control view or a content view. Applications can also create custom views.

AutomationElement objects expose common properties of the UI elements they represent. One of these properties is the control type, which defines its basic appearance and functionality as a single recognizable entity: for example, a button or check box.

In addition, elements expose control patterns that provide properties specific to their control types. Control patterns also expose methods that enable clients to get further information about the element and to provide input.


There is not a one-to-one correspondence between control types and control patterns. A control pattern may be supported by multiple control types, and a control may support multiple control patterns, each of which exposes different aspects of its behavior. For example, a combo box has at least two control patterns: one that represents its ability to expand and collapse, and another that represents the selection mechanism. For specifics, see UI Automation Control Types.

UI Automation also provides information to client applications through events. Unlike WinEvents, UI Automation events are not based on a broadcast mechanism. UI Automation clients register for specific event notifications and can request that specific UI Automation properties and control pattern information be passed into their event handlers. In addition, a UI Automation event contains a reference to the element that raised it. Providers can improve performance by raising events selectively, depending on whether any clients are listening.

See also


You will also like:


UI automation is a programmatic interface to the user interface of your application to external applications that may be interested in programmatic communication with your UI. In fact, UI automation is the prime facilitator for accessibility features in Windows where external applications such as screen readers that have no clue about your application, can still interact with your application easily.

I was acquainted with UI automation during my tenure at Microsoft in 2005 when I started working as software developer in test within Visual Studio Team Architects team on a project called Whitehorse. For those who need some background, Whitehorse consisted of SOA diagrams within Visual Studio (you can learn more about this project in the MSDN magazine here). Testing of visual designers in the product test team was done entirely using an internal UI test automation framework built on top of Microsoft Active Accessibility framework.

The UI automation framework is relatively new to Windows platform and successor to Windows Active Accessibility. The framework provides a unified way to discover UI controls in an application built using any of the following desktop technologies: Win32, Windows Forms or WPF.

UI automation has several benefits when used in applications:

  • Accessibility features – UI automation provides accessibility support in your application. Accessibility support becomes even more important when there are matter of legal compliance in the domain your application is used in (e.g. Government regulations).
  • Automated UI testing – UI automation can automate testing for application UI saving time and costs associated with manual and regression testing. Moreover given a set of use cases, UI automation can be used to verify application behavior via scenario tests.
  • Custom controls – If you are authoring custom controls, UI automation support enables end clients of your control to automate your control in their application UI.

How Does Automation Work?

UI automation provides a common protocol for exchanging information between your application and an external entity such as screen reader or automated UI tests. UI automation provides an API using which an external application can discover controls, navigate the visual tree, access the state of the controls and perform control specific actions.

In WPF, automation object model is provided via instance associated with a (said to be “automation peer” of the control). If you are authoring a custom control, you may want to implement one or more interfaces defined in the UIAutomationProvider.dll under the namespace to support UI automation.

To support UI automation, a control author needs to implement an class from class’ virtual method . is then used at runtime to extract for the . It is important to note that the standard controls in WPF have standard implementation of associated with the controls out of the box. In authoring a derivative of , you may want to subclass a standard implementation rather than deriving directly from and authoring a class from scratch.

The following image shows out of the box derivatives of :


Each must implement one or more “standard control patterns” to expose capabilities that can be used by automation clients such as screen readers. A control may expose one or more control patterns defined by the members of enumeration:


A control can be queried for a specific via ’s method that takes member as a parameter and returns the supported pattern – a derivative of or if the specified value does not correspond to a supported pattern.


has several out of the box derivatives that target standard control patterns:


For example , and controls support and can therefore indicate to the automation clients that control is capable of invoking a command. Similarly an expander control supports to indicate that control is capable of expanding or collapsing content within it.

Control patterns are strictly limited and custom control patterns are not supported. The reason is that automation clients must work on a standardized protocol and cannot interpret custom patterns to figure out the functionality of the controls.

Since UI automation is optional in WPF, you may tag on automation properties on a control via attached properties specified in the class:


For example, if we want to specify automation id on a text box, we could specify it as follows in XAML:

<TextBoxText="{Binding Telephone}"AutomationProperties.Id="ID_TXT_TELEPHONE"/>

Navigating the Automation Tree in WPF

Similar to visual tree navigation, one can navigate UI automation tree in WPF using the instance. However, there is an important distinction – unlike the visual tree, an automation tree is not constructed upfront and is only constructed during automation tree navigation. This is so because unlike visual tree, UI automation is optional in applications. There is more than one way to navigate the automation tree depending on the view of automation tree you would want:

  • Raw view – Raw view refers to non-filtered complete automation tree view. Using class’ property raw view can be navigated.
  • Control view – A control view is a subset of raw view that only contains s that correspond to controls (or in other words have their property set). Using class’ property control view can be navigated.
  • Content view – A content view is a subset of control view that only contains that correspond to elements that represent information in terms of content to the end user (or in other words have their property set). Using class’ property content view can be navigated.
  • Custom view – Using an and a condition (which can be one of , , or ), one can navigate automation tree in a custom manner. To navigate a custom view, a call to the or methods of is made with a tree scope (enumeration specifying the levels in terms of , , , , and or any combination of these flags) and a condition.

The root of the automation tree is represented by property refers to the user’s desktop. One can either navigate an automation tree via or quite often more efficiently from the corresponding to the application window which can be obtained via method by passing in the handle to the application window.

A Real World Example - Automating Windows Calculator

As a real world example, let’s author test cases that use UI automation to test Windows calculator. The aim of the tests is to test the following aspects of calculator:

  • Data entry verification – Validates that a number being typed appears correctly in the calculator
  • Editing options – Validates that copy and paste works as expected in the calculator
  • Calculation validation – Validates given an expression tree, calculator calculates the expression correctly.

Calculator Class

We will start the example by modelling class which will be our interface to Windows calculator. The calculator class shall fire up an instance of Windows calculator upon construction and shall provide methods to manipulate the calculator. Also, this class shall implement interface and shall dispose the calculator process upon test conclusion. Clean calculator instance for every test ensures no d.

Following is the constructor of class:

public Calculator() { _calculatorProcess = Process.Start("Calc.exe"); int ct = 0; do { _calculatorAutomationElement = AutomationElement.RootElement.FindFirst (TreeScope.Children, new PropertyCondition(AutomationElement.NameProperty, "Calculator")); ++ct; Thread.Sleep(100); } while (_calculatorAutomationElement == null && ct < 50); if (_calculatorAutomationElement == null) { thrownew InvalidOperationException("Calculator must be running"); } _resultTextBoxAutomationElement = _calculatorAutomationElement.FindFirst (TreeScope.Descendants, new PropertyCondition (AutomationElement.AutomationIdProperty, "150")); if (_resultTextBoxAutomationElement == null) { thrownew InvalidOperationException("Could not find result box"); } GetInvokePattern(GetFunctionButton(Functions.Clear)).Invoke(); }

The code fires up calculator process and waits for automation element to become available. Then it proceeds to discovering and initializing automation elements that will be used to interact with the calculator. For example, to obtain access to calculator result text box, we do a lookup of automation id of 150 within the main calculator automation element. Automation Ids of elements were obtained using Spy++ utility that ships with Visual Studio by following the steps below:

  • Fire up a calculator instance and then fire up Spy++.
  • Type any number in calculator (I typed 1234) to identify it within Spy++.
  • Press Ctrl + F3 in Spy++ and from the search window drag the cross bow over to the calculator instance. This will highlight the calculator process in Spy ++.
  • Find the window instance corresponding to the pressed number and click to select it.
  • Right click on the selected window and select Properties menu item to fire up the properties window
  • Look up the control Id in the Property Inspector window. This is the automation id for the control in Hex. Convert it into decimal before using it in code. In my case, a hex value of 96 corresponds to 150 in decimal. This is how I got 150 !


method simply terminates the process:

publicvoid Dispose() { _calculatorProcess.CloseMainWindow(); _calculatorProcess.Dispose(); }

To obtain to invoke buttons, we use a utility method for a specified :

public InvokePattern GetInvokePattern(AutomationElement element) { return element.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern; }

for a function button can be retrieved via method that takes in for the function (e.g. “”) as its input:

public AutomationElement GetFunctionButton(string functionName) { AutomationElement functionButton = _calculatorAutomationElement.FindFirst (TreeScope.Descendants, new PropertyCondition (AutomationElement.NameProperty, functionName)); if (functionButton == null) { thrownew InvalidOperationException("No function button found with name: " + functionName); } return functionButton; }

All the function names are defined within the class:

publicclass Functions { publicconststring MemoryClear = "Memory clear"; publicconststring Backspace = "Backspace"; publicconststring MemoryRecall = "Memory recall"; publicconststring ClearEntry = "Clear entry"; publicconststring MemoryStore = "Memory store"; publicconststring Clear = "Clear"; publicconststring DecimalSeparator = "Decimal separator"; publicconststring MemoryAdd = "Memory add"; publicconststring MemoryRemove = "Memory subtract"; publicconststring Equals = "Equals"; }

Similarly, we query for a digit button using method that takes digit as an input and returns the associated :

public AutomationElement GetDigitButton(int number) { if ((number < 0) || (number > 9)) { thrownew InvalidOperationException("number must be a digit 0-9"); } AutomationElement buttonElement = _calculatorAutomationElement.FindFirst (TreeScope.Descendants, new PropertyCondition (AutomationElement.NameProperty, number.ToString())); if (buttonElement == null) { thrownew InvalidOperationException ("Could not find button corresponding to digit" + number); } return buttonElement; }

One thing to note is the use of in querying for the element. This property cannot be seen via Spy++ and I had to open the inspect the object in debugger to find it out (I used to load an element and queried it in the debugger to find the name property).

The result of calculator can be retrieved using property. The setter of this property parses the character from left to right, locates the for the button on the calculator corresponding to the character and then invokes it using ’s method. This is really mimicking user typing numbers in the calculator.

publicobject Result { get { return _resultTextBoxAutomationElement.GetCurrentPropertyValue (AutomationElement.NameProperty); } set { string stringRep = value.ToString(); for (int index = 0; index < stringRep.Length; index++) { int leftDigit = int.Parse(stringRep[index].ToString()); GetInvokePattern(GetDigitButton(leftDigit)).Invoke(); } } }

The , and methods simply evaluate (mimicking pressing of = button), clears (mimicking pressing of C button) and invokes a function respectively:

publicvoid Evaluate() { InvokeFunction(Functions.Equals); } publicvoid Clear() { InvokeFunction(Functions.Clear); } publicvoid InvokeFunction(string functionName) { GetInvokePattern(GetFunctionButton(functionName)).Invoke(); }

method locates the specified calculator menu and returns its :

private ExpandCollapsePattern FindMenu(CalculatorMenu menu) { AutomationElement menuElement = _calculatorAutomationElement.FindFirst (TreeScope.Descendants, new PropertyCondition (AutomationElement.NameProperty, menu.ToString())); ExpandCollapsePattern expPattern = menuElement.GetCurrentPattern (ExpandCollapsePattern.Pattern) as ExpandCollapsePattern; return expPattern; }

and use to obtain for menus and then expand and collapse menus respectively. looks for a menu item within expanded menu and invokes the command using the of the menu.

publicvoid OpenMenu(CalculatorMenu menu) { ExpandCollapsePattern expPattern = FindMenu(menu); expPattern.Expand(); } publicvoid CloseMenu(CalculatorMenu menu) { ExpandCollapsePattern expPattern = FindMenu(menu); expPattern.Collapse(); } publicvoid ExecuteMenuByName(string menuName) { AutomationElement menuElement = _calculatorAutomationElement.FindFirst (TreeScope.Descendants, new PropertyCondition (AutomationElement.NameProperty, menuName)); if (menuElement == null) { return; } InvokePattern invokePattern = menuElement.GetCurrentPattern (InvokePattern.Pattern) as InvokePattern; if (invokePattern != null) { invokePattern.Invoke(); } }

Now that we have exposed functionality of calculator that we would interact with, we can proceed and model expression tree so we can evaluate expressions in the calculator. For the purpose of testing expression evaluation, we need two modes of expression computation: expression evaluation through calculator UI (calculated expression) and evaluation through code (expected evaluation). If calculated expression equals expected expression, we can assert that calculator computes expression in the correct manner. This evaluation option is captured via enumeration:

internalenum EvaluateOption { UIEvaluate, ActualEvaluate }

At the end level, we need to model the operands that have sub-expressions to evaluate. Operands are modelled using interface:

internalinterface IOperand { int Evaluate(EvaluateOption option, Calculator calculator); }

At the fundamental level, an operator is an operand:

internalabstractclass Operator : IOperand { internal Operator(string automationName) { AutomationName = automationName; } publicstring AutomationName { privateset; get; } publicabstractint Evaluate(EvaluateOption option, Calculator calculator); protectedvirtualvoid InvokeOperator(Calculator calculator) { calculator.InvokeFunction(AutomationName); } }

Automation name refers to constants within class which correspond to function names in calculator (i.e. values):

internalclass Operators { publicconststring Negate = "Negate"; publicconststring Divide = "Divide"; publicconststring Multiply = "Multiply"; publicconststring Subtract = "Subtract"; publicconststring Add = "Add"; publicconststring Sqrt = "Square root"; publicconststring Percentage = "Percentage"; publicconststring Reciprocal = "Reciprocal"; }

A binary operator is represented using class:

internalabstractclass BinaryOperator : Operator { public BinaryOperator(string automationName) : base(automationName) { } public IOperand Left { get; set; } public IOperand Right { get; set; } publicoverrideint Evaluate(EvaluateOption option, Calculator calculator) { VerifyEvaluationState(); int result = 0; switch (option) { case EvaluateOption.UIEvaluate: calculator.Clear(); int leftValue = Left.Evaluate(option, calculator); calculator.Clear(); int rightValue = Right.Evaluate(option, calculator); calculator.Clear(); calculator.Result = leftValue; InvokeOperator(calculator); calculator.Result = rightValue; calculator.Evaluate(); result = int.Parse(calculator.Result.ToString()); break; case EvaluateOption.ActualEvaluate: result = Evaluate(Left.Evaluate(option, calculator), Right.Evaluate(option, calculator)); break; } return result; } protectedvoid VerifyEvaluationState() { if ((Left == null) || (Right == null)) { thrownew InvalidOperationException(); } } protectedabstractint Evaluate(int left, int right); }

As can be seen, if is used within function, then UI automation is used to mimic the user’s input in calculator; otherwise evaluation of expression is done in code. The other overload is implemented in derivative classes such as , , and . class is as follows (rest of the operators are very similar differing only in actual computation and automation name in constructor):

internalclass AddOperator : BinaryOperator { public AddOperator() : base(Operators.Add) { } protectedoverrideint Evaluate(int left, int right) { return left + right; } }

is just a number constant in expression tree:

internalclass NumberOperator : Operator { public NumberOperator(int number) : base (null) { _number = number; } publicoverrideint Evaluate(EvaluateOption option, Calculator calculator) { return _number; } privatereadonlyint _number; }

class creates an expression tree from a XML stream via method:

internalstaticclass ExpressionTree { internalstatic IOperand CreateTree(Stream stream) { XDocument doc = XDocument.Load(stream); return CreateOperend(doc.Root); } privatestatic IOperand CreateOperend(XElement operandElement) { XAttribute type = operandElement.Attribute("Type"); IOperand operand = null; switch (type.Value) { case"NumberOperator": operand = new NumberOperator(int.Parse (operandElement.Attribute("Value").Value)); break; default: string qualifyingName = "CalculatorTests." + type.Value; operand = Activator.CreateInstance (Type.GetType(qualifyingName)) as IOperand; List<XNode> childNodes = new List<XNode>(operandElement.Nodes()); if (operand is BinaryOperator) { BinaryOperator binaryOperator = operand as BinaryOperator; binaryOperator.Left = CreateOperend(childNodes[0] as XElement); binaryOperator.Right = CreateOperend(childNodes[1] as XElement); } elseif (operand is UnaryOperator) { UnaryOperator unaryOperator = operand as UnaryOperator; unaryOperator.Operand = CreateOperend(childNodes[0] as XElement); } break; } return operand; } }

calls method that parses the ’s attribute to determine type of operand and depending on whether type is (in which case it looks for attribute) or otherwise (unary or binary operand in which case it looks for child Operand XML elements), it creates and returns an recursively (if needed). Finally the root operand is returned. As an example, the expression (6 - 1) + (7 * 9) is represented using the XML:


Now that we have successfully created the infrastructure for interacting with the calculator and evaluating the expression tree, we can start authoring test cases:

  • Verify data entry of a random number in calculator and verify result shows up correctly in the UI: [TestMethod] publicvoid TypeRandomNumber() { using (Calculator calc = new Calculator()) { int number = new Random().Next(100, 10000); string stringRep = number.ToString(); calc.Result = stringRep; Assert.AreEqual(stringRep, calc.Result); } }
  • Verify cut and paste functionality: [TestMethod] publicvoid VerifyCopyPaste() { using (Calculator calc = new Calculator()) { string stringRep = new Random().Next(100, 10000).ToString(); calc.Result = stringRep; calc.OpenMenu(Calculator.CalculatorMenu.Edit); calc.ExecuteMenuByName("Copy"); calc.Clear(); calc.OpenMenu(Calculator.CalculatorMenu.Edit); calc.ExecuteMenuByName("Paste"); Assert.AreEqual(stringRep, calc.Result); } }
  • Verification of expression tree computation from embedded XML resource files: [TestMethod] publicvoid VerifyExpressionTrees() { string[] files = new[] { "CalculatorTests.Resources.SimpleNumberOperator.xml", "CalculatorTests.Resources.SimpleAdditionOperator.xml", "CalculatorTests.Resources.MixedOperators.xml" }; using (Calculator calc = new Calculator()) { foreach (string file in files) { calc.Clear(); IOperand expression = LoadExpressionTreeFromFile(file); Assert.AreEqual(expression.Evaluate (EvaluateOption.ActualEvaluate, calc), expression.Evaluate(EvaluateOption.UIEvaluate, calc)); } } } private IOperand LoadExpressionTreeFromFile(string resourceFileName) { return ExpressionTree.CreateTree (this.GetType().Assembly.GetManifestResourceStream(resourceFileName)); }

You may run the tests using Visual Studio test manager and will see calculator instances popping up and controls being manipulated as per the test cases followed by termination of calculator process (remember the implementation in class that does the teardown).

So that’s it, you have just seen an automation client in action! Stay tuned for my next blog where I will demonstrate how to implement automation support during the development of custom controls.

You may also want to look at project White on CodePlex which is based on UI automation framework and aims to simplify UI automation programming model.


  • 1st January, 2011: Initial post

1464 1465 1466 1467 1468