Test Robot

Users continually complain that everything runs slow, but is it true? If so, why? End-to-end monitoring solutions like Alyvix [1], the monitoring tool I look at in this article, measure the performance and availability of business-critical applications from the user's point of view. Quality checks thus take place when the program is running in production mode. The objective is to ensure not only that users are satisfied, but that the admin can detect potential problems in advance on the basis of performance data.

This method also assumes that monitoring individual components of an IT service does not provide meaningful results. Databases, web servers, or hosts might work fine in and of themselves, but the application that builds on them still might be unusable because the individual parts do not interact smoothly. In this case, admins can detect performance hits by measuring the time required for actions that users would normally carry out. Admins thus measure certain data (e.g., responsiveness or failed interactions) that represents the user experience.

End-to-end monitoring [2] distinguishes between two techniques: Active (synthetic) monitoring versus passive (user) monitoring. Active monitoring tools simulate typical user behavior and record performance data. Passive monitoring tools save the performance data while the user is performing the various tasks. The benefit of the active approach is that it draws your attention to sources of interference before they affect the system. In principle, though, both approaches provide meaningful data.

Performance problems prolong individual processes, reduce staff efficiency, and cause frustration. If the end user is a customer, the negative effects are usually greater and, in the worst case, can end in termination of an agreement, so end-to-end monitoring is an important component in comprehensive IT service management.

Another aspect of performance monitoring is that the data can be used as important starting values for capacity planning. On this basis, admins can plan targeted scale-outs themselves, thus achieving a useful effect at a low cost.

QoS from the User's Viewpoint

Alyvix is an engine for synthetic application monitoring that supports monitoring of applications from the user's point of view by simulating transaction sequences and measuring and logging the corresponding performance data. The solution, based on Computer Vision [3], checks whether users can perform a desired action at a specific time in various branch offices and whether the agreed Quality of Service (QoS) is guaranteed.

Computer Vision refers to the detection of objects in an application's GUI (e.g., buttons, toolbars, selection lists, dialog boxes) with which the engine interacts as a user would; that is, it clicks on buttons and types characters in dialog boxes. The advantage of this approach is that Alyvix does not rely on the existence of application programming interfaces (APIs) for testing but acts like a human user directly through the user interface.

Through interaction with the application interface, Alyvix can automate all web and desktop applications and measure the resulting transaction times. The collected performance data can then be visualized in all common monitoring systems. Alyvix also generates HTML reports containing details of the individual steps and actions.

Open Source Simulation

Alyvix is an open source module for Python that relies on OpenCV [4] and the Tesseract OCR engine [5] to identify GUI objects. With the help of these tools, the software is able to determine the exact position (x , y coordinates) of all graphical objects, and the engine can interact with the identified objects using the keyboard and mouse.

As a Python module, Alyvix integrates with the Robot Framework [6], which is designed for automating software tests with a specific focus on acceptance testing. It uses a simple table-like structure to manage the test data and is based on keyword-driven testing. The test cases are stored as text or HTML files.

The Robot Framework also provides a GUI known as the Robot Integrated Development Environment (RIDE) [7], which allows for additional features such as code completion, simplified keyword management, and much more. The Alyvix installer automatically provides the Robot Framework, allowing you to create individual test scenarios by using the GUI.

If you run the defined sequence of actions at definable intervals (e.g., every five minutes), you enable automated performance, availability, and responsiveness testing of all the applications. Additionally, you can archive, process, and visualize the performance data on the central monitoring system.

Alyvix is open source and thus offers familiar benefits at no charge, such as adaptability and extensibility by the user, extensive testing before roll-out, and ongoing community development.

The Functional Principle

As mentioned, the Alyvix Wizard is integrated with RIDE (Figure 1), offering several tools for defining arbitrary interactions with the desired application as a transaction. For this to work, you need to define Alyvix keywords; that is, you select a graphical element on the screen and set the interaction type (e.g., click, drag and drop, double-click, etc.). Depending on the type of element the monitoring tool is to recognize and use for the interaction, the Alyvix IDE provides various so-called Finders (Table 1).

Figure 1: Alyvix simulates user behavior thanks to open source tools.

In a search, you also need to define the order in which the objects are to be found and in which area of the screen they can occur. A spatial restriction is important, especially for text recognition, because OCR image recognition over a large area demands a great deal of CPU power.

The wizard creates Python functions you invoke by calling their names in the RIDE Wizard. In addition to the Alyvix objects, you can use all other existing RIDE libraries and commands.

A simple example opens an editor, enters some text, and closes the editor. In everyday business operations, procedures of this type can be extended with any application to make them as complex as needed.

With the Create Process function in the IDE, Alyvix can launch any program, such as the Windows text editor Notepad (Figure 2). Using a second keyword, the Editor window is maximized so that the entire screen area can be used for text input. This is followed by the Send Keys function, which simulates keyboard input.

Figure 2: A test case launches the Notepad editor on Windows.

The Alyvix Screenshot keyword tells Alyvix to create a screenshot, which it then embeds in the resulting HTML log. In the context of the screenshot, Alyvix defines the wizard objects Click Menu File , Click Menu File Exit , and Click Dont Save ; other keywords from the Alyvix library and from the standard Robot Framework libraries are also available.

Because the keyword names are self explanatory, I will only look more closely at one: Click Menu File , which is defined in the Object Finder (Figure 3). In the Alyvix Wizard, you first define two elements:

Figure 3: The Click Menu File keyword is built-in to the Object Finder.

• One that uses the Image Finder designed to find the Notepad icon at top left.
• One in the Text Finder, which should find the "File" text.

In the Object Finder, you then define a composite object that (1) looks for the icon, (2) investigates a small area below the icon to find the "File" text, and (3) clicks on the text with the mouse. This sequence is titled Click Menu File and is inserted at the appropriate position in the list of keywords.

When all the keywords are defined, the complete test case is ready to run. In the background, a Python command created by Alyvix lets you call the test case at the command line, and Nagios/Icinga-compliant output is delivered (with exit code, text, and performance characteristics). Moreover, it creates an HTML page that contains the collected log data (Figure 4). If an error occurs, the page displays the failed action as a GIF animation, helping you identify the source of the error quickly.

Figure 4: The HTML page with the log of the completed test scenario.