Sikuli Integration with Selenium Webdriver

Sikuli is a tool for automating Graphical User Interfaces (GUI) using the Visual Image Match method. All the web elements are considered as an image and stored inside the project. Sikuli will then trigger GUI interactions based on the image visual match, which is passed as the parameter and all the methods.

Sikuli can be very useful for automating flash objects (which do not have ID or name). It can be useful when we have a stable GUI (i.e., GUI components not changing).

Even the Window-based applications can be automated using Sikuli. Sikuli provides a very user-friendly Sikuli-script.jar, which can be easily used together with Selenium WebDriver. You can even automate Adobe Video/Audio player, Flash Games on the website using Sikuli. With simple API also makes coding easier.

Advantages of Sikuli Graphical User Interface Tool:

• It is an open-source tool for automation.
• Easily integrate sikuli with selenium.
• It can automate Desktop / Windows applications.
• It easily automates Flash objects – Flash Testing.
• Sikuli can be used on any platform such as Windows/Linux/Mac/Mobile.
• Because of its image recognition technique, we could automate even though there is no code access. It also allows us to automate anything we see on the screen.

Sikuli Setup to Integrate Sikuli with Selenium:

To integrate sikuli with selenium, we need to follow the below steps.

Step 1: Download Sikuli jars from the URL:

https://launchpad.net/sikuli/+download

Step 2: Double-click on “sikulixsetup-1.1.1.jar” to do the setup.

After having installed Sikuli on your system, a jar file “sikulixapi.jar” generate.

Step 3: Open Eclipse IDE and create a project.

Step 4: Include “sikulixapi.jar.”

Right-click on the project – Go to Build path – libraries tab – click on ‘Add external jars’ and add the following jar file and click on OK.

Add this “sikulixapi.jar” file.

Some Sikuli Methods:

#1) Creating an Object for the Screen Class:

The screen is a base class provided by the Sikuli. We first need to create an object for this screen class first, then only we can access all the methods provided by Sikuli.

Syntax:

Screen s=new Screen();

2) Click On An Element

This is used to Click on the specific image present on the screen.

Syntax:

s.click("<<image name>>");
Example,
s.click("tst.png");

3) Right Click On An Element

This is used to right-click on the specific image present on the screen.

Syntax:

s.rightClick("<<image name>>");
Example,
s.rightClick("tst.png");

4) Find An Element

This is used to find a specific element present on the screen.

Syntax:

s.find("<<image name>>");
Example,
s.find("tst.png");

5) Double Click on An Element

This is used to trigger a double-click event on a specific image present on the screen.

Syntax:

s.doubleClick("<<image name>>");
Example,
s.doubleClick("tst.png");

6) Check whether an Element is present on the Screen:

This is used to check whether the specified element is present on the screen.

Syntax:

s.exists("<<image name>>");
Example,
s.exists("tst.png");

7) Type a string on a Textbox

This is used to enter the specified text on the Text box.

Syntax:

s.type("<<image name>>","String to be typed");
Example,
s.type("tst.png","HELLO!!");

8) Wheeling on a particular image

This is used to perform wheeling action on the element image.

Syntax:

s.wheel("<<image name>>",<<int position>>,<<int direction>>);
Example,
s.wheel("tst.png",25,0);

9) Drag and Drop a Image/Element

This is used to drag and drop a specified image from a source position to a target position.

Syntax: 

s.dragDrop("<<source image name>>","<<target image name>>");
Example,     
s.dragDrop("tst.png","test1.png");

10) Roll Hover on a particular image

This is used to perform a roll hover event on the specified image.

Syntax:

s.hover("<<image name>>");
Example,
s.hover("tst.png");

11) Paste Copied String

This is used to paste text on the specified textbox.

Syntax:

s.paste("<<image name>>","test");
Example,
s.paste("tst.png","test");

Examples of Sikuli:

#1) YouTube Video – Pause/Play A Video

Step 1) Open any YouTube video link and then Capture play and pause element images using the screen capture tool.

Pause button (filename is pause.png)

Play button (filename is play.png)

Copy the above images inside the project.

Step 2) Create one package inside the Sikuli java project and create a class named “Youtube.”

Step 3) Type the below code inside that class.

package com.test; 
import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen; 
public class Youtube { 
public static void main(String arg[]) throws FindFailed, InterruptedException {
Screen s=new Screen();
 s.find("pause.png"); //identify pause button
 s.click("pause.png"); //click pause button
 System.out.println("pause button clicked");
s.find("play.png"); //identify play button
 s.click("play.png"); //click play button
 }
}

2) Open Notepad And Type Some Text

Step 1) Capture the image of the notepad icon on the desktop on the screen.

notepad_icon.png

notepad.png                                              

Step 2) Copy the above images inside your project.

Step 3) Create a class named NotepadExample inside your project and type the below code.

package com.test; 
import org.sikuli.script.FindFailed;
 import org.sikuli.script.Screen; 
public class NotepadExample { 
public static void main(String arg[]) throws FindFailed { 
Screen s=new Screen();
 s.click("notepad_icon.png");
 s.find("notepad.png");
 s.type("notepad.png","This is Nice Sikuli Tutorial!!!!");
 } 
}