Leap Motion in Eclipse 4

The Leap Motion device is out! Great news for all who waited for it.

For me as an Eclipse 4 enthusiast and a technology lover it was obvious to bring both worlds together. Controlling an Eclipse 4 application with gestures. What nice opportunities this could bring up? Well the future will tell.

But first the main challenge needs to be solved, bringing the native libraries for the Leap Motion device into the OSGi context of an Eclipse 4 application. As a registered Leap Motion developer I was able to get my hands on that and solve it a while ago. Unfortunately I didn’t had time to write about it earlier. But now the Leap Motion device is out, I definitely should do so. So here is the blog post on how to bring the Leap Motion device into an Eclipse 4 application, so you are able to start control it with gestures.

The main idea I had was to create an OSGi service which allows to inject the Leap Motion Controller instance whereever necessary. Knowing about the ExtendedObjectSupplier, creating the OSGi service is rather easy. Of course there is a tutorial about it written by Lars Vogel you can find here.

First we need to specify the annotation that should be used to inject the Controller. Let’s name it LeapController

package org.fipro.leapmotion;

import java.lang.annotation.Documented;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

import javax.inject.Qualifier;

@Qualifier
@Documented
@Target({ElementType.FIELD, ElementType.PARAMETER})
@Retention(RetentionPolicy.RUNTIME)
public @interface LeapController {

}

The next step is to create the ExtendedObjectSupplier for the annotation.

package org.fipro.leapmotion.impl;

import org.eclipse.e4.core.di.suppliers.ExtendedObjectSupplier;
import org.eclipse.e4.core.di.suppliers.IObjectDescriptor;
import org.eclipse.e4.core.di.suppliers.IRequestor;

import com.leapmotion.leap.Controller;

public class LeapControllerObjectSupplier extends ExtendedObjectSupplier {

	static {
		//load the native libraries for the Leap Motion
		//device in the specified order
		System.loadLibrary("Leap");
		System.loadLibrary("LeapJava");
	}

	/**
	 * The Leap Motion Controller that will be provided by this
	 * object supplier.
	 */
	private Controller controller;

	@Override
	public Object get(
		IObjectDescriptor descriptor, IRequestor requestor,
			boolean track, boolean group) {

		if (this.controller == null) {
			this.controller = new Controller();
		}
		return this.controller;
	}
}

As the libraries for Leap Motion are native libraries, they need to be loaded when the LeapControllerObjectSupplier is loaded by the classloader. This is done in the static init block as suggested in the wiki.

Now we register our service as an OSGi declarative service. To do this we create the file OSGI-INF/leapcontrollersupplier.xml with the following content.

<?xml version="1.0" encoding="UTF-8"?>
<scr:component
   xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0"
   name="org.eclipse.leapmotion.leapcontrollersupplier">
   <implementation class="org.eclipse.leapmotion.impl.LeapControllerObjectSupplier"/>
   <property
      name="dependency.injection.annotation"
      type="String"
      value="org.eclipse.leapmotion.LeapController"/>
   <service>
      <provide interface="org.eclipse.e4.core.di.suppliers.ExtendedObjectSupplier"/>
   </service>
</scr:component>

Now it is time to talk about the plugin project setup. There are four settings to be made in the MANIFEST.MF you need to set in order to make things work:

1. Add the LeapJava.jar to the Bundle-ClassPath so the Leap Motion Controller can be resolved
2. Add the Service-Component parameter that points to the leapcontrollersupplier.xml
3. Add javax.inject and org.eclipse.e4.core.di as required bundles
4. Set the Bundle-ActivationPolicy to lazy so the libraries get loaded correctly when everything necessary for the OSGi services are ready.

To find out about the last setting cost me a lot of time. Finally Lars Vogel pointed out to that fact, so big thanks again for the tipp.

After adding all those settings to the MANIFEST.MF it should look similar to this

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Leapmotion
Bundle-SymbolicName: com.beone.leapmotion;singleton:=true
Bundle-Version: 1.0.0.qualifier
Bundle-Vendor: BeOne Stuttgart GmbH
Bundle-RequiredExecutionEnvironment: JavaSE-1.7
Bundle-ClassPath: LeapJava.jar,
 .
Export-Package: com.leapmotion.leap,
 com.beone.leapmotion
Service-Component: OSGI-INF/leapcontrollersupplier.xml
Bundle-ActivationPolicy: lazy
Require-Bundle: javax.inject,
 org.eclipse.e4.core.di

Also ensure that the necessary files are added in your build.properties so they get exported.

Now you might ask yourself “Where are the native libraries located that we need to load?”. Well, for a clean separation we create separate fragment projects for each platform. This is similar to SWT. As an example we create a fragment project for a platform running on an x86 architecture, using a win32 operating system and the win32 windowing system, and name it com.beone.leapmotion.win32.win32.x86. In the top level of the project we need to put the DLL files for that platform out of the LeapMotion SDK. These are the ones we load in the static initializer of the LeapControllerObjectSupplier.

In the MANIFEST.MF of the fragment project we need to set the Eclipse-PlatformFilter to the corresponding platform. This way we ensure that the fragment is only resolved if the application is running on the matching platform (for more information have a look in the Eclipse Help). For the platform specified above, the MANIFEST.MF could look like this

Manifest-Version: 1.0
Eclipse-PlatformFilter: (& (osgi.ws=win32) (osgi.os=win32) (osgi.arch=x86))
Bundle-ManifestVersion: 2
Bundle-Name: %fragmentName
Bundle-SymbolicName: com.beone.leapmotion.win32.win32.x86;singleton:=true
Bundle-Version: 1.0.0.qualifier
Bundle-Vendor: %providerName
Fragment-Host: com.beone.leapmotion;bundle-version="1.0.0"
Bundle-RequiredExecutionEnvironment: JavaSE-1.7

You can find the possible values for the Eclipse-PlatformFilter in the OSGi Specification.

Now you are able to create Eclipse 4 based applications that are controllable with the Leap Motion device. Your product simply needs to add the Plugins and Fragments created before. Also you need to ensure to start the plugins org.eclipse.core.runtime and org.eclipse.equinox.ds with a start level lower than 4. This way it is ensured the OSGi services are started correctly.

The following example is a simple part with a label. The label gets updated if you perform the swipe gesture, simply showing in which direction the swipe was performed.

package org.fipro.leapmotion.e4.example.part;

import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import javax.inject.Inject;

import org.eclipse.e4.core.contexts.IEclipseContext;
import org.eclipse.e4.ui.di.Focus;
import org.eclipse.e4.ui.di.UISynchronize;
import org.eclipse.swt.SWT;
import org.eclipse.swt.widgets.Composite;
import org.eclipse.swt.widgets.Label;

import org.fipro.leapmotion.LeapController;

import com.leapmotion.leap.Controller;
import com.leapmotion.leap.Frame;
import com.leapmotion.leap.Gesture;
import com.leapmotion.leap.GestureList;
import com.leapmotion.leap.Listener;
import com.leapmotion.leap.SwipeGesture;

public class LeapMotionExamplePart  {

	private Listener listener;

	private Label label;

	@Inject
	private UISynchronize uiSync;

	@PostConstruct
	public void postConstruct(Composite parent,
				@LeapController Controller leapController,
				IEclipseContext context) {
		this.label = new Label(parent, SWT.NONE);
		this.listener = new LeapMotionListener();
		leapController.addListener(this.listener);
	}

	@PreDestroy
	public void preDestroy(@LeapController Controller leapController) {
		leapController.removeListener(this.listener);
	}

	@Focus
	public void onFocus() {
		if (this.label != null) {
			this.label.setFocus();
		}
	}

	private class LeapMotionListener extends Listener {

    		String labelText = "";

		@Override
		public void onInit(Controller controller) {
			System.out.println("Initialized");
		}

		@Override
		public void onConnect(Controller controller) {
			System.out.println("Connected");
			controller.enableGesture(Gesture.Type.TYPE_SWIPE);
		}

		@Override
		public void onDisconnect(Controller controller) {
			System.out.println("Disconnected");
		}

		@Override
		public void onExit(Controller controller) {
			System.out.println("Exited");
		}

		@Override
		public void onFrame(Controller controller) {
			// Get the most recent frame and report some basic information
			Frame frame = controller.frame();
			GestureList gestures = frame.gestures();
			for (int i = 0; i < gestures.count(); i++) {
				Gesture gesture = gestures.get(i);
				switch (gesture.type()) {
					case TYPE_SWIPE:
						SwipeGesture swipe = new SwipeGesture(gesture);
						float x = swipe.startPosition().getX() - swipe.position().getX();
						labelText = "Swipe drawed in direction " + (x > 0 ? "left" : "right");
	                    			break;
	                		default:
						System.out.println("Unknown gesture type.");
						break;
				}

				uiSync.syncExec(new Runnable() {
					@Override
					public void run() {
						label.setText(labelText);
					}
				});
			}
		}
	}
}

Note that the UI updates need to be performed in the UI thread!

I would love to share my whole project infrastructure in Git or somewhere else. But I’m not aware of the policies for sharing the Leap Motion SDK  out of the Leap Motion Developer Portal. If someone from the Leap Motion Team reads this post and could provide me with informations on how to share my project, I would love to do so. It would be even greater if the OSGi bundles containing the native libraries for several platforms would be provided by Leap Motion directly, so not every developer needs to create the plugins himself. Of course I would love to provide any input you need to do so.

You can find the sources to this blog post in my GitHub repository