Using Leap Motion and PhoneGap (Cordova)
Hi there. Today I want to introduce one of my new digital gadget, Leap Motion. Maybe many of the gadget-lovers already know, Leap Motion is a small stick-size device that captures your hands motion and send it to the host computer. It is like a Kinect, but have simpler SDK, and more affordable.
The usual case to use Leap Motion is to plug it to your PC, and drive some apps that supports it. Leap Motion allows third party developers to build apps that support Leap Motion. It includes, traditional C++ based SDK for Windows and Mac, and a JavaScript SDK for Web app developers.
JavaScript? Yes. It means, combined with PhoneGap, you can develop a mobile app to access to this new device with your favorite technology. So, this article will explain the overview and some sample application with Leap Motion.
How can a PhoneGap app connect to Leap Motion?
Leap Motion will be connected via USB to the host computer. So you are actually connecting to the PC from your PhoneGap app. When you install Leap Motion driver, it will automatically setup a daemon program running on your PC that opens and listens to its WebSocket protocol. Then, your app will connect using HTML5 WebSocket client to the host, via Leap Motion JavaScript API.
Since WebSocket client is required for the connection, Android devices cannot meet this requirement. Therefore, throughout this article I am using my iPad to run sample apps.
Leap Motion SDK
Leap Motion JavaScript SDK is open-source, and can be found in official website. I suggest you to read through the getting started guide, since it describes important aspect of how it is communicating with your app. In a nutshell, through WebSocket connection, it will send user’s interaction data at the specified frequency (default: 60fps), including the raw coordinates and the gesture commands calculated by the hardware. The recognizable gestures include Key Tap (like tapping to your keyboard), Screen Tap, Swipe, and Circle. Also, the device is smart enough to detect each individual fingers and its movement (like velocity).
Okey, now you are ready to create a new Leap Motion enabled app. In this article, I use Monaca for development environment. If you first hear about Monaca, it is a browser-based development environment for PhoneGap, which provides all-in-one IDE and cloud services. It is free to use, so please register an account if you don’t have one yet.
Hello Leap Motion!
Now let’s create very first example. Create a new project from “Minimum Template”, and IDE will display like below.
Then change your index.html as follows:
<html>
<head>
<script src="plugins/plugin-loader.js"></script>
<link rel="stylesheet" href="plugins/plugin-loader.css">
<style>
#motion {
box-sizing: border-box;
position: absolute;
top: 30%; left: 20%; width: 60%;
opacity: 0.9;
background-color: #fff;
padding: 30px;
border: 1px solid #ccc;
border-radius: 20px;
text-align: center;
font-size: 30px;
}
</style>
<script src="http://js.leapmotion.com/0.2.0-beta1/leap.min.js"></script>
<script>
monaca.viewport({"width": "640"});
Leap.loop({
enableGestures: true,
host: '10.0.5.130' // CHANGE HERE!!
}, function(obj) {
if (obj.gestures) {
obj.gestures.forEach(function(gesture) {
if (gesture.state == "start") {
$("#motion").text(gesture.type).show();
lastGestureType = gesture.type;
addList(gesture);
} else if (gesture.state == "stop") {
$("#motion").fadeOut();
}
});
}
});
function addList(json) {
var $list = $("#gestures");
$list.prepend("<li class='ui-li ui-li-static'><b>Starting " + json.type + "</b> <div style='font-size: 80%'>" + json + "</div></li>")
}
</script>
</head>
<body>
<div data-role="page">
<div data-role="header" data-position="fixed"><h1>Leap Motion Gestures</h1></div>
<div class="ui-listview" id="gestures"></div>
</div>
<div id="motion" style="display: none"></div>
</body>
</html>
Now, please change the host address (mentioned as // CHANGE HERE!! in the source code) to your computer’s IP address. The server should be running if Leap Motion driver is installed.
The program code should be self-explanatory. The function “Leap.loop” is the most important function, where the callback function will be triggered at maximum of 60 times per second, and display the output of the gesture found.
If you are first-comer to Monaca, you just download Monaca Debugger via App Store to execute this program. Please try.
Here is the screenshot of the result of this program. It dumps out the raw data, and displays motion while detecting in front.
More Complex Example
Now, let’s create a more complex example. Replace index.html with following code:
<html>
<head>
<title>DOM Visualizer - Leap</title>
<script src="http://js.leapmotion.com/0.2.0-beta1/leap.min.js"></script>
<script>
function moveFinger(Finger, posX, posY, posZ, dirX, dirY, dirZ) {
Finger.style.webkitTransform = "translateX("+posX+"px) translateY("+posY+"px) translateZ("+posZ+"px) rotateX("+dirX+"deg) rotateY(0deg) rotateZ("+dirZ+"deg)";
}
function moveSphere(Sphere, posX, posY, posZ, rotX, rotY, rotZ) {
Sphere.style.webkitTransform = "translateX("+posX+"px) translateY("+posY+"px) translateZ("+posZ+"px) rotateX("+rotX+"deg) rotateY(0deg) rotateZ(0deg)";
}
var fingers = {};
var spheres = {};
Leap.loop({
enableGestures: true,
host: '10.0.5.130' // CHANGE HERE!!
}, function(frame) {
var fingerIds = {};
var handIds = {};
if (frame.hands === undefined ) {
var handsLength = 0
} else {
var handsLength = frame.hands.length;
}
for (var handId = 0, handCount = handsLength; handId != handCount; handId++) {
var hand = frame.hands[handId];
var posX = (hand.palmPosition[0]*3);
var posY = (hand.palmPosition[2]*3)-200;
var posZ = (hand.palmPosition[1]*3)-400;
var rotX = (hand._rotation[2]*90);
var rotY = (hand._rotation[1]*90);
var rotZ = (hand._rotation[0]*90);
var sphere = spheres[hand.id];
if (!sphere) {
var sphereDiv = document.getElementById("sphere").cloneNode(true);
sphereDiv.setAttribute('id',hand.id);
sphereDiv.style.backgroundColor='#'+Math.floor(Math.random()*16777215).toString(16);
document.getElementById('scene').appendChild(sphereDiv);
spheres[hand.id] = hand.id;
} else {
var sphereDiv = document.getElementById(hand.id);
if (typeof(sphereDiv) != 'undefined' & & sphereDiv != null) {
moveSphere(sphereDiv, posX, posY, posZ, rotX, rotY, rotZ);
}
}
handIds[hand.id] = true;
}
for (handId in spheres) {
if (!handIds[handId]) {
var sphereDiv = document.getElementById(spheres[handId]);
sphereDiv.parentNode.removeChild(sphereDiv);
delete spheres[handId];
}
}
for (var pointableId = 0, pointableCount = frame.pointables.length; pointableId != pointableCount; pointableId++) {
var pointable = frame.pointables[pointableId];
var posX = (pointable.tipPosition[0]*3);
var posY = (pointable.tipPosition[2]*3)-200;
var posZ = (pointable.tipPosition[1]*3)-400;
var dirX = -(pointable.direction[1]*90);
var dirY = -(pointable.direction[2]*90);
var dirZ = (pointable.direction[0]*90);
var finger = fingers[pointable.id];
if (!finger) {
var fingerDiv = document.getElementById("finger").cloneNode(true);
fingerDiv.setAttribute('id',pointable.id);
fingerDiv.style.backgroundColor='#'+Math.floor(Math.random()*16777215).toString(16);
document.getElementById('scene').appendChild(fingerDiv);
fingers[pointable.id] = pointable.id;
} else {
var fingerDiv = document.getElementById(pointable.id);
if (typeof(fingerDiv) != 'undefined' & & fingerDiv != null) {
moveFinger(fingerDiv, posX, posY, posZ, dirX, dirY, dirZ);
}
}
fingerIds[pointable.id] = true;
}
for (fingerId in fingers) {
if (!fingerIds[fingerId]) {
var fingerDiv = document.getElementById(fingers[fingerId]);
fingerDiv.parentNode.removeChild(fingerDiv);
delete fingers[fingerId];
}
}
document.getElementById('showHands').addEventListener('mousedown', function() {
document.getElementById('app').setAttribute('class','show-hands');
}, false);
document.getElementById('hideHands').addEventListener('mousedown', function() {
document.getElementById('app').setAttribute('class','');
}, false);
});
</script>
<style>
*,*:before,*:after {
margin: 0;
padding: 0;
border: 0;
-webkit-box-sizing: border-box;
}
button {
padding: .5em;
}
#app {
position: absolute;
width: 100%;
height: 100%;
font-size: 200%;
overflow: hidden;
background-color: #101010;
-webkit-perspective: 1000;
}
#scene,
#scene:before {
position: absolute;
left: 50%;
top: 50%;
width: 40em;
height: 40em;
margin: -20em 0 0 -20em;
border: 4px solid #A0A0A0;
background-color: rgba(255,255,255,.1);
background-image:
-webkit-linear-gradient(rgba(255,255,255,.4) .1em, transparent .1em),
-webkit-linear-gradient(0deg, rgba(255,255,255,.4) .1em, transparent .1em),
-webkit-linear-gradient(rgba(255,255,255,.3) .05em, transparent .05em),
-webkit-linear-gradient(0deg, rgba(255,255,255,.3) .05em, transparent .05em);
background-size: 5em 5em, 5em 5em, 1em 1em, 1em 1em;
background-position: -.1em -.1em, -.1em -.1em, -.05em -.05em, -.05em -.05em;
-webkit-transform-style: preserve-3d;
-webkit-transform: rotateX(75deg);
}
#scene {
-webkit-transform: rotateX(75deg);
}
#scene:before {
content: '';
-webkit-transform: rotateX(90deg) translateZ(19.5em) translateY(20em);
}
.cube {
background-color: red;
-webkit-transform-style: preserve-3d;
-webkit-transform: translateX(19.5em) translateY(19.5em) translateZ(0em);
}
.finger,
.sphere {
position: absolute;
left: 50%;
top: 50%;
width: 1em;
height: 1em;
margin: -.5em 0 0 -.5em;
-webkit-transform-style: preserve-3d;
-webkit-transform: translateX(14.5em) translateY(14.5em) translateZ(0);
}
.finger {
opacity: .8;
height: 3em;
}
.sphere {
opacity: .3;
display: none;
font-size: 100px;
}
.show-hands .sphere {
display: block;
}
.face {
position: absolute;
width: 1em;
height: 1em;
background-color: inherit;
-webkit-transform-style: preserve-3d;
-webkit-transform-origin: 0 0;
-webkit-box-shadow: inset 0 0 0 1px rgba(255,255,255,.9);
}
.cube .face.tp { -webkit-transform: translateZ(1em); }
.cube .face.ft { -webkit-transform: rotateX(90deg) translateZ(-1em); }
.cube .face.bk { -webkit-transform: rotateX(90deg); }
.cube .face.lt { -webkit-transform: rotateY(90deg) translateX(-1em); }
.cube .face.rt { -webkit-transform: rotateY(90deg) translateX(-1em) translateZ(1em); }
.finger .face.tp { -webkit-transform: translateZ(1em); height: 3em; }
.finger .face.ft { -webkit-transform: rotateX(90deg) translateZ(-3em); }
.finger .face.bk { -webkit-transform: rotateX(90deg); }
.finger .face.lt { -webkit-transform: rotateY(90deg) translateX(-1em); height: 3em;}
.finger .face.rt { -webkit-transform: rotateY(90deg) translateX(-1em) translateZ(1em); height: 3em;}
</style>
</head>
<body>
<div id="app" class="show-hands">
<button id="showHands">Show Hands</button>
<button id="hideHands">hide Hands</button>
<div id="scene">
<div id="cube" class="cube">
<div class="face tp"></div>
<div class="face lt"></div>
<div class="face rt"></div>
<div class="face ft"></div>
<div class="face bk"></div>
</div>
<div id="finger" class="cube finger">
<div class="face tp"></div>
<div class="face lt"></div>
<div class="face rt"></div>
<div class="face ft"></div>
<div class="face bk"></div>
</div>
<div id="sphere" class="cube sphere">
<div class="face tp"></div>
<div class="face lt"></div>
<div class="face rt"></div>
<div class="face ft"></div>
<div class="face bk"></div>
</div>
</div>
</div>
</body>
</html>
I post the example movie taken by this sample. You will notice Leap Motion is detecting five individual fingers seamlessly.
This sample app is a slightly modification version of this page.
From Here
It is up to you to make this cool technology integrated with your mobile apps. The only pitfall is that you need a PC running as a server. I hope next Leap Motion will have its own WiFi connection and a WebSocket server.
You can actually go far beyond the examples here. If you have made a Leap Motion PhoneGap app with Monaca, please inform us so that we can feature on our blog.
Happy coding!
