LEGO Mindstorms NXT Robots Meets the Xbox Kinect!

Open Kinect Penguin    Send article as PDF   
InThe Microsoft Xbox 360 Kinect sensor, is a horizontal bar connected to a small base with a motorized pivot, and is designed to be positioned lengthwise below the video display. The device features an “RGB camera, depth sensor and multi-array microphone running proprietary software”, which provides full-body 3D motion capture, facial recognition, and voice recognition capabilities.


According to information supplied to retailers, the Kinect sensor outputs video at a frame rate of 30 Hz, with the RGB video stream at 32-bit color VGA resolution (640×480 pixels), and the monochrome video stream used for depth sensing at 16-bit QVGA resolution (320×240 pixels with 65,536 levels of sensitivity). The Kinect sensor has a practical ranging limit of 1.2–3.5 metres (3.9–11 ft) distance. The sensor has an angular field of view of 57° horizontally and a 43° vertically, while the motorized pivot is capable of tilting the sensor as much as 27° either up or down. The microphone array features four microphone capsules, and operates with each channel processing 16-bit audio at a sampling rate of 16 kHz.

Microsoft Can’t Envisaged the Power of Open-Source!

Bounty offered for Open-Source Xbox Kinect Driver

In January 2011, Menlo Park technology started sponsoring a contest for the most interesting new use of the Kinect’s depth sensor with a robot. The top prize is $3,000, so get your entries ready to roll. Early entries have shown inventors controlling robots with their arms, making the robots do push-ups and other tasks.

C-NET News: November 4, 2010 12:50 PM PDT

The first person or team to come up with an open-source driver for Kinect could win $1,000 from Adafruit Industries.

The first person who figures out how to build an open-source driver for Microsoft’s much-hyped new Kinect motion controller could win a $2,000 bounty offered by a leading open-source hardware developer.

Kinect, which launched today, is currently available solely for Microsoft’s Xbox 360 and may well someday be extended to the Windows platform. But for New York-based Adafruit Industries, that’s not enough.

And that’s why Adafruit–led by MIT Media Lab alum Limor Fried and Make magazine Senior Editor Phillip Torrone–is offering two grand to someone who can figure out how to decouple the hot new device from Microsoft’s gaming machine.

The bounty will go to the “first person or group to upload code and examples under an open-source license to (social-coding site) GitHub.”

“It’s amazing hardware that shouldn’t just be locked up for Xbox 360,” Torrone told CNET by e-mail. “Its ‘radar camera’ being able to get video and distance as a sensor input from commodity hardware is huge.”

But Microsoft isn’t taking kindly to the bounty offer. “Microsoft does not condone the modification of its products,” a company spokesperson told CNET. “With Kinect, Microsoft built in numerous hardware and software safeguards designed to reduce the chances of product tampering. Microsoft will continue to make advances in these types of safeguards and work closely with law enforcement and product safety groups to keep Kinect tamper-resistant.”

Still, according to Torrone, Adafruit is hoping someone will figure out how to use Kinect in education, robotics, or “fun outside the Xbox.”

“We think First Robotics could use this,” Torrone said. “We think educators could use this. Look at all the cool stuff people did with the Wii remote.”

Torrone added that while Adafruit could likely come up with a Kinect driver itself, “we don’t have the time. We can offer a bounty…and we think we’ll see a million ideas flourish once anyone can plug this in to whatever they want.

Update at 5:07 p.m. PT: Adafruit has raised the bounty from $1,000 to $2,000 in response to Microsoft’s position on the offer.

“The Secret”

I actually have a secret to share on this topic. Back in the late Summer of 2010, trying to argue for the most basic level of PC support for Kinect from within Microsoft, to my immense disappointment, turned out to be really grinding against the corporate grain at the time (for many reasons I won’t enumerate here). When my frustration peaked, I decided to approach AdaFruit to put on the Open Kinect contest…Johnny Chung Lee’s Blog February 21, 2011


Openkinect logo driver(s) released – Winner will use $3000 for more hacking
– PLUS an additional $2000 goes for their Effort!

Tux & OpenKinect

Openkinect logo is an open community of people interested in making use of the amazing Xbox Kinect hardware with their PCs and other devices. They are working on free, open source libraries that will enable the Kinect to be used with Windows, Linux, and Mac. There primary focus is currently the libfreenect software. All Code contributed to OpenKinect where possible is made available under an Apache20 or optional GPL2 license.

Openkinect icon OpenKinect API Documentation:
Openkinect icon OpenKinect Wrappers:
Openkinect icon OpenKinect Utilities:
  • Record – Dumps Kinect data to PPM, PGM, and a bin dump for RGB, Depth, and Accel respectively.
  • Fakenect – libfreenect simulator/mock interface that lets you use the kinect demos without having a kinect (plays back data from Record)
[wpdevart_wpdevart_youtube]rKhW-cvpkks[/wpdevart_wpdevart_youtube] Just 3 Hours after the Xbox Kinects Release…. An Open-Source Driver!

Since it’s release the Microsoft Kinect has enabled scientists to create hologram-like images, build 3-D models of homes and even help NASA scientists teleconference in three dimensions.

Interactive Puppet Prototype with hacked Kinect. The Power of Open-Source!

We’ve always had sensors like these available, but they were very expensive,” according to Tim Conley, the contest’s senior software engineer at Willow Garage. When a Sensor is cheap and almost disposable, you can play with it in ways you wouldn’t with something that’s much more expensive. You aren’t worried about endanger it.”

Controlling a Humanoid Robot with Kinect

To date, much of the hacking has involved the 3-D camera, built by the Israeli company PrimeSense, which Microsoft licensed for the Kinect. When attached to a robot, the camera allows its host to perceive depth accurately – an ability essential for robots to be able to navigate.

Controlling a Helicopter with Kinect

The accuracy of Kinect is pretty amazing. Here are results of using kinect_node_subset.bag from in rviz. Look at the end of the video for the shape of the head.

Kinect on Linux: The accuracy of Kinect is amazing!



The LEGO Mindstorms NXT Controlled by the Microsoft® Xbox 360 Kinect® Sensor, using some neat coding! And not an actual XBox in sight….

Mauro Brunato was among the first with his LEGO Mindstorms NXT Robot that moves on two independent wheels.,to use the Kinect. The user controls the two motors by moving there hands forward and backwards, as if pushing and pulling two levers. You drive the Robot similar to that of a tank, or of a caterpillar dozer. The speed of the motor is proportional to the displacement of the hands from a pre-defined “neutral” position.

Mauro’s Version of a LEGO NXT Robot Controlled with a Kinect Sensor

After seeing the video posted on Mauro Brunato’s website, Xander Soldaat envisaged all sorts of ideas for other cool things to do with the NXT and an Xbox Kinect.

Xander’s Version of a LEGO NXT Robot Controlled with a Kinect Sensor

Xander Soldaat will be making something cool with his NXT’s and the Xbox Kinect for LEGOWorld 2011 in Zwolle , Netherlands, so make sure you’re there!

And we can’t forget using the Xbox Kinect with the late, but great Lego Mindstorms RCX!

LEGO RCX Robot Football Controlled with a Kinect Sensor



The Future:

From Mindsensor’s NXT-Cam®, to a NXT-Kinect Sensor for the NXT Platform.

The future of combining the Microsoft Xbox Kinect with the LEGO Mindstorms NXT platform offers a world of possibilities. Taking the Mindsensor NXT-Cam as a starting point, the obvious evolution in my mind is to extend it to the next level for Object Recognition which has been proved to be achievable with with Kinect.

Teaching Kinect to recognize objects on the PC

Like the Mindsensor NXT-Cam, the NXT-Kinect will have to rely on a PC to process the data supplied by the Konect, and then relayed back to the NXT. With Dexter Industries new Wifi for the Lego Mindstorms NXT module under development, this data exchange may negate the need for the Kinect to be tethered to your PC via a USB Cable. An alternative, may be a USB-to-WiFi adapter to connect the Kinect to the PC in a wireless manor.

Object recognition using Kinect on the PC

Using the Microsoft Xbox Kinect with the LEGO Mindstorms NXT platform also offers the possibility of excellent control in navigation and object avoidance. Mapping, and hence the possibility of autonomous route planning could be achievable with your NXT Robots.

Kinect Enabled Autonomous Mini Robot Car Navigation

The world of Open-Source, Microsoft Xbox Kinect and the LEGO Mindstorms NXT platform have a bright future in my minds eye. With these ingredients, some Imagination and perseverance will produce some amazing LEGO Mindstorms NXTRobots in the not too distant future.

Kinect Enabled Autonomous Personal Robot Navigation

The Microsoft Xbox Kinect has only been available for 5 months and it appears from my research that a very large percentage of the units sold have/will never be used in the manor Microsoft envisaged!

Quadrotor Autonomous Flight and Obstacle Avoidance with Kinect Sensor

May be this not so humble piece of technology will have the same effect that the LEGO Mindstorms RCX and it’s Hackers had on the LEGO Company. That is to force Microsoft in the direction of Open-Source and Open-Hardware. At the least, an Open-Hardware policy by Microsoft would benefit them and the community as a whole!

And the only losers will be all those poor co-operate and copyright Lawyers……….

Live on an Island – Evolve as an Islander!


Please Keep me posted on your ideas and achievements?

Previous Post

Lego Mindstorms NXT Speech File Collections

I have gathered together some collections of Lego Mindstorms NXT Speech Files which I commonly use with my Robots. All the Speech files are in Lego Mindstorms NXT compressed 8KHz ... Read more

Next Post

New Improved Rotacaster Omni-wheels

I received a a new batch of Rotacaster Omni-wheels today which look and work fantastic. The new Colour Palette, matches that of the LEGO Mindstorms NXT Kits. They will look ... Read more

Short URL:

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Do NOT follow this link or you will be banned from the site!