Sunday, December 16, 2012

Hand Tracking (IMPROVED)

Finally fixed some of the issues in the previous hand tracker, and now it works almost as good as the one provided by NITE.

The algorithm is actually very light in resource, and I've already ported it to run on the Android. Stay tuned. =)

Thursday, December 13, 2012


Just received the Odroid-X2 from today. I've to first thank HardKernel for supporting my work by sending me the board along with the accessories. They are just simply amazing when it comes to making these decisions of giving out hardware to developers. Quick and simple.

Here I will show the difference between the X2 and X1 and the potential of such powerful devices, particularly in the field of wearable computing.

Stay tuned... =)

Monday, December 3, 2012

Amazing 20 fps on Odroid-U2

The Odroid-U2 (2Ghz!) rocks =) Thanks hardkernel for making this demo video of the real-time blob tracking + 3D rendering apps with OpenNI cameras. I'll be updating the tracker to perform as good as OpenNI hand tracker next. Still have a few bugs to address.

The source is actual opened to public! Feel free to commit or make changes to the code as you wish.

The compilation may takes a long while to setup (see my previous blogs and README), but you can always download the apk and the openni driver to your own Android devices, and try it out.


Joy! and Thanks HardKernel for reserving the Odroid-U2! =) Donation is the only way to keep this project going.

Wednesday, November 28, 2012

Odroid-X + Android 4.0.4 + OpenNI + OpenCV + PCL

Finally, I've ported everything to Android! Now it even runs on Odroid-X! ;)
Check out the videos. It runs @ 30fps! with tracking! ;) Cannot ask for more from such a small machine. I will write a tutorial on how to setup these on Odroid-X next. It would be useful?

Tuesday, September 25, 2012

Using Accelerometer with Kinect/Xtion on Tegra 3

A trivial solution to adding accelerometer to Xtion or Kinect is:

1. Firmly attach the Xtion to the Tegra 3, hot glue is recommended :)
2. Initialize the Accelerometer callbacks (the events) and set it to 30Hz. Can go higher if we would like to smooth the data.
3. Apply a simple lowpass filter to the raw accelerometer data, and use it for the tracking next.

You can found these new implementation on my svn repository! Have fun!

Demo Video:
Coming up next!


Monday, September 24, 2012

Enable stdout and stderr for debugging Android?

$ adb shell stop
$ adb shell setprop log.redirect-stdio true
$ adb shell start

It is often a pain to output to the terminal using C++ on Android. Here is the best solution I came across!


I/stdout  ( 7105): Saved 5 data points to input:
I/stdout  ( 7105):     0.162692 -0.85557 0.94647
I/stdout  ( 7105):     0.0343666 0.713827 -0.191865
I/stdout  ( 7105):     0.42557 -0.339379 -0.858582
I/stdout  ( 7105):     -0.289159 -0.268988 -0.126301
I/stdout  ( 7105):     0.64959 0.285136 -0.438644
I/stdout  ( 7105): size:5
I/stdout  ( 7105): Transformed 5 data points:
I/stdout  ( 7105):     0.862692 -0.85557 0.94647
I/stdout  ( 7105):     0.734367 0.713827 -0.191865
I/stdout  ( 7105):     1.12557 -0.339379 -0.858582
I/stdout  ( 7105):     0.410841 -0.268988 -0.126301
I/stdout  ( 7105):     1.34959 0.285136 -0.438644
I/stdout  ( 7105): has converged:1 score: 1.28964e-13
I/stdout  ( 7105):            1  2.38419e-07  1.93715e-07          0.7
I/stdout  ( 7105):   2.5332e-07            1 -2.19792e-07  -1.2368e-07
I/stdout  ( 7105): -7.45059e-08 -3.68804e-07     0.999999 -1.38581e-07
I/stdout  ( 7105):            0            0            0            1

P.S. Now Iterative Closest Point (ICP) algorithm is running on Android properly! Slow but it works! :) 3D reconstruction on Android is coming up next!


OpenCV + PCL = FLANN conflicts?

When we try to integrate the PCL and OpenCV library, often times you will run into this problem:
../../pcl-superbuild/build/CMakeExternals/Install/flann-android/include/flann/util/saving.h:113: error: reference to 'flann' is ambiguous../../pcl-superbuild/build/CMakeExternals/Install/flann-android/include/flann/defines.h:76: error: candidates are: namespace flann { }../../OpenCV-2.4.2-android-sdk/sdk/native/jni/include/opencv2/flann/miniflann.hpp:55: error:                 namespace cv::flann { }

The solution to this is simple. Take away the namespace will do the trick.

using namespace cv;
using namespace pcl;

Of course, then you have to add cv:: and pcl:: too the rest of your code, but it is simple.

Wednesday, September 19, 2012

ASUS Xtion Pro Live + Tegra 3

Now we are officially supporting ASUS Xtion Pro Live and also ASUS Xtion Pro (no RGB camera).

ASUS didn't reply to my email, and didn't even donate me anything :( Well... No support so far, but I'm supporting ASUS :(!!

Source code: 

cd ~/NVPACK/TDK_Samples/tegra_android_native_samples_v10p00/sample_apps/
svn co -r 256 OpenNIRecorder

Monday, September 17, 2012

Important: multitouch -> OpenNIRecorder

Now, the project is renamed to


It is important that you use this directory name or else Eclipse will complain.

Additionally, I would drop the support for Kinect because it is 'not' portable anyways.

Instead, we will be using

!ASUS? Give me some hardware?

Thursday, September 13, 2012

Kinect and OpenNI Support

It's been my nightmare trying to figure out why the Kinect wasn't working properly with the OpenNI. The frames just get corrupted from time to time, and eventually 'segmentation fault'...

/DEBUG   (  104): stack:I/DEBUG   (  104):     6079dd38  5d275c8b  /system/lib/libXnDeviceSensorV2.soI/DEBUG   (  104):     6079dd3c  5d2ece84  /system/lib/libXnDeviceSensorV2.soI/DEBUG   (  104):     6079dd40  01f429fc  [heap]I/DEBUG   (  104):     6079dd44  000000f0  I/DEBUG   (  104):     6079dd48  6079dd64  I/DEBUG   (  104):     6079dd4c  fffffd81  I/DEBUG   (  104):     6079dd50  fffffd82  I/DEBUG   (  104):     6079dd54  0000027f  I/DEBUG   (  104):     6079dd58  00003c7e  I/DEBUG   (  104):     6079dd5c  00000000  I/DEBUG   (  104):     6079dd60  00000786  I/DEBUG   (  104):     6079dd64  000000f0  I/DEBUG   (  104):     6079dd68  00000000  I/DEBUG   (  104):     6079dd6c  01f42948  [heap]I/DEBUG   (  104):     6079dd70  df0027ad  I/DEBUG   (  104):     6079dd74  00000000  I/DEBUG   (  104): #00 6079dd78  01f42948  [heap]I/DEBUG   (  104):     6079dd7c  00000774  I/DEBUG   (  104):     6079dd80  01ee1efc  [heap]I/DEBUG   (  104):     6079dd84  01ee1efc  [heap]I/DEBUG   (  104):     6079dd88  00000774  I/DEBUG   (  104):     6079dd8c  5d2ebd94  /system/lib/libXnDeviceSensorV2.soI/DEBUG   (  104):     6079dd90  5d2ec620  /system/lib/libXnDeviceSensorV2.soI/DEBUG   (  104):     6079dd94  60971ee4  I/DEBUG   (  104):     6079dd98  01f42a94  [heap]I/DEBUG   (  104):     6079dd9c  5b1bfd17  /system/lib/libOpenNI.soI/DEBUG   (  104): #01 6079dda0  01f42a94  [heap]I/DEBUG   (  104):     6079dda4  5d2999b1  /system/lib/
Fortunately, I've confirmed that OpenNI works perfectly fine on an ASUS Xtion Live.

HOWEVER, today!, I still don't have the ASUS Xtion with me... !! Guys, donations? 

Wednesday, September 12, 2012

Tegra 3 OpenNIRecorder for Android

Here we go! The OpenNIRecorder is now ready. We have a very simple GUI for you to enable or disable the rendering / visualization and to record the RGB / depth data to your SD card. To get the best performance, you may want to turn off the rendering for both rgb and depth. Also, you shall get a very fast SD card to reduce the I/O latency! It can be very very slow if the SD card is blocking!

Also, there are a few significant changes to the path and other variables. Please make you you check out the source to:

cd ~/NVPACK/TDK_Samples/tegra_android_native_samples_v10p00/sample_apps
svn co OpenNIRecorder

Lastly, I have received several emails from students who are interested in this work. Again, thank you for your interest in our work. But please read through my entire blog (i.e., from the very beginning) and try to understand the basic about the Android NDK etc... It is very unlikely that I can give you a full tutorial over an email. Download the source code, read it through, and if you run into problems or you have some design issues? Then feel free to drop me a line here and there. If you ask me to help you setting up everything from scratch? Here you go :)! read the blogs :)

Wednesday, September 5, 2012

Multithreading on Android 4.0.2 + Tegra 3

Multithreading is one of the most commonly used technique for accelerating your application. Of course, we assume that your application is parallelizable at the first place, or else you would be looking into NEON instruction set and other ways to accelerate your code.

Apparently, Tegra 3 was designed for multi-tasking as it comes with 4 (FOUR!) cores!, but how do we take advantages of such without spending nights and days debugging? 

When we perform multiple tasks in parallel, the first problem we have to handle is the synchronization issue. These are often handles by using 'locks'. The detail of how to handle race condition, using locks, conditional variables, and so are left for your reading (see below). Additionally, creating threads for every task can be time consuming, and eventually you may even run out of thread id. Thus, we often implement a thread pool, a set of threads/workers that waits for commands. 

In this coming up tutorial, I will show you how to multithread your application in 10 mins, and which would give you a potential 3x speed up. We will demonstrate how you can easily obtain real-time results by utilizing the cores of a mobile processor! 

Using the cThreadPool
Sometimes, simplicity is just gold. I've found this little implementation that handles the thread pool. And here is how I have implemented that to our flow.
     static int counter = 0;
     char my_path[512];
     RGBpack *filepack = (RGBpack*)(malloc(sizeof(RGBpack)));
     //filepack->rgb_data = (unsigned char*)malloc(IMAGE_WIDTH*IMAGE_HEIGHT*3*sizeof(unsigned char));
     filepack->depth_data = (unsigned short*)malloc(IMAGE_WIDTH*IMAGE_HEIGHT*sizeof(unsigned short));
      return false;
     int ret = threadpool_add_task(pool,fast_task,filepack,1); //this will also free the memory

      __android_log_write(ANDROID_LOG_INFO, "THREAD POOL:", "POOL ERROR?\n");
      __android_log_write(ANDROID_LOG_INFO, "THREAD POOL:", "FAILED to add task, pool full?\n");
As we can see, we will be packaging the frame we receive from the Kinect and add each to the task list. The thread pool will automatically pick up these whenever they are free, and our job is done. One thing we have to watch out for is memory usage, it does get memory hungry if we allocate too many tasks at the same time! To no surprise, we can now achieve real-time capturing with the Tegra 3! Check it out :)

Raw PNG outputs from the Kinect:

Excuse my dance moves. Scroll down quickly to see what we have captured in 2 seconds. 

Some more randomly selected from the set:

Coming up: 

Source code:

svn co multitouch

Note: the source code at the repository could be broken for time to time. Please email me if you run into any problem.


Linux Tutorial: