Monday, June 18, 2012

Creating OpenNI '.oni' files from OpenCV/Jpeg Images

I had been thinking of tweaking OpenNI samples to work with modified data. It could be as a result of post processing on the Depth/Rgb images in Matlab or OpenCV. Mostly we record data from OpenNI in the form of '.oni' file and then want to process them in Matlab or OpenCV. For this we need to convert the images to OpenCV images and then can save to disk. Later load these images in Matlab or OpenCV and do processing and then resave them as '.oni' files to run the OpenNI samples on the modified data. 

For the first part you can either use OpenNI and convert them to OpenCV images and write them to disk. 

You can convert images to OpenCV images in the following way. 

         cv::Mat colorArr[3];
        cv::Mat colorImage;
        const XnRGB24Pixel* pImageRow;
        const XnRGB24Pixel* pPixel;

        imageGen.SetPixelFormat(XN_PIXEL_FORMAT_RGB24 );  //
        xn::ImageGenerator imageGen;
        imageGen.GetMetaData(imageMD);    //xn::ImageMetaData imageMD;
        pImageRow = imageMD.RGB24Data();

        colorArr[0] = cv::Mat(imageMD.YRes(),imageMD.XRes(),CV_8U);
        colorArr[1] = cv::Mat(imageMD.YRes(),imageMD.XRes(),CV_8U);
        colorArr[2] = cv::Mat(imageMD.YRes(),imageMD.XRes(),CV_8U);

        for (int y=0; y<imageMD.YRes(); y++)
              pPixel = pImageRow;
              uchar* Bptr = colorArr[0].ptr<uchar>(y);
              uchar* Gptr = colorArr[1].ptr<uchar>(y);
              uchar* Rptr = colorArr[2].ptr<uchar>(y);
              for(int x=0;x<imageMD.XRes();++x , ++pPixel)
                        Bptr[x] = pPixel->nBlue;
                        Gptr[x] = pPixel->nGreen;
                        Rptr[x] = pPixel->nRed;
              pImageRow += imageMD.XRes();

The second way is to directly use the Matlab Toolboox for kinect and save the images from there. 

For the second part i.e how to compile these images as an '.oni' file , I used the OpenNI program  NiRecordSynthetic. So I provide a dummy .oni file as an input and then change its data according to my new depth maps and record a new .oni file out of it. This is what I do in the transformMD() function of this program


   char filename[100];

        static int count = 1;

        sprintf(filename,"filename.png",count++); // here i assume that images are numbered in a sequence
        IplImage* img=0;
         img=cvLoadImage(filename,CV_LOAD_IMAGE_ANYDEPTH |      CV_LOAD_IMAGE_ANYCOLOR); // Reading 16 bit depth maps

          if(!img) printf("Could not load image file: %s\n",filename); /
        DepthMap& depthMap = depthMD.WritableDepthMap();

for (XnUInt32 y = 0; y < depthMap.YRes(); y++)
for (XnUInt32 x = 0; x < depthMap.XRes(); x++)

CvScalar s;
              s=cvGet2D(img,y,x); // getting the value in s 
             depthMap(x,y) = s.val[0]; // setting the value to the modified depth pixel 



Wednesday, May 2, 2012

Installing Skeltrack on Ubuntu

Skeltrack  is an Open Source skeletal tracker and tracks up to 7 joints currently: head shoulders, elbows, and hands .Openni and Microsoft SDK trackers are not open source, so this might provide developers a good platform to tweak the algorithm and parameters according to their application :)

How to install skeltrack on ubuntu 11.10

#need clutter 1.8 or greater
# make sure the glut-dev is there, I forget the package name
sudo apt-get install git-core cmake  pkg-config build-essential libxmu-dev libxi-dev libusb-1.0-0-dev
git clone git://
cd libfreenect
mkdir build
cd build
cmake ..
sudo make install
sudo ldconfig /usr/local/lib64/
#sudo glview
cd ../.. #back to src
sudo aptitude install gobject-introspection gtk-doc-tools

git clone git://
cd gfreenect
git clean -f
libtoolize -v --copy --install
sudo make install
sudo ldconfig /usr/local/lib/
cd ..

sudo aptitude install libclutter-1.0-dev
git clone
cd Skeltrack
git clean -f
libtoolize -v --copy --install
./configure --enable-examples=yes
sudo make install
cd examples

If you encounter the following error

error while loading shared libraries: 
cannot open shared object file: No such file or directory

You need to refresh your ldconfig cache. The easiest way to do this is to create
 a file usr-local-libs.conf (or whatever name you wish) with the following lines:
Switch to root account and move it to /etc/
Then update the ldconfig cache:
$ su root
$ mv ~/usr-local-libs.conf /etc/
$ /sbin/ldconfig -v


Friday, April 20, 2012

ROS ppl_detection

ppl_detection is a ROS package for detecting and tracking people based on kinect data. Today i gave it a try and only encountered one problem.
To make it running do the following steps.

Checkout the source code in a directory {make sure this directory is added to the ROS_PATH}
Change the following line in kinect.launch 
<node pkg="rviz" name="rviz" type="rviz" args="--display-config $(find trainer)/kinect_detect.vcg"/>


<node pkg="rviz" name="rviz" type="rviz" args="--display-config $(find ppl_detection)/kinect_detect.vcg"/>

Change the folllowing line in kinect_detect.cpp 
"svm_models/" + model_name + ".model"

 "./svm_models/" + model_name + ".model"
Build the package
  • rosmake ppl_detection
  • roslaunch ppl_detection kinect_launch
  • rosrun ppl_detection kinect_detect
Hopefully you will see the rgb point cloud in rviz and if a person is in the scene you will see a bounding box with the text 'human'

Thursday, April 19, 2012

Running ROS Openni Tracker (Offline) with Recorded data '.oni' files

It is always more convenient to run the tracker on a recorded data.  I recorded an '.oni' file using Openni Sample 'NiViewer'. Initialized the Context manually rather than from the 'xml' file.

You need to make the following changes in your openni_tracker.cpp

I had created a ros package mimicking the openni_tracker {i.e with same dependencies}. Then i created an Eclipse project for it. It makes debugging and changing the code easier.

// Declare .oni player

xn::Player   g_Player;  

Instead of

    string configFilename = ros::package::getPath("openni_tracker") + "/openni_tracker.xml";
    XnStatus nRetVal = g_Context.InitFromXmlFile(configFilename.c_str());
    CHECK_RC(nRetVal, "InitFromXml");


    // added for Reading '.oni' file

    XnStatus nRetVal = g_Context.Init();
    CHECK_RC(nRetVal, "Init");
    nRetVal = g_Context.OpenFileRecording("your .oni file", g_Player);
    if (nRetVal != XN_STATUS_OK)
    printf("Can't open recording %s\n", xnGetStatusString(nRetVal));
    return 1;

  • make
  •  run openni_tracker
  • rosrun rviz rviz 
Hopefully you will see the skeleton.
Following video shows ros openni tracker on recorded data :)


Running OpenNi tracker on ROS Bag files/ Subscribed Depth topics
/ / OpenNI Context
xn::Context g_context;
g_context.Init ();
// OR initialize from an XML file
nRetVal = nRetVal = xnLogInitFromXmlFile(SAMPLE_XML_PATH);
/ / Depth generator
xn::DepthGenerator depth_generator;
depth_generator.Create (g_context);
/ / Mock depth generator
xn::MockDepthGenerator g_mock_depth;
g_mock_depth.CreateBasedOn (depth_generator, "the mock-depth" );
/ / Create user generator using mock depth generator
xn::Query g_Query;
g_Query.AddNeededNode ( "mock-depth" );
xn::UserGenerator g_user;
g_user.Create (g_context, & g_Query);
//following runs in a while loop
while (ros::ok())
/ / Update data
g_context.WaitOneUpdateAll (depth_generator);
/ / Get original depth map
xn::DepthMetaData depthMD;
depth_generator.GetMetaData (depthMD);
/ / Make data writable and modify
depthMD.MakeDataWritable ();

/ / Modify data ....
//write a function which changes the depth data in your original depth map
transformDepthMD(DepthMetaData& depthMD)
/ / set the data of the mock-depth generator and pose will tracking will run on this
g_mock_depth.SetData (depthMD);
//rest of the code
//the function for modifying depth data could like this
void transformDepthMD(DepthMetaData& depthMD)
          xn::Depthmap & depthMap = depthMD.WritableDepthMap ();
          for ( unsigned int y = 0; y < depthMap.YRes (); + + y)
                   for ( unsigned int x = 0; x < depthMap.XRes (); + + x)
                           depthMap (x, y) = 0; // replace the values of the depth map by recorded depth map

Wednesday, April 18, 2012

Building OpenNI from Source on Linux

You will find lot of tutorials on building OpenNI from source on ubuntu.
I encountered only one error and had to go through number of blogs to figure it out.

Error is something like this

/usr/bin/ld: ./x64-Release/NiSimpleViewer.o: undefined reference to symbol 'glEnd'
/usr/bin/ld: note: 'glEnd' is defined in DSO /usr/lib/ so try adding it to the linker command line
/usr/lib/ could not read symbols: Invalid operation
collect2: ld returned 1 exit status


Goto OpenNI/Platform/Linux/Build/Samples/

In each sample try to find the line

USED_LIBS += glut

and replace it with

 USED_LIBS += glut GL

hope it helps :)

Wednesday, April 11, 2012

Openni Tracker with ROS

I have read on many posts that guys sometimes have problem using the openni_tracker, to me also it appeared at first that openni tracker is not working properly or something. There are few things to keep in mind.
Openni_tracker will not show anything except console messages, like Calibration started, Tracking started. To visualize the tracker run rviz. And add TF.
TF transforms will not be published until the program has started tracking the user. Openni_tracker works fine if the camera can see the whole body or till knees at least.

Kinect Driver and Samples Installation on Ubuntu 11.04

Nice tutorial for the drivers installation on Ubuntu

Converting ROS messages to Open CV images

Following tutorial provides a comprehensive explanation and implementation details about converting ros messages to opencv images.

This tutorial works with RGB images, and you can easily extend it to depth images
 Make sure you change the subscribed topic and image encodings e.g 16UC1 for raw depth data, following code snippet might be of help :)

 for (int i=0;i<cv_ptr->image.rows;i++)
         for (int j=0;j<cv_ptr->image.cols;j++)
             pixel_depth = cv_ptr->[i*cv_ptr->image.cols+j];
             if (pixel_depth == 0)
               // do your stuff 

Saturday, April 7, 2012

Eclipse with ROS

If you want to write scripts for ROS using Eclipse editor and project settings, you can do so by following some ordered steps.
  1. Create a ROS package  # Create a ros package in your ros-workspace
  2. roscd package_name     # Goto the directory where you created your package 
  3. make eclipse-project      # This is an important step to tell eclipse that we will be using our package in eclipse as a project
  4. Goto Import ->package_directory->apply
  5. Add the executable name to CMakelists.txt file
  6. rosmake package_name
  7. Goto Run Configurations {give path to your binary} -> Apply
  8. Run
If you get an error while doing step 3, try the following

For roscpp, your project needs to follow the standard package template, where the Makefile looks like:
 include $(shell rospack find mk)/
And nothing more (i.e. the rest of your bulid is configured via CMake). The above file declares the eclipse-project target.
If you don't wish to use the file, then you can grab the necessary Makefile rule from mk/ and adapt it to your needs.

Enjoy your project in eclipse now :)

Creating new python project
Make sure you first create a source folder in the project by right clicking on your project name and then New-> Source folder.
The source folder created by the rospkg does not work.
Then go on to add the new file {pydev module}.
By default the project name will be weird like containing '@' symbols so make sure to renaming your project to something nice before you try running it.

Passing Runtime arguments in Eclipse-Python
import sys
sys.argv = raw_input('Enter command line arguments: ').split()

Tuesday, April 3, 2012

Testing Openni Kinect with ROS

Openni Kinect with ROS

 ROS is an operating system developed at Stanford Artificial Intelligence Lab in 2008 .

Once you have followed the instructions provided here and installed everything accordingly, you might still face some problems, e.g
  • Rviz crashes with "Bad Drawable" command. 
[Fix]:  Remove the ".rviz" folder by typing "rm -rf .rviz" assuming you are in your home directory.

Frame Rate drop
You might come across this problem if you have just installed your linux and started running openni_kinect programs. You might notice that a considerable frame drop if you add a 'point cloud' display along with an 'image/camera' display in 'rviz'.
[Fix]: Check that you have installed the Graphics driver on your system :)

Playing Depth Data with Rosbag
You might have a problem playing 'depth' data rosbag. You might want to follow following steps.
Recording data
rosbag record -O depth_data.bag /camera/depth/image_raw

Playing data 
rosbag play depth_data.bag

Visualizing data
[1] rosrun rviz rviz   'or'
[2] rosrun image_view image_view image:=/camera/depth/image_raw

Running python API for Rosbag
Make sure you have set the path variable and add these two lines to the top of your main python script to avoid any dependency errors.

import roslib; roslib.load_manifest('your_package_name')

then write your rest of the script. Please keep in mind that this does not need to be in every python script but just in the main entry point.
Depth and RGB synchoronization 
RGB and depth data from Kinect are not synchronized  so you have to enable it. Thanks to ROS who does it for us. You just need to run the following
[1] Run the node which publishes the topics e.g roslaunch openni_launch openni.launch
[2] rosrun dynamic_reconfigure reconfigure_gui
[3] Goto the /Camera/drivers and then check the depth registration check box.

Now your both RGB and depth data are synchronized :)

Openni User Tracker on Linux Demo

Recording Point Cloud

When you record large amount of data published by some topics such as point clouds the system's memory gets exhausted.
What you might want to do is :
If you don't need all of this data, you can use the topic tools to drop or throttle messages, or the rostopic --filter to get rid of some messages.
Recording bag files in  a distributed way
Note: For those who want to record data from on board camera of a quadcopter

You can run the quadcopter on one PC and the recorder {camera driver on the other so that the load on recording machine is not that much }
export ROS_MASTER_URI=http://ip_address:port
Remember to do it in every terminal if you are using multiple
run camera launch on the one where you want to record the data

Subscribing data to a bag file publishers