ECE497 Project BeagleBoard Kinect with Usable Framerates

Team members: Aaron Bamberger, Kyle Green, Ty Strayer

Executive Summary
There is currently no good solution for using a Kinect sensor with a BeagleBoard and a Linux build suited for embedded use (we are using Angstrom). Most current examples use a desktop-grade Linux build (Ubuntu chief among them) and run at horrendously slow frame rates (< 5 FPS is normal). We believe most of the problem is due to the Kinect drivers not being optimized to utilize the hardware the BeagleBoard has to offer. This project aims to improve the performance of the Kinect video display demo on the BeagleBoard by porting it to use the SGX graphics accelerator on the BeagleBoard using OpenGL ES. The Kinect video display demo uses a framework called GLUT (The OpenGL Utility Toolkit) to handle the creation of operating system shells and keyboard and mouse interaction. Unfortunately, OpenGL ES and the BeagleBoard do not support GLUT. We will need to replace the functionality of GLUT with the functionality of PVR, a toolkit with similar intentions to GLUT that is provided by the SGX hardware manufacturer and will run on the BeagleBoard.

Our progress to date includes configuring Angstrom with the proper video display mode so that the SGX graphics driver can run, installing the Kinect drivers under Angstrom and demonstrating that we have connectivity with the Kinect (through manipulation of its motors and LED), and getting some of the pre-compiled SGX demo programs to successfully run.

The issues still facing us include resolving some header and library dependencies in order to get the SGX demo programs compiling successfully on the BeagleBoard. Once we have succeeded with this, the only remaining step will be to modify the Kinect video display demo source code, replacing all GLUT functionality with its equivalent PVR functionality.

In conclusion, we believe we are very close to having a working Kinect video display demo running on the BeagleBoard with increased performance owing to its use of the BeagleBoards hardware graphics acceleration.

Kinect Drivers--libfreenect
Full information about libfreenect and its installation can be found at its site, here

Here I've reproduced just the sections that are useful for getting the drivers set up and working on the BeagleBoard

 Clone the git repository with all of the libfreenect source code in it git clone git://github.com/OpenKinect/libfreenect.git If you don't have git already installed on your BeagleBoard, install it with the following command opkg install git Get all of the packages you need for building the drivers To get the basic drivers compiled, you'll need the cmake and libusb-1.0-0 packages. Use the following command to install these packages to your BeagleBoard opkg install cmake libusb-1.0-0 NOTE: For some reason on our system the cmake package failed to install the first time, it's some issue with ncurses and dependent libraries. We just reran the command to install cmake and it worked the second time. Run the cmake utility to generate the necessary makefiles Go into your libfreenect directory and create a subdirectory named build cd libfreenect mkdir build Run cmake cmake .. Make the tiltdemo make tiltdemo NOTE: If you try to make everything, by just running make like it says in the libfreenect instructions, your make will fail with a bunch of linker errors. This is because there is no GLUT library for the BeagleBoard, so the video demos fail when they try to link against GLUT. We'll be solving this problem later by replacing GLUT calls with PVR calls, but for now, go ahead and make the tiltdemo to be sure you have connectivity with the Kinect Run the tiltdemo Go to the bin folder and ./tiltdemo NOTE: You may have to  chmod +x tiltdemo to get it to run. Now you should see the Kinect tilting up and down and cycling through the various LED options (Red, Green, Blink, Off). Press ctrl+c to exit the demo 

SGX Graphics Drivers
The PowerVR SGX graphics drivers are already installed as part of the SpED BeagleBoard image. We need to install the TI development package to get access to the development headers and example code. This package can either be installed on your host computer or on the BeagleBoard directly. When installed, it defaults to being set up to cross-compile, but for this project we are going to natively compile on the BeagleBoard to make our lives a bit simpler. This requires a couple of changes in the Makefiles, which will be detailed later in the instructions.  Download the TI Omap SGX drivers Get them from This Site. NOTE: You may need to create a login if you haven't already. Don't worry, it's free! Get whatever the latest release is. At the time of this writing it is Graphics_SDK_setuplinux_4_03_00_02.bin

Make the installer executable</li> chmod +x Graphics_SDK_setuplinux_4_03_00_02.bin

Install the Graphics SDK</li> ./Graphics_SDK_setuplinux_4_03_00_02.bin This will install a large source tree containing the drivers and lots of example code, both for OpenGL ES 1.0 and OpenGL ES 2.0. We're interested in the 1.0 stuff for this project, which can be found under the GFX_Linux_SDK/OGLES/SDKPackage subdirectory

Explore the source tree a bit</li> Poke around the source tree a bit to get a feel for what's included. Pay attention especially to the TrainingProgram sub-folder. This is where all of the example code is located. Also, the PDF located at the top level is a user's manual that contains some useful information. </ol>

Configuring the SpED Image to Work with the SGX Drivers
One slightly annoying feature of the SpED image is that even though it comes with the SGX drivers pre-configured, it defaults to a video mode that isn't compatible with the drivers, specifically it starts with 720p HD resolution with 32-bit color-depth. As far as we can tell, the SGX drivers won't run correctly unless the video mode is 640x480 resolution with 16-bit color depth. Clearly, this is an area that needs a bit more research, but for now it's just easier to set it up so that SpED boots at 640x480 with 16-bit color, so that the SGX drivers will work correctly.

There are two main ways of getting this set up.  Modify the boot.scr script to set up the appropriate video mode at boot</li> If you look in the boot partition of your SD card, you should see a couple of files named boot.scr and user.scr. These are u-boot boot scripts that control the way the kernel boots, including passing kernel command line arguments that set up the video mode. u-boot runs boot.scr by default if it's there, and it runs user.scr if the user button is pressed while booting up. We want to mess with boot.scr, so we don't have to worry about dealing with the user button. The boot scripts are stored in a mostly text-based format that has some binary signing at the top for u-boot's benefit. I wasn't able to open it in gedit, but I could under windows with notepad++. Then I just ignored the binary garbage at the top and pulled out the script text. We'll modify it and re-sign it later. The textual contents of the original boot script is reproduced below: bootargs=console=ttyS2,115200n8 mem=80M@0x80000000 mem=384M@0x88000000 mpurate=1000 buddy=none camera=lbcm3m1 vram=16M omapfb.vram=0:8M,1:4M,2:4M omapfb.mode=dvi:hd720 omapdss.def_disp=dvi root=/dev/mmcblk0p2 rw rootfstype=ext3 rootwait mmc init setenv dvimode hd720 setenv vram 16M omapfb.vram=0:8M,1:4M,2:4M if test "${beaglerev}" = "AxBx"; then setenv optargs mem=80M@0x80000000 musb_hdrc.fifomode=5 elif test "${beaglerev}" = "Cx"; then setenv optargs mem=80M@0x80000000 mem=128M@0x88000000 musb_hdrc.fifomode=5 else setenv optargs mem=80M@0x80000000 mem=384M@0x88000000 fi run loaduimage run mmcboot The parts we're interested in are the section of the bootargs that reads omapfb.mode=dvi:hd720, and the dvimode environment variable that is being set to hd720. We want to change these to read omapfb.mode=dvi:640x480MR-16@60 setenv dvimode 640x480MR-16@60 For those who are curious and haven't figured it out yet, 640x480 is the resolution, 16 is the color-depth in bits, and 60 is the refresh rate in Hz. I'm not sure what MR stands for Save this text file (I saved it as boot.cmd), and now it's time to turn it into a script that u-boot will recognize. For this we will use the mkimage utility. If you don't have this installed, it's in the uboot-mkimage package, so just use your package manager to grab that and then you'll be good to go. To turn our boot script into a u-boot format, we run the command mkimage -A arm -O linux -T script -C none -a 0 -e 0 -n 'Some Descriptive Text Here' -d boot.cmd boot.scr This will sign the boot script and produce the boot.scr file. Then you can just replace the boot.scr file in the boot partition of the SD card with your new one, and the BeagleBoard should boot up with the proper video mode enabled. NOTE: The new boot.scr will cause a bunch of errors to be spewed to the console early during boot-up. We haven't been able to figure out why these show up, but they don't appear to have any adverse effects on the boot process and the video mode gets set correctly, so just go ahead and ignore them. If we figure out how to fix this, we'll post the update here

Get rid of the boot scripts and let the defaults take over</li> The other way to get the proper video mode set at boot is to just get rid of both the boot.scr and user.scr files (or rename them so u-boot can't find them). It turns out that if u-boot can't find the boot scripts it's looking for, it will boot with some default, lowest common denominator settings, which at least for video mode happen to be 640x480 resolution with 16-bit color. If you go this route, u-boot will complain on boot that it can't find boot.scr or user.scr, but then it will keep booting normally and the video mode should be set correctly. This method has the advantage that it avoids all of the mysterious console error messages that get thrown with method 1, and it's also easier. The downside is you have less control over the boot process than you do when you use the boot scripts. </ol>

Once you've done either of the above methods, you can check if the mode got set properly by looking at the kernel command line. Go to /proc and cat cmdline. Somewhere in the command line you should see omapfb.mode=dvi:640x480-16@60. If you do, you should have the right video mode. As a final check, you can try and run one of the pre-compiled demo programs that came with the SGX drivers installation. Go to SDKPackage/Binaries/CommonX11/Demos and try running one of the demos you find there. I suggest ChameleonMan, because he runs like a chameleon and doesn't afraid of anything. If you can get one of the demos to run, congratulations, you have your video mode set up correctly.

Modifying the Kinect Display Program to Use PVR and SGX
This section is a work in progress. It will be updated as progress is made

User Instructions
Once the Kinect display program has been successfully ported from GLUT to PVR, using the program should be as simple as running the display program.

Highlights
This is a work in progress. Will hopefully be updated when we have something cool to show.

Theory of Operation


The above functional overview diagram shows the various hardware and software components of our project, and how they all fit together. The two areas of special note are highlighted in red and green. The box highlighted in red shows the PVR windowing and input toolkit. We are using this as a drop in replacement for the GLUT windowing and input toolkit, as GLUT does not run on the BeagleBoard. The box highlighted in green shows the glview Kinect display demo. We are modifying this piece of software to use the PVR toolkit instead of the GLUT toolkit

Major Tasks
Get libfreenect source and compile demos sans graphics--done

Get SGX drivers and demo code installed--done

Get BeagleBoard video mode set up correctly for SGX drivers--done

Get SGX demo programs running--done

Get SGX demo code compiling on BeagleBoard--started

Modify Kinect display demo to use PVR instead of GLUT--started

Conclusions
We believe that porting the Kinect display code to the BeagleBoard's onboard graphics accelerator will result in a significant performance boost, but we won't be sure by how much until we have it completely working. Some possible avenues for additional performance tweaks would be to identify parts of the low-level Kinect driver code that are DSP intensive, and port those parts to run on the BeagleBoard's embedded DSP using c6run.