Difference between revisions of "ECE497 Project Web Face Recognition"

From eLinux.org
Jump to: navigation, search
m
 
(23 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:ECE497 |PW]]
+
[[Category:ECE497Fall2017|W]]
 
{{YoderHead}}
 
{{YoderHead}}
  
 
Team members: [[user:Wyvette0542|Yvette Weng]], [[user:Wangc|John Wang]]
 
Team members: [[user:Wyvette0542|Yvette Weng]], [[user:Wangc|John Wang]]
  
== Grading Template ==
 
I'm using the following template to grade.  Each slot is 10 points.
 
0 = Missing, 5=OK, 10=Wow!
 
 
<pre style="color:red">
 
00 Executive Summary
 
00 Installation Instructions
 
00 User Instructions
 
00 Highlights
 
00 Theory of Operation
 
00 Work Breakdown
 
00 Future Work
 
00 Conclusions
 
00 Demo
 
00 Late
 
Comments: I'm looking forward to seeing this.
 
 
Score:  10/100
 
 
</pre>
 
</pre>
  
Line 27: Line 9:
  
 
== Executive Summary ==
 
== Executive Summary ==
 
+
[[File:Face Rec.jpg|thumbnail|System Overview]]
The goal of this project is to use webcam to capture face and compare with the database to identify the person. We will use Microsoft's face API to compare the photos. The database will contain students' pictures and can be modified at any time.  
+
The goal of this project is to use webcam to capture face and compare with the database to identify the person. Besides that, we also aim to create an easy-to-use adding face mechanism and error handling system. We will use Microsoft's face API to generate faceID and compare photos. The database will contain students' pictures and can be modified at any time.  
  
 
There are two buttons connected to the bone. One is for capturing and identifying the student's face and the other one is to add new face to the database.  
 
There are two buttons connected to the bone. One is for capturing and identifying the student's face and the other one is to add new face to the database.  
  
Here is the picture of the overall system.  
+
Here is the picture of the overall system.
 
 
 
 
  
 
== Packaging ==
 
== Packaging ==
There is no special packaging for this project.  
+
There is no special packaging for this project. Although, as shown in the system overview picture, the wiring is simple and clear, most of interesting stuffs are under the hood.
  
 
== Installation Instructions ==
 
== Installation Instructions ==
 +
Here is the github repository:  [https://github.com/wyvette0542/ECE497_Final github].
  
Give step by step instructions on how to install your project. 
 
 
Here is the github repository:  [https://github.com/wyvette0542/ECE497_Final github].
 
* Be sure your README.md is includes an up-to-date and clear description of your project so that someone who comes across you git repository can quickly learn what you did and how they can reproduce it.
 
* Include a Makefile for you code.
 
* Include any additional packages installed via '''apt'''.
 
* Include kernel mods.
 
* If there is extra hardware needed, include links to where it can be obtained.
 
 
In this project we use Playstation Eye Camera and ili9341 LCD screen. The LCD screen should be included in your lab kit, and you can borrow the camera from Dr. Yoder or ECE instrument room.
 
In this project we use Playstation Eye Camera and ili9341 LCD screen. The LCD screen should be included in your lab kit, and you can borrow the camera from Dr. Yoder or ECE instrument room.
  
Line 91: Line 64:
 
  uri_base = 'eastus.api.cognitive.microsoft.com'
 
  uri_base = 'eastus.api.cognitive.microsoft.com'
  
You can get yours from [https://azure.microsoft.com/en-us/try/cognitive-services/?api=face-api Microsoft Azure Face API] webpage for free. One thing to notice here is free trial key will only allow up to 20 transactions per minutes. We are using a paid version here for faster transaction.
+
You can get yours from [https://azure.microsoft.com/en-us/try/cognitive-services/?api=face-api Microsoft Azure Face API] for free. One thing to notice here is free trial key will only allow up to 20 transactions per minutes. We are using a paid version here for faster transaction. You can, of course, try our key. But keep in mind this key will expire in 20 days, so there is very little chance they will still work when you try.  
  
 
Finally, make sure you have saved your change and run:
 
Finally, make sure you have saved your change and run:
 
  bone$ ./setup.sh
 
  bone$ ./setup.sh
  
You are now good to try the face recognition system.
+
You are now good to try the face recognition system by running main.py. The instructions will be showed on LCD screen.
  
 
== User Instructions ==
 
== User Instructions ==
Line 104: Line 77:
 
Our group used C for capturing with webcam and we used python for face identification. You can do multiple tings with the code base that is in Github.  
 
Our group used C for capturing with webcam and we used python for face identification. You can do multiple tings with the code base that is in Github.  
  
'''database'''
+
'''Database'''
  
 
The database folder contains all students' pictures and all the names are stored in face.dat.  
 
The database folder contains all students' pictures and all the names are stored in face.dat.  
Line 125: Line 98:
 
4-2. In order to make face identification:  
 
4-2. In order to make face identification:  
 
  push the red button, wait for the program to identify, see your name shows up on LCD screen.
 
  push the red button, wait for the program to identify, see your name shows up on LCD screen.
 +
 +
'''Image Grabber'''
 +
 +
For this specific webcam, PlayStation Eye camera, you MUST set the resolution to be 320*280 or lower. Otherwise this camera won't save any picture or video clip at all.
  
 
== Highlights ==
 
== Highlights ==
  
This project works with both pictures and humans. The program can successfully identify a person even if the person's face is partially covered or out of the frame.  
+
This project works with both face pictures and human faces. The program can successfully identify a person even if the person's face is partially covered or out of the frame. In most cases, the program can even recognize a person with or without accessories like hat and glasses. For the sake of simplicity, we didn't show this feature in video demo but was presented during the physical demo session.
  
You can see a pretty cool, using Justin Bieber's pictures, demo [https://youtu.be/AneM-AGpXlc here].
+
You can see a pretty cool demo [https://youtu.be/AneM-AGpXlc here]. We used two pictures from Justin Bieber where one was added to the database and another was used to verify the identity. Although there are some similarities shared between two pictures, they are visually distinguishable with different face angles, shadow levels and facial expressions. After we pressed the green button and typed the user name, the first picture was put into database. We then presented camera the second picture, after running verification, the result was shown on screen.
  
 
== Theory of Operation ==
 
== Theory of Operation ==
 +
The program can be divided into three parts: image grabber, face detect, face verification. Press both buttons will run the image grabber and save picture to program's root directory. The default camera output file type is pmm file which can't be display or process easily. Therefore, the pmm file will then be converted to jpg file whose name depends of work type. If user presses green button(add new face), the file will be named as the number of pictures plus one and move into database. In another scenario where user presses red button(verify face), the file will be named as 1.jpg. The detailed face detection, verification and error handling are showed in flowchart. One thing to notice is that, the input for face verification is not picture but faceIDs that returned from face detection. All the pmm and jpg files in root directory will be removed when verification, detection finish or an error is detected.
  
Give a high level overview of the structure of your software.  Are you using GStreamer?  Show a diagram of the pipeline. Are you running multiple tasks?  Show what they do and how they interact.
+
 
 +
A high resolution pdf version of our operation flowchart can be found here: [[File:FaceRec ToO.pdf|thumbnail|right|Operating Flowchart]]
 +
[[File:FaceRec ToO.jpg|thumbnail|right|Operation Flowchart]]
  
 
== Work Breakdown ==
 
== Work Breakdown ==
Line 160: Line 140:
  
 
You could also add an error handler when adding the new faces. Right now the program adds a face no matter the captured picture contains a face or not, so if there is a "bad" picture, one needs to go to the database and delete the photo manually. An error handler could simplify this process by automatically deleting a "bad" picture when it detects no face in the photo.
 
You could also add an error handler when adding the new faces. Right now the program adds a face no matter the captured picture contains a face or not, so if there is a "bad" picture, one needs to go to the database and delete the photo manually. An error handler could simplify this process by automatically deleting a "bad" picture when it detects no face in the photo.
 +
 +
You can, if you really want to, use openCV instead of commercial face recognition services like Microsoft Face API. We will not recommend that as face verification requires huge amount of face training which are not the focus of this class. 
  
 
One cool thing we came up at the very end of this project was you can use the touch screen to replace two physical buttons. This need more research and experiment on using the LCD screen as an input device but will definitely make the whole system more compact and useful.
 
One cool thing we came up at the very end of this project was you can use the touch screen to replace two physical buttons. This need more research and experiment on using the LCD screen as an input device but will definitely make the whole system more compact and useful.
Line 165: Line 147:
 
== Conclusions ==
 
== Conclusions ==
  
The design featured in this project is a good foundation for IoT devices like smartlock. We finished the first part which is analysis and verification face. Next step we can focus on the Internet part. We have looked into Blynk which is an Internet of Things platform with a drag-n-drop mobile application builder and saw great potential in it. By utilizing the command line display function and streaming function we can shrink the physical size one step further. Besides that, as we mentioned in previous section, we haven't released full potential of LCD touch screen. Currently we only use it as a display. There are still much to do in this project and can be a good topic for future classes.
+
The design featured in this project is a good foundation for IoT devices like smartlock. We finished the first part which is analysis and verification face. Next step we can focus on the Internet part. We have looked into Blynk which is an Internet of Things platform with a drag-n-drop mobile application builder and saw great potential in it. By utilizing the command line display function and streaming function we can shrink the physical size one step further. Besides that, as we mentioned in previous section, we haven't released full potential of LCD touch screen. Currently we only use it as a display. A good amount of hour were spent to figure out how to take picture as the sample code from Derek Molloy won't work directly for our camera. Putting this main problem aside, our exploration on sending request, parsing JSON response, forming and organizing database went pretty smoothly. We also put lots of effort into optimizing user instructions. Our efforts do pay off as these instructions help users go through confusing parts without our help during demo session. Although we achieved a high degree of completion in this project, there are much more to do in it and can still be a good topic for future classes.
  
 
{{YoderFoot}}
 
{{YoderFoot}}

Latest revision as of 16:14, 23 May 2019

thumb‎ Embedded Linux Class by Mark A. Yoder


Team members: Yvette Weng, John Wang

(Inline Comment)

Executive Summary

System Overview

The goal of this project is to use webcam to capture face and compare with the database to identify the person. Besides that, we also aim to create an easy-to-use adding face mechanism and error handling system. We will use Microsoft's face API to generate faceID and compare photos. The database will contain students' pictures and can be modified at any time.

There are two buttons connected to the bone. One is for capturing and identifying the student's face and the other one is to add new face to the database.

Here is the picture of the overall system.

Packaging

There is no special packaging for this project. Although, as shown in the system overview picture, the wiring is simple and clear, most of interesting stuffs are under the hood.

Installation Instructions

Here is the github repository: github.

In this project we use Playstation Eye Camera and ili9341 LCD screen. The LCD screen should be included in your lab kit, and you can borrow the camera from Dr. Yoder or ECE instrument room.

To install required library and environment follow the instructions below. These installation instructions are for Debian and Ubuntu distributions:

Please make sure you have python3 installed. You can find the instruction here.

1 - Update your Operating System and install the dependencies:

bone$ sudo apt-get update
bone$ sudo apt-get install v4l-utils libv4l-dev fbset fbi imagemagick

2 - After installation, some of v4l2 utilities should be available to use, you can confirm by issuing the following command:

bone$ v4l2-ctl -h

Next we are going to make sure the camera is suitable for the job. You can find a detailed video instruction from Derek Molloy here.

3 - Make sure the camera is plugged in and run:

bone$ lsusb

You should be able to see your camera listed on there. To know more about your camera, run:

bone$ v4l2-ctl --all

If v4l2 gives you an error or it won't recognize your camera, sorry buddy, you have to use a different camera that supported by v4l2.

4 - Install hardware To install the ili9341 LCD screen, you need to connect it to GPIO0 and SP1 as shown in HW6

Besides that, the red button is connected to GP1-3, the green button is connected to GP1-4.

5 - Setup program Now we are good to setup the main program. To do this, first go to the file:

bone$ cd ECE497_Final
bone$ ls -l

You should see a file called apitest.py . Open it with your favorite editor. In this case, we simply use nano:

bone$ nano apitest.py

Once you open up the python file, you need to change the subscription_key and uri_base. For example, ours are:

# Replace the subscription_key string value with your valid subscription key.
subscription_key = '7ffe7a3ee1844bc2aa3211c4f02bbc55'
uri_base = 'eastus.api.cognitive.microsoft.com'

You can get yours from Microsoft Azure Face API for free. One thing to notice here is free trial key will only allow up to 20 transactions per minutes. We are using a paid version here for faster transaction. You can, of course, try our key. But keep in mind this key will expire in 20 days, so there is very little chance they will still work when you try.

Finally, make sure you have saved your change and run:

bone$ ./setup.sh

You are now good to try the face recognition system by running main.py. The instructions will be showed on LCD screen.

User Instructions

Programming Tools and Instructions

Our group used C for capturing with webcam and we used python for face identification. You can do multiple tings with the code base that is in Github.

Database

The database folder contains all students' pictures and all the names are stored in face.dat.

How to Run

1. Connect your BeagleBone Blue to your PC using the USB cable.

2. ssh into your Bone and make sure it has Internet connection. You can test this by ping google:

bone$ ping -c2 google.com

This program won't work without Internet connection.

3. Run the following line:

bone$ sudo ./main.py

4-1. In order to add new face to the database:

push the green button, enter your name, wait for the program to finish adding. 

4-2. In order to make face identification:

push the red button, wait for the program to identify, see your name shows up on LCD screen.

Image Grabber

For this specific webcam, PlayStation Eye camera, you MUST set the resolution to be 320*280 or lower. Otherwise this camera won't save any picture or video clip at all.

Highlights

This project works with both face pictures and human faces. The program can successfully identify a person even if the person's face is partially covered or out of the frame. In most cases, the program can even recognize a person with or without accessories like hat and glasses. For the sake of simplicity, we didn't show this feature in video demo but was presented during the physical demo session.

You can see a pretty cool demo here. We used two pictures from Justin Bieber where one was added to the database and another was used to verify the identity. Although there are some similarities shared between two pictures, they are visually distinguishable with different face angles, shadow levels and facial expressions. After we pressed the green button and typed the user name, the first picture was put into database. We then presented camera the second picture, after running verification, the result was shown on screen.

Theory of Operation

The program can be divided into three parts: image grabber, face detect, face verification. Press both buttons will run the image grabber and save picture to program's root directory. The default camera output file type is pmm file which can't be display or process easily. Therefore, the pmm file will then be converted to jpg file whose name depends of work type. If user presses green button(add new face), the file will be named as the number of pictures plus one and move into database. In another scenario where user presses red button(verify face), the file will be named as 1.jpg. The detailed face detection, verification and error handling are showed in flowchart. One thing to notice is that, the input for face verification is not picture but faceIDs that returned from face detection. All the pmm and jpg files in root directory will be removed when verification, detection finish or an error is detected.


A high resolution pdf version of our operation flowchart can be found here: File:FaceRec ToO.pdf

Operation Flowchart

Work Breakdown

1. Obtain and test the Face API. John Wang & Yvette Weng Completed
2. Face API works with local pictures using the picture's path. Yvette Weng Completed
3. Webcam is able to capture pictures. John Wang Completed
4. Set up database. John Wang & Yvette Weng Completed
5. LCD screen displays pictures and messages. John Wang Completed
6. Use button to send signal to capture pictures. Yvette Weng Completed

Future Work

There are a lot of things you can do with this project.

You could make the program auto-run, so that each time you power up your Bone, the program will start running.

You could also add an error handler when adding the new faces. Right now the program adds a face no matter the captured picture contains a face or not, so if there is a "bad" picture, one needs to go to the database and delete the photo manually. An error handler could simplify this process by automatically deleting a "bad" picture when it detects no face in the photo.

You can, if you really want to, use openCV instead of commercial face recognition services like Microsoft Face API. We will not recommend that as face verification requires huge amount of face training which are not the focus of this class.

One cool thing we came up at the very end of this project was you can use the touch screen to replace two physical buttons. This need more research and experiment on using the LCD screen as an input device but will definitely make the whole system more compact and useful.

Conclusions

The design featured in this project is a good foundation for IoT devices like smartlock. We finished the first part which is analysis and verification face. Next step we can focus on the Internet part. We have looked into Blynk which is an Internet of Things platform with a drag-n-drop mobile application builder and saw great potential in it. By utilizing the command line display function and streaming function we can shrink the physical size one step further. Besides that, as we mentioned in previous section, we haven't released full potential of LCD touch screen. Currently we only use it as a display. A good amount of hour were spent to figure out how to take picture as the sample code from Derek Molloy won't work directly for our camera. Putting this main problem aside, our exploration on sending request, parsing JSON response, forming and organizing database went pretty smoothly. We also put lots of effort into optimizing user instructions. Our efforts do pay off as these instructions help users go through confusing parts without our help during demo session. Although we achieved a high degree of completion in this project, there are much more to do in it and can still be a good topic for future classes.




thumb‎ Embedded Linux Class by Mark A. Yoder