ECE434 Project - Mandroid

From eLinux.org
Revision as of 12:17, 2 November 2020 by Turnerdm (talk | contribs) (Packaging)
Jump to: navigation, search


Team members: Dylan Turner

Executive Summary

MandroidHead.jpg

A humanoid talking robot head.

The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.

Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!

An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.

Packaging

Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.

The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.

The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.

Installation Instructions

External Hardware

Obviously, there's a beaglebone

The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.

Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.

The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.

Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.

You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.

Software

First, make sure you have Python 3.7

If you don't, then you can install it with:

sudo apt install libpython3.7-dev

If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions here.

Afterwards, you're ready to install the program

Here's the installation commands for installing the Mandroid software:

sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac
pip3 install PyAudio
pip3 install SpeechRecognition
git clone https://github.com/blueOkiris/python-duckduckgo
cd python-duckduckgo
sudo python3 setup.py install
cd ..
git clone https://github.com/blueOkiris/man-droid
cd man-droid
make
sudo make install

Here's the explanation:

  • Install Dependencies:
    • SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`
    • The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`
    • The python speech recognition library is called in C++ using pybind: `pybind11-dev`
    • Flac for audio input
    • python-duckduckgo
      • Download custom duckduckgo library (for search)
      • Go into the directory
      • Install it
      • Leave the directory
  • Download main project from git
  • Go into the project folder
  • Build it with make
  • Install system service for running at start

User Instructions

Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.

Consider making it autostart for full credit.

Highlights

Here is where you brag about what your project can do.

Include a YouTube demo the audio description.

Theory of Operation

Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.

Work Breakdown

As the only team member, I did all of the work.

The project can be broken up into four main sections with subsections.

  • Servo/Head control
    • PWM Control
  • Speech Recognition
  • Speech Synthesis
    • Recording sound files
    • IPA map to sound files
    • Synthesis object with instance of Servo
  • Brain (Chat-bot)
    • Tie it all together
    • Process inputs and produce sentences in IPA as response

The only thing really unfinished is the brain, though the synthesis can also be improved.

Future Work

Suggest addition things that could be done with this project.

Conclusions

Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.