https://elinux.org/api.php?action=feedcontributions&user=Turnerdm&feedformat=atomeLinux.org - User contributions [en]2024-03-28T18:36:25ZUser contributionsMediaWiki 1.31.0https://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=538106ECE434 Project - Mandroid2020-11-10T04:08:25Z<p>Turnerdm: /* Conclusions */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package (easier, but not recommended) ====<br />
<br />
Base install:<br />
<br />
wget https://github.com/blueOkiris/man-droid/releases/download/1.0/mandroid-arm.deb<br />
sudo dpkg -i mandroid.deb<br />
<br />
Then to make easier to access (i.e. run it by calling "mandroid" in a terminal):<br />
<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
At this moment in time, Jorge the Mandroid can do three things:<br />
<br />
* Respond to "hello"<br />
* Respond to "goodbye" and quit the program<br />
* Respond to "tell me about _____" which will cause him to duckduckgo search a small paragraph about a topic<br />
<br />
[https://www.youtube.com/watch?v=8WY1pg1DUaE Here is a Youtube video demonstrating]<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
The project isn't amazing. It definitely is a good framework, but pretty much all aspects could be improved to some degree.<br />
I definitely feel like I've created a system that can be easily expanded upon in the future with more time.<br />
<br />
As far as future additions go, for one thing, more typical assistant tools like weather, calendar, music playing, etc.<br />
Also maybe a hand/body for moving and grabbing things playing with the idea of an android.<br />
<br />
Overall I'm happy with how it turned out, but recognize it isn't wholly what I set out to achieve.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=538101ECE434 Project - Mandroid2020-11-10T04:05:18Z<p>Turnerdm: /* Highlights */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package (easier, but not recommended) ====<br />
<br />
Base install:<br />
<br />
wget https://github.com/blueOkiris/man-droid/releases/download/1.0/mandroid-arm.deb<br />
sudo dpkg -i mandroid.deb<br />
<br />
Then to make easier to access (i.e. run it by calling "mandroid" in a terminal):<br />
<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
At this moment in time, Jorge the Mandroid can do three things:<br />
<br />
* Respond to "hello"<br />
* Respond to "goodbye" and quit the program<br />
* Respond to "tell me about _____" which will cause him to duckduckgo search a small paragraph about a topic<br />
<br />
[https://www.youtube.com/watch?v=8WY1pg1DUaE Here is a Youtube video demonstrating]<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537911ECE434 Project - Mandroid2020-11-05T19:13:35Z<p>Turnerdm: /* Software */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package (easier, but not recommended) ====<br />
<br />
Base install:<br />
<br />
wget https://github.com/blueOkiris/man-droid/releases/download/1.0/mandroid-arm.deb<br />
sudo dpkg -i mandroid.deb<br />
<br />
Then to make easier to access (i.e. run it by calling "mandroid" in a terminal):<br />
<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537796ECE434 Project - Mandroid2020-11-05T01:58:39Z<p>Turnerdm: /* Installing from Package */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package ====<br />
<br />
Base install:<br />
<br />
wget https://github.com/blueOkiris/man-droid/releases/download/1.0/mandroid-arm.deb<br />
sudo dpkg -i mandroid.deb<br />
<br />
Then to make easier to access (i.e. run it by calling "mandroid" in a terminal):<br />
<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537706ECE434 Project - Mandroid2020-11-03T18:58:28Z<p>Turnerdm: /* Installing from Package */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package ====<br />
<br />
THIS IS THE BETA RELEASE AND IS NOT COMPLETE YET!<br />
<br />
Base install:<br />
<br />
wget https://github.com/blueOkiris/man-droid/releases/download/beta-1.0/mandroid-arm.deb<br />
sudo dpkg -i mandroid.deb<br />
<br />
Then to make easier to access (i.e. run it by calling "mandroid" in a terminal):<br />
<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537701ECE434 Project - Mandroid2020-11-03T18:58:07Z<p>Turnerdm: /* Installing from Package */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package ====<br />
<br />
THIS IS THE BETA RELEASE AND IS NOT COMPLETE YET!<br />
<br />
Base install:<br />
<br />
wget https://github.com/blueOkiris/man-droid/releases/download/beta-1.0/mandroid-arm.deb<br />
sudo dpkg -i pkg/mandroid.deb<br />
<br />
Then to make easier to access (i.e. run it by calling "mandroid" in a terminal):<br />
<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537696ECE434 Project - Mandroid2020-11-03T18:50:22Z<p>Turnerdm: /* Software */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program.<br />
<br />
==== Installing from Package ====<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make mandroid.deb<br />
sudo dpkg -i pkg/mandroid.deb<br />
<br />
Then to make easier to access:<br />
ln -s /opt/mandroid/mandroid /usr/bin/mandroid<br />
<br />
And finally, to startup on boot:<br />
sudo systemctl enable mandroid<br />
<br />
==== Building from Source ====<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537691ECE434 Project - Mandroid2020-11-03T17:11:30Z<p>Turnerdm: /* Theory of Operation */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
It should be noted that I've made an attempt to never let the program crash, and to keep retrying or simply move on if something fails.<br />
<br />
In cases of failure or dysfunction, when installed a service, the program outputs to /var/mandroid.log.<br />
<br />
Speaking of the service it installs the program and necessary files to /opt/mandroid/ and creates a symlink to /usr/bin/mandroid.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537551ECE434 Project - Mandroid2020-11-03T03:55:37Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|640px|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537546ECE434 Project - Mandroid2020-11-03T03:49:07Z<p>Turnerdm: /* Future Work */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
Also, it may be possible to power the beaglebone from the same power source as the servo.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537541ECE434 Project - Mandroid2020-11-03T03:48:04Z<p>Turnerdm: /* User Instructions */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do <br />
<br />
sudo make install<br />
<br />
after building, the program should autostart upon reboot.<br />
<br />
If you opt-out of that, you can start it with start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537536ECE434 Project - Mandroid2020-11-03T03:32:47Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|480px]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537531ECE434 Project - Mandroid2020-11-03T03:32:16Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png|frame]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537526ECE434 Project - Mandroid2020-11-03T03:31:53Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, and it's described in the fritzing diagram below. The transistors used are NPN B547.<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see how it physically attaches, view the images in the above section.<br />
<br />
Fritzing diagram:<br />
<br />
[[File:Mandroid-diagram bb.png]]<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=File:Mandroid-diagram_bb.png&diff=537521File:Mandroid-diagram bb.png2020-11-03T03:31:35Z<p>Turnerdm: Fritzing diagram for mandroid project</p>
<hr />
<div>== Summary ==<br />
Fritzing diagram for mandroid project</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537496ECE434 Project - Mandroid2020-11-03T02:54:37Z<p>Turnerdm: /* Work Breakdown */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see the attachments, go to the above section.<br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
** Building structure to support jaw (T-structure)<br />
** Attaching motor to jaw and to support structure<br />
* Speech Recognition<br />
** Link to Python speech library<br />
** Attach USB microphone<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
** Attach speaker through USB<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
** Create main dowel "tower" to place brain on top and rest mask over parts<br />
<br />
The main areas of improvement are improving the sound of the speech synthesis and the brain's complexity.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537491ECE434 Project - Mandroid2020-11-03T02:50:35Z<p>Turnerdm: /* Packaging */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="400px" heights="400px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see the attachments, go to the above section.<br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537486ECE434 Project - Mandroid2020-11-03T02:50:13Z<p>Turnerdm: /* Packaging */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
<gallery mode=nolines widths="480px" heights="480px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see the attachments, go to the above section.<br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537481ECE434 Project - Mandroid2020-11-03T02:49:57Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
To see the attachments, go to the above section.<br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537476ECE434 Project - Mandroid2020-11-03T02:49:31Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
Hardware images:<br />
<br />
<gallery mode=nolines widths="480px" heights="480px"><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537471ECE434 Project - Mandroid2020-11-03T02:45:20Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
The T-shaped dowel structure is attached to a servo at the angle. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
Hardware images:<br />
<br />
<gallery><br />
Mandroid-usb-hardware.jpg|The central dowel with attached usb microphone and speaker<br />
Mandroid-beagle-brain.jpg|The location of the beaglebone with servo power circuit on top within the head of the Mandroid<br />
Mandroid-t-bar.jpg|The T shaped dowel structure with attached servo within the mask of the Mandroid<br />
</gallery><br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=File:Mandroid-beagle-brain.jpg&diff=537466File:Mandroid-beagle-brain.jpg2020-11-03T02:43:25Z<p>Turnerdm: The location of the beaglebone with servo power circuit on top within the head of the Mandroid</p>
<hr />
<div>== Summary ==<br />
The location of the beaglebone with servo power circuit on top within the head of the Mandroid</div>Turnerdmhttps://elinux.org/index.php?title=File:Mandroid-t-bar.jpg&diff=537461File:Mandroid-t-bar.jpg2020-11-03T02:41:29Z<p>Turnerdm: The T shaped dowel structure with attached servo within the mask of the Mandroid</p>
<hr />
<div>== Summary ==<br />
The T shaped dowel structure with attached servo within the mask of the Mandroid</div>Turnerdmhttps://elinux.org/index.php?title=File:Mandroid-usb-hardware.jpg&diff=537456File:Mandroid-usb-hardware.jpg2020-11-03T02:38:37Z<p>Turnerdm: The usb connections on the primary dowel of the Mandroid</p>
<hr />
<div>== Summary ==<br />
The usb connections on the primary dowel of the Mandroid</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537451ECE434 Project - Mandroid2020-11-03T02:26:17Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below (though it will be in the fritzing diagram as well). The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
Fritzing diagram:<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537441ECE434 Project - Mandroid2020-11-02T20:42:45Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537436ECE434 Project - Mandroid2020-11-02T20:42:31Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|MandroidHead.jpg]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537431ECE434 Project - Mandroid2020-11-02T20:42:20Z<p>Turnerdm: /* Theory of Operation */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|Bot.png]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537426ECE434 Project - Mandroid2020-11-02T20:42:03Z<p>Turnerdm: /* Theory of Operation */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
There is also a set of test functions (Tests.hpp) which go through the various functionalities.<br />
<br />
Hierarchy:<br />
<br />
[[File:Bot.png|thumb|]]<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=File:Bot.png&diff=537421File:Bot.png2020-11-02T20:41:19Z<p>Turnerdm: Class hierarchy for Mandroid</p>
<hr />
<div>== Summary ==<br />
Class hierarchy for Mandroid</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537416ECE434 Project - Mandroid2020-11-02T20:19:09Z<p>Turnerdm: /* Theory of Operation */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
At the top level, there is the Mandroid object (Brain.hpp/.cpp). It is an abstract class in C++ and has instances of two other abstract classes: a SpeechRecognizer as its ears (Listen.hpp/.cpp) and a SpeechSynthesizer as its mouth (Speech.hpp/.cpp).<br />
<br />
The created instance of a Mandroid is currently a child called IfElseBot (Brain.hpp/.cpp). This implementation is based on if and if-else statements, the most barebones way to program a chat bot. Another implementation could utilize a natural language processing library or machine learning to be more "real," but as it stands, the only implementation is the IfElseBot.<br />
<br />
As an implementation of the Mandroid class, the IfElseBot utilizes a SpeechRecognizer and a SpeechSynthesizer. The specific children of these abstract classes utilized by the IfElseBot are a PythonSpeechRecognizer (Listen.hpp/.cpp) which calls a python function from the C++ code to process language into an std::string and a ClipBasedSpeechSynthesizer (Speech.hpp/.cpp) which loads audio clips and pieces them together to produce sound.<br />
<br />
The ClipBasedSpeechSynthesizer also makes use of a Servo (Servo.hpp/.cpp) to physically move a mouth. This Servo makes a system call to launch a python program that initializes the PWM pin (for some reason it was the only way to make it work). It then uses the sysfs interface to control the duty cycle driven into the physical servo.<br />
<br />
Back to the top-level, now that it can speak and hear, the IfElseBot is able to process speech and produce a result. One of its operations also makes use of a python library that grabs information about a topic from Duck Duck Go. The other operation exits the program.<br />
<br />
Different implementations of speech are possible as long as they produce methods for producing sound from IPA and converting English text to IPA. Different implementations for listening are possible as long as they have a listen method for producing a string represent heard speech. Different implementations of the Brain are possible as long as they have a respond function which produces a boolean for if the program should quit or not.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537411ECE434 Project - Mandroid2020-11-02T20:03:08Z<p>Turnerdm: /* User Instructions */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537406ECE434 Project - Mandroid2020-11-02T20:01:57Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- |/<br />
| |______|<br />
\| |\ <br />
|--- P9_22 V<br />
/| |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537401ECE434 Project - Mandroid2020-11-02T19:34:07Z<p>Turnerdm: /* Software */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- /<br />
| |______|<br />
\ \ <br />
|--- P9_22 V<br />
/ |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile (all instances of 3.7 become 3.x or whatever), or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537396ECE434 Project - Mandroid2020-11-02T19:32:52Z<p>Turnerdm: /* Future Work */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- /<br />
| |______|<br />
\ \ <br />
|--- P9_22 V<br />
/ |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
The biggest improvement is in the chat bot, giving it natural language processing and more commands it can do.<br />
<br />
Afterwards, you'd have to improve the speech synthesis. It sounds like a speak and spell, and the dictionary of known pronunciations is small, so it has to guess pronunciations a lot.<br />
<br />
Both of these are based on abstract classes, so both additions could integrate well with the system.<br />
<br />
Beyond that, the power circuitry can be improved so there's less power cords, a USB wifi adapter would make it simpler to initialize, the hardware could be better hidden, the physical structure could be more robust, and more motion could be added to the face like moving eyes and multiple "muscles" for better facial movement.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537391ECE434 Project - Mandroid2020-11-02T19:29:50Z<p>Turnerdm: /* User Instructions */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- /<br />
| |______|<br />
\ \ <br />
|--- P9_22 V<br />
/ |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
When you do `sudo make install`, the program should autostart upon reboot.<br />
<br />
Otherwise, start it with:<br />
<br />
cd <Location of Repo><br />
./mandroid<br />
<br />
<br />
Once running, you can talk to the robot and it will respond.<br />
<br />
Currently, you can only say "bye" or "goodbye" to end the program and "tell me about `x`" to get web info on `x`<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537386ECE434 Project - Mandroid2020-11-02T19:27:25Z<p>Turnerdm: /* External Hardware */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone. A T-shaped structure created from thin dowels attaches at the mask's ears and nose, and the motor rests under their joint, attached with hot-glue.<br />
<br />
Another dowel is attached to a servo. I used a 5V 20kg servo because of the weight. This central dowel is connected to the jaw of the head, which has been cut where it meets the chin and on the sides, so the mouth can open more easily.<br />
<br />
There is a small circuit to provide a large current to the servo without voltage drops (as the servo can be driven with a 3.3V signal which can be provided by the Beaglebone, but requires 4.8-6.8V with a decent current at its source). This circuit is dead simple, so I'll make it with ASCII art below. The two transistors are B547 NPN.<br />
<br />
/<br />
/| 5V<br />
|<br />
*-------------------<br />
| |<br />
Z 1kOhm Z 1kOhm<br />
Z Z<br />
| |<br />
| *---------- Servo Control Pin<br />
*----------- /<br />
| |______|<br />
\ \ <br />
|--- P9_22 V<br />
/ |<br />
V |<br />
| ---<br />
--- -<br />
-<br />
<br />
Attached to the central pole is a downward facing Rock-band microphone and an outward facing mini-speaker. The microphone is a USB mic, and the speaker attaches to a 3.5mm cable which thin goes into a USB adapter. Then both USB devices are plugged into a hub which hangs down from the beagle bone.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537381ECE434 Project - Mandroid2020-11-02T19:17:18Z<p>Turnerdm: /* Packaging */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
Everything is placed inside the mask and uses wooden dowels glued together with hot glue to hold it with support.<br />
<br />
The base is connected to a thick wooden dowel which has a second platform on its top. On this secondary platform rests the BeagleBone black, and on top of the black is a tiny breadboard with a circuit to control the servo.<br />
<br />
The servo is attached to the support structure within the mask whereas all other components are attached to the primary wooden dowel and simply *covered* by the mask. The mask itself (with servo) is completely removable to allow tinkering with the internals.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537376ECE434 Project - Mandroid2020-11-02T19:15:16Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, speech recognition, the mouth, and a rudimentary chat bot all working!<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537371ECE434 Project - Mandroid2020-11-02T19:14:44Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:MandroidHead.jpg|frameless|]]<br />
<br />
Image of mask. To be replaced with actual project near completion<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=File:MandroidHead.jpg&diff=537366File:MandroidHead.jpg2020-11-02T19:14:09Z<p>Turnerdm: The built mandroid with beaglebone inside</p>
<hr />
<div>== Summary ==<br />
The built mandroid with beaglebone inside</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537361ECE434 Project - Mandroid2020-11-02T19:01:44Z<p>Turnerdm: </p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:Mask-temp.png|frameless|Temporary picture of mask used until full project picture can be taken]]<br />
<br />
Image of mask. To be replaced with actual project near completion<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
git clone https://github.com/blueOkiris/python-duckduckgo<br />
cd python-duckduckgo<br />
sudo python3 setup.py install<br />
cd ..<br />
git clone https://github.com/blueOkiris/man-droid<br />
cd man-droid<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
** python-duckduckgo<br />
*** Download custom duckduckgo library (for search)<br />
*** Go into the directory<br />
*** Install it<br />
*** Leave the directory<br />
* Download main project from git<br />
* Go into the project folder<br />
* Build it with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537261ECE434 Project - Mandroid2020-11-01T19:07:06Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[File:Mask-temp.png|frameless|Temporary picture of mask used until full project picture can be taken]]<br />
<br />
Image of mask. To be replaced with actual project near completion<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Download from git<br />
** Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
* Build with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537256ECE434 Project - Mandroid2020-11-01T19:06:11Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
[[Mask-temp.png]]<br />
<br />
Image of mask. To be replaced with actual project near completion<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Download from git<br />
** Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
* Build with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537251ECE434 Project - Mandroid2020-11-01T19:05:49Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
<gallery><br />
Mask-temp.png|Image of mask. To be replaced with actual project near completion<br />
</gallery><br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Download from git<br />
** Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
* Build with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=File:Mask-temp.png&diff=537246File:Mask-temp.png2020-11-01T19:05:20Z<p>Turnerdm: Temporary image for my project, ECE434 - Mandroid. Will be replaced when project nears completion</p>
<hr />
<div>== Summary ==<br />
Temporary image for my project, ECE434 - Mandroid. Will be replaced when project nears completion</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537241ECE434 Project - Mandroid2020-11-01T19:02:59Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
/* Insert Image Here */<br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Download from git<br />
** Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
* Build with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537236ECE434 Project - Mandroid2020-11-01T19:02:25Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
<gallery><br />
Temp-mask-image.png|Temporary image to be replaced<br />
</gallery><br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Download from git<br />
** Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
* Build with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdmhttps://elinux.org/index.php?title=File:Temp-mask-image.png&diff=537231File:Temp-mask-image.png2020-11-01T19:01:54Z<p>Turnerdm: A temporary image to be replaced when my project is finished</p>
<hr />
<div>== Summary ==<br />
A temporary image to be replaced when my project is finished</div>Turnerdmhttps://elinux.org/index.php?title=ECE434_Project_-_Mandroid&diff=537226ECE434 Project - Mandroid2020-11-01T19:00:26Z<p>Turnerdm: /* Executive Summary */</p>
<hr />
<div>[[Category:ECE497 |Pm]]<br />
[[Category:ECE434Fall2020 |Mandroid]]<br />
<br />
Team members: [[user:Turnerdm|Dylan Turner]]<br />
<br />
== Executive Summary ==<br />
<br />
<gallery><br />
Example.jpg|Temporary image to be replaced<br />
</gallery><br />
<br />
A humanoid talking robot head.<br />
<br />
The head listens to you with a microphone, formulates a response, and then replies using speech synthesis while moving its mouth.<br />
<br />
Currently, I have PWM working on the beaglebone along with servo control (with a circuit for a high-torque servo), a rudimentary speech synthesizer, and speech recognition.<br />
<br />
The primitive chat-bot and the mouth are not working.<br />
<br />
An example of a much more complex version of what I'm aiming for can be found here. At a minimum, this robot will respond to some speech input, move its mouth a bit, and output some speech output. If I can't build a complex chat bot in time, that's okay with me. I plan to focus on the other three parts more than anything else.<br />
<br />
== Packaging ==<br />
<br />
If you have hardware, consider [http://cpprojects.blogspot.com/2013/07/small-build-big-execuition.html Small Build, Big Execuition] for ideas on the final packaging.<br />
<br />
== Installation Instructions ==<br />
<br />
=== External Hardware ===<br />
<br />
Obviously, there's a beaglebone<br />
<br />
The dowels and wooden base are used to make a framework for holding up the mask as well as holding the beaglebone.<br />
<br />
Another dowel is attached to a servo, which is attached to the central dowel. I used a 5V 20kg servo because of the weight.<br />
<br />
The motor's dowel is also connected at the other end to the base of the jaw so that the mouth can be opened.<br />
<br />
Beyond the structure itself, one needs a microphone. I used a Logitech usb microphone that was made for the game rockband.<br />
<br />
You also need a speaker. It can't be usb or use bluetooth over usb as the microphone is using that.<br />
<br />
=== Software ===<br />
<br />
First, make sure you have Python 3.7<br />
<br />
If you don't, then you can install it with:<br />
<br />
sudo apt install libpython3.7-dev<br />
<br />
If it gets deprecated, you can update the versions in the Makefile, or build it from source using the instructions [https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/ here].<br />
<br />
Afterwards, you're ready to install the program<br />
<br />
Here's the installation commands for installing the Mandroid software:<br />
<br />
git clone https://github.com/blueOkiris/man-droid<br />
sudo apt install -y libsdl2-dev libsdl2-mixer-dev python3-pyaudio pybind11-dev flac<br />
pip3 install PyAudio<br />
pip3 install SpeechRecognition<br />
make<br />
sudo make install<br />
<br />
Here's the explanation:<br />
<br />
* Download from git<br />
** Install Dependencies:<br />
** SDL2_mixer is required for speech synthesis: `libsdl2-dev libsdl2-mixer-dev`<br />
** The Python pip libraries `PyAudio` and `SpeechRecognition` is required for speech recognition. It relies on: `python3-pyaudio`<br />
** The python speech recognition library is called in C++ using pybind: `pybind11-dev`<br />
** Flac for audio input<br />
* Build with make<br />
* Install system service for running at start<br />
<br />
== User Instructions ==<br />
<br />
Once everything is installed, how do you use the program? Give details here, so if you have a long user manual, link to it here.<br />
<br />
Consider making it autostart for full credit.<br />
<br />
== Highlights ==<br />
<br />
Here is where you brag about what your project can do.<br />
<br />
Include a [http://www.youtube.com/ YouTube] demo the audio description.<br />
<br />
== Theory of Operation ==<br />
<br />
Give a high level overview of the structure of your software. Are you using GStreamer? Show a diagram of the pipeline. Are you running multiple tasks? Show what they do and how they interact.<br />
<br />
== Work Breakdown ==<br />
<br />
As the only team member, I did all of the work.<br />
<br />
The project can be broken up into four main sections with subsections.<br />
<br />
* Servo/Head control<br />
** PWM Control<br />
* Speech Recognition<br />
* Speech Synthesis<br />
** Recording sound files<br />
** IPA map to sound files<br />
** Synthesis object with instance of Servo<br />
* Brain (Chat-bot)<br />
** Tie it all together<br />
** Process inputs and produce sentences in IPA as response<br />
<br />
The only thing really unfinished is the brain, though the synthesis can also be improved.<br />
<br />
== Future Work ==<br />
<br />
Suggest addition things that could be done with this project.<br />
<br />
== Conclusions ==<br />
<br />
Give some concluding thoughts about the project. Suggest some future additions that could make it even more interesting.</div>Turnerdm