ECE597 Project NodeRed Voice

From eLinux.org
Jump to: navigation, search

Px thumb‎ Embedded Linux Class by Mark A. Yoder


Team members: Dmitry , Matt Olson

Grading Template

I'm using the following template to grade. Each slot is 10 points. 0 = Missing, 5=OK, 10=Wow!

00 Executive Summary
00 Installation Instructions 
00 User Instructions
00 Highlights
00 Theory of Operation
00 Work Breakdown
00 Future Work
00 Conclusions
00 Demo
00 Late
Comments: I'm looking forward to seeing this.

Score:  10/100

(Inline Comment)

Executive Summary

Our project has branched out along several paths since it's inception. The goal was to create a personal assistant that could listen to commands, talk back, and perform them. For modularity, our original goal was to use IBM's Node Red to accomplish this. However, Node Red is not designed for the complicated and workflows that we needed, and extensions were not scale-able. In this document we are going to describe three things.

  • A simple node red speech recognition and synthesis system
  • Using the open source jasper project on Raspberry Pi
  • Using the open source Jasper project on Beaglebone Black

The simple node red speech synthesis will essentially listen to the user, and the repeat back what is said. Currently the speech synthesis is working, and we are trying to integrate pocket sphinx for the speech recognition.

Jasper was developed with the Raspberry Pi, and as a result has a very simple set up procedure. Our goal is too use this platform to begin developing modules. We currently have working versions of jasper installed and have 2 modules developed.

There are no reliable installation instructions for setting up Jasper on the Beaglebone black. We have done the research, and are hoping to be able to provide concise installation instructions and port over our work.

Installation Instructions

List of required hardware

Raspberry Pi Beaglebone Black
USB mic, Edimax USB wi-fi USB audiocard, USB/Audio-In mic
Network, power supply, Speakers
  • Note: Internet connectivity is an important part of the project as we are using the Google Speech API for voice recognition in the current revision.
  • Note 2: Additional hardware is required for some custom modules to function.

Setting up Network on RPi

There are two options for the internet access: wired connection and wi-fi connection. Both are very easy to manage with wicd connection manager with wicd-curses user interface.

To get wicd installed, plug the ethernet cord (assuming the initial raspbian installation it should work right out of the box) and then run:

sudo apt-get update
sudo apt-get install wicd
sudo apt-get install wicd-curses 

Alternatively /etc/network/interfaces file should be edited to include all of the following:

auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0

iface wlan0 inet dhcp
        wpa-ssid "ssid"
        wpa-psk "password"

This also allows auto-connect to a wireless network specified on a startup.


Installing Network on Beaglebone Black

Being processed

Setting up Network on Beaglebone

Complete the following instructions, which will guide you through the manual steps for installing Jasper:

http://teslafly.wordpress.com/2014/09/22/installing-jasper-on-the-beaglebone-black/

After you have completed and tested the installation, run the following commands to install pip and the adafruit GPIO library

sudo apt-get update
sudo apt-get install build-essential python-dev python-setuptools python-pip python-smbus -y
sudo pip install Adafruit_BBIO

Installing our custom Modules to demonstrate Jasper

git clone https://github.com/dmitryvv/woodoo
cd woodoo
chmod +x install_modules_BBB.sh
./install_modules_BBB.sh

Installing Jasper on raspberry Pi

Follow the instructions provided on the documentation page to install JASPER

http://jasperproject.github.io/documentation/software/

The modules we have implemented use the python RPI.gpio library, run the following commands to install it

sudo apt-get update
sudo apt-get install python-dev
sudo apt-get install python-rpi.gpio

Installing our custom Modules to demonstrate Jasper

git clone https://github.com/dmitryvv/woodoo
cd woodoo
chmod +x install_modules_RPi.sh
./install_modules_RPi.sh

Setting up wireless light control with MRF24J40MA

Section needs work, skip for now One of the custom modules is wireless light control based on 802.15.4 IEEE standard commonly used for home automation purposes.

For the proper functioning module there should be transmitter and receiver pieces assembled. The hardware and software instructions:

Rx Unit

Rx Unit is currently based on Arduino Duemilanove, MRF24J40MA wireless module, relay, power regulator and a number of resistors and capacitors. See figure 1 for the proper schematics. To load the right firmware to Arduino:

  1. Install Arduino IDE
  2. Open Arduino IDE and add mrf24j library using Sketch -> Import Library, the library itself lives in the Wireless_Light/arduino as a zip file.
  3. Open the arduino file from Wireless_Light/arduino/mrf-rx-tx and upload it to the board.
  4. Receiver should be ready to go.

Tx Unit

Tx Unit is the same MRF24J40MA transceiver but on the RPi side. Follow the steps below:

  1. Configure RPi to see SPI plus python I2C extension
  2. Run Wireless_Light/rpi/mrf24j_radio.py from the repo to test everything.
  3. Enable the module.

Setting up wireless light control with Xbee

This is a tested module that works correctly.

ZigBee Configuration

Use some FTDI Cable to conntect to XBee

  1. Download XCTU utility
  2. ZigBee needs to be burned with latest RS 232 ADAPTER firmware for the right board family
  3. Make sure that API Mode is enabled in the register list in XCTU
  4. Flow control is optional

Rx Unit

This Rx Unit is based on Arduino Duemilanove, XBee X24 wireless module, RGB LED and a number of resistors and capacitors. To load the right firmware to Arduino:

  1. Install Arduino IDE
  2. Open Arduino IDE and add xbee library using Sketch -> Import Library, the library itself lives in the Wireless_Light/arduino as a zip file.
  3. Open the arduino file from Wireless_Light/arduino/xbee and upload it to the board.
  4. Receiver should be ready to go.

Tx Unit

Tx Unit is the same XBee transceiver but on the RPi side. Follow the steps below:

  1. Install_script_RPi.py will do everything for you.

Advanced Configuration

By default Jasper uses pocketsphinx as a voice recognition platform and it was found to produce a low quality recognition so we decided to move to Google Speech.

  1. Follow the directions here to get API public key for the jasper.
  2. From the jasper directory navigate to clients directory and run python populate.py
  3. Provide all the information and when prompted to enter STT enter 'google' and then enter your API key when asked.
  4. Check that everything is good to go by running jasper. It is common to have a few errors during the first startup but don't panic, as there is a very helpful Google Group and FAQ maintained by Jasper authors.

User Instructions

To start jasper, navigate to the jasper directory and run the following command

python jasper.py

Jasper will initialize everything, and if everything is connected properly, should prompt you for a command. The typical use case is to say Jasper for it to begin listening, wait for a high pitched beep, say your command, and then wait for another lower pitched beep for a confirmation.

You: Jasper
Jasper: *BEEP*
You: What time is it.
Jasper: *BEEP*
Jasper: It's currently now.

Highlights

Packaging

Jerry Talyor of Mansfield, IN is going to make a fantastic wooden laser cut and laser engraved box for us.

Supported Voice Commands

  1. <color> ONE and <color> ZERO where color = [RED, GREEN, BLUE]
  2. ROSE SCHEDULE will tell you the next class
  3. TOGGLE ON <pin number> and TOGGLE OFF <pin number>

Theory of Operation

Our project uses the opensource Jasper project as it's backbone. The system is constantly listening to it's surroundings, and when it finds a discrete sound block sends it to the google voice API for analysis. Google voice returns text blocks of what it thought the sound click was of. Jasper then runs it across modules looking for a hit, and then executes a command if it finds one.

Completed Tasks

Matt Olson

  • Wi-fi dongle working - 10/22/2014
  • Created Module for Jasper - 10/28/2014
  • Created install script for installing modules on RPi - 11/4/2014
  • Created simple GPIO Module - 11/8/2014
  • Created Rose-Schedule Module - 11/11/2014
  • Created Wireless Light Module 11/16/2014

Dmitry Votinstev

  • Speech recognition software installed - 10/20/2014
  • Sound recording/payback with arecord aplay works 10/22/2014
  • Ported Jasper to Work on the Google voice API 11/12/2014
  • Wireless Light control module transmitter and receiver modules 11/16/2014
  • Jasper Wireless Light Module debug 11/16/2014
  • Install script mods to setup RPi as wireless light server 11/17/2014

Tasks Remaining

  • Hook wireless light to an actual light - Dmitry
  • Develop Additional testing modules - Matt, Dmitry
  • Get Jasper working on BBB - Matt, Dmitry

Future Work

The most immediate extension to this project would to fully port all of the code and features over to the Beaglebone. Although we do provide instructions for installing jasper on the beaglebone, some of the scripts that we wrote are not compatible for the Beaglebone. This will mostly require a simple port of the code, and replacing some of the libraries that are being used.

It would also be useful to see a way to integrate NodeRed and Jasper. We created a module that simply writes everything that jasper hears to a file. In NodeRed, that file can be tailed and used in NodeRed. A native way to interface the two would be useful for decreasing latency and hopefully improve compatibility.

Conclusions

Node Red does not provide a very good architecture for acting as a speech recognition system and dispatcher, which resulted in our choice to investigate other options.




thumb‎ Embedded Linux Class by Mark A. Yoder