Difference between revisions of "BeagleBoard/GSoC/2021 Proposal/ALSA plugin for BELA"

From eLinux.org
Jump to: navigation, search
(Created page with "Category: BeagleBoard Category: GSoC Category: GSoCProposal2021 =[ALSA plugin for BELA] = About ''Student'': [https://elinux.org/User:Jduchniewicz Jakub Duchniewi...")
 
Line 28: Line 28:
  
 
===Description===
 
===Description===
// todo: add links
+
 
BELA is a cape designed for BB Black which features real-time audio processing via usage of Xenomai threads. Apart from being a hardware solution, BELA supplies its own operating system based on Debian Linux distribution and a full-fledged IDE allowing for seamless audio development experience. BELA provides its own library for interfacing with the hardware, however it does not provide any unified interface via ALSA, JACK or PulseAudio. Therefore, it is currently impossible to use BELA like a regular Linux audio device and it has to be done by utilizing its API calls.
+
[https://learn.bela.io/ BELA] is a cape designed for BB Black which features real-time audio processing via usage of [https://source.denx.de/Xenomai/xenomai/-/wikis/home Xenomai] threads. Apart from being a hardware solution, BELA supplies its own operating system based on Debian Linux distribution and a full-fledged IDE allowing for seamless audio development experience. BELA provides its own library for interfacing with the hardware, however it does not provide any unified interface via ALSA, JACK or PulseAudio. Therefore, it is currently impossible to use BELA like a regular Linux audio device and it has to be done by utilizing its API calls.
  
 
The main premise of this project is to enable the unified access by means of ALSA plugin. This plugin will allow for tying user-provided functions for regular system calls alsa-lib API uses for operating on its devices. Since such need may arise for any other real-time ALSA devices, this plugin would be a valuable addition to the ALSA ecosystem and would be mainline'able. This way, users can call regular ALSA API's for interacting with the device and still profit from all the real-time benefits it offers.
 
The main premise of this project is to enable the unified access by means of ALSA plugin. This plugin will allow for tying user-provided functions for regular system calls alsa-lib API uses for operating on its devices. Since such need may arise for any other real-time ALSA devices, this plugin would be a valuable addition to the ALSA ecosystem and would be mainline'able. This way, users can call regular ALSA API's for interacting with the device and still profit from all the real-time benefits it offers.
+
 
// add graphs
 
 
// add the analog etc interpretation stuff?
 
// add the analog etc interpretation stuff?
 +
The project will also focus on writing all necessary components for interfacing with this plugin, such as an exemplary userspace application and instructions on how to use the ALSA API with BELA.
 +
  
 +
===Implementation===
 +
The first part - ALSA plugin will be realized basing on already existent ALSA plugins, like the [https://github.com/alsa-project/alsa-lib/blob/master/src/pcm/pcm_file.c file plugin] which allows interaction with arbitrary files as if they were regular ALSA devices.
  
  
===Implementation===
 
 
In order to understand why  
 
In order to understand why  
 
Because the Xenomai thread runs with a higher priority than any other Linux thread,  
 
Because the Xenomai thread runs with a higher priority than any other Linux thread,  
  
 +
 +
the job done by the PRU is simple but has tight time constrains: every 11us it has to access the SPI bus, the McASP bus and the GPIOs and shift data to/out of RAM.. So you effectively need a computing core that does just that. It doesn't have to be powerful but whatever runs on it cannot be preempted by anything else ever. So yes in principle you could run it on one of the two RISCV cores but that would seem like a huge waste (though in principle doable!)! By having all the low-level access to the peripherals delegated to an external microcontroller, then the data could be shifted to the RISCV core via a single DMA and the 2 CPU cores would be free all of the time!
 +
 +
// add graphs
 +
The system overview is described in the above diagram.
 +
 +
Subsequent
  
  
Line 61: Line 70:
  
 
===Deliverables===
 
===Deliverables===
 +
* The ALSA plugin itself
 +
* Userspace example app
 +
* Guidelines on using ALSA with BELA
  
 
===Timeline===
 
===Timeline===
Line 131: Line 143:
  
 
Apart from that, I have completed my Bachelors degree at Warsaw University of Technology successfully defending my thesis titled: [https://jduchniewicz.com/FPGA-synth.pdf | FPGA Based Hardware Accelerator for Musical Synthesis for Linux System]. In this system I created a polyphonic musical synthesizer capable of producing various waveforms in Verilog code and deployed it on a De0 Nano SoC FPGA. Additionally I wrote two kernel drivers - one encompassed ALSA sound device and was responsible for proper synchronization of DMA transfers.  
 
Apart from that, I have completed my Bachelors degree at Warsaw University of Technology successfully defending my thesis titled: [https://jduchniewicz.com/FPGA-synth.pdf | FPGA Based Hardware Accelerator for Musical Synthesis for Linux System]. In this system I created a polyphonic musical synthesizer capable of producing various waveforms in Verilog code and deployed it on a De0 Nano SoC FPGA. Additionally I wrote two kernel drivers - one encompassed ALSA sound device and was responsible for proper synchronization of DMA transfers.  
 +
 +
The ALSA part proved to be very time consuming and difficult to debug, but after hours of a wild goose chase I understand how ALSA works at a level allowing me for various kernel driver creation and userspace plugin extesion.
  
 
I am familiar with Deep Learning concepts and basics of Computer Vision. During my studies at UTU I achieved the maximal grades for my subjects, excelling at Navigation Systems for Robotics and Hardware accelerators for AI.  
 
I am familiar with Deep Learning concepts and basics of Computer Vision. During my studies at UTU I achieved the maximal grades for my subjects, excelling at Navigation Systems for Robotics and Hardware accelerators for AI.  
 
I have some experience working with OpenGL, mostly learning it for the programming engine needs and for personal benefit. Since this project does not require in-depth knowledge of it, but rather to create an abstraction over the OpenGL ES bindings and perform the necessary data conversions and extractions inside the API. This requires skills I do already possess and have proficiency using.
 
  
 
In my professional work, many times I had to complete various tasks under time pressure and choose the proper task scoping. Basing on this experience I believe that this task is deliverable in the mentioned time-frame.
 
In my professional work, many times I had to complete various tasks under time pressure and choose the proper task scoping. Basing on this experience I believe that this task is deliverable in the mentioned time-frame.
Line 143: Line 155:
 
If all fails I may be able be forced to change my approach and backtrack, but this will not be a big problem, because the knowledge won't be lost and it will only make my future approaches better. Alternatively, I can focus on documenting my progress in a form of blogposts and videos while waiting for my mentor to come back to cyberspace.
 
If all fails I may be able be forced to change my approach and backtrack, but this will not be a big problem, because the knowledge won't be lost and it will only make my future approaches better. Alternatively, I can focus on documenting my progress in a form of blogposts and videos while waiting for my mentor to come back to cyberspace.
  
 +
In case of this problem, there exists a high risk of problems without obvious solutions and in this case I might need to directly email the creators of ALSA (or join the linux kernel mailing list) - Jaroslav Kysela and Takashi Iwai. Because the plugin development will be in the userspace, regular GDB will be enough for debugging it.
  
 +
During my previous adventures with ALSA i accumulated a list of sound (no pun intended) materials:
 +
* [https://www.kernel.org/doc/html/v4.10/sound/kernel-api/writing-an-alsa-driver.html Writing a kernel driver] - explains what is happening inside ALSA from the drivers perspective
 +
* [https://jan.newmarch.name/LinuxSound/ Overview of Linux Sound] - just a good reference
 +
* [https://www.volkerschatz.com/noise/alsa.html Describing ALSA plugins]
 +
* [https://www.alsa-project.org/main/index.php/FramesPeriods Vital element of understanding the ring buffer in ALSA]
 +
* [https://github.com/alsa-project/alsa-lib/ The source code of ALSA lib]
 +
* [http://eecs.qmul.ac.uk/~andrewm/mcpherson_aes2015.pdf Internal workings of BELA]
 +
* [https://www.geeklan.co.uk/files/oshug63-bela.pdf Overview of BELA]
  
 +
===Benefit===
 +
Having an ALSA plugin for BELA would allow interfacing with the devices in a unified manner and it will remove the onus of knowing what API BELA uses for communication. Moreover, such plugin can be mainline'able as it will allow for substituting various Linux syscalls the ALSA API makes for the device interfacing with this plugin, such as substituting the `read()` syscall with the Xenomai `cobalt_read()` making the operation run in real-time.
 +
 +
currently for each different application that a user want to run, the audio backend of the application has to be changed from using ALSA/portaudio/RTAudio/jack to using the Bela API. This could be mitigated by having portaudio/RTAudio/jack wrappers (what I put in the original project idea), but I figured if it could be just an ALSA driver (perhaps an ALSA plugin, which runs all in userspace I think?) then things could be more maintainable
 +
~ Giulio Moro
  
===Benefit===
 
  
  
 
==Misc==
 
==Misc==
 
The qualification PR is available [https://github.com/jadonk/gsoc-application/pull/148 here].
 
The qualification PR is available [https://github.com/jadonk/gsoc-application/pull/148 here].

Revision as of 00:33, 12 April 2021


[ALSA plugin for BELA]

About Student: Jakub Duchniewicz
Mentors: Giulio Moro
Code: not yet created!
Wiki: https://elinux.org/index.php?title=BeagleBoard/GSoC/2021_Proposal/GPGPU_with_GLES
GSoC: [1]

Status

Discussing the implementation ideas with Giulio Moro and others on #beagle-gsoc IRC.

About you

IRC: jduchniewicz
Github: JDuchniewicz
School: University of Turku/KTH Royal Institute of Technology
Country: Finland/Sweden/Poland
Primary language: Polish
Typical work hours: 8AM-5PM CET
Previous GSoC participation: Participating in GSoC, especially with BeagleBoard would further develop my software and hardware skills and help me apply my current knowledge for the mutual benefit of the open source community. I planned to do the YOLO project, but after spending several days researching and preparing the proposal I found it is impossible to do on current BBAI/X15.

About your project

Project name: ALSA plugin for BELA

Description

BELA is a cape designed for BB Black which features real-time audio processing via usage of Xenomai threads. Apart from being a hardware solution, BELA supplies its own operating system based on Debian Linux distribution and a full-fledged IDE allowing for seamless audio development experience. BELA provides its own library for interfacing with the hardware, however it does not provide any unified interface via ALSA, JACK or PulseAudio. Therefore, it is currently impossible to use BELA like a regular Linux audio device and it has to be done by utilizing its API calls.

The main premise of this project is to enable the unified access by means of ALSA plugin. This plugin will allow for tying user-provided functions for regular system calls alsa-lib API uses for operating on its devices. Since such need may arise for any other real-time ALSA devices, this plugin would be a valuable addition to the ALSA ecosystem and would be mainline'able. This way, users can call regular ALSA API's for interacting with the device and still profit from all the real-time benefits it offers.

// add the analog etc interpretation stuff? The project will also focus on writing all necessary components for interfacing with this plugin, such as an exemplary userspace application and instructions on how to use the ALSA API with BELA.


Implementation

The first part - ALSA plugin will be realized basing on already existent ALSA plugins, like the file plugin which allows interaction with arbitrary files as if they were regular ALSA devices.


In order to understand why Because the Xenomai thread runs with a higher priority than any other Linux thread,


the job done by the PRU is simple but has tight time constrains: every 11us it has to access the SPI bus, the McASP bus and the GPIOs and shift data to/out of RAM.. So you effectively need a computing core that does just that. It doesn't have to be powerful but whatever runs on it cannot be preempted by anything else ever. So yes in principle you could run it on one of the two RISCV cores but that would seem like a huge waste (though in principle doable!)! By having all the low-level access to the peripherals delegated to an external microcontroller, then the data could be shifted to the RISCV core via a single DMA and the 2 CPU cores would be free all of the time!

// add graphs The system overview is described in the above diagram.

Subsequent


Alternative ideas/Stretch goals

API Overview

Detailed Implementation

Expected Performance

Action Items

Deliverables

  • The ALSA plugin itself
  • Userspace example app
  • Guidelines on using ALSA with BELA

Timeline

During 25.07.21-08.08.21 I have a summer camp from my study programme and will be probably occupied for a half of the day. The camp will most likely be held online though.

Date Milestone Action Items
13.04.21-17.05.21 Pre-work
  • Expand knowledge OpenGL ES
  • Run the OpenGL computation acceleration on a regular PC
  • Learn about various scientific computations which can be done best in the GPU
18.05.21-07.06.21 Community Bonding
  • Familiarize myself with the community
  • Experiment with BB
14.06.21 Milestone #1
  • Introductory YouTube video
  • Setup the environment
  • Write first blog entry - "Part 1: Game plan"
21.06.21 Milestone #2
  • setup SGX and run simple shaders
  • write the array addition
28.06.21 Milestone #3
  • setup the basic API
  • write the FIR convolution
  • write the matrix multiplication
5.07.21 Milestone #4
  • write additional API (for continuous usage)
  • write broadcast math operations
  • benchmark the functions with various sizes
12.07.21 Milestone #5
  • Write second blog post - "Part 2: Implementation"
  • Evaluate the mentor
19.07.21 Milestone #6
  • document everything
26.07.21 Milestone #7
  • Summer camp, mostly collect feedback for the project
  • Less brain-intensive tasks, documentation, benchmarking
31.07.21 Milestone #8
  • Write third blog post - "Part 3: Optimization and Benchmarks"
  • Polish the implementation
7.8.21 Milestone #9
  • Polish the API and make it extensible
  • Prepare materials for the video
  • Start writing up the summary
14.8.21 Milestone #10
  • Finish the project summary
  • Final YouTube video
24.08.21 Feedback time
  • Complete feedback form for the mentor
31.08.21 Results announced
  • Celebrate the ending and rejoice

Experience and approach

I have strong programming background in the area of embedded Linux/operating systems as a Junior Software Engineer in Samsung Electronics during December 2017-March 2020. Additionally I have developed a game engine (| PolyEngine) in C++ during this time and gave some talks on modern C++ during my time as a Vice-President of Game Development Student Group "Polygon".

Apart from that, I have completed my Bachelors degree at Warsaw University of Technology successfully defending my thesis titled: | FPGA Based Hardware Accelerator for Musical Synthesis for Linux System. In this system I created a polyphonic musical synthesizer capable of producing various waveforms in Verilog code and deployed it on a De0 Nano SoC FPGA. Additionally I wrote two kernel drivers - one encompassed ALSA sound device and was responsible for proper synchronization of DMA transfers.

The ALSA part proved to be very time consuming and difficult to debug, but after hours of a wild goose chase I understand how ALSA works at a level allowing me for various kernel driver creation and userspace plugin extesion.

I am familiar with Deep Learning concepts and basics of Computer Vision. During my studies at UTU I achieved the maximal grades for my subjects, excelling at Navigation Systems for Robotics and Hardware accelerators for AI.

In my professional work, many times I had to complete various tasks under time pressure and choose the proper task scoping. Basing on this experience I believe that this task is deliverable in the mentioned time-frame.

Contingency

Since I am used to tackling seemingly insurmountable challenges, I will first of all keep calm and try to come up with alternative approach if I get stuck along the way. The internet is a vast ocean of knowledge and time and again I received help from benevolent strangers from reddit or other forums. Since I believe that humans are species, which solve problems in the best way collaboratively, I will contact #beagle, #beagle-gsoc and relevant subreddits (I received tremendous help on /r/FPGA, /r/embedded and /r/askelectronics in the past).

If all fails I may be able be forced to change my approach and backtrack, but this will not be a big problem, because the knowledge won't be lost and it will only make my future approaches better. Alternatively, I can focus on documenting my progress in a form of blogposts and videos while waiting for my mentor to come back to cyberspace.

In case of this problem, there exists a high risk of problems without obvious solutions and in this case I might need to directly email the creators of ALSA (or join the linux kernel mailing list) - Jaroslav Kysela and Takashi Iwai. Because the plugin development will be in the userspace, regular GDB will be enough for debugging it.

During my previous adventures with ALSA i accumulated a list of sound (no pun intended) materials:

Benefit

Having an ALSA plugin for BELA would allow interfacing with the devices in a unified manner and it will remove the onus of knowing what API BELA uses for communication. Moreover, such plugin can be mainline'able as it will allow for substituting various Linux syscalls the ALSA API makes for the device interfacing with this plugin, such as substituting the `read()` syscall with the Xenomai `cobalt_read()` making the operation run in real-time.

currently for each different application that a user want to run, the audio backend of the application has to be changed from using ALSA/portaudio/RTAudio/jack to using the Bela API. This could be mitigated by having portaudio/RTAudio/jack wrappers (what I put in the original project idea), but I figured if it could be just an ALSA driver (perhaps an ALSA plugin, which runs all in userspace I think?) then things could be more maintainable

~ Giulio Moro


Misc

The qualification PR is available here.