Difference between revisions of "BeagleBoard/GSoC/2020 Projects/Media IP Streaming"

From eLinux.org
Jump to: navigation, search
(Benefit)
m (Jkridner moved page BeagleBoard/GSoC/MediaIpStreaming to BeagleBoard/GSoC/2020 Projects/Media IP Streaming: Move it under the accepted projects page)
(25 intermediate revisions by 2 users not shown)
Line 6: Line 6:
 
{{#ev:youtube|Jl3sUq2WwcY||right|BeagleLogic}} <!-- latest video will go here -->
 
{{#ev:youtube|Jl3sUq2WwcY||right|BeagleLogic}} <!-- latest video will go here -->
  
This project will equip the Beagleboard AI with Media IP Streaming capabilities, by porting the sound card drivers for [https://hackaday.io/project/9634-ctag-face-and-beast-multichannel-audio-systems CTAG face2|4 Audio Card] to the Beaglebone AI. Additionally the AVB protocol stack is ported from [https://elinux.org/BeagleBoard/GSoC/BeagleBoneAVB Beagle Bone AVB] to the Beaglebone AI.
+
This project will equip the Beagleboard AI with Media IP Streaming capabilities, by porting the sound card drivers for [https://hackaday.io/project/9634-ctag-face-and-beast-multichannel-audio-systems CTAG face2|4 Audio Card] and the AVB protocol stack from [https://elinux.org/BeagleBoard/GSoC/BeagleBoneAVB BeagleBone AVB] to the BeagleBone AI.  
  
''Student'': [http://elinux.org/User:nwan]<br>
+
''Student'': [http://elinux.org/User:nwan nwan]<br>
''Mentors'': [https://elinux.org/User:Rma]<br>
+
''Mentors'': [https://elinux.org/User:Rma rma]<br>
''Code'': [N/A]<br>
+
''Code'': https://github.com/NiklasWan/linux<br>
 +
''Progress and Documentation/Research Results'': https://niklaswan.github.io/GSoC-Overview <br>
 
''Wiki'': http://elinux.org/BeagleBoard/GSoC/MediaIpStreaming<br>
 
''Wiki'': http://elinux.org/BeagleBoard/GSoC/MediaIpStreaming<br>
 
''GSoC'': [N/A]<br>
 
''GSoC'': [N/A]<br>
Line 34: Line 35:
  
 
===Description===
 
===Description===
The BeagleBone AI is equipped with a high amount of processing power due to the Dual Core ARM Cortex-A15 chip as a main computing unit and accompanying co-processors.  
+
The BeagleBone AI is equipped with a high amount of processing power due to the Dual Core ARM Cortex-A15 chip as a main computing unit and its accompanying co-processors.  
 
This makes the AI a perfect fit for highly demanding applications regarding CPU consumption, like audio applications which have extremely strong realtime constraints.  
 
This makes the AI a perfect fit for highly demanding applications regarding CPU consumption, like audio applications which have extremely strong realtime constraints.  
 
Professional audio/video studios have to guarantee for small latencies when transmitting media signals between different devices and different media channels in a transmitted stream need to be synchronized.  
 
Professional audio/video studios have to guarantee for small latencies when transmitting media signals between different devices and different media channels in a transmitted stream need to be synchronized.  
Latency and Snychronicity is extremely important when transmitting e.g. a video channel together with the accompanying audio channel.  
+
Latency and snychronicity are both extremely important when transmitting e.g. a video channel together with the accompanying audio channel.  
 
Those two channels have to be transmitted in a manner, that lip  synchronicity can be guaranteed because humans are extremely sensitive to voice offset to accompanying video signals.
 
Those two channels have to be transmitted in a manner, that lip  synchronicity can be guaranteed because humans are extremely sensitive to voice offset to accompanying video signals.
  
To bring media ip streaming capabilities to the Beaglebone AI, the following steps are planned:
+
To bring media ip streaming capabilities to the BeagleBone AI, the following steps are planned:
 
A previous GSoC project ported a sound card driver from the BeagleBone Green/Black to the BeagleBoard-X15 (https://summerofcode.withgoogle.com/archive/2016/projects/5351212496977920/).  
 
A previous GSoC project ported a sound card driver from the BeagleBone Green/Black to the BeagleBoard-X15 (https://summerofcode.withgoogle.com/archive/2016/projects/5351212496977920/).  
This port will now be ported to the BegleBone AI. With the sound card driver successfully ported, the next step would be to port the AVB protocol driver stack from [https://elinux.org/BeagleBoard/GSoC/BeagleBoneAVB Beagle Bone AVB] enabling media streaming over the network.  
+
This port will now be ported to the BegleBone AI. With the sound card driver successfully ported, the next step would be to port the AVB protocol driver stack from [https://elinux.org/BeagleBoard/GSoC/BeagleBoneAVB BeagleBone AVB] enabling media streaming over the network.  
This would allow to use the Beaglebone AI as a media streaming device in professional audio/media applications and bring audio stream synchronisation features to the Beaglebone AI.  
+
This would allow to use the BeagleBone AI as a media streaming device in professional audio/media applications and bring audio stream synchronization features to the BeagleBone AI.  
 
Thus allowing for tight synchronization between different audio and video streams which are transmitted over the network.
 
Thus allowing for tight synchronization between different audio and video streams which are transmitted over the network.
 +
Additionally for people who don't own the CTAG Face 2|4 cape HDMI audio output should be realized.
 +
 +
Practical Use:
 +
AVB is primarily used in large scale media productions, like sports venues, broadcasting studios or concert halls. Basically AVB can be used everywhere where media data has to be transmitted over larger distances in a local network.
 +
Implementing this on a BeagleBone AI would allow for a low cost alternative for proprietary hardware and further allow for customization by the Beagleboard.org community.
  
 
===Timeline===
 
===Timeline===
Line 52: Line 58:
 
| Mar 30 || Proposal complete, Submitted to https://summerofcode.withgoogle.com
 
| Mar 30 || Proposal complete, Submitted to https://summerofcode.withgoogle.com
 
|-
 
|-
| Apr 27 || Proposal accepted or rejected
+
| May 4  ||  
|-
+
Proposal accepted or rejected Community Bonding Period starts.
| May 18 || Pre-work complete, Coding officially begins!
+
* Learn about embedded linux structure
|-
+
* Learn about Linux kernel driver development
| May 25 || Milestone #1, Introductory YouTube video
+
* Set up general development environment for embedded Linux systems and required periphery
 +
* Work through current code base on CTAG drivers
 +
* Work through current code base on AVB drivers
 +
* Learn about ALSA SoC driver development
 +
* Learn about Beaglebone AI hardware structure
 
|-
 
|-
| June 1 || Milestone #2
+
| June 1 || Pre-work complete, Coding officially begins!
 
|-
 
|-
| June 8 || Milestone #3
+
| June 8 || Milestone #1, Introductory YouTube video, review of existing drivers for ctag face audio interface, identifying challenges for porting drivers to Beagle AI and selection of appropriate kernel, basis for drivers is https://elinux.org/BeagleBoard/GSoC/2016_Projects#Project:_Porting_the_CTAG_face2.7C4_multichannel_soundcard_drivers_to_BeagleBoard-X15_.28AM5728_SoC.29._Create_library_to_make_use_of_AM5728_DSPs_.28C66x.29.
 
|-
 
|-
| June 15 18:00 UTC || Milestone #4, Mentors and students can begin submitting Phase 1 evaluations
+
| June 15 || Milestone #2 Implementation / porting of ALSA audio drivers for ctag face to Beagle AI --> toolchain setup, driver adoptions, coding
 
|-
 
|-
| June 19 18:00 UTC || Phase 1 Evaluation deadline
+
| June 22 || Milestone #3 Port of sound card drivers, testing, performance check
 
|-
 
|-
| June 22 || Milestone #5
+
| July 3 18:00 UTC || Milestone #4 (Phase 1 evaluations), finalzing port of ctag face audio card driver to Beagle AI and getting pull request to Beagleboard Mainline
 
|-
 
|-
| June 29 || Milestone #6
+
| July 10 || Milestone #5 Review of existing AVB network driver architecture for real-time audio streaming, basis is https://elinux.org/BeagleBoard/GSoC/2017_Projects#Project:_BeagleBone_AVB_Stack , identifying challenges for porting to Beagle AI
 
|-
 
|-
| July 6 || Milestone #7
+
| July 17 || Milestone #6 Implementation / porting of ALSA AVB network drivers to Beagle AI --> toolchain setup, driver adoptions
 
|-
 
|-
| July 13 18:00 UTC || Milestone #8, Mentors and students can begin submitting Phase 2 evaluations
+
| July 24 || Milestone #7 AVB ALSA drivers implementation for Beagle AI
 
|-
 
|-
| July 17 18:00 UTC || Phase 2 Evaluation deadline
+
| July 31 18:00 UTC || Milestone #8 (Phase 2 evaluations), Getting ALSA AVB network drivers finished and document everything till now
 
|-
 
|-
| July 20 || Milestone #9
+
| August 3 || Milestone #9 Joining AVB ALSA drivers with ctag face audio card drivers
 
|-
 
|-
| July 27 || Milestone #10
+
| August 10 || Milestone #10 Performance and integration testing of driver ports
 
|-
 
|-
| August 3 || Milestone #11, Completion YouTube video
+
| August 17 || Milestone #11, Completion YouTube video, pull request of driver architecture for mainline
 
|-
 
|-
| August 10 - 17 18:00 UTC || Final week: Students submit their final work product and their final mentor evaluation
+
| August 24 - 31 18:00 UTC || Final week: Students submit their final work product and their final mentor evaluation
 
|-
 
|-
| August 17 - 24 18:00 UTC || Mentors submit final student evaluations
+
| August 31 - September 7 18:00 UTC || Mentors submit final student evaluations
 
|}
 
|}
  
 
===Experience and approach===
 
===Experience and approach===
During my bachelor's degree in information technology I had several courses like programming in c, programming in c++, operating systems and embedded system programming which layed down the basis for  
+
During my bachelor's degree in information technology I had several courses like programming in C, programming in C++, operating systems and embedded system programming which layed down the basis for  
 
developing embedded software. Due to my additional bachelor's degree in audio production I have additional experience in audio applications and audio and media codecs, which will help me to understand the theory behind the different needed algorithms.
 
developing embedded software. Due to my additional bachelor's degree in audio production I have additional experience in audio applications and audio and media codecs, which will help me to understand the theory behind the different needed algorithms.
With my previous development work for the [https://github.com/ctag-fh-kiel/ctag-straempler Strämpler project] I already have experience in working on complex embedded c projects and which potential pitfalls could occur.
+
With my previous development work for the [https://github.com/ctag-fh-kiel/ctag-straempler Strämpler project] I already have experience in working on complex embedded C projects and which potential pitfalls could occur.
  
 
===Contingency===
 
===Contingency===
Line 100: Line 110:
  
 
===Benefit===
 
===Benefit===
Equipping the Beaglebone AI with media ip streaming capabilities would allow the Beaglebone.org community to use those capabilities to implement the system in professional media applications.
+
Equipping the BeagleBone AI with media ip streaming capabilities would allow the Beagleboard.org community to use those capabilities to implement the system in professional media applications.
 
The community could also implement further media protocols like AES/Ravenna to allow the usage of the AI for even more media streaming tasks.
 
The community could also implement further media protocols like AES/Ravenna to allow the usage of the AI for even more media streaming tasks.
  
Line 106: Line 116:
 
Link to pull request [https://github.com/jadonk/gsoc-application/pull/139 #139].
 
Link to pull request [https://github.com/jadonk/gsoc-application/pull/139 #139].
  
===Suggestions===
+
===References===
Is there anything else we should have asked you?
+
#[1]    „4.3. PTP — Processor SDK Linux Documentation“. https://software-dl.ti.com/processor-sdk-linux/esd/docs/06_02_00_81/linux/Industrial_Protocols_PTP.html (accessed March 30, 2020).
 +
#[2] 1733-2011 IEEE Standard for Layer 3 Transport Protocol for Time-Sensitive Applications in Local Area Networks. ///.
 +
#[3] M. A. Yoder und J. Kridner, BeagleBone cookbook, First edition. Sebastopol, CA: O’Reilly Media, Inc, 2015.
 +
#[4] C. Hallinan, Embedded Linux primer: a practical real-world approach, 2nd ed. Upper Saddle River, NJ: Prentice Hall, 2011.
 +
#[5] A. Liberal de los Ríos, Linux driver development for embedded processors: Learn to develop Linux embedded drivers with kernel 4.9 LTS, Second edition. .
 +
#[6] R. Love, Linux kernel development, 3rd ed. Upper Saddle River, NJ: Addison-Wesley, 2010.
 +
#[7] E. White, Making embedded systems: design patterns for great software, 1. ed. Beijing: O’Reilly, 2012.
 +
#[8] D. Molloy, Molloy_exploring BeagleBone 2e. Indianapolis, NY: John Wiley and Sons, 2018.
 +
#[9] „The Linux Kernel documentation — The Linux Kernel documentation“. https://www.kernel.org/doc/html/latest/index.html (accessed March 26, 2020).
 +
 
 +
===PTP Overview===
 +
* is used by the AVB protocol to achieve synchronization between devices
 +
* based on IEEE 1588v2
 +
* up to nanoseconds accuracy
 +
* sharing timestamps over the network for synchronization of devices
 +
* uses master/slave hierarchy
 +
* slave retrieves time from master ==> network dely has to be taken into account
 +
 
 +
===PTP Clocks:===
 +
====Ordinary Clock====
 +
* normally endpoint of the network
 +
* single port
 +
* BMCA (best master clock algorithm) determines which clock is used as master (the one with the highest accuracy)
 +
 
 +
====Grandmaster Clock====
 +
* is used as an endpoint master and has extremely high accuracy (normally timed by GPS or NTP)
 +
* there can be more than one in a network to achieve redundancy
 +
====Boundary Clock====
 +
* mutli port
 +
* a network switch with master/slave ports
 +
====Transparent Clock====
 +
* accounts for queuing delays when a standrad switch is used and thus improves accuracy
 +
 
 +
===Delay mechanisms:===
 +
=====E2E:=====
 +
* calculate network delay End-To-End
 +
* no need of PTP equipment but this results in added cost in accuracy
 +
====P2P:====
 +
* calculate network delay Peer-To-Peer
 +
* results in high accuracy, but all devices in the network need to be PTP enabled

Revision as of 16:31, 22 May 2020


Proposol of equipping the Beaglebone AI with Media IP Streaming capabilities

{{#ev:youtube|Jl3sUq2WwcY||right|BeagleLogic}}

This project will equip the Beagleboard AI with Media IP Streaming capabilities, by porting the sound card drivers for CTAG face2|4 Audio Card and the AVB protocol stack from BeagleBone AVB to the BeagleBone AI.

Student: nwan
Mentors: rma
Code: https://github.com/NiklasWan/linux
Progress and Documentation/Research Results: https://niklaswan.github.io/GSoC-Overview
Wiki: http://elinux.org/BeagleBoard/GSoC/MediaIpStreaming
GSoC: [N/A]

Status

This project is currently just a proposal.

Proposal

All requirements have been fullfilled, the Pull Request can be found here #139

About you

IRC: nwan
Github: NiklasWan
School: Kiel University of Applied Sciences
Country: Germany
Primary languages: German, English
Typical work hours: 8AM-5PM CET
Previous GSoC participation: I want to participate at GSoC because I want to gather experience in working within an open source community and try to apply theoretical knowledge into the practical domain. Also I hope to learn new awesome things. This would be my first time participating in GSoC.

About your project

Project name: Media Ip Streaming

Description

The BeagleBone AI is equipped with a high amount of processing power due to the Dual Core ARM Cortex-A15 chip as a main computing unit and its accompanying co-processors. This makes the AI a perfect fit for highly demanding applications regarding CPU consumption, like audio applications which have extremely strong realtime constraints. Professional audio/video studios have to guarantee for small latencies when transmitting media signals between different devices and different media channels in a transmitted stream need to be synchronized. Latency and snychronicity are both extremely important when transmitting e.g. a video channel together with the accompanying audio channel. Those two channels have to be transmitted in a manner, that lip synchronicity can be guaranteed because humans are extremely sensitive to voice offset to accompanying video signals.

To bring media ip streaming capabilities to the BeagleBone AI, the following steps are planned: A previous GSoC project ported a sound card driver from the BeagleBone Green/Black to the BeagleBoard-X15 (https://summerofcode.withgoogle.com/archive/2016/projects/5351212496977920/). This port will now be ported to the BegleBone AI. With the sound card driver successfully ported, the next step would be to port the AVB protocol driver stack from BeagleBone AVB enabling media streaming over the network. This would allow to use the BeagleBone AI as a media streaming device in professional audio/media applications and bring audio stream synchronization features to the BeagleBone AI. Thus allowing for tight synchronization between different audio and video streams which are transmitted over the network. Additionally for people who don't own the CTAG Face 2|4 cape HDMI audio output should be realized.

Practical Use: AVB is primarily used in large scale media productions, like sports venues, broadcasting studios or concert halls. Basically AVB can be used everywhere where media data has to be transmitted over larger distances in a local network. Implementing this on a BeagleBone AI would allow for a low cost alternative for proprietary hardware and further allow for customization by the Beagleboard.org community.

Timeline

Provide a development timeline with a milestone each of the 11 weeks and any pre-work. (A realistic timeline is critical to our selection process.)

Mar 30 Proposal complete, Submitted to https://summerofcode.withgoogle.com
May 4

Proposal accepted or rejected Community Bonding Period starts.

  • Learn about embedded linux structure
  • Learn about Linux kernel driver development
  • Set up general development environment for embedded Linux systems and required periphery
  • Work through current code base on CTAG drivers
  • Work through current code base on AVB drivers
  • Learn about ALSA SoC driver development
  • Learn about Beaglebone AI hardware structure
June 1 Pre-work complete, Coding officially begins!
June 8 Milestone #1, Introductory YouTube video, review of existing drivers for ctag face audio interface, identifying challenges for porting drivers to Beagle AI and selection of appropriate kernel, basis for drivers is https://elinux.org/BeagleBoard/GSoC/2016_Projects#Project:_Porting_the_CTAG_face2.7C4_multichannel_soundcard_drivers_to_BeagleBoard-X15_.28AM5728_SoC.29._Create_library_to_make_use_of_AM5728_DSPs_.28C66x.29.
June 15 Milestone #2 Implementation / porting of ALSA audio drivers for ctag face to Beagle AI --> toolchain setup, driver adoptions, coding
June 22 Milestone #3 Port of sound card drivers, testing, performance check
July 3 18:00 UTC Milestone #4 (Phase 1 evaluations), finalzing port of ctag face audio card driver to Beagle AI and getting pull request to Beagleboard Mainline
July 10 Milestone #5 Review of existing AVB network driver architecture for real-time audio streaming, basis is https://elinux.org/BeagleBoard/GSoC/2017_Projects#Project:_BeagleBone_AVB_Stack , identifying challenges for porting to Beagle AI
July 17 Milestone #6 Implementation / porting of ALSA AVB network drivers to Beagle AI --> toolchain setup, driver adoptions
July 24 Milestone #7 AVB ALSA drivers implementation for Beagle AI
July 31 18:00 UTC Milestone #8 (Phase 2 evaluations), Getting ALSA AVB network drivers finished and document everything till now
August 3 Milestone #9 Joining AVB ALSA drivers with ctag face audio card drivers
August 10 Milestone #10 Performance and integration testing of driver ports
August 17 Milestone #11, Completion YouTube video, pull request of driver architecture for mainline
August 24 - 31 18:00 UTC Final week: Students submit their final work product and their final mentor evaluation
August 31 - September 7 18:00 UTC Mentors submit final student evaluations

Experience and approach

During my bachelor's degree in information technology I had several courses like programming in C, programming in C++, operating systems and embedded system programming which layed down the basis for developing embedded software. Due to my additional bachelor's degree in audio production I have additional experience in audio applications and audio and media codecs, which will help me to understand the theory behind the different needed algorithms. With my previous development work for the Strämpler project I already have experience in working on complex embedded C projects and which potential pitfalls could occur.

Contingency

If I get stuck and my mentor is not around I will follow the following steps in displayed order:

  1. Search the internet for the problem.
  2. Serach through literature acquired during milestone #1.
  3. Ask in the GSoC IRC, if fellow students know a solution to the specific problem.
  4. If the problem is still not solved, postpone the problem until mentor is available again and work on another part of the project.

Benefit

Equipping the BeagleBone AI with media ip streaming capabilities would allow the Beagleboard.org community to use those capabilities to implement the system in professional media applications. The community could also implement further media protocols like AES/Ravenna to allow the usage of the AI for even more media streaming tasks.

Misc

Link to pull request #139.

References

  1. [1] „4.3. PTP — Processor SDK Linux Documentation“. https://software-dl.ti.com/processor-sdk-linux/esd/docs/06_02_00_81/linux/Industrial_Protocols_PTP.html (accessed March 30, 2020).
  2. [2] 1733-2011 IEEE Standard for Layer 3 Transport Protocol for Time-Sensitive Applications in Local Area Networks. ///.
  3. [3] M. A. Yoder und J. Kridner, BeagleBone cookbook, First edition. Sebastopol, CA: O’Reilly Media, Inc, 2015.
  4. [4] C. Hallinan, Embedded Linux primer: a practical real-world approach, 2nd ed. Upper Saddle River, NJ: Prentice Hall, 2011.
  5. [5] A. Liberal de los Ríos, Linux driver development for embedded processors: Learn to develop Linux embedded drivers with kernel 4.9 LTS, Second edition. .
  6. [6] R. Love, Linux kernel development, 3rd ed. Upper Saddle River, NJ: Addison-Wesley, 2010.
  7. [7] E. White, Making embedded systems: design patterns for great software, 1. ed. Beijing: O’Reilly, 2012.
  8. [8] D. Molloy, Molloy_exploring BeagleBone 2e. Indianapolis, NY: John Wiley and Sons, 2018.
  9. [9] „The Linux Kernel documentation — The Linux Kernel documentation“. https://www.kernel.org/doc/html/latest/index.html (accessed March 26, 2020).

PTP Overview

  • is used by the AVB protocol to achieve synchronization between devices
  • based on IEEE 1588v2
  • up to nanoseconds accuracy
  • sharing timestamps over the network for synchronization of devices
  • uses master/slave hierarchy
  • slave retrieves time from master ==> network dely has to be taken into account

PTP Clocks:

Ordinary Clock

  • normally endpoint of the network
  • single port
  • BMCA (best master clock algorithm) determines which clock is used as master (the one with the highest accuracy)

Grandmaster Clock

  • is used as an endpoint master and has extremely high accuracy (normally timed by GPS or NTP)
  • there can be more than one in a network to achieve redundancy

Boundary Clock

  • mutli port
  • a network switch with master/slave ports

Transparent Clock

  • accounts for queuing delays when a standrad switch is used and thus improves accuracy

Delay mechanisms:

E2E:
  • calculate network delay End-To-End
  • no need of PTP equipment but this results in added cost in accuracy

P2P:

  • calculate network delay Peer-To-Peer
  • results in high accuracy, but all devices in the network need to be PTP enabled