Cloud Native Computing Foundation Cloud Native Rust Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Streaming Video from WebXR content using Servo - Alan Jeffry, Servo Project

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Streaming Video from WebXR content using Servo - Alan Jeffry, Servo Project

Servo is an embeddable web engine written in Rust, which for a long time was The Other Big Rust Program. One of the strengths of Servo is its ability to deliver immersive content using WebXR and WebGL. This talk introduces the Servo GStreamer plugin, which allows web content to be streamed to low-end devices, including streaming 360 video from WebXR content.

A

Hi there, my name is alan jeffrey and I'm the technical chair of the servo project and I'm here today to talk about one of the applications of servo, which is in streaming, video using g streamer um from web xlr content.

A

So what is servo well servo's a browser engine, and so that's the core of a web browser, that's responsible for downloading content off of the internet and then displaying it to the user. And traditionally this has been part of the 2d web, using standards such as html or css or javascript.

A

Our old friends and more recently, though, there's been efforts to allow 3d content um to be displayed using standards such as webgl and webxr, and so servo was originally developed by mozilla um and is now part of the linux foundation.

A

um And the original motivation for for a survey was could twofold, so one of which was finding out whether components could be developed in rust and and then deployed in florifox and the other ways to experiment with and newer web technologies, such as webxr and the experiment of developing web components. Using rust is very successful and so technologies got transferred out of servo and into firefox such as the stylo css engine and the web.

A

Renderer, graphics, backend, and but also my server, like I said, was being used as an experimental platform um for finding out what you could do with an embeddable uh web engine um and what kinds of experiences this would enable, and I'm going to be talking about one of those today. So the next bit of the technology stack that we need to look at today is gstreamer.

A

So g streamer is a streaming media, plugin architecture and where plugins can be sources um for video or audio uh filters, such as transcoders um or syncs, and such as your file system or streaming over the internet um and the way that content providers use g streamer is by stringing together um these uh plugins into a pipeline where media is consumed at one end, is them formed by various filters and then is produced and in a sync um and one of the things that's interesting about g streamer from our point of view is that it has quite strong connections into the rust world.

A

um So there are rough bindings um for all of the g streamer apis and and also uh it's possible to develop um g streamer plugins using rust um and that's very useful for servo and which is primarily implemented in and gstreamer is used inside servo as the media subsystem. But that's not actually what we're looking at today.

A

What we're looking today is using servo to provide um a source, so servo is designed to be an embeddable web engine and one of the places we've embedded um is into gstreamer, so there's a servo source plug-in and that you can use to take web content and put it into a juice tumor pipeline.

A

So, for example, you could use this to lay out the title sequence for a video using html and css and then use the server source plugin to produce that as part of your g streamer pipeline and one of the nice features about this plugin um is that it's entirely using gl memory. So if the rest of your pipeline supports gl memory, this means that the video content never actually has to be in the main memory of your machine and can instead reside entirely on the gpu.

A

And if you're familiar with the webcam embedding into gstream or the wpe plugin- and this is doing a very similar job for server and like I said you can use this- to render traditional 2d content um into a g-stream or pipeline, but you can also use it to render webex so webxr is the w3c api for developing virtual reality and augmented reality.

A

Applications uh using web technologies- and these can be both 2d experiences such as holding a phone up and seeing a virtual object in your living room and immersive 3d experiences for users wearing appropriate headsets and for servo we've got a web xr implementation with back ends for both the hololens um and the magic leap, um augmented reality headset and as part of that, uh the servo development team was quite actively involved with the w3c standards effort.

A

So plugging these three technologies together, so webex are server and g-streamer, and this means we can take web xlr experiences um and put them into a g-stream of pipeline, and this means you can do interesting things that I don't think the um authors of the web xr experiences um intended. So, for example, we've got a back end for servo.

A

The renders to red cyan uh 3d, so if you put on your uh red blue glasses- and you can actually then watch a 3d experience at home without a headset um or a we've, also got a another back end and the renders, rather than rendering stereoscopic views of the 3d scene, which you would for a headset. um We render uh six views of the scene to a cube map, and then we take that cube map and we project it into 360 video.

A

And so this means we have the ability to record 360 video out of an arbitrary web xor scene and as long as that scene hasn't hardwired. The idea that people only have two eyes um so we've been able to take webex our content, and that was intended for viewing a headset um and we've now managed to actually reuse it um for in scenarios that the original authors didn't intend- and this is part of the philosophy of the web- is that it should support this kind of accidental.

A

A