Rust Programming Language EuroRust 2022, 4 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The art of programmatic videos with Rust - Dmitriy Kovalenko - EuroRust 2022

Description

Dmitriy Kovalenko
Lead Software Engineer at Lightsource

This is going to be a release of https://fframes.studio. Dmitriy has spent mostly 2 years to actually built it and dived into the video programming a lot. Videos are actually a sequence of images properly encoded. But how you can make your own video from scratch with only the code?

Here is a list of things you will know after attending this talk:

1. How videos works under the hood? What are codecs and how they work?
2. How to make a video with a code?
3. Why using rust?
3.1 Interop with libav aka ffmpeg
3.2 Memory efficient frame rendering
3.3 GPU support
4. Rendering frames problem. Browser vs Rust in svg rendering
5. Audio creation remixing and blending
6. GPU for video rendering

And also you will have everything to start making videos programatically with Rust. Gonna be 🔥

A

Thanks guys, first of all, I have a question: do you enjoy the conference so far? Yeah wow yeah, that's actually better for me, because it makes me responsible to not mess up your impression. uh Okay. So let's dive in and I will say that I am like the last boss and I know that I separate you from the closing dinner, so I won't use my superpower of delay in the talk for an hour and I'll start right now. Yeah, my name is Dimitri I'm doing a lot of open source and I am working at lightforce.am.

A

We are doing something really interesting, we're doing the whole end to end supply chain and procurement, stack in rust, from graphql, endpoint and API to artificial intelligence and like data processing and analyzation. So if you're interested just let me know, you know where to find me: okay, but today I'm going to talk about my different passion and it's actually videos I'm, really obsessed with everything that you can touch still visualize or hear in the programming, and actually videos are really important type of content for current Association right.

A

According to statistics that IO people are spending more than 10 hours per week, it's dozens or hours of billions hours watching the videos and by the way, did you understand how we appealed in the world where this is the correct type of the video, the vertical one, I, don't, and actually somebody is making this video somebody making a tons of videos, but the Technologies behind the videos are completely outdated and I'm, even not talking about like professional software, but we should actually we're talking about some free to use software.

A

Maybe mobile API is full of videos, and this there is a big problem and the problem is Javascript. Thank you.

A

Yeah I have no idea how JavaScript became the popular for video rendering, but people are really making uh videos in browsers, but actually that's not all the Adobe After Effects, the most popular software, the most professional software for uh making animations and Graphics actually runs plugins over JavaScript and that's crazy, like people spending 20 000 Euros on Pro Mac computers to have a bottleneck and run in JavaScript, that's like yeah, that's bad and we we together are responsible to change that because it's time for us it's it's yeah.

A

Thanks a lot and always has been because uh yeah memory management is the most important part for videos and for media programming. You just can use garbage collected language and you can't have memory errors because they're really hard to spot, and today, I'm gonna talk about my journey of learning, how to use rust for media programming.

A

How to and a majority of building framework for this and I'll show you everything, but I'll first start from the scratch from what is the video, how it actually works and what is inside the video and uh to be honest, videos are not really hard to understand. That's actually a sequence of images. Nothing more! So you can say that we have an mp4 file right right here like here and MP4 is not a codec. It's an important statement.

A

It's just a media container that somehow contains an information, and here we can see that media container can contain couple streams like image stream, where each green pill is an image, but it's already encoded image with two numbers. That's all you need to know about images. The first number is presentation timestamp. The second number is the decoding. Timestamp, so basically, that's just a giant binary file was where every image is also a buffer and your codec is responsible to read it and decode it correctly.

A

So, for example, if we have decoding times template they're, not in order, so the codec may need to encode like, for example. Here we can see uh the force frame being needed to decode it before the sword, one and so on and so forth. That's a magic of codex, we'll return back here and we have also an audio stream. An audio stream is completely different scene.

A

We capture audio in samples, and we have like you know this 44 100 samples per second, and we also break it inside the frames which are usually like 1000, uh two by yeah, 1024 or 512 uh inside the frame, and it all also has a protection decoding sometimes, and when the player reads this file, it synchronizes the content and it actually plays nothing really uh hard until you figure out how much codecs there are and which one is correct.

A

So, basically, if you want to include your video, you need to use a codec and there are a lot of different codecs. Someone is like useful for professional video, editing somewhere useful for um streaming. Someone useful for something. Do you hear me better? Okay, perfect, so, uh okay, I'll grab my clicker I forgot it. So you just need to somehow encode the encode, your images into the video and how you will do this, because the codecs are not like software. It's actually a specification. So we have the high efficiency.

A

Video, Codec or h.265 has a specification over 700 pages, and you don't want to deal with it directly right. You don't want to implement this because you take. It will take 10 years for me to make this on weekends, but you can use the project called FFM back and you have all used this I'm I'm very much sure- and this is enough- and this is what people are using to make in videos to make videos. So we can generate a bunch of images.

A

Somehow an example is browser which just runs the Headless browser and uh then saving the screenshots with a specific pattern. Like pick, then the number of frame and the PNG, so what is y UV image? It's the most popular image that stands for uh video production for decades and what is it? It's just like regular image where the pixels are splitted to the Lumen channel, the brightness and the red and blue projections. When you combine it, you get the real image and yeah. That's just an interesting story where it came from.

A

It came from analog television, where the companies discovered that they want to sell people the color of Television. They didn't want to make a breaking chain. They didn't want to introduce something new, so they created an algorithm, a new video structure that sends two additional channels. So people who want to use calorie television only needs to additional channels, and that was that really work. Well, we all know that color television succeed, but right now for everything that touches the video. We need to do something like this for each frame and it becomes a problem.

A

So this is a copy from my framework that takes the RGB image and sends it and converts it to the why?

A

U v or cbcr channels, there's some some magic inside and if you want to copy it, just make sure that you follow in examples, because sea libraries may be crazy and they just can sometimes read every sequence by by some sometimes read every Force byte from the data source. So we need to be exactly sure that we are using it as designed, but right now we know that you can actually take any kind of image. You can decode it.

A

You can get the raw image sources converted to yuv, send it to the encoder and then get a video okay, that's for you! But right now we need to actually somehow render images and that's becoming a problem, because uh how you will render images? Maybe somebody have some proposals?

A

No, not a lot of proposals uh and the problem again. Yeah fragment Shader! That's uh that's a nice proposal, but it stands for um yeah. Let's get back to this later, I I'm talking about this. So basically, if we want to render some images, the most popular today is browser.

A

um The browser is stands today for everything. People are using browsers for everything, but that's not our option and the problem is that, like it's, it's becoming really crazy. Can you spot the similarity between these two images? The first one is the obvious streaming that streams on Twitch and has some notifications on the UI, the last one. It's my GitHub repository uh and Sarah like image for generated by GitHub.

A

They both render it in browser the OBS used in browser to render all the UI when streaming and the GitHub is also using browser to render this previews. But that makes literally no sense. Why do you need so much browser everywhere? I have no idea so I started thinking of what we can use instead of browser and what we need to. We need to actually have and fix it.

A

Animatable friendly for developers GPU, first and important to understand that it should be understanded by developers, debuggable and clear, because nobody wants to learn new additional language in order to generate some video, and it always has been. It always has been a free informant that perfectly supports old age cases. It's SVG and do not be scary. Svg is not what you're thinking about.

A

Thanks so SVG can contain not only the scale rate pauses, with a lot of like weird things that even I cannot understand and I made a parser for SVG. uh It contains, like literally all the required Primitives, some basic shapes, like rectangle ellipse Circle text, with a different font, families, images and even for rain objects that you can actually customize as you need. So uh the Adventist like it's just perfect fit for video rendering, because SVG defines how to render images and we actually need it.

A

And, let's let me introduce you my framework, it's ever frames, I'm working on it for 1.5 years and right now, I will try to describe how I combine all of this like knowledge together to make it end to end rust solution to actually produce the final video using SVG and all of this stuff. We just learned. So let's do it. So, basically what you want to, if you want to make a program that makes a video you want to have some input. Let's see it's a very simple example: you have a slug of a string.

A

You have the implementation of trade for the video, which has FPS width and height the dimensions and the speed of the rendering, the duration, which is 30 seconds. You have no audio and you have some function that renders this particular frame and the function that renders this particular frame looks very much like SVG. You can actually just copy SVG somewhere and pass it here. What's important important here. It's this part.

A

So basically, you can see that you have here the rectangle that starts on the zero zero position and take the whole bits and height of the canvas and it animates over. Like timeline every five seconds from specific color with a bunch of different easings like the liner, the Apple like spring animations, uh you have the font family of a text that goes next.

A

The DM stands that will be resolved from your computer or from from your media folder, and you have the Jared brains monophone for another text which stands for displaying the index frame and the correct current second, and that's how you define the video and then what you are getting the result. That's a development editor which gives you an ability to preview the uh the actual video it works like in 60 FPS. It displays you the whole timeline. I bet, you don't see the colors different here, but it actually it's there.

A

You see the resources, the two phones and you see uh the preview of the editor. But what is more important that you can actually open devtools and you can debug your SVG and that's actually an intention. The browser is like really the best thing to develop. Svg today and preview. Things in the browser makes it lightweight and make it easy to debug. So you can actually just debug your SVG.

A

You can change the arbit, your values here to experiment or you can copy the SVG from the browser and pass it into figma and design it in Sigma and then copy it back and pass again into the rust code. uh Yeah, let's go to the next slide, and here is how it works. So the idea behind of the editor, the video creation app, is always uh the problem is that you need to somehow preview it, and there was some bridge that makes the editor live.

A

It actually works perfectly, so we basically take the video definition that you, since, above and given it to the some front, end client-side app that renders you preview with all the information and with all the calculation that you made inside the render frame function. So you actually make any kind of calculation. You can pass any kind of inputs and you also can use couple of core Alpha frames apis. All like like animations, you see and also for sure we need another one back end.

A

We need a renderer backend which actually renders you the whole file and yeah. So the actually, uh how you will include the awesome Bridge you just pop up the macro is for everything rate you just includes macro includes some inputs. I forgot to put here the slag, but yeah. That's how it is pretty much simple, but here is probably the most important part of this talk. It's the rendering problem, and if you want to avoid something like browser for Android, it's becoming really tricky because you need to somehow analyze all the SVG.

A

You need to somehow parse it and you need to actually render it and apparently rust. Community is so awesome.

A

So when I started the project, there was already a red SVG library that had 1.5 000 tests that uh it was built by rather Falcon, and they just do what I needed to do so right now, if you don't want to use like Frameworks, like my implementations, you can just take it and use it directly, but it doesn't have any kind of caching in the rendering of video is very much like dependent on what you are, um how you will reuse the rendered process between the frame, um so I actually implemented something over the uh oh, my god.

A

Let's back it back over this library and uh created my custom, renderer backend, which is like one of for CPU and one for GPU.

A

The most important part here is that we need to somehow also parallelize the rendering, and this is really hard, because uh the rendering encoders are doing a lot of mass based on the sequence of the frames. It is important, so you for you to supply all the frames in order, so you can just use Rion uh split all the power Eater over all the frames render them in Supply. That uh will have literally no sense, you will not work, and you will also have mostly no cash heat between the rendering.

A

So basically, however, frames is doing uh the effort frames actually takes, uh creates like in quarters by the number of your cores, in this case, like 12 encoders and mostly renders 12 different files in a specific order, so that, after all, it just mostly concatenating the frames without any actual re-encoding. So basically, if you have like 12 cores, you will randomly 12 uh 12 videos into the file system and then basically read the whole buffer of the file system and concatenate it with another buffer and render the final file and that's becoming really really performant.

A

uh So, in order to render something with uh effort frames, you are basically creating the hello world video, which defines how you render frame you pass the output file. You pass some render options uh like bus, Which, logger you want to use which backend you want to use in the private codec and here is an example. I promise you it wasn't rendered with frames, it's the real screen screencast from my computer, and it renders a full HD 30.

A

Second image with where every frame is animated, with some color, where you have on every frame where you have the different text per each frame in four seconds, which is actually faster than the browser will start.

A

uh But you know that I know that you probably expect right now to see the demo of the video, but if I will put the the previous video that we defined, that would be really really boring, so be a little bit. Passionate and I will show you something more and more interesting in a second okay, but the problem with CPU rendering is about these things.

A

So, basically, if you want to oh it's not displaying at all, but if you want to have some Shadows where you have to use some gradients or you have to blur something. Cpu physically cannot render correctly this type of content, because it involves a lot of one by one one by one calculation for each specific pixel. So you need to like or smooth the color or you need to make it go simpler, which is really hard or you need to do everything together and that's where GPU actually helps a lot.

A

It's it's some somewhat useless, I would say because nobody will use GPU backend, that's my personal assumption, but that's really cool and that's implemented from scratch. So the idea of GPU here is some graphs over rendering of the VPS like frames, render it per second for CPU and GPU, and for this like Hello World video that we've seen definition couple slides ago, the CPU wins because GPU can actually take one core and the parallelizability of uh rust actually makes CPU even faster. But if we make it, 4K CPU actually makes slowdown.

A

And if you have some blues and gradients here, GPU wins perfectly. Unfortunately, supporting GPU involves a lot of a lot of trickiness, because right now we need not to only analyze the SVG and map it directly to the buffers we need to. Firstly, simplify SVG into vertices and indices and the important part uh the the shaders and to involve the actual GPU Shader over the vertices and indices then get the resources, then get the texture and map it directly to the encoder, which makes it hard and yeah, because it was not implemented.

A

I needed to do this from scratch and it's like really bad supported the whole spec. But what is really cool is that because of where he's in Rust- and we are using everything, we're building everything from scratch- I actually build an average performance crash. We are benefit we are getting like benefits from using the whole end-to-end stack the basically because we know how you are parsing rust code.

A

We can introducing the, which is probably not the best name, but imagine that you have some Paths of SVG, which is really one and because SVG is always the flat structure and your uh all the elements like elements cannot be nested. You always know that the next element overflow, the previous, so we have the pass. For example, we have the image and we have some rectangle- that animates with some animation. What we can do, we can actually uh compile it.

A

We can actually run simplification and render it to the uh Pixma or to the vertices and indices ahead of time it right inside the proc macro and that what makes like a durandering uh really as easy as render like couple pixels on a Pixma and that's what makes effort frames really fast and really uh like ahead of time-ish ahead of time. Ish yeah- something like this. So basically, instead of having to render everything you are run, caching, independent layers or even shrinking couple layers and having only the base Pixma.

A

That only renders the last animatable element and that's really easy, because rust provides the really awesome, Brock marker tool, and we can understand when it is linked to this variable, uh where it is a constant which we can ignore, or it's something uh that we should animate okay. So uh you know that videos are not only about the actual images, it's also about the audio and we can actually go crazy with audio and we can do something like this. The pure Mass be a very cool thing.

A

We can just assumpt that Mars uh we can just like calculate the Hertz for the specific, like the frequency for the specific time point, and you can create even music with code, we can be really creative, but what we really want to have is something like this: we want to have the preview that is development mode. Where you have the old, your audio tracks, They will mix it together. Then simple, SVG yeah, nothing, really, nothing really uh surprising!

A

Supposedly, you get all the audio tracks, maybe some music background, maybe some voice, uh maybe some audio sound and combine it together, using the simple rust API and, as a result, you're getting the one audio stream with the mixed audio. But what is really awesome here that, because we always lowered in the audio, we can provide user the API for gorgeous video, audio analyzation and Spectrum analyzation to create audio visualization, just like new iPhone Dynamic element, and that is really easy as well. There was actually the very first intention to build a project.

A

I had the podcast at times and I wanted to have this awesome, audio visualization and what I've done so? Basically, you just asking for frames to give you the audio data for the specific frame, and then it gives you the vector of the frequencies which you somehow smooth in this case I'm using the simple disables the logarithmic decibels, uh smoothing and render it as a rectangle with a specific heat, nothing more and it works pretty seamlessly and we had like podcast visualization, rendering running for 15 minutes.

A

uh That is really interesting and cool and videos are really interesting and cool. I hope you think the same, and uh but you may ask me, but we you talking about like videos today, you talked about the rust, but where is the art? Where is the art? That's the main question so I ask you to have an attention to the screen, and that is the art of programmatic videos in Rust and a few frames in particular.

A

Welcome to Alpha frames beta the very first rasp-based framework for motion, graphic videos, the interested in order to try it out sign up to our Discord and send your GitHub to the beta Channel. You'll, get an invitation to our GitHub repository with all this source code and couple examples to get started.

A

Thanks a lot and I will feel a little bit like Steve Jobs. You can pre-order it starting from no. No, no, the beta starts right now. The public beta starts right now, and the instructions available in videos are okay are correct. You can right now sign up to the Discord chat and get an access to the GitHub repository of everframe straight out or maybe play around with it, and I will be happy to help you support. You support your work and everything related to the video production and rest thanks a lot for your attention.